Summary: Let's use more references when we can
Reviewed By: krallin
Differential Revision: D24161694
fbshipit-source-id: 5cf7edf843fa8dcf0d24ca114c3d520263d92a3b
Summary:
This change enables the filler job to work on all repos available instead of a
single one. We are still going to be able to dedicate the job to a certain repo
(by crafting a config with a single repo enabled) but we can put the entire
long tail for low-traffic repos under a single job.
This requires D24110335 to land in configerator to work.
Reviewed By: krallin
Differential Revision: D24136239
fbshipit-source-id: 4b77d1667c37cc55f11c3087b02a09dbae29db0f
Summary: Allow bookmark to be optional - again, will be used in the next diffs
Reviewed By: ahornby
Differential Revision: D24163608
fbshipit-source-id: e037731117181d0b1bbe4eb273301245142b507d
Summary: This functionality will be used in the next diffs.
Reviewed By: ahornby
Differential Revision: D24163517
fbshipit-source-id: 36e5c9646e21913f0e0d79d77dd11862f5aa5331
Summary:
This diff fixes how syncing of merge commits decides on the `CommitSyncConfigVersion` to use. Old and incorrect behavior just always uses current version from `LiveCommitSyncConfig`. The desired behavior is to reuse the version with which parent commits are synced, and manually sync commits when version changes are needed.
For merges it is more interesting, as merges have multiple parents. The overarching idea is to force all of the parents to have the same version and bail a merge if this is not the case. However, that is an ideal, and we are not there yet, because:
- there are `NotSyncCandidate` parents, which can (and should at the moment) be safely excluded from the list of parents of the synced commit.
- there are `Preserved` parents (which will turn into the ones synced with a `noop` version)
- there are `RewrittenAs` and `EquivalentWorkingCopy` parents, which don't have an associated version.
So until the problems above are solved:
- absent `RewrittenAs`/`EquivalentWorkingCopy` versions are replaced with the current version
- `Preserved` merge parents cause merge sync to fail.
Reviewed By: StanislavGlebik
Differential Revision: D24033905
fbshipit-source-id: c1c98b3e7097513af980b5a9f00cc62d248fc03b
Summary:
Our higher-level goal is to get rid of `CommitSyncOutcome::Preserved` altogether. This diff is a step in that direction. Specifically, this diff removes the creation of "accidental" Preserved commits: the ones where the hashes are identical, although a `Mover` of some version have been applied. There are a few sides to this fix:
- `get_commit_sync_outcome` now returns `Preserved` only when the source and target hashes are identical, plus stored version is `None` (previously it would only look at hashes).
- `sync_commit_no_parents` now records the `Mover` version it used to rewrite the commit (previously it did not, which would sometimes create `Preserved` roots)
- there are now just two ways to sync commits as `Preserved`:
- `unsafe_preserve_commit` (when the caller explicitly asks for it). The idea is to only remove it once we remove the callers of this methods, of course.
- `sync_commit_single_parent` when the parent is also `Preserved`. Note that automatically upgrading from `Preserved` parent to a rewritten changeset is incorrect for now: `Preserved` does not have an associated version by definition, so we would have to use a current version, which may corrupt the repo. Once we get rid of `Preserved`, this case will naturally go away.
- as we now have `update_mapping_with_version` and `update_mapping` (which consumes current version), we need to add explicit `update_mapping_no_version` for preserved commits we are still creating (again, recording a current version is a mistake here, same reason as above)
NB: I've added/changed a bunch of `println`s in tests, leaving them here, as they are genuinely useful IMO and not harmful.
Reviewed By: StanislavGlebik
Differential Revision: D24142837
fbshipit-source-id: 2153d3c5cc406b3410eadbdfca370f79d01471f9
Summary:
There's a bug in Thrift-py3 streaming support, where interrupting
iterating over an async stream leaves Thrift objects in a broken
state. Futures get dropped (and warnings are printed to the console)
but the destructors hang. Don't even try to garbage collect the heap
in that case.
Reviewed By: genevievehelsel
Differential Revision: D24032229
fbshipit-source-id: 5f16667fe6cfd27de1b39cc2974028729e18b214
Summary: Thrift-py3 supports streaming, so give the new client access to APIs from streamingeden.thrift.
Reviewed By: wez
Differential Revision: D24032144
fbshipit-source-id: 44f350b5cfa943154084b8d64f6c696e315e6b88
Summary: Eden mounts are identified by paths, not by strings, so fix the Thrift signature.
Reviewed By: wez
Differential Revision: D23774513
fbshipit-source-id: c0fb82c48eee5ce4e8fbffef5623f9016ef76e40
Summary:
We're seeing an issue on the hg servers where the filecache assertion
that if a value is in obj.__dict__ it's also in obj._filecache is broken. This
occurred about 10% of the time in sandcastle jobs. The diff that caused this
went in in April (D21148446 (73c5cb89de)), so it's unclear why it's only cropping up now.
This is caused by the following steps:
1. repo._bookmarks is accessed while _bookmarks is in the _filecache but not in
the __dict__
2. This causes construction of _bookmarks, before it can set it to __dict__
3. Construction of _bookmarks calls repo.invalidate(clearfilecache=True), which
deletes _bookmarks from _filecache.
4. _bookmarks construction completes, and gets set to __dict__ (but now it's
missing from _filecache, so the invariant will fail next time someone checks).
5. Someone accesses _bookmarks later, and the assertion fires.
The fix is to just not clear the filecache during bookmark construction. The
main purpose of this invalidate was to let the changelog be reloaded, and I
think that will still happen since, if there are any new commits in the
changelog, the file size and time will change, triggering a reload next time the
_filecache entry is checked.
Reviewed By: quark-zju
Differential Revision: D24182914
fbshipit-source-id: fb49137e28d9224c6617d9c84faaf2f9de363aaf
Summary:
Adds a subcommand to `eden debug prefetch_profile` to fetch all the files
for a list of profiles or all the active profiles for a checkout.
These profiles (files) can contain lists of file names, or globs of files, eden
will be able to resolve them since this uses the existing prefetch code.
I opted to put this command under `eden debug prefetch_profile` instead
of `eden prefetch` since the command may change to call into eden with out
the list of files (letting eden resolve the active profiles and names). Then
it will no longer resemble prefetch and so long term it may be better homed
here.
Reviewed By: genevievehelsel
Differential Revision: D23771483
fbshipit-source-id: 12af81d40cc495efd381e3c3a2df645d72053ccd
Summary:
Before I make further changes to the Journal, improve the comments and
refactor a few small things.
Reviewed By: kmancini
Differential Revision: D24089530
fbshipit-source-id: de9da2c1e6b1c87b6587781cfa55ae7cc4085eeb
Summary:
Scanning through the functionality provided in ServiceFrameworkLight,
it looks like none of it really applies to the edenfs daemon, so break
the dependency entirely. Removing this complexity would have prevented
the regression where we stopped recording Thrift call statistics.
This should also improve our build times a bit, and maybe resource
consumption.
Reviewed By: genevievehelsel
Differential Revision: D24094784
fbshipit-source-id: fefd1a648c7ecba8484296527ff8100269c176b6
Summary: It turns out that perf_buffer API is only available in libbpf 0.2.0, which hasn't been released yet. To unblock katran, I grabbed the latest libbpf revision and created a temporary manifest. To avoid relying on this "beta" version libbpf on other projects only katran uses it. I'll make sure to remove it in favor of the official libbpf 0.2.0 once it's released.
Reviewed By: anakryiko
Differential Revision: D24156655
fbshipit-source-id: 32f6e04079a862fbfe96fd5475678cfa4ae1b3db
Summary:
Implement BlobstorePutOps for S3Blob. This uses is_present to check the various put behaviours
While implementing this I noticed get_sharded_key could be updated to take a reference, so I did that as well.
Differential Revision: D24079253
fbshipit-source-id: 16e194076dbdb4da8a7a9b779e0bd5fb60f550a6
Summary: Now that fileblob and memblob support put behaviour logic, update the overwrite test to check the overwrite result.
Differential Revision: D24021167
fbshipit-source-id: d9578630205cf5d79999a459cc29481968d5717d
Summary: Update memblob to be PutBehaviour aware by changing implementation from Blobstore to BlobstoreOps
Differential Revision: D24021166
fbshipit-source-id: 04dd25c5535769ea507120c1886592b808a7bbc6
Summary: Update Memblob::new callsites to ::default() in preparation for adding arguments to ::new() to specify the put behaviour desired
Differential Revision: D24021173
fbshipit-source-id: 07bf4e6c576ba85c9fa0374d5aac57a533132448
Summary: Add put behaviour handling to fileblob so that it can prevent overwrites if requested.
Differential Revision: D23933228
fbshipit-source-id: 8e74ac96b232be841174f6ad2bd2fccf92aaa90d
Summary:
Add put behaviour to BlobstoreOptions in preparation for passing in the put behaviour through blobstore_factory.
Later in the stack a command line option is added to set this non-None so that we can turn on overwrite logging for particular jobs.
Reviewed By: StanislavGlebik
Differential Revision: D24021169
fbshipit-source-id: 5692e2d3912ebde07b0d7bcce54b79df188a9f16
Summary: Add a reliable, lightweight TraceBus class for publishing events to a background thread. Subscribers can be registered for observing events or computing telemetry about them.
Reviewed By: wez
Differential Revision: D23404525
fbshipit-source-id: 3539466421b0821ffb918ea862168d3cccd19b15
Summary: To be more Windows compatable, we should move away from using `sh -c`. We don't use shell=True because that is susceptible to shell injection. As long as we don't close stdin until we're done, using Popen raw should be fine.
Reviewed By: xavierd
Differential Revision: D24151425
fbshipit-source-id: c0bcc883af948491862e8ce0cee56bcbe98e04f1
Summary: Add a new "user" column to the EdenAPI server's Scuba data.
Reviewed By: krallin
Differential Revision: D24153765
fbshipit-source-id: 95a3177d9283e5b0e3f47e7f42a1de5142049b99
Summary: Add a method to get the client's username from a client identity. This is helpful for logging, etc.
Reviewed By: krallin
Differential Revision: D24153766
fbshipit-source-id: 2ecf85e5de72918aeb292ce88539a991da4de900
Summary: Allow source control service clients to set pushvars.
Reviewed By: krallin
Differential Revision: D24136870
fbshipit-source-id: 34f9176ec66ca319b363c91015dae9b59a55a244
Summary:
Add the possibilty of setting `pushvars` when deleting bookmarks. This makes
it consistent with the other write operations.
Reviewed By: krallin
Differential Revision: D24136869
fbshipit-source-id: f98b74c6c731e50552184000ad697d04748711fd
Summary:
Previously all mutable_counters reads were going to leader. This might be
useful for some cases, but in the case of blobimport warmer this opens too
many connections to sql leader.
Let's read from replica instead
Reviewed By: krallin
Differential Revision: D24160315
fbshipit-source-id: 0cecde3c54a00bbea215a1e0fa63d4a7c3dc9eaa
Summary:
This seems to have broken as a result of a land race between D23999804 (6421dca639) and
D23455274 (bdff69b747). Let's fix it.
Reviewed By: ikostia
Differential Revision: D24158809
fbshipit-source-id: 1d733e2c93eb8a0803395d409fbb15e2e2146bdd
Summary: Adds version of `bounded_traversal_stream` where unfold returns a stream over children instead of an iterator. This function also applies back pressure on children iteration when we have too many unscheduled items.
Reviewed By: krallin
Differential Revision: D23931035
fbshipit-source-id: 2e2806653782d4e646dcdf4b2d4e624fd6543da8
Summary:
Our stdin/stdout bytes/str manipulations caused input() to print
warnings about buffered not being supported. The only reason we do those
manipulations to handle the case in tests where the prompt answer doesn't come
from stdin, so let's just handle that case via readline instead of prompt.
This is what upstream Mercurial does.
Reviewed By: quark-zju
Differential Revision: D24122909
fbshipit-source-id: ab9d989a66d39990b688c65a1fae80bd48b0f42e
Summary: Add `--debug` flag to `read_res cat` command for debug printing entire entry rather than just the data blob.
Reviewed By: kulshrax
Differential Revision: D23999804
fbshipit-source-id: 6955854edab2643cffbe5fae484a398716b48055
Summary:
Initial support for a backend using edeanpi.
Note this is just the first step. Most code paths are not updated to the
streaming API to get commit data, so they will error out with commit not found
errors.
Confirm that commit data can be fetched via edenapi:
$ RUST_LOG=debug lhg dbsh --config experimental.lazy-commit-data=1
In [1]: master= repo['master'].node()
In [3]: cl.inner.getcommitrawtext(master) is None
Out[3]: True
In [4]: s=cl.inner.streamcommitrawtext(repo.nodes('master~10::master'))
In [5]: it=iter(s)
...
[2020-09-25T02:09:16.793Z DEBUG hgcommits::hybrid] >> resolve_remote input=[e6c4e203b66f1416e08dc597a2d63b91e93b1466, bfb610989e9dd701e785b4a3a5998e76d9709cab, 68bbfc79602a153895b761089e9479dd8fa33351, 5366fe39ad538463abae6c648eb5150bbb79d4c7, 5ea45d8ab0f8203837ca1736f36ded4a492571b4, 722da0a32eae12de5e85078beea2ae4b7aafe4a4, 4dbe3eab10d13b30697e1762eb7b9ff3ad0cf630, 430ae91aab8028b6572ccef89f8396dafec622c4, 5abd96c5420f0d512c63e768f8cea83f1c6691c9, c84ab3412cebfade730e95a1bc5ebc9b1dd0747b, 790ed2d40e4a0b08fb22fe9b4246fec0165f8a87]
[2020-09-25T02:09:16.793Z DEBUG hgcommits::hybrid] << resolve_remote input=[e6c4e203b66f1416e08dc597a2d63b91e93b1466, bfb610989e9dd701e785b4a3a5998e76d9709cab, 68bbfc79602a153895b761089e9479dd8fa33351, 5366fe39ad538463abae6c648eb5150bbb79d4c7, 5ea45d8ab0f8203837ca1736f36ded4a492571b4, 722da0a32eae12de5e85078beea2ae4b7aafe4a4, 4dbe3eab10d13b30697e1762eb7b9ff3ad0cf630, 430ae91aab8028b6572ccef89f8396dafec622c4, 5abd96c5420f0d512c63e768f8cea83f1c6691c9, c84ab3412cebfade730e95a1bc5ebc9b1dd0747b, 790ed2d40e4a0b08fb22fe9b4246fec0165f8a87]
...
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] >> Zstore::contains id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] << Zstore::contains id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] >> Zstore::insert data_len=1010 id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] << Zstore::insert data_len=1010 id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
...
[2020-09-25T02:09:16.959Z INFO zstore::zstore] >> Zstore::flush
[2020-09-25T02:09:16.959Z DEBUG indexedlog::log] >> Log::sync dirty_bytes=7048
[2020-09-25T02:09:16.959Z DEBUG indexedlog::log] << Log::sync dirty_bytes=7048
[2020-09-25T02:09:16.959Z INFO zstore::zstore] << Zstore::flush
In [6]: list(it)
Out[6]: [...]
The logs about `hgcommits::hybrid ... resolve_remote` shows the remote fetching
is working. The logs about `Zstore::insert` and `Zstore::flush` shows the
commit data were written to disk.
Reviewed By: sfilipco
Differential Revision: D23924148
fbshipit-source-id: a3d77999e29395ce5c603fe66412936947456534
Summary:
Support constructing the "hybrid" commits backend, which is similar to
"doublewrite" but read commit text from edenapi via the `streamcommitrawtext`
method.
Reviewed By: sfilipco
Differential Revision: D23924149
fbshipit-source-id: cb15ee4be7953af7798d460557ba2ae2d4f24a52
Summary:
The hybrid backend is similar to the doublewrite backend, except that it does
not use revlog to read commit data, but uses EdenAPI instead.
Note:
- The non-stream API will not fetch commit data from EdenAPI.
- The commit hashes are not lazy yet.
Reviewed By: sfilipco
Differential Revision: D23924147
fbshipit-source-id: eb2cf8d3a7e1704b4efb13ad3ad86f8b6a1b31d0
Summary:
This can be used like:
In [1]: s=cl.inner.streamcommitrawtext(repo.nodes('.%%master')) # repo.nodes returns a generator, becomes stream
In [2]: s
Out[2]: <stream at 0x7f5eec742df0>
In [3]: list(s)
Out[3]: [{'vertex': ..., 'raw_text': ...}, ...]
In [4]: s.typename()
Out[4]: 'cpython_ext::convert::Serde<hgcommits::ParentlessHgCommit>'
Reviewed By: sfilipco
Differential Revision: D23911870
fbshipit-source-id: f54959a551d446ed5b8086a2235fe74e47b29e70
Summary:
This makes it convertible to `PyObject` via `cpython_ext::convert::Serde`
without additional code or dependencies.
Reviewed By: sfilipco
Differential Revision: D23966993
fbshipit-source-id: 74d83524a7c0701cde7aa6d61bb930ff4a1c90f5
