Summary:
We might have a somewhat weird case - a file was replaced with a directory and
then in the next the same file was deleted again (even though this file doesn't exist
anymore). In that case we need to make sure these two commits are in two
different stacks of commits, however previously we weren't doing that. This
diff fixes it.
Reviewed By: markbt
Differential Revision: D31168174
fbshipit-source-id: 4b9986e615ec98b6452ff81b113124d14f236382
Summary:
This works more reliably and fully restores `test-commitcloud-sync-race.t` to
pre-D28595292 (c72cd2333f) state.
Reviewed By: markbt
Differential Revision: D30974286
fbshipit-source-id: 729e20f23cb5d8aacdbcef1c869fc9a73ac4d4d4
Summary:
Change the state of visibleheads from the in-memory Python `_heads` variable
to the `svfs.metalog["visibleheads"]`. This changes a few things:
- No need to manually invalidate or reload the `_heads` state on transaction
close/reload, since metalog gets reloaded on transaction boundaries.
- No need to use features from the old transaction framework, such as
`addfilegenerator`, keeping `journal.visibleheads` for transaction
rollback. No need to track `dirty`.
This probably solves issues where `hg pull` hides visible heads unexpectedly.
See P458576970 for example reported by chadaustin where pull runs right after
a cloud sync and hides 5a8c51b193 unexpectedly, but metalog parent shows that
pull got the state after the cloud sync. See also
https://fb.workplace.com/groups/scm/posts/4114332378616351/ for a similar
report from dtolnay.
In theory other states (bookmarks, remotenames) might have similar issues.
But this diff only focuses on visibleheads.
Reviewed By: markbt
Differential Revision: D30974289
fbshipit-source-id: 85d81fd2e2d85ed22ac144f2cb663eb0423955fb
Summary:
`test-cross-repo-commit-validator.t` seems to take longer to run with the next
change. Bump the timeout to make it pass.
Reviewed By: markbt
Differential Revision: D31148285
fbshipit-source-id: 2c815d988b323eb08cf06256ee666130eeebf9a6
Summary:
It tests the Python pushrebase server logic which is no longer relevant.
The next change breaks it and it seems easier deleting the test.
Reviewed By: markbt
Differential Revision: D31121918
fbshipit-source-id: ee5619b35ad4aa16f0227e563ed531e879d1c8d7
Summary:
Store unchanged `self.heads` in a local variable. This avoids some overhead
if `self.heads` is going to be a bit more expensive.
Reviewed By: markbt
Differential Revision: D30974287
fbshipit-source-id: baaffb8f41da4b57e4ac94c305e5ad490a3c3135
Summary:
This is subtle. But fbhistedit (providing `exec` support) depends on it
invaliding everything related to repo to trigger state reloading after
executing a command (which could be `hg`).
Reviewed By: markbt
Differential Revision: D30974284
fbshipit-source-id: b033d81565dcf61104e4d30ecd7d48c33d6d79a4
Summary:
In a future change we'll require `svfs.metalog`. Let's move metalog fix to
before other stuff and attach fixed metalog to `svfs`.
Reviewed By: markbt
Differential Revision: D30974285
fbshipit-source-id: 3be89d1f1cda3d29dd5746940959ee47c1dd674d
Summary:
This allows doctor to construct changelog without requiring valid visibleheads
data. doctor cannot fix visibleheads first, because fixing visibleheads requires
changelog.
Reviewed By: markbt
Differential Revision: D30974288
fbshipit-source-id: 5bcf0f1918809fc0c7db3c89c70e0d17f961dc2c
Summary:
To make metalog replace more features supported by the transaction framework,
there is a need to expose pending metalog states to sub-processes. This diff
makes it so.
Reviewed By: markbt
Differential Revision: D30970502
fbshipit-source-id: 84192a14f4cef0765e4e361b61ab630311fd2dff
Summary: .drain() retains the drained container and its heap allocation for reuse, but as we're not reusing the container, moving the contents into_iter() makes the intent clearer
Reviewed By: StanislavGlebik
Differential Revision: D31149817
fbshipit-source-id: 07cc8b7cabc9b1d522daee8b13cfa6eeb96e2d30
Summary: .drain() retains the drained container and its heap allocation for reuse, but as we're not reusing the container, moving the contents into_iter() by for..in makes the intent clearer.
Reviewed By: StanislavGlebik
Differential Revision: D31149816
fbshipit-source-id: 63c7bba8a457e62a37944aecd8ec8c42dac8deaa
Summary:
No need to keep the Bytes live after compact_protocol::deserialize, can move them into it instead.
Makes it clearer bytes aren't reused, and should have some small effect on peak memory usage during deserialization by freeing the Bytes buffer earlier
Reviewed By: StanislavGlebik
Differential Revision: D31149815
fbshipit-source-id: 858914d2d8e3d91b5e863053dfeeb5d5ec37b9eb
Summary:
optimization for edenapi upload
Lookup for filenodes and trees can be done in parallel. Usually we have a small number of trees to check and a small number of filenodes, it is better to send them in a single lookup request, so they all can be checked in parallel. Parallelism limit for the lookup request is few thousands, so almost always if we merge the requests here, they will be parallelised.
Reviewed By: yancouto
Differential Revision: D31127401
fbshipit-source-id: 8014b27a2ba9d082babe2e0cd7bebf43c8b46082
Summary:
add scuba metrics for stages of EdenApi Uploads
add cloud sync reason for manual run
This is an effort to improve our Eden Api Uploads metrics and Commit Cloud metrics, so we can analyse and improve its performance.
Reviewed By: markbt
Differential Revision: D31109948
fbshipit-source-id: ee5a449e2652ea1798997ae2c52c4672f55e3eae
Summary:
I recently added this feature but it had a bug - when DontAggregate mode was
used it compared file changes of a new commit with the previous commit only
instead of all changes in the stack.
Since FileAggregation is broken let's remove it and collect both file changes
for the whole stack and for a given commit
Reviewed By: mitrandir77
Differential Revision: D31145055
fbshipit-source-id: 99dbedb919fb9edbdfaeaa658d49a08d008bd282
Summary:
The `ObjectFetchContext::Origin::FromBackingStore` is widely interpreted as
meaning that a network fetch was performed, but for some backing stores, this
isn't true. The Mercurial backing store for instance can either read data from
its on-disk cache, or from the network. Since both have very different
characteristics we shouldn't bundle them in the same enum value.
Since the backing store knows how data was obtained, let's have the backing
store return how it was obtained to enable the ObjectStore to properly record
this information. The `FromBackingStore` is also renamed to make it clearer
what its purpose is.
Reviewed By: zhengchaol
Differential Revision: D31118906
fbshipit-source-id: ee42a0c9d221f870742de07c0df7c732bc79d880
Summary:
we are passing some bytes into Popen and shlex.quote. shlex.quote expects a
string not bytes. fsencode gives us bytes fsdecode gives us string. Let's used
fsdecode instead.
Reviewed By: zhengchaol
Differential Revision: D31129335
fbshipit-source-id: 7792bdcd4dd833a4946daf8ec75576cfe4fc24af
Summary:
Derived data manager now doesn't allow deriving a batch of commits if all
ancestors weren't derived yet (and that's a good idea to do this check).
But it started to break benchmark if --batch-size, --backfill and --parallel
options are set, because in the very
beginning of the function we mark all commits as not derived, and when we start
deriving the second batch the first batch is assumed to not be derived, and
this triggers derived data manager check.
Let's instead mark only commits that we are about to derive as not derived, and
clear this check once we are done.
Reviewed By: mitrandir77
Differential Revision: D31140464
fbshipit-source-id: fc74d58dc3c4a3ad70e8e2527f7d6dfc8fde8a9c
Summary:
I'd like to reuse them in the next diff, so let's refactor it a bit.
Note - in D30837581 (315a8b311d) markbt suggested a good idea for refactoring
backfill_derived_data. I liked the idea, but when I tried to approach this
refactoring it turned out to be tricky to do so (FWIW, it might be easier to
rewrite everything from scratch). So for now I did the smallest possible
refactoring that's needed to add validation in the next diff, but this small
refactoring can probably be used for a larger refactoring later.
Reviewed By: mitrandir77
Differential Revision: D31115979
fbshipit-source-id: f0b4d70454186a023cd9e12cd645768af1b716e8
Summary:
I'd like to use it in the next diffs to add a way to validate that derived data
is the same after rederivation. A lot of the code in `benchmark.rs` is useful
for doing this validation, so let's rename `benchmark.rs` so that it's ok to
use it from two different subcommands.
Reviewed By: mitrandir77
Differential Revision: D31115981
fbshipit-source-id: 86439534d8e49a4022086cb27918b7bcd7befc5c
Summary: Share the megarepo api object as it contains repo configs which are quite large
Reviewed By: StanislavGlebik
Differential Revision: D31144245
fbshipit-source-id: bc87bac8d51d255d3f0a2d6dfbacf97ec89a565d
Summary:
We update bookmarks and insert sync mappings in separate transactions. Let's
adjust for that (see comment for details)
Reviewed By: StanislavGlebik
Differential Revision: D31113203
fbshipit-source-id: 95264bac0dfffee3fef82d475ccf7c4a62b81f24
Summary:
This tests legacy bookmark pull behavior which we no longer use. Let's
delete it.
Reviewed By: quark-zju
Differential Revision: D31032053
fbshipit-source-id: c1ab3b2e17a81871eae377f02947fecc438d7209
Summary:
This test tests a lot of legacy repo formats and a bunch of
peer-to-peer clones/pushes, etc which are no longer supported. Let's delete it.
Reviewed By: quark-zju
Differential Revision: D31032056
fbshipit-source-id: bc3ed9226e1a2d4656630ea5f99a904673268da6
Summary:
This test is fragile with respect to .hg store changes. We don't have
strong requirements of .hg permissions these days since we don't share repos
between people really. So Let's delete this test.
Reviewed By: quark-zju
Differential Revision: D31032058
fbshipit-source-id: 2d6b1ffed064ad41fda8938cfabc64291f509015
Summary:
This tests details of the getbundle wireproto command. This is becoming
less and less relevant as we move to edenapi, and the test itself is basically
impossible to maintain since it's heavily dependent on hashes. Let's just delete
it.
Reviewed By: quark-zju
Differential Revision: D31032062
fbshipit-source-id: 31f007ab989ebcd93424f2aea09168edc368a6c0
Summary:
Previously newserver/clone supported ssh and mononoke. Now let's
support eagerrepo as well
Reviewed By: quark-zju
Differential Revision: D31032061
fbshipit-source-id: 5016bb74b317523611c645c040da92c8cbf897c1
Summary:
This is part of removing filepeer. I also enabled treemanifest and
modernclient (i.e. lazy changelog) on a few tests.
Reviewed By: quark-zju
Differential Revision: D31032060
fbshipit-source-id: cb45831892a6c75d89a553c23f3318e188f9489f
Summary:
Removes the tests usage of server logic, by migrating it to
modernclient.
Reviewed By: quark-zju
Differential Revision: D31032057
fbshipit-source-id: d7f5d2cf4fae4d0edd364ad435d17e4083c3297f
Summary:
Removes usage of server logic from test-bookmarks.t.
This test is long and convoluted. I decided that the latter half of the test
wasn't really relevant/important to our usage of Mercurial so I deleted it
instead of trying to fix it.
Reviewed By: quark-zju
Differential Revision: D31032059
fbshipit-source-id: ac32d8179eb1caaa6383015ec3f6f67b2fc9106e
Summary:
We want to get rid of server logic in Mercurial, and to do so we want
to get rid of filepeer. To make porting legacy tests easier, let's create some
helper functions to simulate clones, pushes, and pulls.
Reviewed By: quark-zju
Differential Revision: D31004444
fbshipit-source-id: def6fc4fc3d43c313eb4f2a6b0adcba7ddb5e00f
Summary:
We want to remove filepeer as part of removing server logic in the
client. To start with, let's disable it by default and only enable it in tests
that need it. The next step will be to update those tests.
Reviewed By: quark-zju
Differential Revision: D30977765
fbshipit-source-id: c56016f017e894a15bf43fb7a8d3a0a417663ad9
Summary:
The duality of this function is a bit awkward, especially for backing stores
that want metadata caching, but not blob caching. This makes the code in
ObjectStore more complicated that it needs to be.
This will also be used in a future diff.
Reviewed By: chadaustin
Differential Revision: D31090782
fbshipit-source-id: cb7d7fc44d8780f86abad166d0f099675d29e5e7
Summary:
Similarly to blobs, fetching trees is converting data from HgImportRequest to
individual hashes and proxy hashes vector, making copies of these. This is
inefficient and it makes the code harder to read and understand. Passing the
batch of HgImportRequest directly avoid these copies and makes the code easier
to read.
Reviewed By: fanzeyi
Differential Revision: D30583567
fbshipit-source-id: e85952975141c92f9524095c62418baabf8fefcd
Summary:
Both the caller and the function itself are copying data to make sure that the
next function gets the data in the format it desires. This makes the code
complicated to read as well as inefficient as each conversion ends up being a
copy.
We can simply pass the HgImportRequest directly to avoid both of these issues
which removes all the copies.
A similar change will be done for the getTreeBatch function, after which the
HgDatapackStore will be folded onto the HgBackingStore code.
Reviewed By: fanzeyi
Differential Revision: D30563706
fbshipit-source-id: bb392f89e691c22ff9ad4df0d365ddb62077e657
Summary:
We can configure chaos blobstore and a bunch of other options. But sometimes
it's useful to be able to add an artificial delay to the blobstore.
Reviewed By: mitrandir77
Differential Revision: D31110811
fbshipit-source-id: 7b3a4d1b85047a5daff9183df8aa5db8971a7d9d
Summary:
The test does not discover new issues but it seems nice to have protecting
behaviors.
Reviewed By: StanislavGlebik
Differential Revision: D31106124
fbshipit-source-id: 4060b16f2dac34ff29c0d4cee1ce19a9a61014ee
Summary:
There are 2 issues:
- Previously it only follows parent -> child relations when parent is the head
of a segment. It's also possible that we need to follow parent -> child when
parent is in the middle of a flat segment. The added ASCII graph in
`iddag.rs` shows when that happens.
- Previously there is no check about following p2 (second parent) for linear
history. The check needs to be added and p2 of merges cannot be resolved.
This change makes `x~0` result an error. Some `x~n` are changed to alternative
ones because more child segments are visited (see also D30919581 (3ef445c0a6)).
Reviewed By: StanislavGlebik
Differential Revision: D30919579
fbshipit-source-id: eaa69d31612a2006b0b46132a797025536288624
Summary:
The endpoint "hash to location" cannot really expose individual errors
correctly. The type `CommitHashToLocationResponse` allows exposing individual
errors for each hgid but the implementation will just take one error and report
such error for all hgids. This is because the use of
`collect::<Result<ContainerType>>()?` collapses multiple individual "ok" and "err"
values into one "err".
Change the stack so we can preserve individual errors.
Reviewed By: StanislavGlebik
Differential Revision: D31106125
fbshipit-source-id: 46b1e8b2fcee00b8d8dda0935e2a005ee8aef0ee
Summary:
Those two commands allow us to easily make another job pick-up unfinished
request and finish a request with error.
aborting will actually make the worker stop working on the requests the next
time worker tries to update the heartbeat.
Reviewed By: StanislavGlebik
Differential Revision: D30965078
fbshipit-source-id: 5b683ea09c7dd74ff903416f2a1d393f9b55c588
Summary: Fix bug in httpenabled check. Only local repo has edenapi attribute and not remote.
Reviewed By: quark-zju
Differential Revision: D31059979
fbshipit-source-id: c77f1455949f28b081f8fef86aeb83ecd1c3ceba
Summary: Move conditionals for using edenapi commit graph endpoint to a common method. This fixes the conditional check in _pullchangeset which should match the one in pull.
Reviewed By: quark-zju
Differential Revision: D31030783
fbshipit-source-id: e665067bcf12bc8211850d547df42bc48727de48
Summary: The url for commit graph on the server is repo/commit/graph and repo/commit/commit_graph on the client. Update client to match server url.
Reviewed By: quark-zju
Differential Revision: D31030784
fbshipit-source-id: fb58c53a9264498f0e488abadd5f2b5480f0c672
Summary:
The `unfiltered` field was to support legacy clients that are months old. Drop
support for it to clean the code a bit.
Reviewed By: StanislavGlebik
Differential Revision: D31106176
fbshipit-source-id: 7cfd6827812258776d6d9eac512b497777fecb70
Summary:
Previously it just passes "failed to compute the common descendant and distance
for X", the outmost layer (context) of the error to the client. Make it contain
more messages about the inner error and heads explicitly.
Reviewed By: StanislavGlebik
Differential Revision: D31071741
fbshipit-source-id: ec4f161491e14ae0e0c422f5e92a3849a5a29b56
