Summary:
add scuba metrics for stages of EdenApi Uploads
add cloud sync reason for manual run
This is an effort to improve our Eden Api Uploads metrics and Commit Cloud metrics, so we can analyse and improve its performance.
Reviewed By: markbt
Differential Revision: D31109948
fbshipit-source-id: ee5a449e2652ea1798997ae2c52c4672f55e3eae
Summary:
I recently added this feature but it had a bug - when DontAggregate mode was
used it compared file changes of a new commit with the previous commit only
instead of all changes in the stack.
Since FileAggregation is broken let's remove it and collect both file changes
for the whole stack and for a given commit
Reviewed By: mitrandir77
Differential Revision: D31145055
fbshipit-source-id: 99dbedb919fb9edbdfaeaa658d49a08d008bd282
Summary:
The `ObjectFetchContext::Origin::FromBackingStore` is widely interpreted as
meaning that a network fetch was performed, but for some backing stores, this
isn't true. The Mercurial backing store for instance can either read data from
its on-disk cache, or from the network. Since both have very different
characteristics we shouldn't bundle them in the same enum value.
Since the backing store knows how data was obtained, let's have the backing
store return how it was obtained to enable the ObjectStore to properly record
this information. The `FromBackingStore` is also renamed to make it clearer
what its purpose is.
Reviewed By: zhengchaol
Differential Revision: D31118906
fbshipit-source-id: ee42a0c9d221f870742de07c0df7c732bc79d880
Summary:
we are passing some bytes into Popen and shlex.quote. shlex.quote expects a
string not bytes. fsencode gives us bytes fsdecode gives us string. Let's used
fsdecode instead.
Reviewed By: zhengchaol
Differential Revision: D31129335
fbshipit-source-id: 7792bdcd4dd833a4946daf8ec75576cfe4fc24af
Summary:
Derived data manager now doesn't allow deriving a batch of commits if all
ancestors weren't derived yet (and that's a good idea to do this check).
But it started to break benchmark if --batch-size, --backfill and --parallel
options are set, because in the very
beginning of the function we mark all commits as not derived, and when we start
deriving the second batch the first batch is assumed to not be derived, and
this triggers derived data manager check.
Let's instead mark only commits that we are about to derive as not derived, and
clear this check once we are done.
Reviewed By: mitrandir77
Differential Revision: D31140464
fbshipit-source-id: fc74d58dc3c4a3ad70e8e2527f7d6dfc8fde8a9c
Summary:
I'd like to reuse them in the next diff, so let's refactor it a bit.
Note - in D30837581 (315a8b311d) markbt suggested a good idea for refactoring
backfill_derived_data. I liked the idea, but when I tried to approach this
refactoring it turned out to be tricky to do so (FWIW, it might be easier to
rewrite everything from scratch). So for now I did the smallest possible
refactoring that's needed to add validation in the next diff, but this small
refactoring can probably be used for a larger refactoring later.
Reviewed By: mitrandir77
Differential Revision: D31115979
fbshipit-source-id: f0b4d70454186a023cd9e12cd645768af1b716e8
Summary:
I'd like to use it in the next diffs to add a way to validate that derived data
is the same after rederivation. A lot of the code in `benchmark.rs` is useful
for doing this validation, so let's rename `benchmark.rs` so that it's ok to
use it from two different subcommands.
Reviewed By: mitrandir77
Differential Revision: D31115981
fbshipit-source-id: 86439534d8e49a4022086cb27918b7bcd7befc5c
Summary: Share the megarepo api object as it contains repo configs which are quite large
Reviewed By: StanislavGlebik
Differential Revision: D31144245
fbshipit-source-id: bc87bac8d51d255d3f0a2d6dfbacf97ec89a565d
Summary:
We update bookmarks and insert sync mappings in separate transactions. Let's
adjust for that (see comment for details)
Reviewed By: StanislavGlebik
Differential Revision: D31113203
fbshipit-source-id: 95264bac0dfffee3fef82d475ccf7c4a62b81f24
Summary:
This tests legacy bookmark pull behavior which we no longer use. Let's
delete it.
Reviewed By: quark-zju
Differential Revision: D31032053
fbshipit-source-id: c1ab3b2e17a81871eae377f02947fecc438d7209
Summary:
This test tests a lot of legacy repo formats and a bunch of
peer-to-peer clones/pushes, etc which are no longer supported. Let's delete it.
Reviewed By: quark-zju
Differential Revision: D31032056
fbshipit-source-id: bc3ed9226e1a2d4656630ea5f99a904673268da6
Summary:
This test is fragile with respect to .hg store changes. We don't have
strong requirements of .hg permissions these days since we don't share repos
between people really. So Let's delete this test.
Reviewed By: quark-zju
Differential Revision: D31032058
fbshipit-source-id: 2d6b1ffed064ad41fda8938cfabc64291f509015
Summary:
This tests details of the getbundle wireproto command. This is becoming
less and less relevant as we move to edenapi, and the test itself is basically
impossible to maintain since it's heavily dependent on hashes. Let's just delete
it.
Reviewed By: quark-zju
Differential Revision: D31032062
fbshipit-source-id: 31f007ab989ebcd93424f2aea09168edc368a6c0
Summary:
Previously newserver/clone supported ssh and mononoke. Now let's
support eagerrepo as well
Reviewed By: quark-zju
Differential Revision: D31032061
fbshipit-source-id: 5016bb74b317523611c645c040da92c8cbf897c1
Summary:
This is part of removing filepeer. I also enabled treemanifest and
modernclient (i.e. lazy changelog) on a few tests.
Reviewed By: quark-zju
Differential Revision: D31032060
fbshipit-source-id: cb45831892a6c75d89a553c23f3318e188f9489f
Summary:
Removes the tests usage of server logic, by migrating it to
modernclient.
Reviewed By: quark-zju
Differential Revision: D31032057
fbshipit-source-id: d7f5d2cf4fae4d0edd364ad435d17e4083c3297f
Summary:
Removes usage of server logic from test-bookmarks.t.
This test is long and convoluted. I decided that the latter half of the test
wasn't really relevant/important to our usage of Mercurial so I deleted it
instead of trying to fix it.
Reviewed By: quark-zju
Differential Revision: D31032059
fbshipit-source-id: ac32d8179eb1caaa6383015ec3f6f67b2fc9106e
Summary:
We want to get rid of server logic in Mercurial, and to do so we want
to get rid of filepeer. To make porting legacy tests easier, let's create some
helper functions to simulate clones, pushes, and pulls.
Reviewed By: quark-zju
Differential Revision: D31004444
fbshipit-source-id: def6fc4fc3d43c313eb4f2a6b0adcba7ddb5e00f
Summary:
We want to remove filepeer as part of removing server logic in the
client. To start with, let's disable it by default and only enable it in tests
that need it. The next step will be to update those tests.
Reviewed By: quark-zju
Differential Revision: D30977765
fbshipit-source-id: c56016f017e894a15bf43fb7a8d3a0a417663ad9
Summary:
The duality of this function is a bit awkward, especially for backing stores
that want metadata caching, but not blob caching. This makes the code in
ObjectStore more complicated that it needs to be.
This will also be used in a future diff.
Reviewed By: chadaustin
Differential Revision: D31090782
fbshipit-source-id: cb7d7fc44d8780f86abad166d0f099675d29e5e7
Summary:
Similarly to blobs, fetching trees is converting data from HgImportRequest to
individual hashes and proxy hashes vector, making copies of these. This is
inefficient and it makes the code harder to read and understand. Passing the
batch of HgImportRequest directly avoid these copies and makes the code easier
to read.
Reviewed By: fanzeyi
Differential Revision: D30583567
fbshipit-source-id: e85952975141c92f9524095c62418baabf8fefcd
Summary:
Both the caller and the function itself are copying data to make sure that the
next function gets the data in the format it desires. This makes the code
complicated to read as well as inefficient as each conversion ends up being a
copy.
We can simply pass the HgImportRequest directly to avoid both of these issues
which removes all the copies.
A similar change will be done for the getTreeBatch function, after which the
HgDatapackStore will be folded onto the HgBackingStore code.
Reviewed By: fanzeyi
Differential Revision: D30563706
fbshipit-source-id: bb392f89e691c22ff9ad4df0d365ddb62077e657
Summary:
We can configure chaos blobstore and a bunch of other options. But sometimes
it's useful to be able to add an artificial delay to the blobstore.
Reviewed By: mitrandir77
Differential Revision: D31110811
fbshipit-source-id: 7b3a4d1b85047a5daff9183df8aa5db8971a7d9d
Summary:
The test does not discover new issues but it seems nice to have protecting
behaviors.
Reviewed By: StanislavGlebik
Differential Revision: D31106124
fbshipit-source-id: 4060b16f2dac34ff29c0d4cee1ce19a9a61014ee
Summary:
There are 2 issues:
- Previously it only follows parent -> child relations when parent is the head
of a segment. It's also possible that we need to follow parent -> child when
parent is in the middle of a flat segment. The added ASCII graph in
`iddag.rs` shows when that happens.
- Previously there is no check about following p2 (second parent) for linear
history. The check needs to be added and p2 of merges cannot be resolved.
This change makes `x~0` result an error. Some `x~n` are changed to alternative
ones because more child segments are visited (see also D30919581 (3ef445c0a6)).
Reviewed By: StanislavGlebik
Differential Revision: D30919579
fbshipit-source-id: eaa69d31612a2006b0b46132a797025536288624
Summary:
The endpoint "hash to location" cannot really expose individual errors
correctly. The type `CommitHashToLocationResponse` allows exposing individual
errors for each hgid but the implementation will just take one error and report
such error for all hgids. This is because the use of
`collect::<Result<ContainerType>>()?` collapses multiple individual "ok" and "err"
values into one "err".
Change the stack so we can preserve individual errors.
Reviewed By: StanislavGlebik
Differential Revision: D31106125
fbshipit-source-id: 46b1e8b2fcee00b8d8dda0935e2a005ee8aef0ee
Summary:
Those two commands allow us to easily make another job pick-up unfinished
request and finish a request with error.
aborting will actually make the worker stop working on the requests the next
time worker tries to update the heartbeat.
Reviewed By: StanislavGlebik
Differential Revision: D30965078
fbshipit-source-id: 5b683ea09c7dd74ff903416f2a1d393f9b55c588
Summary: Fix bug in httpenabled check. Only local repo has edenapi attribute and not remote.
Reviewed By: quark-zju
Differential Revision: D31059979
fbshipit-source-id: c77f1455949f28b081f8fef86aeb83ecd1c3ceba
Summary: Move conditionals for using edenapi commit graph endpoint to a common method. This fixes the conditional check in _pullchangeset which should match the one in pull.
Reviewed By: quark-zju
Differential Revision: D31030783
fbshipit-source-id: e665067bcf12bc8211850d547df42bc48727de48
Summary: The url for commit graph on the server is repo/commit/graph and repo/commit/commit_graph on the client. Update client to match server url.
Reviewed By: quark-zju
Differential Revision: D31030784
fbshipit-source-id: fb58c53a9264498f0e488abadd5f2b5480f0c672
Summary:
The `unfiltered` field was to support legacy clients that are months old. Drop
support for it to clean the code a bit.
Reviewed By: StanislavGlebik
Differential Revision: D31106176
fbshipit-source-id: 7cfd6827812258776d6d9eac512b497777fecb70
Summary:
Previously it just passes "failed to compute the common descendant and distance
for X", the outmost layer (context) of the error to the client. Make it contain
more messages about the inner error and heads explicitly.
Reviewed By: StanislavGlebik
Differential Revision: D31071741
fbshipit-source-id: ec4f161491e14ae0e0c422f5e92a3849a5a29b56
Summary: This will be used by upcoming changes.
Reviewed By: DurhamG
Differential Revision: D30919581
fbshipit-source-id: 5e4c13004c577d4a87948c1985bdf48ebd1379a3
Summary: This method will be used in the next change.
Reviewed By: DurhamG
Differential Revision: D30919580
fbshipit-source-id: a5cab7fad0dda0de15244a9d7c1887534f061f3a
Summary:
After D26360561 (aa9dfeff2e), `IdDag<Store>` derefs to `Store` for algorithms. There is no
need to duplicate the methods in `IdDag<Store>`.
Reviewed By: yancouto
Differential Revision: D30911148
fbshipit-source-id: d476e318ab565369b93f4bd8467989a2a2e76f82
Summary:
Added checks so the `x` in the returned `x~n` is a parent of a merge that is an
ancestor of requested heads.
Reviewed By: DurhamG
Differential Revision: D30911149
fbshipit-source-id: 09fd6a01ea346a53c8d2ff6042951c8094cb8b04
Summary:
After migration to Mononoke & EdenAPI supporting different destinations don't make sense.
The destination is always either default path or edenapi endpoint.
Reviewed By: yancouto
Differential Revision: D31081193
fbshipit-source-id: 49a02899173690cfcb55bf8fc6aec13b05377ca6
Summary: It used to be flaky. Perhaps ncat exited by itself in some cases so there was nothing to kill. Just ignore the kill command's output. As long as we get the cats header the test is fine.
Reviewed By: HarveyHunt
Differential Revision: D31108514
fbshipit-source-id: e2791610c49bacdc31c26710f087cbe49f4d734e
Summary: It can be useful if we want to rebase a lot of commits at once
Reviewed By: mitrandir77
Differential Revision: D31106975
fbshipit-source-id: 26bb7f3fd166b483c44a4991ede4f0a9a462fff7
Summary: Follow the D30313559, this diff will enable ```download_directory``` with eden option
Reviewed By: Skory
Differential Revision: D30549600
fbshipit-source-id: f87326bd7c17abd975eab823c62ee0ffe3bf7443
Summary: Add support for fetching file aux data from BackingStore via scmstore.
Reviewed By: andll
Differential Revision: D31048574
fbshipit-source-id: ccb783272d0ffe32277debdaac45c00fc4df9ddb
Summary:
We've never really supported this and it has caused weird behavior in
the past. Let's delete it.
Reviewed By: quark-zju
Differential Revision: D31027574
fbshipit-source-id: 297668e35db3c4d5ea17d2a18fc0127ca694cc45
Summary: This is no longer used, now that configerator does it's sync via SCS.
Reviewed By: quark-zju
Differential Revision: D31006839
fbshipit-source-id: ddf7923a3c80ad47519bce0e5cb3895f62e7ae2e
Summary:
The server_realhostname log field no longer contains just the server hostname,
but instead was changed to the format `HOSTNAME session SESSIONID`, so that the
client knows the server session id.
Redefine the response format to `HOSTNAME (INFOKEY INFOVALUE)*`, so that
session is just an info field from the peer, and parse the response and log its
components.
Reviewed By: quark-zju
Differential Revision: D31081922
fbshipit-source-id: 28f963b4a44f7e9bdd75393ac9b14a2d1a4b5bd0
Summary:
Create rust progress bar model in python whenever any bar is created, as opposed to only when the bar is rendered. This makes progress metadata always available in rust, regardless of whether progress ends up getting displayed or not. In particular, this enables upcoming "global progress" work to generically expose all hg invocations' progress.
This change necessitated rust to respect more cases where progress should not be displayed. For example, "quiet" mode was previously only respected in python by creating a type of bar that did not end up being registered in rust. Now since all bars are registered, rust needs to know not to render output in this case and others.
Note that I discovered a config discrepancy between python and rust in the norepo=True command mode. Work around for now by changing debugprogress to optionalrepo.
Reviewed By: quark-zju
Differential Revision: D31005075
fbshipit-source-id: 0fa6fb594a15b34b3ae260343526496f8f8b05f8
Summary:
Add rust utility function to test for HGPLAIN and HGPLAINEXCEPT, analagous to python's ui.plain() method.
I need this so rust can expand its logic of whether to render progress bars or not.
Reviewed By: quark-zju
Differential Revision: D31070840
fbshipit-source-id: 0c394234b9c4337bbddaa08d7a9b1d8ed59dfec0
