Summary:
Readdir tries to be smart and prefetch the metadata for each of the children.
But this uses the old path to read metadata for files which can cause eden to
download the blob. When metadata prefetching is turned on in the backing store
it is better to leave this to metadata prefetching to the backing store.
Reviewed By: wez
Differential Revision: D23476876
fbshipit-source-id: 41cc5e6f423f19adb18581564c069c12621b6c1b
Summary:
This diff fixes a bug on Windows when the redirection target is a non-empty directory. As seen in P141872812
This doesn't make the exception go away but generate a more meaningful error message so the user can act on it.
Reviewed By: xavierd
Differential Revision: D23605233
fbshipit-source-id: 2d2bde0e9cd94323a6537ebcec29a4c15868806d
Summary: Once we start to move the bookmark for the large repo commits, small repo commits should also start to appear for the dependent systems (e.g. Phabricator) through back-syncing. This diff adds this functionality to see if the commits have been recognised by the tools.
Reviewed By: StanislavGlebik
Differential Revision: D23566994
fbshipit-source-id: 2f6f3b9099bb864fec6a488064abe1abe7f06813
Summary: Useful when looking into blobstore corruption - you can compare all the blobstore versions by manual fetchees.
Reviewed By: krallin
Differential Revision: D23604436
fbshipit-source-id: 7b56947b0188536499514bae6615c6e81b9106c3
Summary: Going to add more features, so simplify by asyncifying first
Reviewed By: krallin
Differential Revision: D23604437
fbshipit-source-id: 52b2b372e4d3fbf1d59168c6c11311d9edf4ff0f
Summary: When we're scrubbing blobstores, it's not actually a success state if a scrub fails to write. Report this back to the caller - no-one will usually be scrubbing unless they expect repair writes to succeed, and a failure is a sign that we need to investigate further
Reviewed By: mitrandir77
Differential Revision: D23601541
fbshipit-source-id: d328935af9999c944719a6b863d0c86b28c54f59
Summary:
One test was fixed earlier by switching MacOS to use modern version of bash, the other is fixed here by installing "nmap" and using "ncat" from within it on both linux and mac.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/50
Reviewed By: krallin
Differential Revision: D23599695
Pulled By: lukaspiatkowski
fbshipit-source-id: e2736cee62e82d1e9da6eaf16ef0f2c65d3d8930
Summary:
Fixes a few issues with Mononoke tests in Python 3.
1. We need to use different APIs to account for the unicode vs bytes difference
for path hash encoding.
2. We need to set the language environment for tests that create utf8 file
paths.
3. We need the redaction message and marker to be bytes. Oddly this test still
fails with jq CLI errors, but it makes it past the original error.
Reviewed By: quark-zju
Differential Revision: D23582976
fbshipit-source-id: 44959903aedc5dc9c492ec09a17b9c8e3bdf9457
Summary:
For repositories that have the old-style LFS extension enabled, the pointers
are stored in packfiles/indexedlog alongside with a flag that signify to the
upper layers that the blob is externally stored. With the new way of doing LFS,
pointers are stored separately.
When both are enabled, we are observing some interesting behavior where
different get and get_meta calls may return different blobs/metadata for the
same filenode. This may happen if a filenode is stored in both a packfile as an
LFS pointers, and in the LFS store. Guaranteeing that the revisionstore code is
deterministic in this situation is unfortunately way too costly (a get_meta
call would for instance have to fully validate the sha256 of the blob, and this
wouldn't guarantee that it wouldn't become corrupted on disk before calling
get).
The solution take here is to simply ignore all the lfs pointers from
packfiles/indexedlog when remotefilelog.lfs is enabled. This way, there is no
risk of reading the metadata from the packfiles, and the blob from the
LFSStore. This brings however another complication for the user created blobs:
these are stored in packfiles and would thus become unreadable, the solution is
to simply perform a one-time full repack of the local store to make sure that
all the pointers are moved from the packfiles to to LFSStore.
In the code, the Python bindings are using ExtStoredPolicy::Ignore directly as
these are only used in the treemanifest code where no LFS pointers should be
present, the repack code uses ExtStoredPolicy::Use to be able to read the
pointers, it wouldn't be able to otherwise.
Reviewed By: DurhamG
Differential Revision: D22951598
fbshipit-source-id: 0e929708ba5a3bb2a02c0891fd62dae1ccf18204
Summary:
hg-http's built client should provide integration with Mercurial's stats
collection mechanisms.
Reviewed By: kulshrax
Differential Revision: D23577867
fbshipit-source-id: 93c777021bc347511322269d678d6879710eed3e
Summary:
Add `with_stats_reporting` to HttpClient. It takes a closure that will be
called with all `Stats` objects generated. We then use this function in
the hg-http crate to integrate with the metrics backend used in Mercurial.
Reviewed By: kulshrax
Differential Revision: D23577869
fbshipit-source-id: 5ac23f00183f3c3d956627a869393cd4b27610d4
Summary: Rust based metrics so that even Rust libraries can write metrics.
Reviewed By: quark-zju
Differential Revision: D23577870
fbshipit-source-id: b19904968d9372c8ce19775fb37c7af53a370ea5
Summary:
We start off simple here. Python only really has counters so we only implement
counters. There are a lot of options on how to improve this and things get
slightly complicated when we look at the how ecosystem and fb303. Anyway,
simple start.
Reviewed By: quark-zju
Differential Revision: D23577874
fbshipit-source-id: d50f5b2ba302d900b254200308bff7446121ae1d
Summary:
Slash is probably the standard metric delimiter nowadays. Since we don't have
that many metrics I think that it makes sense to look at slash as the
standard metric delimiter going forward.
This diff updates parsing of metric names to treat both '_' and '/' as
delimiters.
Reviewed By: quark-zju
Differential Revision: D23577876
fbshipit-source-id: 03997b1285df9c52d6e2837b5af5372deb69b133
Summary:
The command is easier to use than `hg cloud join --switch`.
Also highlight the workspace name in the output of `hg cloud status`
Reviewed By: mitrandir77
Differential Revision: D23601507
fbshipit-source-id: 74eb17c9366a9dbe96881c8e3e0705619fadb3d6
Summary:
typed_hash only implements serialize. Because of this, if we want to serialize a struct that contains e.g changesetid(s), we can't deserialize it later. This diff adds deserialize implementation for typed_hashes.
Implementation is similar to HgNodeHash's: https://fburl.com/diffusion/r3df5iga
Reviewed By: krallin
Differential Revision: D23598925
fbshipit-source-id: 4d48b75eb8a01028e6e2d9bcc1ae20051a97b7fb
Summary:
Streaming clone implementation did not check that received files have the corrects. This change addresses it.
Before this change if connection was interrupted for whatever reason client would treat fetch of changeset as successful and proceed with cloning operations, but later checks would report corruption of internal state of hg data. This is based on user [report](https://fb.workplace.com/groups/scm/permalink/3177150312334567/)
Reviewed By: quark-zju, krallin
Differential Revision: D23572058
fbshipit-source-id: d740b45ca217cd6db0a65e01aabc2ba9a4835221
Summary: The Mercurial codebase uses hyphens in crate names rather than underscores. This is similar to the convention favored by the larger Rust community, though it is different from Mononoke, which uses underscores. While we'll probably need to eventually settle on a consistent convention for all of projects in the Eden SCM repo, for now, `http_client` should be made consistent with the adjacent crates.
Reviewed By: sfilipco
Differential Revision: D23585721
fbshipit-source-id: d2e690d86815be02d7b8d645198bcd28e8cbd6e0
Summary:
While it's unlikely to work properly (it uses /bin/sh), it compiles properly on
Windows, so let's include it in the build.
Reviewed By: wez
Differential Revision: D23520368
fbshipit-source-id: 267ba04f98f5dacc81e1772f86f5ad43c846815d
Summary:
We are using older version of tokio which spawns as many threads as we have
physical cores instead of the number of logical cores. It was fixed in
https://github.com/tokio-rs/tokio/issues/2269 but we can't use it yet because
we are waiting for another fix to be released -
https://github.com/rust-lang/futures-rs/pull/2154.
For now let's hardcode it in mononoke
Reviewed By: krallin
Differential Revision: D23599140
fbshipit-source-id: 80685651a7a29ba8938d9aa59770f191f7c42b8b
Summary:
This change move logic associated with mercurial changeset derivation to `mercurial_derived_data` crate.
NOTE: it is not converted to derived data infrastructure at this point, it is a preparation step to actually do this
Reviewed By: farnz
Differential Revision: D23573610
fbshipit-source-id: 6e8cbf7d53ab5dbd39d5bf5e06c3f0fc5a8305c8
Summary:
The Mac integration test workflow already installs a modern curl that fixes https://github.com/curl/curl/issues/4801, but it does so after "hg" is built, so "hg" uses the system curl libraries, which fails when used with a certificate not present in keychain.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/53
Reviewed By: krallin
Differential Revision: D23597285
Pulled By: lukaspiatkowski
fbshipit-source-id: a7b8b6ae55ce338bfb9946a852cbb6b929e73203
Summary:
There are blobs that fail to scrub and terminate the process early for a variety of reasons; when this is running as a background task, it'd be nice to get the remaining keys scrubbed, so that you don't have a large number of keys to fix up later.
Instead of simply outputting to stdout, write keys to one of three files in the format accepted on stdin:
1. Success; you can use `sort` and `comm -3` to remove these keys from the input dat, thus ensuring that you can continue scrubbing.
2. Missing; you can look at these keys to determine which blobs are genuinely lost from all blobstores, and fix up.
3. Error; these will need running through scrub again to determine what's broken.
Reviewed By: krallin
Differential Revision: D23574855
fbshipit-source-id: a613e93a38dc7c3465550963c3b1c757b7371a3b
Summary:
With three blobstores in play, we have issues working out exactly what's wrong during a manual scrub. Make the error handling better:
1. Manual scrub adds the key as context for the failure.
2. Scrub error groups blobstores by content, so that you can see which blobstore is most likely to be wrong.
Reviewed By: ahornby, krallin
Differential Revision: D23565906
fbshipit-source-id: a199e9f08c41b8e967d418bc4bc09cb586bbb94b
Summary:
Sorting bookmark names can be expensive for the MySQL server. As we
don't rely on the ordering of bookmark names when requesting all bookmarks,
remove the sorting.
I've not modified the `Select.*After` queries as they are used for pagination,
which does rely on the order of bookmark names. Further, any queries for
bookmarks that have a limit other than `std::u64::MAX` will remain sorted.
Reviewed By: ahornby
Differential Revision: D23574741
fbshipit-source-id: 79e07b64bb8bb34229c429bdf885c5144963f140
Summary:
The test-blobimport.t creates few files that are conflicting in a case insensitive file system, so make them differ by changing number of underscores in one of the files.
test-pushrebase-block-casefolding.t is directly testing a feature of case sensitive file system, so it cannot be really tested on MacOS
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/49
Reviewed By: farnz
Differential Revision: D23573165
Pulled By: lukaspiatkowski
fbshipit-source-id: fc16092d307005b6f0c8764c1ce80c81912c603b
Summary:
Previously we were not logging a redacted access if previous access was logged
less < MIN_REPORT_TIME_DIFFERENCE_NS ago. That doesn't work well with our
tests.
Let's instead add a sampling tunable.
Reviewed By: krallin
Differential Revision: D23595067
fbshipit-source-id: 47f6152945d9fdc2796fd1e74804e8bcf7f34940
Summary: Moving some of the functionality (which is required for mercurial changeset derivation) into a separate crate. This is required to convert mercurial changeset to derived data to avoid circular dependency it would create otherwise.
Reviewed By: StanislavGlebik
Differential Revision: D23566293
fbshipit-source-id: 9d30b4b3b7d8a922f72551aa5118c43104ef382c
Summary:
In some very rare scenario, we're seeing bind fail due to WSAEADDRINUSE, and
I'm suspecting that a connect race is going on, where connect creates the file
on disk while a bind is ongoing. Since at this point there is no way for
another EdenFS process to be alive due to the fs lock being held, there is no
way for the socket to be already bound to another EdenFS process. Thus, we can
remove the special case.
Reviewed By: genevievehelsel
Differential Revision: D23591009
fbshipit-source-id: 094a7926aa9ac45ea9dea5646000aa287a9b5b7a
Summary: No more tokio-core! More `async/await`.
Reviewed By: kulshrax
Differential Revision: D23586509
fbshipit-source-id: b2e766ddb7575bc96963432f0c8582b4370b19aa
Summary: These are now not working on mac due to the Catalina upgrade, so fixing here to be multi platform compatible by just following our same steps for path resolution.
Reviewed By: wez
Differential Revision: D23582015
fbshipit-source-id: e813986f5523bb721a4f01679559380c2348b98c
Summary: This fixes the `The program issued a command but the command length is incorrect` error we have been seeing in EdenFS Windows.
Reviewed By: wez
Differential Revision: D23589264
fbshipit-source-id: 72478a653ff07fca81c163bab12c0fd33f6a0a67
Summary:
This diff adds a `SocketTransport` implementation that no longer uses legacy `tokio-core` based futures but `tokio-tower` and `tower-service` for processing Thrift requests.
The old implementation is renamed to `SocketTransportLegacy` for better transitioning.
Reviewed By: dtolnay
Differential Revision: D20019196
fbshipit-source-id: 3bee684e9254bf1a81669ef0d2c2262a55e75daa
Summary: This is needed in a later diff that requires "codec" feature from `future-util`.
Reviewed By: dtolnay
Differential Revision: D23575630
fbshipit-source-id: e9cdf11b6ec05e5f2744da6b6efd8cb7bf08b212
Summary:
One of the difference between linux/macOS and Windows is that `edenfsctl` needs
to be used while `eden` works on the other platforms. This forces both users to
change their habit, and all the scripts at FB to be changed to take edenfsctl
into consideration.
Reviewed By: chadaustin
Differential Revision: D23550567
fbshipit-source-id: de2b0853137409e595a0012ca9286c37208b98a1
Summary:
This method is a future of synced-commit-mapping: there can be multiple query
results and we should make a decision of whether it is acceptable for the
business logic in the business logic, rather than pick a random one.
In later diffs I will introduce the consumers for this method.
Reviewed By: mitrandir77
Differential Revision: D23574165
fbshipit-source-id: f256f82c9848f54e5096c6e50d42600bfd260081
Summary:
Another preparatory step for the actuall mapping model fix. This just renames
`get` method into a `get_one` to emphasize it's use-case and to ease the search later.
At the end of this change, I expect there to be no use-cases for `get_one` and expect is to be gone.
Reviewed By: mitrandir77
Differential Revision: D23574116
fbshipit-source-id: f5015329b15f3f08961006607d0f9bf10f499a88
Summary: This is just preparatory extraction to make further work more convenient.
Reviewed By: mitrandir77
Differential Revision: D23574077
fbshipit-source-id: 352ca8ac62bae4fd8fcb980da05c95ce477a414e
