Summary:
Add a new `HandlerInfo` struct that can be inserted into the request's `State` by each handler to provide information about the handler to post-request middleware. (This includes things like which handler ran, which repo was queried, etc.)
A note on design: I did consider adding these fields to `RequestContext` (similar to how it's done in the LFS server), but that proved somewhat problematic in that it would require using interior mutability to avoid forcing each handler to borrow the `RequestContext` mutably (and therefore prevent it from borrowing other state data). The current approach seems preferable to adding a `Mutex` inside of `RequestContext`.
Reviewed By: krallin
Differential Revision: D23619077
fbshipit-source-id: 911806b60126d41e2132d1ca62ba863c929d4dc9
Summary:
The corpus rev that biggrep has indexed may not be available in the
local client. Later on in the function it will pull that revision, but earlier
in the function the new logic I added a few weeks ago is just crashing.
That logic was trying to diff against the earlier revision, but that's pretty
arbitrary. Let's just diff against one of the revs at random
(deterministically) and get rid of the need for the hash to exist in the repo
early in the command.
Reviewed By: sfilipco
Differential Revision: D23635801
fbshipit-source-id: 1c284d710b8df9539a696e900183bc10d5d71869
Summary:
The AccessCount fields were recently renamed to start with fsChannel instead of
fuse to be more platform independant. This broke edenfsctl due to it using the
old names.
Reviewed By: wez
Differential Revision: D23633574
fbshipit-source-id: 2a5fc73c47d2f0a6db407ecfeaf85992b7932c10
Summary:
Readdir tries to be smart and prefetch the metadata for each of the children.
But this uses the old path to read metadata for files which can cause eden to
download the blob. When metadata prefetching is turned on in the backing store
it is better to leave this to metadata prefetching to the backing store.
Reviewed By: wez
Differential Revision: D23476876
fbshipit-source-id: 41cc5e6f423f19adb18581564c069c12621b6c1b
Summary:
This diff fixes a bug on Windows when the redirection target is a non-empty directory. As seen in P141872812
This doesn't make the exception go away but generate a more meaningful error message so the user can act on it.
Reviewed By: xavierd
Differential Revision: D23605233
fbshipit-source-id: 2d2bde0e9cd94323a6537ebcec29a4c15868806d
Summary: Once we start to move the bookmark for the large repo commits, small repo commits should also start to appear for the dependent systems (e.g. Phabricator) through back-syncing. This diff adds this functionality to see if the commits have been recognised by the tools.
Reviewed By: StanislavGlebik
Differential Revision: D23566994
fbshipit-source-id: 2f6f3b9099bb864fec6a488064abe1abe7f06813
Summary: Useful when looking into blobstore corruption - you can compare all the blobstore versions by manual fetchees.
Reviewed By: krallin
Differential Revision: D23604436
fbshipit-source-id: 7b56947b0188536499514bae6615c6e81b9106c3
Summary: Going to add more features, so simplify by asyncifying first
Reviewed By: krallin
Differential Revision: D23604437
fbshipit-source-id: 52b2b372e4d3fbf1d59168c6c11311d9edf4ff0f
Summary: When we're scrubbing blobstores, it's not actually a success state if a scrub fails to write. Report this back to the caller - no-one will usually be scrubbing unless they expect repair writes to succeed, and a failure is a sign that we need to investigate further
Reviewed By: mitrandir77
Differential Revision: D23601541
fbshipit-source-id: d328935af9999c944719a6b863d0c86b28c54f59
Summary:
One test was fixed earlier by switching MacOS to use modern version of bash, the other is fixed here by installing "nmap" and using "ncat" from within it on both linux and mac.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/50
Reviewed By: krallin
Differential Revision: D23599695
Pulled By: lukaspiatkowski
fbshipit-source-id: e2736cee62e82d1e9da6eaf16ef0f2c65d3d8930
Summary:
Fixes a few issues with Mononoke tests in Python 3.
1. We need to use different APIs to account for the unicode vs bytes difference
for path hash encoding.
2. We need to set the language environment for tests that create utf8 file
paths.
3. We need the redaction message and marker to be bytes. Oddly this test still
fails with jq CLI errors, but it makes it past the original error.
Reviewed By: quark-zju
Differential Revision: D23582976
fbshipit-source-id: 44959903aedc5dc9c492ec09a17b9c8e3bdf9457
Summary:
For repositories that have the old-style LFS extension enabled, the pointers
are stored in packfiles/indexedlog alongside with a flag that signify to the
upper layers that the blob is externally stored. With the new way of doing LFS,
pointers are stored separately.
When both are enabled, we are observing some interesting behavior where
different get and get_meta calls may return different blobs/metadata for the
same filenode. This may happen if a filenode is stored in both a packfile as an
LFS pointers, and in the LFS store. Guaranteeing that the revisionstore code is
deterministic in this situation is unfortunately way too costly (a get_meta
call would for instance have to fully validate the sha256 of the blob, and this
wouldn't guarantee that it wouldn't become corrupted on disk before calling
get).
The solution take here is to simply ignore all the lfs pointers from
packfiles/indexedlog when remotefilelog.lfs is enabled. This way, there is no
risk of reading the metadata from the packfiles, and the blob from the
LFSStore. This brings however another complication for the user created blobs:
these are stored in packfiles and would thus become unreadable, the solution is
to simply perform a one-time full repack of the local store to make sure that
all the pointers are moved from the packfiles to to LFSStore.
In the code, the Python bindings are using ExtStoredPolicy::Ignore directly as
these are only used in the treemanifest code where no LFS pointers should be
present, the repack code uses ExtStoredPolicy::Use to be able to read the
pointers, it wouldn't be able to otherwise.
Reviewed By: DurhamG
Differential Revision: D22951598
fbshipit-source-id: 0e929708ba5a3bb2a02c0891fd62dae1ccf18204
Summary:
hg-http's built client should provide integration with Mercurial's stats
collection mechanisms.
Reviewed By: kulshrax
Differential Revision: D23577867
fbshipit-source-id: 93c777021bc347511322269d678d6879710eed3e
Summary:
Add `with_stats_reporting` to HttpClient. It takes a closure that will be
called with all `Stats` objects generated. We then use this function in
the hg-http crate to integrate with the metrics backend used in Mercurial.
Reviewed By: kulshrax
Differential Revision: D23577869
fbshipit-source-id: 5ac23f00183f3c3d956627a869393cd4b27610d4
Summary: Rust based metrics so that even Rust libraries can write metrics.
Reviewed By: quark-zju
Differential Revision: D23577870
fbshipit-source-id: b19904968d9372c8ce19775fb37c7af53a370ea5
Summary:
We start off simple here. Python only really has counters so we only implement
counters. There are a lot of options on how to improve this and things get
slightly complicated when we look at the how ecosystem and fb303. Anyway,
simple start.
Reviewed By: quark-zju
Differential Revision: D23577874
fbshipit-source-id: d50f5b2ba302d900b254200308bff7446121ae1d
Summary:
Slash is probably the standard metric delimiter nowadays. Since we don't have
that many metrics I think that it makes sense to look at slash as the
standard metric delimiter going forward.
This diff updates parsing of metric names to treat both '_' and '/' as
delimiters.
Reviewed By: quark-zju
Differential Revision: D23577876
fbshipit-source-id: 03997b1285df9c52d6e2837b5af5372deb69b133
Summary:
Update the `README.md` file in the fbcode_builder subdirectory to document the
current getdeps.py build system rather than the older fbcode_builder scripts
that required each project to keep their own python-based build files in their
project directory.
Reviewed By: chadaustin, wez
Differential Revision: D23591356
fbshipit-source-id: 75a099a10793e68a2b59696682010c4dff47ec69
Summary:
The command is easier to use than `hg cloud join --switch`.
Also highlight the workspace name in the output of `hg cloud status`
Reviewed By: mitrandir77
Differential Revision: D23601507
fbshipit-source-id: 74eb17c9366a9dbe96881c8e3e0705619fadb3d6
Summary:
Update FindGflags.cmake to work on recent CentOS and RedHat distributions.
On these distributions the CMake package configuration installed by the
gflags-devel RPM is slightly broken, and sets `gflags_INCLUDE_DIR` to a
directory that does not exist. This happens because `/lib64` is symlinked to
`/usr/lib64`, and CMake ends up searching `/lib64` before `/usr/lib64` by
default when searching for packages. Therefore it finds `gflags-config.cmake`
via the `/lib64` symlink. However, `gflags-config.cmake` computes the include
directory relative to where its config file was found, and this relative path
computation only works when using the actual `/usr/lib64` path where it was
installed. When found via `/lib64` it returns a bogus `//include` path that
does not exist.
This updates `FindGflags.cmake` to verify if the `gflags_INCLUDE_DIR` path
actually exists, and set it to a sane location instead.
This also updates the code to use the `gflags-shared` target that is exported
by the default gflags package configuration if it was only built as a shared
library.
Pull Request resolved: https://github.com/facebook/folly/pull/1409
Reviewed By: yfeldblum
Differential Revision: D23588288
fbshipit-source-id: b68a717953ae0521f568d7bcfd05ca33cd6dc578
Summary:
typed_hash only implements serialize. Because of this, if we want to serialize a struct that contains e.g changesetid(s), we can't deserialize it later. This diff adds deserialize implementation for typed_hashes.
Implementation is similar to HgNodeHash's: https://fburl.com/diffusion/r3df5iga
Reviewed By: krallin
Differential Revision: D23598925
fbshipit-source-id: 4d48b75eb8a01028e6e2d9bcc1ae20051a97b7fb
Summary:
Streaming clone implementation did not check that received files have the corrects. This change addresses it.
Before this change if connection was interrupted for whatever reason client would treat fetch of changeset as successful and proceed with cloning operations, but later checks would report corruption of internal state of hg data. This is based on user [report](https://fb.workplace.com/groups/scm/permalink/3177150312334567/)
Reviewed By: quark-zju, krallin
Differential Revision: D23572058
fbshipit-source-id: d740b45ca217cd6db0a65e01aabc2ba9a4835221
Summary: The Mercurial codebase uses hyphens in crate names rather than underscores. This is similar to the convention favored by the larger Rust community, though it is different from Mononoke, which uses underscores. While we'll probably need to eventually settle on a consistent convention for all of projects in the Eden SCM repo, for now, `http_client` should be made consistent with the adjacent crates.
Reviewed By: sfilipco
Differential Revision: D23585721
fbshipit-source-id: d2e690d86815be02d7b8d645198bcd28e8cbd6e0
Summary:
While it's unlikely to work properly (it uses /bin/sh), it compiles properly on
Windows, so let's include it in the build.
Reviewed By: wez
Differential Revision: D23520368
fbshipit-source-id: 267ba04f98f5dacc81e1772f86f5ad43c846815d
Summary:
We are using older version of tokio which spawns as many threads as we have
physical cores instead of the number of logical cores. It was fixed in
https://github.com/tokio-rs/tokio/issues/2269 but we can't use it yet because
we are waiting for another fix to be released -
https://github.com/rust-lang/futures-rs/pull/2154.
For now let's hardcode it in mononoke
Reviewed By: krallin
Differential Revision: D23599140
fbshipit-source-id: 80685651a7a29ba8938d9aa59770f191f7c42b8b
Summary:
This change move logic associated with mercurial changeset derivation to `mercurial_derived_data` crate.
NOTE: it is not converted to derived data infrastructure at this point, it is a preparation step to actually do this
Reviewed By: farnz
Differential Revision: D23573610
fbshipit-source-id: 6e8cbf7d53ab5dbd39d5bf5e06c3f0fc5a8305c8
Summary:
The Mac integration test workflow already installs a modern curl that fixes https://github.com/curl/curl/issues/4801, but it does so after "hg" is built, so "hg" uses the system curl libraries, which fails when used with a certificate not present in keychain.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/53
Reviewed By: krallin
Differential Revision: D23597285
Pulled By: lukaspiatkowski
fbshipit-source-id: a7b8b6ae55ce338bfb9946a852cbb6b929e73203
Summary:
There are blobs that fail to scrub and terminate the process early for a variety of reasons; when this is running as a background task, it'd be nice to get the remaining keys scrubbed, so that you don't have a large number of keys to fix up later.
Instead of simply outputting to stdout, write keys to one of three files in the format accepted on stdin:
1. Success; you can use `sort` and `comm -3` to remove these keys from the input dat, thus ensuring that you can continue scrubbing.
2. Missing; you can look at these keys to determine which blobs are genuinely lost from all blobstores, and fix up.
3. Error; these will need running through scrub again to determine what's broken.
Reviewed By: krallin
Differential Revision: D23574855
fbshipit-source-id: a613e93a38dc7c3465550963c3b1c757b7371a3b
Summary:
With three blobstores in play, we have issues working out exactly what's wrong during a manual scrub. Make the error handling better:
1. Manual scrub adds the key as context for the failure.
2. Scrub error groups blobstores by content, so that you can see which blobstore is most likely to be wrong.
Reviewed By: ahornby, krallin
Differential Revision: D23565906
fbshipit-source-id: a199e9f08c41b8e967d418bc4bc09cb586bbb94b
