Summary:
To allow EdenFS to get aux manifest data from Mononoke without needing to derive fsnodes, provide
a mechanism to list a manifest using the hg manifest id that returns the size and content hashes
of each of the files.
NOTE: this is temporary until the EdenAPI server is fully online and serving this data.
Reviewed By: krallin
Differential Revision: D22975967
fbshipit-source-id: 0a25da6d74534d42fc3b5f38ba3b72107b209681
Summary: It served no purpose. Also as a bonus, remove an unecessary std::endl.
Reviewed By: fanzeyi
Differential Revision: D22954057
fbshipit-source-id: bcaca833cdef8b643b16fbda2a0d576b655c2857
Summary: This is unused, no need to add to the build time.
Reviewed By: fanzeyi
Differential Revision: D22967814
fbshipit-source-id: 91a5ed9f03128947af9cb69bca62ed75b75e7e66
Summary:
One of the example is to delete a workspace that doesn't exist.
For service error remove host/port info.
These verbose details are not important for users.
Reviewed By: markbt
Differential Revision: D22976512
fbshipit-source-id: b8437f5b3c0e21e23183270d07ea158404598810
Summary: Previously it was opened twice, even though there were no reason to do it.
Reviewed By: krallin
Differential Revision: D22976149
fbshipit-source-id: 426858da4548f1eaffe1d989e5424937af2583a5
Summary:
Factor out the walkers state internals to BuildStateHasher and StateMap
This change keeps the defaults the same using DashMap and ahash::RandomState and uses the same ahash version that DashMap defaults to internally.
This is in preparation for the next diff the where the ahash dependency is updated to 0.4.4. Though it was clearer not to combine the refactoring and the update of the hasher used in the same diff.
Reviewed By: ikostia
Differential Revision: D22851585
fbshipit-source-id: 84fa0dc73ff9d32f88ad390243903812a4a48406
Summary:
Only emit NodeData from walker if required to save some memory. Each of the walks can now specify which NodeData it is interested in observing in the output stream.
We still need to emit Some as part of the Option<NodeData> in the output stream as it is used in things like the final count of loaded objects. Rather than stream over Option<Option<NodeData>> we instead add a NodeData::NotRequired variant
Reviewed By: markbt
Differential Revision: D22849831
fbshipit-source-id: ef212103ac2deb9d66b017b8febe233eb53c9ed3
Summary:
Extract verify_working_copy_inner function, which lets directly specify
source/target repo, hash and movers. It can be useful to verify equivalence of
two commits even if they are not in commit equivalence mapping.
Reviewed By: krallin
Differential Revision: D22950840
fbshipit-source-id: ab30be7190e29db3343b846b48333d7c7339d043
Summary:
We have reports from users about errors but the errors don't have stack
traces uploaded to scuba because they are considered handled. Let's just upload
traces for every single exception that propagates up to the dispatch layer.
Reviewed By: quark-zju
Differential Revision: D22938883
fbshipit-source-id: 525b6f13422ee2aa79de3beb48e58e13405ed199
Summary: fix the problem of printing 0% fsck progress bars when fsck is actually not happening.
Reviewed By: genevievehelsel
Differential Revision: D22927401
fbshipit-source-id: 868fa188e3c2671b0f6e18e842ec6a23281dc337
Summary: Move it from `'static` BoxFutures to async_trait and lifetimes
Reviewed By: markbt
Differential Revision: D22927171
fbshipit-source-id: 637a983fa6fa91d4cd1e73d822340cb08647c57d
Summary:
Eden runs without a sparse checkout, so "hg prefetch" can be a very expensive
operation on eden. Let's not allow "hg prefetch" on eden if pats are not
specified - this will force users to specify what exactly they want to
prefetch. Obviously they can still request everything, but this diff will make
it harder to do so.
Reviewed By: DurhamG
Differential Revision: D22946092
fbshipit-source-id: 895505bb90980a74b31ded4a75d102c527801652
Summary:
For EdenFS Redirection on Windows, we can simply use symlinks in place of bind mounts on other platforms. This works fine from what I can tell.
NOTE: at this commit EdenFS on Windows is still not able to create these redirections at start up time since that code is located in `EdenMount` and it uses `folly::subprocess`: https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/eden/fs/inodes/EdenMount.cpp. For the same reason we can't enable the EdenFS redirection integration tests.
Reviewed By: xavierd
Differential Revision: D22544200
fbshipit-source-id: b1a33da128c932544660c3e895583574d7891be9
Summary:
The mix use of `os.path` and `pathlib` in `redirect.py` is a little messy, which is making adding Windows support trickier since `os.path` functions do not accept `Path` until 3.8. So before I make the change I think it's better to clean it up.
Since we are target Python 3 nowadays, replacing `os.path` with `pathlib` seems to be better. Basically this diff does the following:
* Replace use of `os.path` functions with the counterpart in `pathlib`.
* Reduce unnecessary conversions from/to `Path` to `str` / `bytes`.
* Only convert `Path` to `str` or `bytes` when interactive with other APIs (Thrift or os)
* Cross-platform APIs: `os.fspath`
* API expecting `str`: `os.fsdecode`
* API expecting `bytes`: `os.fsencode`
Reviewed By: chadaustin
Differential Revision: D22879004
fbshipit-source-id: a247973dc9919c8805daa4046472124310725516
Summary:
This is a backout of D22912569 (34760b5164), which is breaking opt-clang-thinlto builds on platform007 (S206790).
Original commit changeset: 5ffdc48adb1f
Reviewed By: aaronabramov
Differential Revision: D22956288
fbshipit-source-id: 45940c288d6f10dfe5457d295c405b84314e6b21
Summary: Often users run into kerberos certificate issues, which in turn causes permissions issues with mercurial and eden. This issue is not obvious while users are running hg commands, so `eden doctor` should identity this issue and tell the user about it. `hg doctor` runs `eden doctor`, so this should also show itself if a user runs `hg doctor`.
Reviewed By: wez
Differential Revision: D22825882
fbshipit-source-id: 5f51901934862336b0ebc2996da6e1168ea8d8a3
Summary:
Added more logs when running the binary to be able to track the progress more easily.
Saved bonsai hashes into a file. In case we fail at deriving data types, we can still try to derive them manually with the saves hashes and avoid running the whole tool again.
Reviewed By: StanislavGlebik
Differential Revision: D22943309
fbshipit-source-id: e03a74207d76823f6a2a3d92a1e31929a39f39a5
Summary:
Large commits and many hooks can mean checking 100 commits at a time overload
the system. Reduce the default concurrency to something more reasonable.
While we're here, lets use the proper mechanism for default values in clap.
Reviewed By: ikostia
Differential Revision: D22945597
fbshipit-source-id: 0f0a086c3b74bec614ada44a66409c8d2b91fe69
Summary:
Argument names should be `snake_case`. Long options should be `--kebab-case`.
Retain the old long options as aliases for compatibility.
Reviewed By: HarveyHunt
Differential Revision: D22945600
fbshipit-source-id: a290b3dc4d9908eb61b2f597f101b4abaf3a1c13
Summary:
Add `--log-interval` to log every N commits, so that it can be seen to be
making progress in the logs.
The default is set to 500, which logs about once every 10 seconds on my devserver.
Reviewed By: HarveyHunt
Differential Revision: D22945599
fbshipit-source-id: 7fc09b907793ea637289c9018958013d979d6809
Summary: In cases where `stopWorkersOnStopListening` is disabled, `ThriftServer` continues to process inflight and even new requests after `ThriftServer::serve()` has returned. This is unintuitive and error-prone. This diff ensures that when `serve()` returns, there are no outstanding requests and new requests will not be processed.
Reviewed By: yfeldblum, genevievehelsel
Differential Revision: D22827447
fbshipit-source-id: cda35843ee6be084042e1a7c806c77fb472dd114
Summary: Commitcloud fillers use wishlist priority because we want them to wait their turn behind other users; let's also stop them from flooding the blobstore healer queue by making them background priority.
Reviewed By: ahornby
Differential Revision: D22867338
fbshipit-source-id: 5d16438ea185b580f3537e3c4895a545483eca7a
Summary:
Backfillers and other housekeeping processes can run so far ahead of the blobstore sync queue that we can't empty it from the healer task as fast as the backfillers can fill it.
Work around this by providing a new mode that background tasks can use to avoid filling the queue if all the blobstores are writing successfully. This has a side-effect of slowing background tasks to the speed of the slowest blobstore, instead of allowing them to run ahead at the speed of the fastest blobstore and relying on the healer ensuring that all blobs are present.
Future diffs will add this mode to appropriate tasks
Reviewed By: ikostia
Differential Revision: D22866818
fbshipit-source-id: a8762528bb3f6f11c0ec63e4a3c8dac08d0b4d8e
Summary:
This operation is useful immediately after a small repo is merged into a large repo.
See example below
```
B' <- manually synced commit from small repo (in small repo it is commit B)
|
BM <- "big merge"
/ \
... O <- big move commit i.e. commit that moves small repo files in correct location
|
A <- commit that was copied from small repo. It is identical between small and large repos.
```
Immediately after a small repo is merged into a large one we need to tell that a commit B and all of
its ancestors from small repo needs to be based on top of "big merge" commit in large repo rather than on top of
commit A.
The function below can be used to achieve exactly that.
Reviewed By: ikostia
Differential Revision: D22943294
fbshipit-source-id: 33638a6e2ebae13a71abd0469363ce63fb6b014f
Summary:
We were experiencing hangs during lfs http fetches. We've seen similar
issues before when using Hyper, which Reqwest is based off of. Let's switch to
the new Curl-based http_client instead.
Note, this does get rid of the ability to obey hg config http proxy settings.
That said, they shouldn't be used much, and the http proxy environment variables
are respected by libcurl.
Reviewed By: xavierd
Differential Revision: D22935348
fbshipit-source-id: 1c61c04bbb4043e3bde592251f12bf846ab3afd4
Summary:
A future diff does LFS fetches via http_client. Curl has some default
behavior of adding the "Expect: 100-continue" header which causes the server to
send a 100 status code response after the headers have been received but before
the the payload has been received. Since the http_client model only expects a
single response, this breaks the model and we're unable to read the second
response. Let's disable this behavior by manually setting the header to empty
string, which appears to be the official way to handle this.
Add it early so callers can overwrite it.
Reviewed By: quark-zju
Differential Revision: D22935349
fbshipit-source-id: 3009a5eb72f40584b846510f34f121e0e821a2bc
Summary:
hg rage had a couple string issues with Python 3 and subprocess output.
This fixes them.
Reviewed By: quark-zju
Differential Revision: D22937629
fbshipit-source-id: 2b90ada536f152afb2d2662eb7f8e12d787f9544
Summary:
In python 3, curses.unctrl(...) expects a single character string. If
we pass a longer string it will throw an exception. Let's handle that.
Reviewed By: singhsrb
Differential Revision: D22937223
fbshipit-source-id: 00d1a38e48ee7d6bc0aa43eb771619027dc3d802
Summary:
On Windows, this environment variable is required to be set for less to
properly recognize that the input encoding is utf-8.
Reviewed By: kulshrax
Differential Revision: D22925498
fbshipit-source-id: 7969441792e0c1d9e2b93a690f658431f0acf59c
Summary:
Reading the entire file to compute its sha1 can cause EdenFS to OOM on large
files. Instead, let's simpy read 8k at a time, like what we do on Linux.
Reviewed By: chadaustin
Differential Revision: D22924710
fbshipit-source-id: cd472f396a4385049994a1976c7d046cb901337a
Summary: We were using a git snapshot of auto_impl from somewhere between 0.3 and 0.4; 0.4 fixes a bug around Self: 'lifetime constraints on methods that blocks work I'm doing in Mononoke, so update.
Reviewed By: dtolnay
Differential Revision: D22922790
fbshipit-source-id: 7bb68589a1d187393e7de52635096acaf6e48b7e