Summary: In cases where `stopWorkersOnStopListening` is disabled, `ThriftServer` continues to process inflight and even new requests after `ThriftServer::serve()` has returned. This is unintuitive and error-prone. This diff ensures that when `serve()` returns, there are no outstanding requests and new requests will not be processed.
Reviewed By: yfeldblum, genevievehelsel
Differential Revision: D22827447
fbshipit-source-id: cda35843ee6be084042e1a7c806c77fb472dd114
Summary: Commitcloud fillers use wishlist priority because we want them to wait their turn behind other users; let's also stop them from flooding the blobstore healer queue by making them background priority.
Reviewed By: ahornby
Differential Revision: D22867338
fbshipit-source-id: 5d16438ea185b580f3537e3c4895a545483eca7a
Summary:
Backfillers and other housekeeping processes can run so far ahead of the blobstore sync queue that we can't empty it from the healer task as fast as the backfillers can fill it.
Work around this by providing a new mode that background tasks can use to avoid filling the queue if all the blobstores are writing successfully. This has a side-effect of slowing background tasks to the speed of the slowest blobstore, instead of allowing them to run ahead at the speed of the fastest blobstore and relying on the healer ensuring that all blobs are present.
Future diffs will add this mode to appropriate tasks
Reviewed By: ikostia
Differential Revision: D22866818
fbshipit-source-id: a8762528bb3f6f11c0ec63e4a3c8dac08d0b4d8e
Summary:
This operation is useful immediately after a small repo is merged into a large repo.
See example below
```
B' <- manually synced commit from small repo (in small repo it is commit B)
|
BM <- "big merge"
/ \
... O <- big move commit i.e. commit that moves small repo files in correct location
|
A <- commit that was copied from small repo. It is identical between small and large repos.
```
Immediately after a small repo is merged into a large one we need to tell that a commit B and all of
its ancestors from small repo needs to be based on top of "big merge" commit in large repo rather than on top of
commit A.
The function below can be used to achieve exactly that.
Reviewed By: ikostia
Differential Revision: D22943294
fbshipit-source-id: 33638a6e2ebae13a71abd0469363ce63fb6b014f
Summary:
We were experiencing hangs during lfs http fetches. We've seen similar
issues before when using Hyper, which Reqwest is based off of. Let's switch to
the new Curl-based http_client instead.
Note, this does get rid of the ability to obey hg config http proxy settings.
That said, they shouldn't be used much, and the http proxy environment variables
are respected by libcurl.
Reviewed By: xavierd
Differential Revision: D22935348
fbshipit-source-id: 1c61c04bbb4043e3bde592251f12bf846ab3afd4
Summary:
A future diff does LFS fetches via http_client. Curl has some default
behavior of adding the "Expect: 100-continue" header which causes the server to
send a 100 status code response after the headers have been received but before
the the payload has been received. Since the http_client model only expects a
single response, this breaks the model and we're unable to read the second
response. Let's disable this behavior by manually setting the header to empty
string, which appears to be the official way to handle this.
Add it early so callers can overwrite it.
Reviewed By: quark-zju
Differential Revision: D22935349
fbshipit-source-id: 3009a5eb72f40584b846510f34f121e0e821a2bc
Summary:
hg rage had a couple string issues with Python 3 and subprocess output.
This fixes them.
Reviewed By: quark-zju
Differential Revision: D22937629
fbshipit-source-id: 2b90ada536f152afb2d2662eb7f8e12d787f9544
Summary:
In python 3, curses.unctrl(...) expects a single character string. If
we pass a longer string it will throw an exception. Let's handle that.
Reviewed By: singhsrb
Differential Revision: D22937223
fbshipit-source-id: 00d1a38e48ee7d6bc0aa43eb771619027dc3d802
Summary:
On Windows, this environment variable is required to be set for less to
properly recognize that the input encoding is utf-8.
Reviewed By: kulshrax
Differential Revision: D22925498
fbshipit-source-id: 7969441792e0c1d9e2b93a690f658431f0acf59c
Summary:
Reading the entire file to compute its sha1 can cause EdenFS to OOM on large
files. Instead, let's simpy read 8k at a time, like what we do on Linux.
Reviewed By: chadaustin
Differential Revision: D22924710
fbshipit-source-id: cd472f396a4385049994a1976c7d046cb901337a
Summary: We were using a git snapshot of auto_impl from somewhere between 0.3 and 0.4; 0.4 fixes a bug around Self: 'lifetime constraints on methods that blocks work I'm doing in Mononoke, so update.
Reviewed By: dtolnay
Differential Revision: D22922790
fbshipit-source-id: 7bb68589a1d187393e7de52635096acaf6e48b7e
Summary:
The non-Windows part of it returns the full path for the mount, do the same on
Windows. Since this is used in the remove path to unmount the repo, and the
paths expected in unmount are fullpaths, this meant remove would always fail to
unmount the repo.
Reviewed By: chadaustin
Differential Revision: D22915352
fbshipit-source-id: 6739267188c46d5ca1d6cb212a7155e70056f0f7
Summary:
One repack code path would return an error if the pack was already
deleted before the repack started. This is a fine situation, so let's just eat
the error and repack what can be repacked.
Reviewed By: xavierd
Differential Revision: D22873219
fbshipit-source-id: c716a5f0cd6106fd3464702753fb79df0bc7d13f
Summary:
The initialized_ field was never set, which meant that Overlay::close would
never close the sqlite overlay. This had the negative effect of allowing the
unmount thrift call to return before the sqliteoverlay is properly closed which
on Windows meant that the overlay couldn't be deleted.
Reviewed By: chadaustin
Differential Revision: D22915353
fbshipit-source-id: e501b2d0268449d109cc37dfdc62faebf1953e09
Summary:
On Windows, this has the benefit of automatically converting \r\n into \n,
which allows more tests to pass.
Reviewed By: chadaustin
Differential Revision: D22871408
fbshipit-source-id: 02ec1d21dc236175c3b0f3176db9b8c91dee21a4
Summary:
Eden api endpoint for segmented changelog. It translates a path in the
graph to the hash corresponding to that commit that the path lands on.
It is expected that paths point to unique commits.
This change looks to go through the plumbing of getting the request from
the edenapi side through mononoke internals and to the segmented changelog
crate. The request used is an example. Follow up changes will look more at
what shape the request and reponse should have.
Reviewed By: kulshrax
Differential Revision: D22702016
fbshipit-source-id: 9615a0571f31a8819acd2b4dc548f49e36f44ab2
Summary:
This functionality is going to be used in EdenApi. The translation is required
to unblock removing the changelog from the local copy of the repositories.
However the functionality is not going to be turned on in production just yet.
Reviewed By: kulshrax
Differential Revision: D22869062
fbshipit-source-id: 03a5a4ccc01dddf06ef3fb3a4266d2bfeaaa8bd2
Summary:
To start the only configuration available is whether the functionality provided
by this component is available in any shape or form. By default the component
is going to be disabled to all repositories. We will enable it first to
bootstrapped repositories and after additional tooling is added to production
repositories.
Reviewed By: kulshrax
Differential Revision: D22869061
fbshipit-source-id: fbaed88f2f45e064c0ae1bc7762931bd780c8038
Summary: The original function is changing behavior on non-UNIX platforms. Let's honor it in tests too.
Reviewed By: wez
Differential Revision: D22901850
fbshipit-source-id: 7c4e7dbfa0526d482f86794758e7ad39feacce10
Summary:
Add a way to actually try out segmented changelog by setting
`format.use-segmented-changelog`.
The main compatibility issues are assumptions that `0 .. len(repo)` are valid
revision numbers. Segmented changelog uses 2 spans `0 .. len(master_group)`
and `<a large 64-bit int> ..` for the non-master group. Some assumptions
about `len(repo)` were removed.
Segmented changelog also does not support linkrevs since revs can be
re-assigned. Some logic (ex. in treemanifest) that relies on linkrev being
`len(repo)` is changed.
There might be other issues, which will be fixed once we discover them.
Reviewed By: sfilipco
Differential Revision: D22657187
fbshipit-source-id: 688ad718741de0ca15e6aeaf84aced24a20c6b09
Summary:
For example, `master~1000000` should not trigger auto pull of the name
`1000000`. `x~y` is parsed as `(ancestor x y)`.
Reviewed By: singhsrb
Differential Revision: D22905438
fbshipit-source-id: 757ae8856f28126bc6e988d9989a894f83d83eb4
Summary:
- Enumerate API now provided via trait BlobstoreKeySource
- Implementation for Fileblob and ManifoldBlob
- Modified populate_healer to use new api
- Modified fixrepocontents to use new api
Reviewed By: ahornby
Differential Revision: D22763274
fbshipit-source-id: 8ee4503912bf40d4ac525114289a75d409ef3790
Summary: Since multiple mount points can share the same BackingStore object, it is better to start recording profile for each unique backing store instead of get backing stores by mounts.
Reviewed By: chadaustin
Differential Revision: D22904499
fbshipit-source-id: f95508044912503509c4ac0f84b4a061e41ca0f3
Summary:
added `record-profile` and `finish-profile` subcommands to `eden prefetch`
`eden prefetch record-profile` triggers `startRecordingFetch()` in `HgQueuedBackingStore`. `eden prefetch finish-profile` triggers `stopRecordingFetch()` in `HgQueuedBackingStore` and write the returned fetched file names to a text file. We can use the output file to predict what files are fetched for frequently used commands.
Reviewed By: chadaustin
Differential Revision: D22797896
fbshipit-source-id: ff240829b4fb3c47093279a1f2bd453009ff5f73
Summary:
after `startRecordingFetch()` is called by `HgQueuedBackingStore`, record the path for each fetched file. When `stopRecordingFetch()` is called, stop recording and return the collected file paths.
`startRecordingFetch()` and `stopRecordingFetch()` will be used in the next diff.
Reviewed By: chadaustin
Differential Revision: D22797037
fbshipit-source-id: a1fe30424d3c2884ffe139a0062b1e36328fd4fe
Summary: Update all the remaining steps in the walker to use the new early checks, so as to prune unnecessary edges earlier in the walk.
Reviewed By: farnz
Differential Revision: D22847412
fbshipit-source-id: 78c499a1870f97df7b641ee828fb8ec58303ebef
Summary:
This feature should be similar to `hg cloud deletebackup` but for cloud sync
It is needed to deprecate old backups.
Reviewed By: markbt
Differential Revision: D22876332
fbshipit-source-id: 90527bac4f352dc14fadf8b04f6c2df01045f5ce
Summary:
Check whether to emit an edge from the walker earlier to reduce vec allocation of unnecessary edges that would immediately be dropped in WalkVistor::visit.
The VisitOne trait is introduced as a simpler api to the Visitor that can be used to check if one edge needs to be visited, and the Checker struct in walk.rs is a helper around that that will only call the VisitOne api if necessary. Checker also takes on responsibility for respecting keep_edge_paths when returning paths, so that parameter has be removed for migrated steps.
To keep the diff size reasonable, this change has all the necessary Checker/VisitOne changes but only converts hg_manifest_step, with the remainder of the steps converted in the next in stack. Marked todos labelling unmigrated types as always emit types are be removed as part of converting remaining steps.
Reviewed By: farnz
Differential Revision: D22864136
fbshipit-source-id: 431c3637634c6a02ab08662261b10815ea6ce293
Summary:
This tool can be used in tandem with pre_merge_delete tool to merge a one large
repository into another in a controlled manner - the size of the working copy
will be increased gradually.
Reviewed By: ikostia
Differential Revision: D22894575
fbshipit-source-id: 0055d3e080c05f870cfd0026174365813b0eb253
Summary:
There are two reasons to want a write quorum:
1. One or more blobstores in the multiplex are experimental, and we don't want to accept a write unless the write is in a stable blobstore.
2. To reduce the risk of data loss if one blobstore loses data at a bad time.
Make it possible
Reviewed By: krallin
Differential Revision: D22850261
fbshipit-source-id: ed87d71c909053867ea8b1e3a5467f3224663f6a
Summary:
During large prefetches, (say a clone), it is possible that 2 different
filenode actually refer to the same file content, which thus share the same LFS
blob. The code would wrongly prefetch this blob twice which would then fail due
to the `obj_set` only containing one instance of this object.
Instead of using a Vec for the objects to prefetch, we can simply use a
`HashSet` which will take care of de-duplicating the objects.
Reviewed By: DurhamG
Differential Revision: D22903606
fbshipit-source-id: 4983555d2b16639051acbbb591ebb752d55acc2d
Summary:
There was a small but easy to miss mistake when prefetch was changed to return
the keys that couldn't be prefetched. For LFS pointers, the code would wrongly
return that the blob was fetched, which is misleading as the LFS blob isn't
actually downloaded. For LFS pointers, we need to translate them to their LFS
blob content hashes.
Reviewed By: DurhamG
Differential Revision: D22903607
fbshipit-source-id: e86592cd986498d9f4a574585eb92da695de2e27
Summary: A couple of features stabilized, so drop their `#![feature(...)]` lines.
Reviewed By: eugeneoden, dtolnay
Differential Revision: D22912569
fbshipit-source-id: 5ffdc48adb1f57a1b845b1b611f34b8a7ceff216
Summary:
Our error handling looked pretty, but allocating all of these futures
is expensive. Each future is an allocation and some atomics. This diff
buys back some performance which I will soon spend on a new async
event queue.
Reviewed By: xavierd
Differential Revision: D22799737
fbshipit-source-id: 91dcfe974cf8f461109dfaa9dbf75c054ed84f59
Summary:
In several places in `library.sh` we had `--mononoke-config-path
mononoke-config`. This ensured that we could not run such commands from
non-`$TESTTMP` directorires. Let's fix that.
Reviewed By: StanislavGlebik
Differential Revision: D22901668
fbshipit-source-id: 657bce27ce6aee8a88efb550adc2ee5169d103fa
Summary: The more contexts the better. Makes debugging errors much more pleasant.
Reviewed By: StanislavGlebik
Differential Revision: D22890940
fbshipit-source-id: 48f89031b4b5f9b15f69734d784969e2986b926d
Summary:
I've seen the error a couple of times when messing up with my clones, not
having the path makes it a bit difficult to fully understand what's going on,
make sure we log it.
Reviewed By: fanzeyi
Differential Revision: D22899098
fbshipit-source-id: c9a60b71ea20514158e62fe8fa9c409d6f0f37ff
Summary:
An extremely thin wrapper around existing APIs: just a way to create merge commits from the command line.
This is needed to make the merge strategy work:
```
C
|
M3
| \
. \
| \
M2 \
| \ \
. \ \
| \ \
M1 \ \
| \ \ \
. TM3 \ \
. / | |
. D3 (e7a8605e0d) TM2 |
. | / /
. D2 (33140b117c) TM1
. | /
. D1 (733961456f)
| |
| \
| DAG to merge
|
main DAG
```
When we're creating `M2` as a result of merge of `TM2` into the main DAG, some files are deleted in the `TM3` branch, but not deleted in the `TM2` branch. Executing merge by running `hg merge` causes these files to be absent in `M2`. To make Mercurial work, we would need to execute `hg revert` for each such file prior to `hg merge`. Bonsai merge semantics however just creates correct behavior for us. Let's therefore just expose a way to create bonsai merges via the `megarepotool`.
Reviewed By: StanislavGlebik
Differential Revision: D22890787
fbshipit-source-id: 1508b3ede36f9b7414dc4d9fe9730c37456e2ef9
Summary:
This adds a CLI for the functionality, added in the previous diff. In addition, this adds an integration test, which tests this deletion functionality.
The output of this tool is meant to be stored in the file. It simulates a simple DAG, and it should be fairly easy to automatically parse the "to-merge" commits out of this output. In theory, it could have been enough to just print the "to-merge" commits alone, but it felt like sometimes it may be convenient to quickly examine the delete commits.
Reviewed By: StanislavGlebik
Differential Revision: D22866930
fbshipit-source-id: 572b754225218d2889a3859bcb07900089b34e1c
Summary:
This implements a new strategy of creating pre-merge delete commits.
As a reminder, the higher-level goal is to gradually merge two independent DAGs together. One of them is the main repo DAG, the other is an "import". It is assumed that the import DAG is already "moved", meaning that all files are at the right paths to be merged.
The strategy is as follows: create a stack of delete commits with gradually decreasing working copy size. Merge them into `master` in reverse order.
Reviewed By: StanislavGlebik
Differential Revision: D22864996
fbshipit-source-id: bfc60836553c656b52ca04fe5f88cdb1f15b2c18
Summary:
On Windows, paths are separated by \, but the test was comparing them against
/. We can simply ask Mercurial to return / with the slashpath template filter.
Reviewed By: chadaustin
Differential Revision: D22871407
fbshipit-source-id: 421bd14f752f29265b12eb25609d4f65e593dda8
Summary:
Cache invalidation is hard, and on Windows we avoided doing a lot of them. It
turns out, this was the wrong decision as it's fairly easy to find cases where
the filesystem view is different from the manifest state.
Since the Linux code is most likely correct in where the invalidation is done,
let's also do the same on Windows, removing a whole lot of #ifdef. It is very
likely that as a result of this diff we end up invalidating more than needed,
thus slowing down EdenFS, but at this point I'd prefer to err on the side of
correctness, performance will come later.
While invalidating files should use PrjDeleteFile, for directories, we simply
need to mark them as placeholder, as directories created by a user won't have a
placeholder, thus ProjectedFS would bypass EdenFS when listing in.
Reviewed By: chadaustin
Differential Revision: D22833202
fbshipit-source-id: d807557f5e44279c49ab701b7a797253ef1f0717
Summary: While testing something for another change, I came across this overlooked typo.
Reviewed By: wez
Differential Revision: D22894060
fbshipit-source-id: 8aa48ef5da714650c974adcf8a34a542fdd4ed9e
Summary:
Avoid some overhead and complexity by storing BufVec as a
unique_ptr<IOBuf>. The complexity can be reintroduced if we ever find
FUSE splice support to be a performance win for us.
Reviewed By: kmancini
Differential Revision: D22710795
fbshipit-source-id: e58eedc0fb5cea9e9743ccd20d3e4e2b7cc5d198
Summary:
Previously we log a process to Scuba when it does 2000 (fetchThreshold_) fetchs, but then in Scuba all processes have fetch_count = 2000. In order to see how many fetches a process really did approximately, we log the same process to Scuba every time it does 2000 more fetches.
Note: this change could make the total count of fetch-heavy events in Scuba inaccurate, as we log the same process more than once. So when users want to see how many fetch-heavy events happened, instead of setting "type = fetch_heavy", they should set exactly "fetch_count = 2000".
Reviewed By: chadaustin
Differential Revision: D22867679
fbshipit-source-id: ae3c768a8d3b03628db6a77263e715303a814e3d
Summary:
With upcoming write quorum work, it'll be interesting to know all the failures that prevent a put from succeeding, not just the most recent, as the most recent may be from a blobstore whose reliability is not yet established.
Store and return all errors, so that we can see exactly why a put failed
Reviewed By: ahornby
Differential Revision: D22896745
fbshipit-source-id: a3627a04a46052357066d64135f9bf806b27b974
