Summary:
Add `--log-interval` to log every N commits, so that it can be seen to be
making progress in the logs.
The default is set to 500, which logs about once every 10 seconds on my devserver.
Reviewed By: HarveyHunt
Differential Revision: D22945599
fbshipit-source-id: 7fc09b907793ea637289c9018958013d979d6809
Summary: In cases where `stopWorkersOnStopListening` is disabled, `ThriftServer` continues to process inflight and even new requests after `ThriftServer::serve()` has returned. This is unintuitive and error-prone. This diff ensures that when `serve()` returns, there are no outstanding requests and new requests will not be processed.
Reviewed By: yfeldblum, genevievehelsel
Differential Revision: D22827447
fbshipit-source-id: cda35843ee6be084042e1a7c806c77fb472dd114
Summary: Commitcloud fillers use wishlist priority because we want them to wait their turn behind other users; let's also stop them from flooding the blobstore healer queue by making them background priority.
Reviewed By: ahornby
Differential Revision: D22867338
fbshipit-source-id: 5d16438ea185b580f3537e3c4895a545483eca7a
Summary:
Backfillers and other housekeeping processes can run so far ahead of the blobstore sync queue that we can't empty it from the healer task as fast as the backfillers can fill it.
Work around this by providing a new mode that background tasks can use to avoid filling the queue if all the blobstores are writing successfully. This has a side-effect of slowing background tasks to the speed of the slowest blobstore, instead of allowing them to run ahead at the speed of the fastest blobstore and relying on the healer ensuring that all blobs are present.
Future diffs will add this mode to appropriate tasks
Reviewed By: ikostia
Differential Revision: D22866818
fbshipit-source-id: a8762528bb3f6f11c0ec63e4a3c8dac08d0b4d8e
Summary:
This operation is useful immediately after a small repo is merged into a large repo.
See example below
```
B' <- manually synced commit from small repo (in small repo it is commit B)
|
BM <- "big merge"
/ \
... O <- big move commit i.e. commit that moves small repo files in correct location
|
A <- commit that was copied from small repo. It is identical between small and large repos.
```
Immediately after a small repo is merged into a large one we need to tell that a commit B and all of
its ancestors from small repo needs to be based on top of "big merge" commit in large repo rather than on top of
commit A.
The function below can be used to achieve exactly that.
Reviewed By: ikostia
Differential Revision: D22943294
fbshipit-source-id: 33638a6e2ebae13a71abd0469363ce63fb6b014f
Summary:
We were experiencing hangs during lfs http fetches. We've seen similar
issues before when using Hyper, which Reqwest is based off of. Let's switch to
the new Curl-based http_client instead.
Note, this does get rid of the ability to obey hg config http proxy settings.
That said, they shouldn't be used much, and the http proxy environment variables
are respected by libcurl.
Reviewed By: xavierd
Differential Revision: D22935348
fbshipit-source-id: 1c61c04bbb4043e3bde592251f12bf846ab3afd4
Summary:
A future diff does LFS fetches via http_client. Curl has some default
behavior of adding the "Expect: 100-continue" header which causes the server to
send a 100 status code response after the headers have been received but before
the the payload has been received. Since the http_client model only expects a
single response, this breaks the model and we're unable to read the second
response. Let's disable this behavior by manually setting the header to empty
string, which appears to be the official way to handle this.
Add it early so callers can overwrite it.
Reviewed By: quark-zju
Differential Revision: D22935349
fbshipit-source-id: 3009a5eb72f40584b846510f34f121e0e821a2bc
Summary:
hg rage had a couple string issues with Python 3 and subprocess output.
This fixes them.
Reviewed By: quark-zju
Differential Revision: D22937629
fbshipit-source-id: 2b90ada536f152afb2d2662eb7f8e12d787f9544
Summary:
In python 3, curses.unctrl(...) expects a single character string. If
we pass a longer string it will throw an exception. Let's handle that.
Reviewed By: singhsrb
Differential Revision: D22937223
fbshipit-source-id: 00d1a38e48ee7d6bc0aa43eb771619027dc3d802
Summary:
On Windows, this environment variable is required to be set for less to
properly recognize that the input encoding is utf-8.
Reviewed By: kulshrax
Differential Revision: D22925498
fbshipit-source-id: 7969441792e0c1d9e2b93a690f658431f0acf59c
Summary:
Reading the entire file to compute its sha1 can cause EdenFS to OOM on large
files. Instead, let's simpy read 8k at a time, like what we do on Linux.
Reviewed By: chadaustin
Differential Revision: D22924710
fbshipit-source-id: cd472f396a4385049994a1976c7d046cb901337a
Summary: We were using a git snapshot of auto_impl from somewhere between 0.3 and 0.4; 0.4 fixes a bug around Self: 'lifetime constraints on methods that blocks work I'm doing in Mononoke, so update.
Reviewed By: dtolnay
Differential Revision: D22922790
fbshipit-source-id: 7bb68589a1d187393e7de52635096acaf6e48b7e
Summary:
The non-Windows part of it returns the full path for the mount, do the same on
Windows. Since this is used in the remove path to unmount the repo, and the
paths expected in unmount are fullpaths, this meant remove would always fail to
unmount the repo.
Reviewed By: chadaustin
Differential Revision: D22915352
fbshipit-source-id: 6739267188c46d5ca1d6cb212a7155e70056f0f7
Summary:
One repack code path would return an error if the pack was already
deleted before the repack started. This is a fine situation, so let's just eat
the error and repack what can be repacked.
Reviewed By: xavierd
Differential Revision: D22873219
fbshipit-source-id: c716a5f0cd6106fd3464702753fb79df0bc7d13f
Summary:
The initialized_ field was never set, which meant that Overlay::close would
never close the sqlite overlay. This had the negative effect of allowing the
unmount thrift call to return before the sqliteoverlay is properly closed which
on Windows meant that the overlay couldn't be deleted.
Reviewed By: chadaustin
Differential Revision: D22915353
fbshipit-source-id: e501b2d0268449d109cc37dfdc62faebf1953e09
Summary:
On Windows, this has the benefit of automatically converting \r\n into \n,
which allows more tests to pass.
Reviewed By: chadaustin
Differential Revision: D22871408
fbshipit-source-id: 02ec1d21dc236175c3b0f3176db9b8c91dee21a4
Summary:
Eden api endpoint for segmented changelog. It translates a path in the
graph to the hash corresponding to that commit that the path lands on.
It is expected that paths point to unique commits.
This change looks to go through the plumbing of getting the request from
the edenapi side through mononoke internals and to the segmented changelog
crate. The request used is an example. Follow up changes will look more at
what shape the request and reponse should have.
Reviewed By: kulshrax
Differential Revision: D22702016
fbshipit-source-id: 9615a0571f31a8819acd2b4dc548f49e36f44ab2
Summary:
This functionality is going to be used in EdenApi. The translation is required
to unblock removing the changelog from the local copy of the repositories.
However the functionality is not going to be turned on in production just yet.
Reviewed By: kulshrax
Differential Revision: D22869062
fbshipit-source-id: 03a5a4ccc01dddf06ef3fb3a4266d2bfeaaa8bd2
Summary:
To start the only configuration available is whether the functionality provided
by this component is available in any shape or form. By default the component
is going to be disabled to all repositories. We will enable it first to
bootstrapped repositories and after additional tooling is added to production
repositories.
Reviewed By: kulshrax
Differential Revision: D22869061
fbshipit-source-id: fbaed88f2f45e064c0ae1bc7762931bd780c8038
Summary: The original function is changing behavior on non-UNIX platforms. Let's honor it in tests too.
Reviewed By: wez
Differential Revision: D22901850
fbshipit-source-id: 7c4e7dbfa0526d482f86794758e7ad39feacce10
Summary:
Add a way to actually try out segmented changelog by setting
`format.use-segmented-changelog`.
The main compatibility issues are assumptions that `0 .. len(repo)` are valid
revision numbers. Segmented changelog uses 2 spans `0 .. len(master_group)`
and `<a large 64-bit int> ..` for the non-master group. Some assumptions
about `len(repo)` were removed.
Segmented changelog also does not support linkrevs since revs can be
re-assigned. Some logic (ex. in treemanifest) that relies on linkrev being
`len(repo)` is changed.
There might be other issues, which will be fixed once we discover them.
Reviewed By: sfilipco
Differential Revision: D22657187
fbshipit-source-id: 688ad718741de0ca15e6aeaf84aced24a20c6b09
Summary:
For example, `master~1000000` should not trigger auto pull of the name
`1000000`. `x~y` is parsed as `(ancestor x y)`.
Reviewed By: singhsrb
Differential Revision: D22905438
fbshipit-source-id: 757ae8856f28126bc6e988d9989a894f83d83eb4
Summary:
- Enumerate API now provided via trait BlobstoreKeySource
- Implementation for Fileblob and ManifoldBlob
- Modified populate_healer to use new api
- Modified fixrepocontents to use new api
Reviewed By: ahornby
Differential Revision: D22763274
fbshipit-source-id: 8ee4503912bf40d4ac525114289a75d409ef3790
Summary: Since multiple mount points can share the same BackingStore object, it is better to start recording profile for each unique backing store instead of get backing stores by mounts.
Reviewed By: chadaustin
Differential Revision: D22904499
fbshipit-source-id: f95508044912503509c4ac0f84b4a061e41ca0f3
Summary:
added `record-profile` and `finish-profile` subcommands to `eden prefetch`
`eden prefetch record-profile` triggers `startRecordingFetch()` in `HgQueuedBackingStore`. `eden prefetch finish-profile` triggers `stopRecordingFetch()` in `HgQueuedBackingStore` and write the returned fetched file names to a text file. We can use the output file to predict what files are fetched for frequently used commands.
Reviewed By: chadaustin
Differential Revision: D22797896
fbshipit-source-id: ff240829b4fb3c47093279a1f2bd453009ff5f73
Summary:
after `startRecordingFetch()` is called by `HgQueuedBackingStore`, record the path for each fetched file. When `stopRecordingFetch()` is called, stop recording and return the collected file paths.
`startRecordingFetch()` and `stopRecordingFetch()` will be used in the next diff.
Reviewed By: chadaustin
Differential Revision: D22797037
fbshipit-source-id: a1fe30424d3c2884ffe139a0062b1e36328fd4fe
Summary: Update all the remaining steps in the walker to use the new early checks, so as to prune unnecessary edges earlier in the walk.
Reviewed By: farnz
Differential Revision: D22847412
fbshipit-source-id: 78c499a1870f97df7b641ee828fb8ec58303ebef
Summary:
This feature should be similar to `hg cloud deletebackup` but for cloud sync
It is needed to deprecate old backups.
Reviewed By: markbt
Differential Revision: D22876332
fbshipit-source-id: 90527bac4f352dc14fadf8b04f6c2df01045f5ce
Summary:
Check whether to emit an edge from the walker earlier to reduce vec allocation of unnecessary edges that would immediately be dropped in WalkVistor::visit.
The VisitOne trait is introduced as a simpler api to the Visitor that can be used to check if one edge needs to be visited, and the Checker struct in walk.rs is a helper around that that will only call the VisitOne api if necessary. Checker also takes on responsibility for respecting keep_edge_paths when returning paths, so that parameter has be removed for migrated steps.
To keep the diff size reasonable, this change has all the necessary Checker/VisitOne changes but only converts hg_manifest_step, with the remainder of the steps converted in the next in stack. Marked todos labelling unmigrated types as always emit types are be removed as part of converting remaining steps.
Reviewed By: farnz
Differential Revision: D22864136
fbshipit-source-id: 431c3637634c6a02ab08662261b10815ea6ce293
Summary:
This tool can be used in tandem with pre_merge_delete tool to merge a one large
repository into another in a controlled manner - the size of the working copy
will be increased gradually.
Reviewed By: ikostia
Differential Revision: D22894575
fbshipit-source-id: 0055d3e080c05f870cfd0026174365813b0eb253
Summary:
There are two reasons to want a write quorum:
1. One or more blobstores in the multiplex are experimental, and we don't want to accept a write unless the write is in a stable blobstore.
2. To reduce the risk of data loss if one blobstore loses data at a bad time.
Make it possible
Reviewed By: krallin
Differential Revision: D22850261
fbshipit-source-id: ed87d71c909053867ea8b1e3a5467f3224663f6a
Summary:
During large prefetches, (say a clone), it is possible that 2 different
filenode actually refer to the same file content, which thus share the same LFS
blob. The code would wrongly prefetch this blob twice which would then fail due
to the `obj_set` only containing one instance of this object.
Instead of using a Vec for the objects to prefetch, we can simply use a
`HashSet` which will take care of de-duplicating the objects.
Reviewed By: DurhamG
Differential Revision: D22903606
fbshipit-source-id: 4983555d2b16639051acbbb591ebb752d55acc2d
Summary:
There was a small but easy to miss mistake when prefetch was changed to return
the keys that couldn't be prefetched. For LFS pointers, the code would wrongly
return that the blob was fetched, which is misleading as the LFS blob isn't
actually downloaded. For LFS pointers, we need to translate them to their LFS
blob content hashes.
Reviewed By: DurhamG
Differential Revision: D22903607
fbshipit-source-id: e86592cd986498d9f4a574585eb92da695de2e27
