Summary:
While std::variant is convenient, they are both slow to compile, and the
compiler cannot optimize it as well as a manually written tagged union. Since
ImmediateFuture is performance critical for EdenFS, let's use a tagged union
and speed them up by an additional 40%.
Reviewed By: chadaustin
Differential Revision: D31272296
fbshipit-source-id: e34be4489a596d3577b3bd900a1f20d6c7d8b693
Summary:
The max duration would cause UBSAN failures due to folly's SemiFuture code
multiplying the value which understandably cannot be represented. Splitting the
function is easy and avoids the problem entirely.
Reviewed By: genevievehelsel
Differential Revision: D31272297
fbshipit-source-id: c15ca70ad771c11b4f68bb9974422c0986d4928b
Summary:
When Buck is using the EdenFS globber, the searchRoot argument is set, thus
let's add a new argument to the benchmark to simulate a Buck workflow.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D31283399
fbshipit-source-id: 5e32b2aceb6090e26e88cf7f0d163448d56107d4
Summary:
The goal of this stack is to remove Proxy Hash type, but to achieve that we need first to address some tech debt in Eden codebase.
For the long time EdenFs had single Hash type that was used for many different use cases.
One of major uses for Hash type is identifies internal EdenFs objects such as blobs, trees, and others.
We seem to reach agreement that we need a different type for those identifiers, so we introduce separate ObjectId type in this diff to denote new identifier type and replace _some_ usage of Hash with ObjectId.
We still retain original Hash type for other use cases.
Roughly speaking, this is how this diff separates between Hash and ObjectId:
**ObjectId**:
* Everything that is stored in local store(blobs, trees, commits)
**Hash20**:
* Explicit hashes(Sha1 of the blob)
* Hg identifiers: manifest id and blob hg ig
For now, in this diff ObjectId has exactly same content as Hash, but this will change in the future diffs. Doing this way allows to keep diff size manageable, while migrating to new ObjectId right away would produce insanely large diff that would be both hard to make and review.
There are few more things that needs to be done before we can get to the meat of removing proxy hashes:
1) Replace include Hash.h with ObjectId.h where needed
2) Remove Hash type, explicitly rename rest of Hash usages to Hash20
3) Modify content of ObjectId to support new use cases
4) Modify serialized metadata and possibly other places that assume ObjectId size is fixed and equal to Hash20 size
Reviewed By: chadaustin
Differential Revision: D31316477
fbshipit-source-id: 0d5e4460a461bcaac6b9fd884517e129aeaf4baf
Summary:
https://pxl.cl/1Qh3j
This is the most called edenapi endpoint by far. If we sample logging of it, we can increase the retention of the scuba table.
if we wish, it's possible to not change retention for some "non-trivial" requests, but I haven't done that.
Reviewed By: liubov-dmitrieva
Differential Revision: D31277391
fbshipit-source-id: ee19e9daa4cd39c5d3eac1063e82aa40fc108bc7
Summary:
This is used to uniquely identify requests in gotham. It's logged to output, on errors, and on Scuba.
Problem: On scuba, this packs very badly, as it is a large string (36 chars), unique for all requests.
Solution: Let's get a prefix of it, it should reduce size used on scuba. Got a prefix of size 5.
This affects both LFS and EdenApi.
Pros:
- Reduces size
- Very easy fix
Cons:
- More chance of conflict. The space of this id is 16^5 = 10^6. There will surely be conflicts, but maybe that's not a huge deal?
Alternative: Using 8 digits, that's about 4bi ids, which will reduce conflicts significantly in exchange for more space.
Why not use an int id (example: u64), or using other characters in id (not only hex): This would reduce the size of data significantly, but has drawbacks:
- For int, would require a big refactoring, as everything assumes the id to be string. Specially since this goes through client-server, might be complicated.
- Not just getting a prefix means more processing on each request, and means we need to recalculate it everytime.
- Size reduction might not be that big, as scuba already packs stuff pretty well.
Reviewed By: krallin
Differential Revision: D31305547
fbshipit-source-id: 23f6b6cb7de5b7a090864db414d4d71cd68c4946
Summary: D31115820 (ae87b82eaf) updated quickcheck, but there's some stuff we need to fix forward. This diff fixes the remaining failures I could find.
Reviewed By: farnz
Differential Revision: D31305392
fbshipit-source-id: a6684d47833bc0fd933751c13cdd71392cb1833b
Summary:
use `hg cloud upload` command if usehttpupload enabled
For few users who opted out of cloud sync, we should use `hg cloud upload` command instead of `hg cloud backup` if usehttpupload enabled
Reviewed By: markbt
Differential Revision: D31205919
fbshipit-source-id: 7619e7b299e19a7626782e7b3c1a69e7cd7dbc1b
Summary:
Somehow before this fix I've seen us runing out of semaphores and deadlocking
because not freeing semaphores immediately after finishing running the
function requiring git repo.
Reviewed By: mojsarn
Differential Revision: D31310626
fbshipit-source-id: ba12b2d4918ecc30ca0aa6ff011176f7634badf9
Summary: Need this for cargo check/rust-analyzer to work.
Reviewed By: guswynn
Differential Revision: D31319911
fbshipit-source-id: ebd3fa72d8fc3667391a2067f95cab9e5f53301f
Summary: I'm parsing some deeply-nested JSON, and it's running into the limit. This feature enables a potential footgun, but even with the feature enabled you have to add code to reach for said footgun.
Reviewed By: jsgf
Differential Revision: D31284743
fbshipit-source-id: 00ea5d7d7db8bdeb878d48fe390831f39e007409
Summary:
When calling `changelog.idmap`, it takes a snapshot of the IdMap.
When flushing the changelog, the IdMap might change. Some Ids might get
re-assigned.
For code (phases?) that keeps the `nameset` for a while the idmap might become
out-dated after a changelog flush. That might be a cause of removing visible
heads incorrectly.
Let's change `_torev` and `_tonode` to properties so they always take the
"latest" idmap snapshot.
Reviewed By: DurhamG
Differential Revision: D31296057
fbshipit-source-id: 1b3fec5d21649eab772ab3a150a3182a18b94edf
Summary:
It was added on the client side in D30686450 (7eb11cb392) to handle octopus merges correctly,
let's add it on mononoke as well, otherwise new_streaming_clone fails to parse
a revlog.
Reviewed By: mitrandir77
Differential Revision: D31305651
fbshipit-source-id: 976d7fdb8775f859e4732fd8a68f9b28f04ce4f9
Summary: This implements unix socket support for mercurial's HTTPConnection so commitcloud can use it.
Reviewed By: ahornby
Differential Revision: D31229256
fbshipit-source-id: a610c3c34be608ac2d9b41f3a7b6b62b44227b94
Summary:
We want to reduce duplicated work. Since requests will be consistently hashed, each instance of service will receive some set of requests from multiple clients. By storing requests together with shared future derivation, any client can get the state of derivation.
In addition upon receiving requests we can clean up the map to remove already completed futures so the map will not grow indefinitely.
Reviewed By: StanislavGlebik
Differential Revision: D30776322
fbshipit-source-id: 961055f8b3328378451edd677506d7e716a9afd2
Summary:
Now the runlog for an hg command invocation includes any progress bar metadata. I repurposed the existing rust progress thread to also upate the runlog progress (only if it has changed).
To avoid race conditions with the main thread writing the final "exit" runlog entry, updating the runlog progress is a no-op if the runlog's exit code has already been set.
Reviewed By: quark-zju
Differential Revision: D31065260
fbshipit-source-id: 181661cb06ab2910d8a0e41f5aa767528eb234f5
Summary:
The runlog's purpose is to store live information for every hg invocations. Users/VSCode will access the runlog data to see details about active hg commands.
In this initial commit I've added basic start/end updates to the runlog. The only current storage option is JSON files written to ".hg/runlog/<random ID>". Cleanup of the files will be added later. In the future I may look at sqlite as an alternative.
Set runlog.enable=True to turn on the runlog.
Reviewed By: quark-zju
Differential Revision: D31065258
fbshipit-source-id: 3ff29e1b8473f7e0b6b0d02537d1f18c2c5026fb
Summary:
The old message was a little misleading as in some case EdenFS was running while it couldn't mount the repository. Mercurial will still tell the user that EdenFS is not running. It is not accurate.
The new message is trying to cover this case to avoid confusing people.
Reviewed By: zhengchaol
Differential Revision: D31278947
fbshipit-source-id: dd3e599654390269b6cf31d8842105970cb29cc0
Summary:
This updates the following crates to their latest versions:
- `rand`: 0.7 => 0.8
- `quickcheck`: 0.9 => 1.0
Both crates introduced some breaking changes, so affected clients had to be fixed accordingly. Most changes are rather mechanical and shouldn't change the existing logic. In addition, a few buggy property tests were uncovered, presumably due to `quicktest` becoming smarter with its choice of inputs in the newer version, and the fixes are included in this diff.
Reviewed By: yancouto
Differential Revision: D31115820
fbshipit-source-id: 60a61dfac3236fd93cd4f03b86506654d81d330f
Summary: This diff fixes some integration test errors after enabling the new edenfsctl.
Reviewed By: xavierd
Differential Revision: D30789741
fbshipit-source-id: 02d74defc41def4fb6ea0cc4694f944b4c0044e2
Summary:
Some detail polishing.
Incomplete commands are commented out. Help messages are now printed correctly. Fixed a small behavior divergence in `eden config` (`to_string_pretty` uses multi-line string instead of escaping characters).
Reviewed By: xavierd
Differential Revision: D30547011
fbshipit-source-id: 98d323744ce7a7fc989cbf79dd07ed8af3cee09d
Summary: This diff adds the Rust edenfsctl to our open source build.
Reviewed By: xavierd
Differential Revision: D30788685
fbshipit-source-id: 603caa933ecfc5af0ede7e22f6c7911462da3a65
Summary:
The lookup of content ids was not working as expected.
Reasons:
- If content id was provided, we never checked it was actually on the blobstore, and failed when building the metadata for it. This was happening since D30016963 (f64520a312)
This diff fixes that by explicitly checking it exists. I also added some comments to clarify.
Reviewed By: liubov-dmitrieva, StanislavGlebik
Differential Revision: D31268102
fbshipit-source-id: 9801a7f4ce1536e68f44ebe114087e53cf094d7a
Summary:
Functions that only takes boolean arguments are fairly confusing and error
prone. Here, since we only ever pass a single true value to it, we could simply
inline setting the right counter in the caller. This makes the code easier to
read, and less error prone.
Reviewed By: genevievehelsel
Differential Revision: D31188413
fbshipit-source-id: 64c019ff52b1ff5644e5bea11a361e586044403f
Summary:
Changes to edenfs-client seem to be breaking the hgbuild windows job https://www.internalfb.com/intern/sandcastle/job/27021598254894733/
Original commit changeset: 218f06a4e648
Reviewed By: DurhamG
Differential Revision: D31244893
fbshipit-source-id: e9ef7c2142d0a6afca342f84574d553b136b5fdb
Summary: I would like to use httpclient.HTTPConnection client because in the following diffs I am adding unix domain socket support to it and jplopezgu will add use that support for commitcloud.
Reviewed By: ahornby
Differential Revision: D31229252
fbshipit-source-id: 8999f27b68f9c7aa9f725d65c291f4d338d3b813
Summary:
One way to mitigate the skiplist inefficiencies is to just use segmented
changelog if we can.
Currently we can do it only for commits on master bookmarks for most repos but
upcoming defrag work from farnz would allow us to include release branches
there as well. That will cover most of the is_ancestor queries.
NOTE: This is not the end of diffs switching us to use segmented changelog. I'm planning to also do it for other places where we do ancestry checks and lower common ancestor operations.
Reviewed By: StanislavGlebik
Differential Revision: D31169338
fbshipit-source-id: 9d4b27d3fb22016b0239c52d71a9b2d9ae9a103b
Summary: This would allow us to benefit from segmented changelog server-side
Reviewed By: StanislavGlebik
Differential Revision: D31169337
fbshipit-source-id: 3c648ed2f144cee57de7c319692a37b04adf5705
Summary: Previously, all EdenAPI methods supported callback-based progress reporting. With the new HTTP progress bars, this old progress API is no longer used anywhere (except for a test program). Let's clean it up to get rid of the extra parameter for every method.
Reviewed By: andll
Differential Revision: D31184693
fbshipit-source-id: 996959e0d81dd7685fcfaca98f162e7267684306
Summary:
this admin command in D29734333 (3f8de3336a) started depending on innerRepo because it
needed access to ephemeral blobstore. It didn't need other parts of inner repo
so there's no need for that dependency.
Reviewed By: krallin
Differential Revision: D31210293
fbshipit-source-id: 004fb95d17e7e1d3095db0258f3c55dadaf5524c
Summary:
This mode rederives commits and compares that what was rederived is the same to
what has already been derived. It's useful to test any changes to derive data
logic and make sure these changes don't have any bugs
Reviewed By: markbt
Differential Revision: D31143741
fbshipit-source-id: 618dbf12ab444b5686d50f83a590314adc6c5dda
Summary: Remove some more path clones by changing within_restrictions to take Option<&MPath>
Reviewed By: StanislavGlebik
Differential Revision: D31175004
fbshipit-source-id: 92f0b4b594c4b3e30258acd019e7f42d9b3bc5fb
Summary: Remove a couple of clones of path by moving up ChangesetPathContentContext::new_with_fsnode_entry
Reviewed By: StanislavGlebik
Differential Revision: D31175005
fbshipit-source-id: fa686f69087e317877c2c9a9c0cffe05a6006775
Summary:
`self.map.contains_vertex_name` only checks the local `map` without triggering
remote protocols. That means with lazy changelog clone, if `master` points to
a lazy commit, the clone will fail. Fix it by switching to `self.contains_vertex_name`,
or even better, `self.vertex_id_batch` to do proper batching.
Reviewed By: StanislavGlebik
Differential Revision: D31228524
fbshipit-source-id: 229d8a92c5517ac5a1dbfa3f440df88a4ab8e3e6
Summary:
In advance of Thrift servers defaulting the queue timeout to 100 ms,
which is quite low for EdenFS's needs, explicitly set our queue
timeout to 5 seconds.
Reviewed By: zhengchaol
Differential Revision: D31218348
fbshipit-source-id: 35a109fb6848f7c81c4b58d70e2beae90557e1c8
Summary: we can just use getBackingStores like how is done for `startRecordingBackingStoreFetch` and only record non-empty fileAccesses. This will enable fetch logging for LocalCacheBackingStores which use an HgQueuedBackingStore under the hood
Reviewed By: zhengchaol
Differential Revision: D31215109
fbshipit-source-id: 443d28a57144fdcf078bd653ecf5726825f55740
Summary: fix the dynamic casting for getting a tracebus for the trace hg entrypoint. dynamic cast still makes sense at this point since `trace hg` should only be called on hg backed mounts
Reviewed By: chadaustin
Differential Revision: D31214737
fbshipit-source-id: 65e018e6658d934d8ecd3434bdfc3d72f6873d2b
Summary: instead of dynamic casting to find the repo name, all backing stores can return an optional reponame, and can check if the optional is set.
Reviewed By: zhengchaol
Differential Revision: D31214723
fbshipit-source-id: 9d10114ff6bde13254d3a3caaf2401f87d07ffd7
Summary: add more information to the runtime error thrown by the dynamic cast failure in `eden trace hg` and predictive fetch
Reviewed By: zhengchaol
Differential Revision: D31212247
fbshipit-source-id: 982901dfd2eb05db9ca6e7366277a07b6b29872f
Summary:
VC++ 2019 is pickier about which standard library includes include
each other. Be explicit.
Reviewed By: zhengchaol
Differential Revision: D31186916
fbshipit-source-id: 95cfa8848d0e2e312e2024923fa166db5f68dde0
Summary:
This unused feature allowed a sub-process to inherit an hg repo wlock from the parent process. It was apparently intended for merge drivers, but nothing was using it.
I want to move some locking logic into rust, and this stuff was complicating things.
Note that this functionality was also removed upstream in https://phab.mercurial-scm.org/D9053.
Reviewed By: quark-zju
Differential Revision: D31184339
fbshipit-source-id: 92908220d48e2bc55e2f4fca90e647650ca5bef7
Summary:
While debugging the unlinked inode unloading for NFS I have re-added these
logs a couple times. These seem valuable to have in eden so that we don't have
to add them any time we are debugging eden and we can debug a bit in a
production eden rather than dev built eden.
Reviewed By: xavierd
Differential Revision: D30971151
fbshipit-source-id: 58172079dfe4f4e4ba31bae30bf982e2cbe0fd29
Summary:
We run periodic inode unloading for unlinked inodes on NFS because we get no
information from the client on when inodes are no longer needed, and we have to
clean them up at some point for memory and disk reasons. See previous commit
summaries for more details on this (D30144901 (ffa558bf84)).
Let's add some counters on this so we have a bit more visibility into the
process. This counter is meant to mimic the PeriodicUnloadCounter counter.
Reviewed By: chadaustin
Differential Revision: D30966688
fbshipit-source-id: cfc8d769b53073d9f4c0c27b6bee20e222c6c8d2
Summary:
I believe this is the reason for -
https://fb.workplace.com/groups/238845853462687/posts/845939069420026. We used
default config that doesn't do any chunking and puts large files as a single
blobs.
Let's not do that
Reviewed By: farnz
Differential Revision: D31209331
fbshipit-source-id: 43c2d2ab7caac110a1474856da09c119a5e72429
Summary:
EdenApiUploads: eliminate extra lookup if no stacks
In EdenApiUploads we filter heads first and then we filter the commits belonging to these stacks.
However, in some usecases users don't use stacks. If there is no any single stack, the second lookup would be redundant and it would be nice to avoid it completely.
We can pass a flag to the upload code saying that extra filtering is not needed.
For example, in configerator repo users usually don't do stacks.
Reviewed By: markbt
Differential Revision: D31203489
fbshipit-source-id: 0921a01198bfc377afc3af3f7319fd0c5fec04d7
Summary: Plus a minor refactoring to use the io::IsTty trait in edenfs_client::status instead of calling into libc directly.
Reviewed By: quark-zju
Differential Revision: D31156633
fbshipit-source-id: 218f06a4e64836be88b4afac98dcfa140373c730
Summary:
There is no need to read from the LocalStore twice, the tree is either present
in it, or not.
Reviewed By: chadaustin
Differential Revision: D31187972
fbshipit-source-id: 15bdeef9176b51e6ba3f62ed16550032b0024b94
Summary:
Some of EdenFS backing store requires EdenFS to cache objects locally to avoid
potentially expensive network fetches, while others already have some form of
local caching. In the past, all backing store fell in the first category, but
thanks to Mercurial's native backing store implementation the LocalStore
caching has become pure overhead for it. Previously, this was worked around by
configuring the LocalStore to not cache blobs locally, but this wasn't done for
trees. This config also conflicts with the need to cache blobs and trees
locally for backing stores in the first category (such as ReCas).
Since we know at construction time what backing store needs local caching, we
can simply wrap these in the newly introduced LocalStoreCachedBackingStore
store.
For now, since the Mercurial backing store always writes a proxy hash to the
LocalStore, bypassing the LocalStore for trees would be a regression due to the
added disk IO. Once proxy hashes are gone for Mercurial, we can remove the
LocalStoreCachedBackingStore wrapper.
Reviewed By: chadaustin
Differential Revision: D31118905
fbshipit-source-id: 4a2958eafeeb8144ee4421ec44dbd30cedceee29
Summary: Same as D30974102 (91c4748c5b) but for mercurial cs.
Reviewed By: StanislavGlebik
Differential Revision: D31145642
fbshipit-source-id: c1be7b49bf0cbe70b844f1a31de706215a51d1ae
Summary: Same as D30974102 (91c4748c5b) but for fastlog.
Reviewed By: ahornby
Differential Revision: D31142066
fbshipit-source-id: 44a79e8a9db180736324db734b018344a77c070a
Summary:
Same as D30974102 (91c4748c5b) but for deleted manifest.
Needed some changes regarding using `DerivationContext` instead of `BlobRepo`.
Reviewed By: StanislavGlebik
Differential Revision: D31121260
fbshipit-source-id: f37daac320173b0896f12c83bdd8a723d22ec876
Summary:
Same as D30974102 (91c4748c5b) but for fsnodes.
Needed some changes regarding using `DerivationContext` instead of `BlobRepo`.
Reviewed By: StanislavGlebik
Differential Revision: D31113044
fbshipit-source-id: 6e996135f59f26e76e52b0b24ea61917216d1e53
Summary:
Same as D30974102 (91c4748c5b) but for skeleton manifest.
Needed some changes regarding using `DerivationContext` instead of `DerivedDataManager`.
Reviewed By: StanislavGlebik
Differential Revision: D31111484
fbshipit-source-id: eacc1d3247dffac4537745ec2a2071ef0abcbd43
Summary:
Same as D30974102 (91c4748c5b) but for changeset info.
This turned out quite simple, as we already have the bonsai changeset, so there's no need to do any async stuff.
Reviewed By: StanislavGlebik
Differential Revision: D31110319
fbshipit-source-id: 952686ae5583b858361b7a2a67fe914bfe5239d6
Summary: Now that EdenAPI fetching is turned on everywhere, let's make it the default.
Reviewed By: quark-zju
Differential Revision: D31184213
fbshipit-source-id: 450c1167d42ee867b505a2a14b0c636bed81107d
Summary:
It can be surprising when a job suddenly is no longer able to run sudo, or no
longer run as root that all the tests are marked as being successful, despite
the fact that they no longer run. Let's recognize when we run on EdenFS to
allow tests to fail if they can no longer run EdenFS.
Reviewed By: zhengchaol
Differential Revision: D30357402
fbshipit-source-id: c3758d7a5a3c575dd68bd97062ae24abe4124874
Summary:
Now that we might have multiple kernel protocols per mount (i.e. both fuse and
nfs on macOS) let's include them in eden rage.
Reviewed By: xavierd
Differential Revision: D31154042
fbshipit-source-id: 38e7630829d70fe9dd6dbeabacc3b538ee798e0d
Summary: dyn Drop produces warning because everything is Drop
Reviewed By: quark-zju
Differential Revision: D31175376
fbshipit-source-id: 78f55a60c9bb6d51cde9433ab2815ec133b15ecc
Summary:
We might have a somewhat weird case - a file was replaced with a directory and
then in the next the same file was deleted again (even though this file doesn't exist
anymore). In that case we need to make sure these two commits are in two
different stacks of commits, however previously we weren't doing that. This
diff fixes it.
Reviewed By: markbt
Differential Revision: D31168174
fbshipit-source-id: 4b9986e615ec98b6452ff81b113124d14f236382
Summary:
This works more reliably and fully restores `test-commitcloud-sync-race.t` to
pre-D28595292 (c72cd2333f) state.
Reviewed By: markbt
Differential Revision: D30974286
fbshipit-source-id: 729e20f23cb5d8aacdbcef1c869fc9a73ac4d4d4
Summary:
Change the state of visibleheads from the in-memory Python `_heads` variable
to the `svfs.metalog["visibleheads"]`. This changes a few things:
- No need to manually invalidate or reload the `_heads` state on transaction
close/reload, since metalog gets reloaded on transaction boundaries.
- No need to use features from the old transaction framework, such as
`addfilegenerator`, keeping `journal.visibleheads` for transaction
rollback. No need to track `dirty`.
This probably solves issues where `hg pull` hides visible heads unexpectedly.
See P458576970 for example reported by chadaustin where pull runs right after
a cloud sync and hides 5a8c51b193 unexpectedly, but metalog parent shows that
pull got the state after the cloud sync. See also
https://fb.workplace.com/groups/scm/posts/4114332378616351/ for a similar
report from dtolnay.
In theory other states (bookmarks, remotenames) might have similar issues.
But this diff only focuses on visibleheads.
Reviewed By: markbt
Differential Revision: D30974289
fbshipit-source-id: 85d81fd2e2d85ed22ac144f2cb663eb0423955fb
Summary:
`test-cross-repo-commit-validator.t` seems to take longer to run with the next
change. Bump the timeout to make it pass.
Reviewed By: markbt
Differential Revision: D31148285
fbshipit-source-id: 2c815d988b323eb08cf06256ee666130eeebf9a6
Summary:
It tests the Python pushrebase server logic which is no longer relevant.
The next change breaks it and it seems easier deleting the test.
Reviewed By: markbt
Differential Revision: D31121918
fbshipit-source-id: ee5619b35ad4aa16f0227e563ed531e879d1c8d7
Summary:
Store unchanged `self.heads` in a local variable. This avoids some overhead
if `self.heads` is going to be a bit more expensive.
Reviewed By: markbt
Differential Revision: D30974287
fbshipit-source-id: baaffb8f41da4b57e4ac94c305e5ad490a3c3135
Summary:
This is subtle. But fbhistedit (providing `exec` support) depends on it
invaliding everything related to repo to trigger state reloading after
executing a command (which could be `hg`).
Reviewed By: markbt
Differential Revision: D30974284
fbshipit-source-id: b033d81565dcf61104e4d30ecd7d48c33d6d79a4
Summary:
In a future change we'll require `svfs.metalog`. Let's move metalog fix to
before other stuff and attach fixed metalog to `svfs`.
Reviewed By: markbt
Differential Revision: D30974285
fbshipit-source-id: 3be89d1f1cda3d29dd5746940959ee47c1dd674d
Summary:
This allows doctor to construct changelog without requiring valid visibleheads
data. doctor cannot fix visibleheads first, because fixing visibleheads requires
changelog.
Reviewed By: markbt
Differential Revision: D30974288
fbshipit-source-id: 5bcf0f1918809fc0c7db3c89c70e0d17f961dc2c
Summary:
To make metalog replace more features supported by the transaction framework,
there is a need to expose pending metalog states to sub-processes. This diff
makes it so.
Reviewed By: markbt
Differential Revision: D30970502
fbshipit-source-id: 84192a14f4cef0765e4e361b61ab630311fd2dff
Summary: .drain() retains the drained container and its heap allocation for reuse, but as we're not reusing the container, moving the contents into_iter() makes the intent clearer
Reviewed By: StanislavGlebik
Differential Revision: D31149817
fbshipit-source-id: 07cc8b7cabc9b1d522daee8b13cfa6eeb96e2d30
Summary: .drain() retains the drained container and its heap allocation for reuse, but as we're not reusing the container, moving the contents into_iter() by for..in makes the intent clearer.
Reviewed By: StanislavGlebik
Differential Revision: D31149816
fbshipit-source-id: 63c7bba8a457e62a37944aecd8ec8c42dac8deaa
Summary:
No need to keep the Bytes live after compact_protocol::deserialize, can move them into it instead.
Makes it clearer bytes aren't reused, and should have some small effect on peak memory usage during deserialization by freeing the Bytes buffer earlier
Reviewed By: StanislavGlebik
Differential Revision: D31149815
fbshipit-source-id: 858914d2d8e3d91b5e863053dfeeb5d5ec37b9eb
Summary:
optimization for edenapi upload
Lookup for filenodes and trees can be done in parallel. Usually we have a small number of trees to check and a small number of filenodes, it is better to send them in a single lookup request, so they all can be checked in parallel. Parallelism limit for the lookup request is few thousands, so almost always if we merge the requests here, they will be parallelised.
Reviewed By: yancouto
Differential Revision: D31127401
fbshipit-source-id: 8014b27a2ba9d082babe2e0cd7bebf43c8b46082
Summary:
add scuba metrics for stages of EdenApi Uploads
add cloud sync reason for manual run
This is an effort to improve our Eden Api Uploads metrics and Commit Cloud metrics, so we can analyse and improve its performance.
Reviewed By: markbt
Differential Revision: D31109948
fbshipit-source-id: ee5a449e2652ea1798997ae2c52c4672f55e3eae
Summary:
I recently added this feature but it had a bug - when DontAggregate mode was
used it compared file changes of a new commit with the previous commit only
instead of all changes in the stack.
Since FileAggregation is broken let's remove it and collect both file changes
for the whole stack and for a given commit
Reviewed By: mitrandir77
Differential Revision: D31145055
fbshipit-source-id: 99dbedb919fb9edbdfaeaa658d49a08d008bd282
Summary:
The `ObjectFetchContext::Origin::FromBackingStore` is widely interpreted as
meaning that a network fetch was performed, but for some backing stores, this
isn't true. The Mercurial backing store for instance can either read data from
its on-disk cache, or from the network. Since both have very different
characteristics we shouldn't bundle them in the same enum value.
Since the backing store knows how data was obtained, let's have the backing
store return how it was obtained to enable the ObjectStore to properly record
this information. The `FromBackingStore` is also renamed to make it clearer
what its purpose is.
Reviewed By: zhengchaol
Differential Revision: D31118906
fbshipit-source-id: ee42a0c9d221f870742de07c0df7c732bc79d880
Summary:
we are passing some bytes into Popen and shlex.quote. shlex.quote expects a
string not bytes. fsencode gives us bytes fsdecode gives us string. Let's used
fsdecode instead.
Reviewed By: zhengchaol
Differential Revision: D31129335
fbshipit-source-id: 7792bdcd4dd833a4946daf8ec75576cfe4fc24af
Summary:
Derived data manager now doesn't allow deriving a batch of commits if all
ancestors weren't derived yet (and that's a good idea to do this check).
But it started to break benchmark if --batch-size, --backfill and --parallel
options are set, because in the very
beginning of the function we mark all commits as not derived, and when we start
deriving the second batch the first batch is assumed to not be derived, and
this triggers derived data manager check.
Let's instead mark only commits that we are about to derive as not derived, and
clear this check once we are done.
Reviewed By: mitrandir77
Differential Revision: D31140464
fbshipit-source-id: fc74d58dc3c4a3ad70e8e2527f7d6dfc8fde8a9c
Summary:
I'd like to reuse them in the next diff, so let's refactor it a bit.
Note - in D30837581 (315a8b311d) markbt suggested a good idea for refactoring
backfill_derived_data. I liked the idea, but when I tried to approach this
refactoring it turned out to be tricky to do so (FWIW, it might be easier to
rewrite everything from scratch). So for now I did the smallest possible
refactoring that's needed to add validation in the next diff, but this small
refactoring can probably be used for a larger refactoring later.
Reviewed By: mitrandir77
Differential Revision: D31115979
fbshipit-source-id: f0b4d70454186a023cd9e12cd645768af1b716e8
Summary:
I'd like to use it in the next diffs to add a way to validate that derived data
is the same after rederivation. A lot of the code in `benchmark.rs` is useful
for doing this validation, so let's rename `benchmark.rs` so that it's ok to
use it from two different subcommands.
Reviewed By: mitrandir77
Differential Revision: D31115981
fbshipit-source-id: 86439534d8e49a4022086cb27918b7bcd7befc5c
Summary: Share the megarepo api object as it contains repo configs which are quite large
Reviewed By: StanislavGlebik
Differential Revision: D31144245
fbshipit-source-id: bc87bac8d51d255d3f0a2d6dfbacf97ec89a565d
Summary:
We update bookmarks and insert sync mappings in separate transactions. Let's
adjust for that (see comment for details)
Reviewed By: StanislavGlebik
Differential Revision: D31113203
fbshipit-source-id: 95264bac0dfffee3fef82d475ccf7c4a62b81f24
Summary:
This tests legacy bookmark pull behavior which we no longer use. Let's
delete it.
Reviewed By: quark-zju
Differential Revision: D31032053
fbshipit-source-id: c1ab3b2e17a81871eae377f02947fecc438d7209
Summary:
This test tests a lot of legacy repo formats and a bunch of
peer-to-peer clones/pushes, etc which are no longer supported. Let's delete it.
Reviewed By: quark-zju
Differential Revision: D31032056
fbshipit-source-id: bc3ed9226e1a2d4656630ea5f99a904673268da6
Summary:
This test is fragile with respect to .hg store changes. We don't have
strong requirements of .hg permissions these days since we don't share repos
between people really. So Let's delete this test.
Reviewed By: quark-zju
Differential Revision: D31032058
fbshipit-source-id: 2d6b1ffed064ad41fda8938cfabc64291f509015
Summary:
This tests details of the getbundle wireproto command. This is becoming
less and less relevant as we move to edenapi, and the test itself is basically
impossible to maintain since it's heavily dependent on hashes. Let's just delete
it.
Reviewed By: quark-zju
Differential Revision: D31032062
fbshipit-source-id: 31f007ab989ebcd93424f2aea09168edc368a6c0
Summary:
Previously newserver/clone supported ssh and mononoke. Now let's
support eagerrepo as well
Reviewed By: quark-zju
Differential Revision: D31032061
fbshipit-source-id: 5016bb74b317523611c645c040da92c8cbf897c1
Summary:
This is part of removing filepeer. I also enabled treemanifest and
modernclient (i.e. lazy changelog) on a few tests.
Reviewed By: quark-zju
Differential Revision: D31032060
fbshipit-source-id: cb45831892a6c75d89a553c23f3318e188f9489f
Summary:
Removes the tests usage of server logic, by migrating it to
modernclient.
Reviewed By: quark-zju
Differential Revision: D31032057
fbshipit-source-id: d7f5d2cf4fae4d0edd364ad435d17e4083c3297f
Summary:
Removes usage of server logic from test-bookmarks.t.
This test is long and convoluted. I decided that the latter half of the test
wasn't really relevant/important to our usage of Mercurial so I deleted it
instead of trying to fix it.
Reviewed By: quark-zju
Differential Revision: D31032059
fbshipit-source-id: ac32d8179eb1caaa6383015ec3f6f67b2fc9106e
Summary:
We want to get rid of server logic in Mercurial, and to do so we want
to get rid of filepeer. To make porting legacy tests easier, let's create some
helper functions to simulate clones, pushes, and pulls.
Reviewed By: quark-zju
Differential Revision: D31004444
fbshipit-source-id: def6fc4fc3d43c313eb4f2a6b0adcba7ddb5e00f
Summary:
We want to remove filepeer as part of removing server logic in the
client. To start with, let's disable it by default and only enable it in tests
that need it. The next step will be to update those tests.
Reviewed By: quark-zju
Differential Revision: D30977765
fbshipit-source-id: c56016f017e894a15bf43fb7a8d3a0a417663ad9
Summary:
The duality of this function is a bit awkward, especially for backing stores
that want metadata caching, but not blob caching. This makes the code in
ObjectStore more complicated that it needs to be.
This will also be used in a future diff.
Reviewed By: chadaustin
Differential Revision: D31090782
fbshipit-source-id: cb7d7fc44d8780f86abad166d0f099675d29e5e7
Summary:
Similarly to blobs, fetching trees is converting data from HgImportRequest to
individual hashes and proxy hashes vector, making copies of these. This is
inefficient and it makes the code harder to read and understand. Passing the
batch of HgImportRequest directly avoid these copies and makes the code easier
to read.
Reviewed By: fanzeyi
Differential Revision: D30583567
fbshipit-source-id: e85952975141c92f9524095c62418baabf8fefcd
Summary:
Both the caller and the function itself are copying data to make sure that the
next function gets the data in the format it desires. This makes the code
complicated to read as well as inefficient as each conversion ends up being a
copy.
We can simply pass the HgImportRequest directly to avoid both of these issues
which removes all the copies.
A similar change will be done for the getTreeBatch function, after which the
HgDatapackStore will be folded onto the HgBackingStore code.
Reviewed By: fanzeyi
Differential Revision: D30563706
fbshipit-source-id: bb392f89e691c22ff9ad4df0d365ddb62077e657
Summary:
We can configure chaos blobstore and a bunch of other options. But sometimes
it's useful to be able to add an artificial delay to the blobstore.
Reviewed By: mitrandir77
Differential Revision: D31110811
fbshipit-source-id: 7b3a4d1b85047a5daff9183df8aa5db8971a7d9d
Summary:
The test does not discover new issues but it seems nice to have protecting
behaviors.
Reviewed By: StanislavGlebik
Differential Revision: D31106124
fbshipit-source-id: 4060b16f2dac34ff29c0d4cee1ce19a9a61014ee
Summary:
There are 2 issues:
- Previously it only follows parent -> child relations when parent is the head
of a segment. It's also possible that we need to follow parent -> child when
parent is in the middle of a flat segment. The added ASCII graph in
`iddag.rs` shows when that happens.
- Previously there is no check about following p2 (second parent) for linear
history. The check needs to be added and p2 of merges cannot be resolved.
This change makes `x~0` result an error. Some `x~n` are changed to alternative
ones because more child segments are visited (see also D30919581 (3ef445c0a6)).
Reviewed By: StanislavGlebik
Differential Revision: D30919579
fbshipit-source-id: eaa69d31612a2006b0b46132a797025536288624
Summary:
The endpoint "hash to location" cannot really expose individual errors
correctly. The type `CommitHashToLocationResponse` allows exposing individual
errors for each hgid but the implementation will just take one error and report
such error for all hgids. This is because the use of
`collect::<Result<ContainerType>>()?` collapses multiple individual "ok" and "err"
values into one "err".
Change the stack so we can preserve individual errors.
Reviewed By: StanislavGlebik
Differential Revision: D31106125
fbshipit-source-id: 46b1e8b2fcee00b8d8dda0935e2a005ee8aef0ee
Summary:
Those two commands allow us to easily make another job pick-up unfinished
request and finish a request with error.
aborting will actually make the worker stop working on the requests the next
time worker tries to update the heartbeat.
Reviewed By: StanislavGlebik
Differential Revision: D30965078
fbshipit-source-id: 5b683ea09c7dd74ff903416f2a1d393f9b55c588
Summary: Fix bug in httpenabled check. Only local repo has edenapi attribute and not remote.
Reviewed By: quark-zju
Differential Revision: D31059979
fbshipit-source-id: c77f1455949f28b081f8fef86aeb83ecd1c3ceba
Summary: Move conditionals for using edenapi commit graph endpoint to a common method. This fixes the conditional check in _pullchangeset which should match the one in pull.
Reviewed By: quark-zju
Differential Revision: D31030783
fbshipit-source-id: e665067bcf12bc8211850d547df42bc48727de48
Summary: The url for commit graph on the server is repo/commit/graph and repo/commit/commit_graph on the client. Update client to match server url.
Reviewed By: quark-zju
Differential Revision: D31030784
fbshipit-source-id: fb58c53a9264498f0e488abadd5f2b5480f0c672
Summary:
The `unfiltered` field was to support legacy clients that are months old. Drop
support for it to clean the code a bit.
Reviewed By: StanislavGlebik
Differential Revision: D31106176
fbshipit-source-id: 7cfd6827812258776d6d9eac512b497777fecb70
Summary:
Previously it just passes "failed to compute the common descendant and distance
for X", the outmost layer (context) of the error to the client. Make it contain
more messages about the inner error and heads explicitly.
Reviewed By: StanislavGlebik
Differential Revision: D31071741
fbshipit-source-id: ec4f161491e14ae0e0c422f5e92a3849a5a29b56
Summary: This will be used by upcoming changes.
Reviewed By: DurhamG
Differential Revision: D30919581
fbshipit-source-id: 5e4c13004c577d4a87948c1985bdf48ebd1379a3
Summary: This method will be used in the next change.
Reviewed By: DurhamG
Differential Revision: D30919580
fbshipit-source-id: a5cab7fad0dda0de15244a9d7c1887534f061f3a
Summary:
After D26360561 (aa9dfeff2e), `IdDag<Store>` derefs to `Store` for algorithms. There is no
need to duplicate the methods in `IdDag<Store>`.
Reviewed By: yancouto
Differential Revision: D30911148
fbshipit-source-id: d476e318ab565369b93f4bd8467989a2a2e76f82
Summary:
Added checks so the `x` in the returned `x~n` is a parent of a merge that is an
ancestor of requested heads.
Reviewed By: DurhamG
Differential Revision: D30911149
fbshipit-source-id: 09fd6a01ea346a53c8d2ff6042951c8094cb8b04
Summary:
After migration to Mononoke & EdenAPI supporting different destinations don't make sense.
The destination is always either default path or edenapi endpoint.
Reviewed By: yancouto
Differential Revision: D31081193
fbshipit-source-id: 49a02899173690cfcb55bf8fc6aec13b05377ca6
Summary: It used to be flaky. Perhaps ncat exited by itself in some cases so there was nothing to kill. Just ignore the kill command's output. As long as we get the cats header the test is fine.
Reviewed By: HarveyHunt
Differential Revision: D31108514
fbshipit-source-id: e2791610c49bacdc31c26710f087cbe49f4d734e
Summary: It can be useful if we want to rebase a lot of commits at once
Reviewed By: mitrandir77
Differential Revision: D31106975
fbshipit-source-id: 26bb7f3fd166b483c44a4991ede4f0a9a462fff7
Summary: Follow the D30313559, this diff will enable ```download_directory``` with eden option
Reviewed By: Skory
Differential Revision: D30549600
fbshipit-source-id: f87326bd7c17abd975eab823c62ee0ffe3bf7443
Summary: Add support for fetching file aux data from BackingStore via scmstore.
Reviewed By: andll
Differential Revision: D31048574
fbshipit-source-id: ccb783272d0ffe32277debdaac45c00fc4df9ddb
Summary:
We've never really supported this and it has caused weird behavior in
the past. Let's delete it.
Reviewed By: quark-zju
Differential Revision: D31027574
fbshipit-source-id: 297668e35db3c4d5ea17d2a18fc0127ca694cc45
Summary: This is no longer used, now that configerator does it's sync via SCS.
Reviewed By: quark-zju
Differential Revision: D31006839
fbshipit-source-id: ddf7923a3c80ad47519bce0e5cb3895f62e7ae2e
Summary:
The server_realhostname log field no longer contains just the server hostname,
but instead was changed to the format `HOSTNAME session SESSIONID`, so that the
client knows the server session id.
Redefine the response format to `HOSTNAME (INFOKEY INFOVALUE)*`, so that
session is just an info field from the peer, and parse the response and log its
components.
Reviewed By: quark-zju
Differential Revision: D31081922
fbshipit-source-id: 28f963b4a44f7e9bdd75393ac9b14a2d1a4b5bd0
Summary:
Create rust progress bar model in python whenever any bar is created, as opposed to only when the bar is rendered. This makes progress metadata always available in rust, regardless of whether progress ends up getting displayed or not. In particular, this enables upcoming "global progress" work to generically expose all hg invocations' progress.
This change necessitated rust to respect more cases where progress should not be displayed. For example, "quiet" mode was previously only respected in python by creating a type of bar that did not end up being registered in rust. Now since all bars are registered, rust needs to know not to render output in this case and others.
Note that I discovered a config discrepancy between python and rust in the norepo=True command mode. Work around for now by changing debugprogress to optionalrepo.
Reviewed By: quark-zju
Differential Revision: D31005075
fbshipit-source-id: 0fa6fb594a15b34b3ae260343526496f8f8b05f8
Summary:
Add rust utility function to test for HGPLAIN and HGPLAINEXCEPT, analagous to python's ui.plain() method.
I need this so rust can expand its logic of whether to render progress bars or not.
Reviewed By: quark-zju
Differential Revision: D31070840
fbshipit-source-id: 0c394234b9c4337bbddaa08d7a9b1d8ed59dfec0
Summary:
Mononoke blame derivation differs from Mercurial: if a file is copied over
(i.e., it exists in one or more parents, but also has copy-from info for
a different path), then Mononoke treats this as a merge between the parent
content(s), and the copied-from contents, whereas Mercurial ignores the
parents and just continues the blame from the copied-from contents.
Change Mononoke to match Mercurial behaviour. This is a breaking change:
blame_v2 derived before this change will not match blame_v2 derived after
this change. We will rederive blame_v2 for all repositories affected by
this.
Reviewed By: StanislavGlebik
Differential Revision: D31079507
fbshipit-source-id: 33eb4ce54b99712d551ff46e9e96ccd7e5b78d80
Summary:
The intent behind the refresh method is to both read new data from the disk,
but also to flush the in-memory write buffer to disk. The name "flush" is used
in the revisionstore code to mean the latter, thus let's use "flush" in the
rest of the codebase.
Reviewed By: kmancini
Differential Revision: D30947873
fbshipit-source-id: c85a6abe770a47d6ce454d6af1fa73e505194a22
Summary:
measure pull and push time for cloud sync
This is needed to be able to analyse this data in scuba
Reviewed By: yancouto
Differential Revision: D31081328
fbshipit-source-id: 404df394443282d62845fad094153a83d71d3150
Summary: ACE will have to run in situations where Chef has not run, but we'll need to be able to reliably write to the auth logs so Blackbird can properly build detections. So we need these crates so we can build the somewhat foolproof solution to ensure ACE logs all executions.
Reviewed By: farnz
Differential Revision: D31066559
fbshipit-source-id: 9fa3b5778cd2602bdeaac90a9daa758b117babfe
Summary:
Some requests can occasionally cause fetches from the database for lots of
changesets in one go. This causes excessive load on a single database
instance. Chunk these requests into a more reasonable size.
Reviewed By: krallin
Differential Revision: D31056868
fbshipit-source-id: cbc47f7ef40bfbefaca86b1516fb574772cb531e
Summary:
Add support for Phabricator's "committing", "recently failed to land", and
"needs final review" states. These are fake states based on other status
fields in the response from Phabricator.
Reviewed By: quark-zju
Differential Revision: D31055125
fbshipit-source-id: 16b17e1934efcb5d829dd8d4c72ea8465687ab46
Summary:
folly:format is deprecated in lieu of fmt and std::format. Migrate
most of EdenFS to fmt instead.
Differential Revision: D31025948
fbshipit-source-id: 82ed674d5e255ac129995b56bc8b9731a5fbf82e
Summary: It's useful to have this data
Reviewed By: mitrandir77
Differential Revision: D31053459
fbshipit-source-id: cbe9a3fad30563812218b8b9b5d4c3aaeb279285
Summary:
It's useful to know the repo name (we can get it from tw handle name, but
that's less convenient).
Reviewed By: mitrandir77
Differential Revision: D31053458
fbshipit-source-id: fa7e92c510ea6160c52561d4a7a7c44776c528dc
Summary:
Backport: https://github.com/briansmith/ring/pull/1334
This will allow us to unpin Rust compiler to 1.53.0 and update to 1.55.0.
Reviewed By: xavierd
Differential Revision: D31039024
fbshipit-source-id: f6a9c918e836d93d03c34c77c12bbe63cf7cbe09
Summary:
Previously repo and peer instantiation were in one unified path. This
allowed treating repo's and peers somewhat interchangably. We're moving to a
world where peers and local repos are quite different, so let's separate these
two paths.
This will be useful in the next diff where we remote the file peer, but want to
keep the ability to instantiate local file non-peer repos.
Reviewed By: quark-zju
Differential Revision: D30975887
fbshipit-source-id: 5e676b522c7cfdd5449aeb6a750947dcb023183f
Summary:
We don't use this at Facebook, and most of the tests don't even touch
it anymore. Let's delete it. This will also help us remove our tests dependency
on hg having server logic, once we also delete sshpeer and filepeer.
This will mean we can't use FB hg to clone from http bitbucket though, which is
probably fine.
Differential Revision: D30970713
fbshipit-source-id: 76d96edfbcb7db2168b4b11bfaf8b487406d7f3d
Summary:
Switch derivation of `blame` to the `DerivedDataManager`.
This is mostly the same as the existing derivation implementation. The main difference is that `blame` derivation using the backfilling config
will use the backfilling config for the unodes that it depends on, too.
Reviewed By: mitrandir77
Differential Revision: D30974102
fbshipit-source-id: 5f69f8c218806bb7606b2af4b831e2104b8440d6
Summary: Why not, right? Fixes a few build warnings that showed up to me while building.
Reviewed By: kulshrax
Differential Revision: D30933487
fbshipit-source-id: 318fbd2c5697914fd0bfa723e678dc710524dc02
Summary: There were already helpers to make this code less copy-pasty, this diff just uses them.
Reviewed By: markbt
Differential Revision: D30933408
fbshipit-source-id: acc27a0904425eccfc71fee884a8f2035ed0c37f
Summary:
We already have a macro to make it easier to create wire representation of hash types, let's use it on `HgId` to reduce copy-pasting.
Changes:
- Added `Ord` implementations to wire hash types, as `WireHgId` used it.
- Added from/into implementations on `HgId` to byte arrays, which were used by the macro.
- Changed Debug implementation so it prints hex instead of an actual array of bytes
Reviewed By: krallin
Differential Revision: D30933067
fbshipit-source-id: c88911bfc91e44e07f2f658098036b766495d05f
Summary:
I imagine a pretty common case (specially for automation that's trying to keep two clones in sync), will be that you need to restore a snapshot and then restore another snapshot after that.
Currently, this doesn't work very well, as it fails on (some but not all) cases where there is uncommitted changes. It's kind of boring bc to handle that you need to run `hg purge && hg revert -a -C`.
This diff adds a `--clean` option to `hg snapshot restore` that will clean the working copy before updating to given snapshot. Now the command will also fail if you try to update to a snapshot while you have untracked files.
Reviewed By: markbt
Differential Revision: D30903851
fbshipit-source-id: 387eeeee882093389649dc337c861291c35f4b94
Summary:
The `backfill_batch_dangerous` method requires that the caller ensures
that all dependencies of the batch have been derived, otherwise errors,
such as mappings being written out before the things they map to, can
occur.
When the derived data manager takes over batch derivation, it will enforce this
requirement, so that it is no longer dangerous. However, The backfiller tests
were not ensuring the invariant, so the tests will fail with the new derivation
implementation.
Fix the tests by ensuring the parent commits are always derived before a
batch is started. The test is also extended to expose the failure mode
of accidentally deriving batch parents. This will be fixed in the next
commit.
Reviewed By: yancouto
Differential Revision: D30959132
fbshipit-source-id: 8489a5d0b375692a903854294e3810846c9e13de
Summary:
Implement `DerivedUtils` using the `DerivedDataManager`.
This is just for migration. In the future `DerivedUtils` will be replaced by the manager.
Reviewed By: yancouto
Differential Revision: D30944568
fbshipit-source-id: 32376e3b4aeb959e63f66e989a663c21dee30ba5
Summary:
Implement a new version of data derivation in the derived data manager. This is different from the old version in a few ways:
* `derived_data::BonsaiDerivable` is replaced by `derived_data_manager::BonsaiDerivable`. This trait defines both how to perform derivation and how to store and retrieve mapping values. Derivation is performed with reference to the derived data manager, rather than `BlobRepo`.
* The old `Mapping` structs and traits are replaced with a direct implementation in the derived data manager, using the `BonsaiDerivable` trait to handle the derived-data-type-specific parts.
* The new implementation assumes we will stick with parallel derivation, and doesn't implement serial derivation.
Code is copied from the `derived_data` crate, as it is intended to be a replacement once all the derived data types are migrated, and re-using code would create a circular dependency during migration.
This only covers the basic derivation implementation used during production. The derived data manager will also take over backfilling, but that will happen in a later diff.
Reviewed By: yancouto
Differential Revision: D30805046
fbshipit-source-id: b9660dd957fdf762f621b2cb37fc2eea7bf03074
Summary:
The `find_oldest_underived` method of `DerivedUtils` is used outside tests by
exactly one client (the backfiller in tailing mode). Simplify the
`DerivedUtils` trait by extracting this method from the trait, and replacing
with a more general one that will be easier to implement in terms of the
derived data manager.
Reviewed By: yancouto
Differential Revision: D30944567
fbshipit-source-id: a1d408e091d145297241a5eebc02a87155bc3765
Summary:
Split the `BonsaiDerived` type in two:
* `BonsaiDerived` is now just the interface which is used by callers
who want to derive some derived data type. It will be implemented by
both old and new derivation.
* `BonsaiDerivedOld` is the interface that old derivation uses to
determine the default mapping for derivation. This will not be
implemented by new derivation, and will be removed once migration is
complete.
Reviewed By: yancouto
Differential Revision: D30944566
fbshipit-source-id: 5d30a44da22bcf290ed3123844eb712c7b37dea4
Summary:
The builder pattern turned out to be unnecessary, as mappings don't need to be
stored in the manager after all.
Reviewed By: StanislavGlebik
Differential Revision: D30944565
fbshipit-source-id: 4300cdcc871c89f98e42d5b47600ac640b4b94eb
Summary:
Make the derivation process for mercurial filenodes not depend on `BlobRepo`.
Instead, use the repo attributes (`RepoBlobstore` and `Filenodes`) directly.
This will allow us to migrate to using `DerivedDataManager` in preparation
for removing `BlobRepo` from derivation entirely.
The existing use of `changesets` for determining the commit's parents is
changed to use the parents from the bonsai changeset. For normal derivation,
the bonsai changeset is already loaded, so this saves a database round-trip.
For batch derivation we currently need to load the changeset, but it should
be in cache anyway, as other derived data types will also have loaded it.
We still need to keep a `BlobRepo` reference at the moment. This is because
filenodes depend on the mercurial derived data. The recursive derivation is
hidden in the call to `repo.get_hg_from_bonsai_changeset`. When derivation
is migrated to the derived data manager, we can replace this will a direct
derivation.
Reviewed By: StanislavGlebik
Differential Revision: D30765254
fbshipit-source-id: 20cc17c2eb611544869e5f1c15d858663cd60fd1
Summary:
Let's give them a more descriptive names so that it's easier to understand
what's going on.
Reviewed By: markbt
Differential Revision: D31022612
fbshipit-source-id: 8e4f516f3d0b1cd661b1a8fceba80a8f85a2ed4f
Summary:
This is a new option in split_batch_in_linear_stacks - it either aggregates
file changes from all ancestors in the stack or not. Currently all of our
callsites wants Aggregate, but in the next diff we'll add a new callsite that
doesn't
Reviewed By: markbt
Differential Revision: D31022444
fbshipit-source-id: ce0613863855163f26ab18c7f35142ae569eb31a
Summary:
EdenFS would never log anything when mounting via NFS, let's make it more
visible and easier to grep.
Reviewed By: chadaustin
Differential Revision: D31022158
fbshipit-source-id: 99fd3a04c90526eedf9951ac7c2bcd9e18ef8953
Summary:
this relies on local changes to make it so cargo metadata ACTUALLY finds this
binary: https://github.com/tokio-rs/console/pull/146 is where I try to upstream
it
Reviewed By: jsgf
Differential Revision: D30944630
fbshipit-source-id: 5d34a32a042f83eff7e7ae7445e23badf10fffe3
Summary: For the time being we don't have checksums in saved states. As a temporary workaround add the ability to derive the checksum from the naming table.
Differential Revision: D30967637
fbshipit-source-id: 4ac34d988d08c9af6f08f7ce46206f756cf1cf0c
Summary: Watchman is a C++17 project now, so we can use std::optional.
Reviewed By: xavierd
Differential Revision: D30917549
fbshipit-source-id: 95d8ac15d4939a70347336ddfb120ab5025db993
Summary:
Having tons of booleans in a function can be very error prone from a caller
perspective, using a structure to pass in the same information can mitigate
some of this issue.
Reviewed By: kmancini
Differential Revision: D30883743
fbshipit-source-id: dcf38d29bfe2cb5155879f7ae4eab5cea31f798a
Summary: Without this bit of information we can't tell where the sync came from (i.e. from which of two repos) so we can't reliably find a commit "source" for a landed commit.
Reviewed By: StanislavGlebik
Differential Revision: D30902774
fbshipit-source-id: d85d0d028fbd6bfb2d64bce89bc7934bad2e242b
Summary:
During an `eden chown`, EdenFS will try to chown both the repository, and the
redirections. In some cases, chowning the redirection can both take a long time
and be unecessary. Consider the case where some automation temporarily chown a
repository to a service user that needs to access the repository, and then
chown it back to the owner of the repository. In that case, changing the
ownership of the redirection is superfluous and unecessary.
Reviewed By: mrkmndz
Differential Revision: D31010912
fbshipit-source-id: a882948005ac4fe29ff465088f196e0fc2bc10be
Summary:
This is a very basic commands that uses debug-printing to display all the
request details. In the future we might want to make it more ellaborate but
as-it-is it works.
Reviewed By: StanislavGlebik
Differential Revision: D30965076
fbshipit-source-id: 561c64597b94359843e575550be0ae6f39fad7bf
Summary:
This debug command will allow the user to see and interact with currently
running async requests.
Reviewed By: StanislavGlebik
Differential Revision: D30965077
fbshipit-source-id: 259f1af0eb6ade4a34f6004c7b1ad63cd5f0bc9f
Summary:
It makes it a bit hard to do experiments and compare derivation results.
It's easy to compare these types, so let's do it.
Reviewed By: mitrandir77
Differential Revision: D31017823
fbshipit-source-id: 6173bba53c7ee254198e023dde57564fe9c3efed
Summary:
This will be used in the next diffs to add batch derivations for unode.
Also it makes it symmetrical to create_manifest_unode
Reviewed By: mitrandir77
Differential Revision: D31015719
fbshipit-source-id: 65e12901c6a004375c7c0e3b07f1632ac9c6eaa8
Summary:
In some cases (e.g. when master bookmark moves backwards) there might be
commits in segmented changelog that are not ancestors of master. When reseeding
we still want to build segments for these chagnesets, and this is what this
diff does (see D30898955 for more details about why we want to build segments
for these changesets).
Reviewed By: quark-zju
Differential Revision: D30996484
fbshipit-source-id: 864aaaacfc04d6169afd3d04ebcb6096ae2514e5
Summary:
In D29940980 (2e2b9755cf) we used shlex for a redirect subprocess command line.
The list does not always contain strings tho, which is a requirement to use
shlex.quote my guess is that they are paths. We should still str things
before we shlex.quote them.
Differential Revision: D31001622
fbshipit-source-id: 2a270781d7f2d84ad7a9a2f9975500b29306cfa8
Summary:
One of the largest contributor to EdenFS memory usage are the internal
IndexedLog buffers to hold data in memory until a threshold is reached. Since
the main benefit of these buffers is to utilize the disk bandwidth, very large
buffers aren't necessary and much smaller ones will be able to achieve similar
results.
A default 50MB buffer is used which will cap the memory usage to 50MB * 3:
- File IndexedLogDataStore
- Tree IndexedLogDataStore
- File LFS
The aux and history stores are also reduced down to 10MB.
Reviewed By: DurhamG
Differential Revision: D30948343
fbshipit-source-id: 74e789856ac995a5672b6aefe8a68c9580f69613
Summary:
We periodically need to dereference inodes on NFS because we get no other info
from the kernel on when should dereference them.
This means the NFS kernel might have references to inodes after we delete them.
An unknown inode number is not a bug on NFS. It's just stale, so the error should
reflect that.
Reviewed By: xavierd
Differential Revision: D30144898
fbshipit-source-id: 3d448e94aea5acb02908ea443bcf3adae80eb975
Summary:
We periodically need to dereference inodes on NFS because we get no other info
from the kernel on when should dereference them.
It can be disruptive to a users workflow because an open files that were rm'ed
or removed on checkout will no longer have their old content. (on a native
filesystem or fuse applications that had the file open propr to the removal
would still be able to access files.) For most editors this is not a problem
because they read the file on open (seems fine for vim and vscode from testing).
However folks could theoretically have a workflow this does not jive with.
Let's make it configurable how often this runs, so users can control how
much we distupt their workflow.
Reviewed By: xavierd
Differential Revision: D30144899
fbshipit-source-id: 59cf5faea70b3aea216ca2bcb45b96e34f5e72b5
Summary:
NFSv3 has no inode invalidation flow built into the procall. The kernel does not
send us forget messages like we get in FUSE. The kernel also does not send us
notifications when a file is closed. Thus EdenFS can not easily tell when
all handles to a file have been closed.
As is now we never clean up inodes. This is bad for memory & disk usage.
We will never unload an inode so we always keep it in memory once it's created.
Additonally, we never remove a materialized inode from the overlay. This means
we have unbounded memory and disk usage :/
We need to clean up these inodes at somepoint. There are a couple high level
options:
1. Support nfsv4. NFSv4 sends us close message when a file handle is closed.
This would allow us to actually keep track of reference coundts on an inode.
However, This is a lot of work. There is a lot of other things we would have to
support before we can move to nfsv4.
2. Run background inode cleanups.
nfsv4 is probably the right long term solution. But for now we should be able to
get by with periodic unloads.
I considered a couple of options for unloads:
1. Unload inodes immediatly when files are removed.
2. Delay cleaning up inodes until a while after they are removed. (i.e. clean
up inodes n seconds after an `unlink`, `rename`, `rmdir`, or checkout)
3. Run periodic inode unloading. (i.e. once a day unload inodes).
Option 1. feels a bit too hostile to applications that hold files open.
Option 3. means we will build up a lot of cruft over the course of the day. But is
probably the most application friendly.
I decided to try out option 2 first and see if it works well with the common
developer tools. Its seems to work (see below) so I am going with it.
This diff only does inode cleanup after checkout. we might want to run inode
clean up after unlink/remove dir as well, but this would be more expensive.
Batch unloading feels better on checkout seems better to me and should happen
frequently enough to clean up space for people.
There is one known "broken" behavior in this diff. We unload all unlinked
inodes which means we will erase more inodes than we should. Sometimes EdenFS
crashes or bugs and unlinks legit inodes. Normally we let those live in the
overlay so we could go in an recover them. My plan to fix this is to mark inodes
for unloading instead of just unloading all unlinked inodes.
Reviewed By: xavierd
Differential Revision: D30144901
fbshipit-source-id: 345d0c04aa386e9fb2bd40906d6f8c41569c1d05
Summary:
Delete a non-existing file is fine, and also deleting a file when a directory
with the same name just ignores the delete.
This diff adds tests to cover these cases. Overall it seems like a bug, but I'm
not sure it worth fixing - who knows if we have bonsai changesets that rely on
that!
Reviewed By: yancouto
Differential Revision: D30990826
fbshipit-source-id: b04992817469abe2fa82056c4fddac3689559855
Summary:
This method allows to append a value instead of just replacing it.
It will be used in the next diff when we derive manifest for a stack of commits
in one go.
Reviewed By: yancouto
Differential Revision: D30989889
fbshipit-source-id: dd9a574609b4d289c01d6eebcc6f5c76a973a96b
Summary:
The NFS protocol needs to know if a read reached the end-of-file to avoid a
subsequent read and thus reduce the chattyness of the protocol.
On top of avoiding RPC calls, this should also halve the amount of data read
from Mercurial due to the BlobCache freeing the in-memory cached blob when the
FS has read the file in its entirety. This meant that the second READ would
always force the blob to be reloaded from the Mercurial store, which would also
force that blob to be kept in memory until being evicted (due to it not being
fully read).
Reviewed By: kmancini
Differential Revision: D30871422
fbshipit-source-id: 8acf4e21ea22b2dfd7f81d2fdd1b137a6e90cc8f
Summary:
Changes:
- Limit simultainous open git-repo objects by amount of CPUs.
- Put a semaphore limit so we wait inside tokio::task domain instead of tokio::blocking domain (later is more expensive and has a hard upper limit).
Reviewed By: mitrandir77
Differential Revision: D30976034
fbshipit-source-id: 3432983b5650bac6aa5178d98d8fd241398aa682
Summary:
This allows the mononoke_api user to choose whether the skiplists
should be used to spped up the ancestry checks or not.
The skiplists crate is already prepared for the situation where skiplist
entries are missing and traverses the graph then.
Reviewed By: yancouto
Differential Revision: D30958909
fbshipit-source-id: 7773487b78ac6641fa2a427c55f679b49f99ac8d
Summary:
Allow the mononoke_api user to choose whether they want
oprerations to be sped up using WBC or not.
Reviewed By: yancouto
Differential Revision: D30958908
fbshipit-source-id: 038cf77735e7c655f6801d714762e316b6817df5
Summary:
Some crates like mononoke_api depend on warm bookmark cache to speed up the
bookmark operations. This prevents them from being used in cases requiring
quick and low overhead startup like CLIs.
This diff makes it possible to swap out the warm bookmark cache to a
implementation that doesn't cache anything. (See next diffs to see how it's
used in mononoke_api crate).
Reviewed By: yancouto
Differential Revision: D30958910
fbshipit-source-id: 4d09367217a66f59539b566e48c8d271b8cc8c8e
Summary:
This method was added before the more generic list method was added. Let's get
rid of it for simplicity and to discourage listing all the bookmarks.
Reviewed By: yancouto
Differential Revision: D30958911
fbshipit-source-id: f4518da3f34591c313657161f69af96d15482e6c
Summary:
0.4.24 is incompatible with crates that use `deny(warnings)` on a compiler 1.55.0 or newer.
Example error:
```
error: unused borrow that must be used
--> common/rust/shed/futures_ext/src/stream/return_remainder.rs:22:1
|
22 | #[pin_project]
| ^^^^^^^^^^^^^^
|
= note: this error originates in the derive macro `::pin_project::__private::__PinProjectInternalDerive` (in Nightly builds, run with -Z macro-backtrace for more info)
```
The release notes for 0.4.28 call out this issue. https://github.com/taiki-e/pin-project/releases/tag/v0.4.28
Reviewed By: krallin
Differential Revision: D30858380
fbshipit-source-id: 98e98bcb5a6b795b93ed1efd706a1711f15c57db
Summary:
Move optional line handling logic into a separate function and simplify.
This diff is intended to be a pure refactoring with no observable changes in behavior. In particular, all the code dealing with the "optional" list appears to be dead code because if the line is optional, linematch will return "retry", so that branch is never reachable.
Reviewed By: DurhamG
Differential Revision: D30849757
fbshipit-source-id: 17283f9217466b3f85d913da66222b9a6779abe4
Summary:
This line was iterating over a list of files and looking in the
manifest for each one. This results in serial manifest reads which can result in
serial network requests.
Let's instead use manifest.matches() to test them all at once via the underlying
BFS, which does bulk fetching.
Differential Revision: D30938359
fbshipit-source-id: 1af7d417288b82efdd537a4afeaf93c1b55eaf49
Summary:
Demonstrate issues with the vertex to path resolution. Basically, the vertex to
path resolution logic did not check if the "parent of merge" being used is
actually valid (is an ancestor of provided heads) or not.
Reviewed By: DurhamG
Differential Revision: D30911150
fbshipit-source-id: 83d215910d5ba67ac0d5749927018a7aefcc6730
Summary:
The tree metadata fetching evolution goes as follow
(1) (commit, path) scs query
(2) tree manifest scs query [we are here]
(3) eden api manifest query [in development]
Option (1) is no longer used and is the only placed that required scs proxy hash.
Removing it will simplify transition from (2) to (3) and also cleans up bunch of unused code.
It also comes with minor performance improvement, saving about 5% on file access time.
To be precise, this is measured by running fsprobe [this is probably too little to measure in high noise benchmark like running arc focus]:
```
fsprobe.sh run cat.targets --parallel 24
```
Results:
```
W/ scshash:
P24: 0.1044 0.1007 0.1005 (hot) 0.1019 avg
W/o scshash:
P24: 0.0954 0.0964 0.1008 (hot) 0.0975 avg
```
This performance improvement comes from the fact, that even though scs hash was never created or used, we still attempted to load it from scs table, and even though this load always failed it contributed to execution time.
Reviewed By: xavierd
Differential Revision: D30942663
fbshipit-source-id: af84f1e5658e7d8d9fb6853cbb88f02b49cd050b
Summary: File access latency can actually be less then 1 ms, so it's good to show more digits
Reviewed By: DurhamG
Differential Revision: D30942905
fbshipit-source-id: 2fc8d48dbc08c55b89d829d1474ae11c2c3df1c3
Summary:
Since fsprobe itself requires a 'plan' to run, we need separate script to standartize list of plans we think are relevant
This scripts allows to generate fsprobe plans and run them
Reviewed By: DurhamG
Differential Revision: D30908892
fbshipit-source-id: eb722fe1f6d982e42b66614f08bc73345e04f9e6
Summary:
We got errors like:
error.IndexedLogError: "repo/.hg/store/lfs/pointers/meta": when reading LogMetadata
in log::OpenOptions::open(Filesystem("repo/.hg/store/lfs/pointers"))
Caused by 1 errors:
- failed to fill whole buffer
from Sandcastle. There seems no easy way to get a sample of the broken `meta`
file content. Let's include the file content to make progress on debugging.
Reviewed By: DurhamG
Differential Revision: D30939737
fbshipit-source-id: ccd77f6b67e4aaf75af2248118845fd5b3434ff1
Summary: This `allow` is no longer needed.
Reviewed By: yancouto
Differential Revision: D30859520
fbshipit-source-id: 36b810a72a28af25513404739bccf471e380cdf1
Summary: Update TreeStore to use CommonFetchState and update TreeStore and BackingStore to use the other utility types already in use for files (`StoreTree`, `FetchResults`, etc).
Reviewed By: andll
Differential Revision: D30739008
fbshipit-source-id: e210b8d76614c762ba127d5f2e26391681da004f
Summary: Adds a utility method for converting a `StoreTree` to a `manifest-tree::Entry`, which wraps an hg manifest blob and provides methods for parsing it as a tree manifest (and a `TryFrom` impl used to convert it to a pre-parsed `manifest::List`, which is used by BackingStore in the next change in this stack).
Reviewed By: andll
Differential Revision: D30859470
fbshipit-source-id: 411e80a14861e0739b0c398290055002b35e59d3
