Summary: This implements unix socket support for mercurial's HTTPConnection so commitcloud can use it.
Reviewed By: ahornby
Differential Revision: D31229256
fbshipit-source-id: a610c3c34be608ac2d9b41f3a7b6b62b44227b94
Summary:
We want to reduce duplicated work. Since requests will be consistently hashed, each instance of service will receive some set of requests from multiple clients. By storing requests together with shared future derivation, any client can get the state of derivation.
In addition upon receiving requests we can clean up the map to remove already completed futures so the map will not grow indefinitely.
Reviewed By: StanislavGlebik
Differential Revision: D30776322
fbshipit-source-id: 961055f8b3328378451edd677506d7e716a9afd2
Summary:
Now the runlog for an hg command invocation includes any progress bar metadata. I repurposed the existing rust progress thread to also upate the runlog progress (only if it has changed).
To avoid race conditions with the main thread writing the final "exit" runlog entry, updating the runlog progress is a no-op if the runlog's exit code has already been set.
Reviewed By: quark-zju
Differential Revision: D31065260
fbshipit-source-id: 181661cb06ab2910d8a0e41f5aa767528eb234f5
Summary:
The runlog's purpose is to store live information for every hg invocations. Users/VSCode will access the runlog data to see details about active hg commands.
In this initial commit I've added basic start/end updates to the runlog. The only current storage option is JSON files written to ".hg/runlog/<random ID>". Cleanup of the files will be added later. In the future I may look at sqlite as an alternative.
Set runlog.enable=True to turn on the runlog.
Reviewed By: quark-zju
Differential Revision: D31065258
fbshipit-source-id: 3ff29e1b8473f7e0b6b0d02537d1f18c2c5026fb
Summary:
The old message was a little misleading as in some case EdenFS was running while it couldn't mount the repository. Mercurial will still tell the user that EdenFS is not running. It is not accurate.
The new message is trying to cover this case to avoid confusing people.
Reviewed By: zhengchaol
Differential Revision: D31278947
fbshipit-source-id: dd3e599654390269b6cf31d8842105970cb29cc0
Summary:
This updates the following crates to their latest versions:
- `rand`: 0.7 => 0.8
- `quickcheck`: 0.9 => 1.0
Both crates introduced some breaking changes, so affected clients had to be fixed accordingly. Most changes are rather mechanical and shouldn't change the existing logic. In addition, a few buggy property tests were uncovered, presumably due to `quicktest` becoming smarter with its choice of inputs in the newer version, and the fixes are included in this diff.
Reviewed By: yancouto
Differential Revision: D31115820
fbshipit-source-id: 60a61dfac3236fd93cd4f03b86506654d81d330f
Summary: This diff fixes some integration test errors after enabling the new edenfsctl.
Reviewed By: xavierd
Differential Revision: D30789741
fbshipit-source-id: 02d74defc41def4fb6ea0cc4694f944b4c0044e2
Summary:
Some detail polishing.
Incomplete commands are commented out. Help messages are now printed correctly. Fixed a small behavior divergence in `eden config` (`to_string_pretty` uses multi-line string instead of escaping characters).
Reviewed By: xavierd
Differential Revision: D30547011
fbshipit-source-id: 98d323744ce7a7fc989cbf79dd07ed8af3cee09d
Summary: This diff adds the Rust edenfsctl to our open source build.
Reviewed By: xavierd
Differential Revision: D30788685
fbshipit-source-id: 603caa933ecfc5af0ede7e22f6c7911462da3a65
Summary:
The lookup of content ids was not working as expected.
Reasons:
- If content id was provided, we never checked it was actually on the blobstore, and failed when building the metadata for it. This was happening since D30016963 (f64520a312)
This diff fixes that by explicitly checking it exists. I also added some comments to clarify.
Reviewed By: liubov-dmitrieva, StanislavGlebik
Differential Revision: D31268102
fbshipit-source-id: 9801a7f4ce1536e68f44ebe114087e53cf094d7a
Summary:
Functions that only takes boolean arguments are fairly confusing and error
prone. Here, since we only ever pass a single true value to it, we could simply
inline setting the right counter in the caller. This makes the code easier to
read, and less error prone.
Reviewed By: genevievehelsel
Differential Revision: D31188413
fbshipit-source-id: 64c019ff52b1ff5644e5bea11a361e586044403f
Summary:
Changes to edenfs-client seem to be breaking the hgbuild windows job https://www.internalfb.com/intern/sandcastle/job/27021598254894733/
Original commit changeset: 218f06a4e648
Reviewed By: DurhamG
Differential Revision: D31244893
fbshipit-source-id: e9ef7c2142d0a6afca342f84574d553b136b5fdb
Summary: I would like to use httpclient.HTTPConnection client because in the following diffs I am adding unix domain socket support to it and jplopezgu will add use that support for commitcloud.
Reviewed By: ahornby
Differential Revision: D31229252
fbshipit-source-id: 8999f27b68f9c7aa9f725d65c291f4d338d3b813
Summary:
One way to mitigate the skiplist inefficiencies is to just use segmented
changelog if we can.
Currently we can do it only for commits on master bookmarks for most repos but
upcoming defrag work from farnz would allow us to include release branches
there as well. That will cover most of the is_ancestor queries.
NOTE: This is not the end of diffs switching us to use segmented changelog. I'm planning to also do it for other places where we do ancestry checks and lower common ancestor operations.
Reviewed By: StanislavGlebik
Differential Revision: D31169338
fbshipit-source-id: 9d4b27d3fb22016b0239c52d71a9b2d9ae9a103b
Summary: This would allow us to benefit from segmented changelog server-side
Reviewed By: StanislavGlebik
Differential Revision: D31169337
fbshipit-source-id: 3c648ed2f144cee57de7c319692a37b04adf5705
Summary: Previously, all EdenAPI methods supported callback-based progress reporting. With the new HTTP progress bars, this old progress API is no longer used anywhere (except for a test program). Let's clean it up to get rid of the extra parameter for every method.
Reviewed By: andll
Differential Revision: D31184693
fbshipit-source-id: 996959e0d81dd7685fcfaca98f162e7267684306
Summary:
this admin command in D29734333 (3f8de3336a) started depending on innerRepo because it
needed access to ephemeral blobstore. It didn't need other parts of inner repo
so there's no need for that dependency.
Reviewed By: krallin
Differential Revision: D31210293
fbshipit-source-id: 004fb95d17e7e1d3095db0258f3c55dadaf5524c
Summary:
This mode rederives commits and compares that what was rederived is the same to
what has already been derived. It's useful to test any changes to derive data
logic and make sure these changes don't have any bugs
Reviewed By: markbt
Differential Revision: D31143741
fbshipit-source-id: 618dbf12ab444b5686d50f83a590314adc6c5dda
Summary: Remove some more path clones by changing within_restrictions to take Option<&MPath>
Reviewed By: StanislavGlebik
Differential Revision: D31175004
fbshipit-source-id: 92f0b4b594c4b3e30258acd019e7f42d9b3bc5fb
Summary: Remove a couple of clones of path by moving up ChangesetPathContentContext::new_with_fsnode_entry
Reviewed By: StanislavGlebik
Differential Revision: D31175005
fbshipit-source-id: fa686f69087e317877c2c9a9c0cffe05a6006775
Summary:
`self.map.contains_vertex_name` only checks the local `map` without triggering
remote protocols. That means with lazy changelog clone, if `master` points to
a lazy commit, the clone will fail. Fix it by switching to `self.contains_vertex_name`,
or even better, `self.vertex_id_batch` to do proper batching.
Reviewed By: StanislavGlebik
Differential Revision: D31228524
fbshipit-source-id: 229d8a92c5517ac5a1dbfa3f440df88a4ab8e3e6
Summary:
In advance of Thrift servers defaulting the queue timeout to 100 ms,
which is quite low for EdenFS's needs, explicitly set our queue
timeout to 5 seconds.
Reviewed By: zhengchaol
Differential Revision: D31218348
fbshipit-source-id: 35a109fb6848f7c81c4b58d70e2beae90557e1c8
Summary: we can just use getBackingStores like how is done for `startRecordingBackingStoreFetch` and only record non-empty fileAccesses. This will enable fetch logging for LocalCacheBackingStores which use an HgQueuedBackingStore under the hood
Reviewed By: zhengchaol
Differential Revision: D31215109
fbshipit-source-id: 443d28a57144fdcf078bd653ecf5726825f55740
Summary: fix the dynamic casting for getting a tracebus for the trace hg entrypoint. dynamic cast still makes sense at this point since `trace hg` should only be called on hg backed mounts
Reviewed By: chadaustin
Differential Revision: D31214737
fbshipit-source-id: 65e018e6658d934d8ecd3434bdfc3d72f6873d2b
Summary: instead of dynamic casting to find the repo name, all backing stores can return an optional reponame, and can check if the optional is set.
Reviewed By: zhengchaol
Differential Revision: D31214723
fbshipit-source-id: 9d10114ff6bde13254d3a3caaf2401f87d07ffd7
Summary: add more information to the runtime error thrown by the dynamic cast failure in `eden trace hg` and predictive fetch
Reviewed By: zhengchaol
Differential Revision: D31212247
fbshipit-source-id: 982901dfd2eb05db9ca6e7366277a07b6b29872f
Summary:
VC++ 2019 is pickier about which standard library includes include
each other. Be explicit.
Reviewed By: zhengchaol
Differential Revision: D31186916
fbshipit-source-id: 95cfa8848d0e2e312e2024923fa166db5f68dde0
Summary:
This unused feature allowed a sub-process to inherit an hg repo wlock from the parent process. It was apparently intended for merge drivers, but nothing was using it.
I want to move some locking logic into rust, and this stuff was complicating things.
Note that this functionality was also removed upstream in https://phab.mercurial-scm.org/D9053.
Reviewed By: quark-zju
Differential Revision: D31184339
fbshipit-source-id: 92908220d48e2bc55e2f4fca90e647650ca5bef7
Summary:
While debugging the unlinked inode unloading for NFS I have re-added these
logs a couple times. These seem valuable to have in eden so that we don't have
to add them any time we are debugging eden and we can debug a bit in a
production eden rather than dev built eden.
Reviewed By: xavierd
Differential Revision: D30971151
fbshipit-source-id: 58172079dfe4f4e4ba31bae30bf982e2cbe0fd29
Summary:
We run periodic inode unloading for unlinked inodes on NFS because we get no
information from the client on when inodes are no longer needed, and we have to
clean them up at some point for memory and disk reasons. See previous commit
summaries for more details on this (D30144901 (ffa558bf84)).
Let's add some counters on this so we have a bit more visibility into the
process. This counter is meant to mimic the PeriodicUnloadCounter counter.
Reviewed By: chadaustin
Differential Revision: D30966688
fbshipit-source-id: cfc8d769b53073d9f4c0c27b6bee20e222c6c8d2
Summary:
I believe this is the reason for -
https://fb.workplace.com/groups/238845853462687/posts/845939069420026. We used
default config that doesn't do any chunking and puts large files as a single
blobs.
Let's not do that
Reviewed By: farnz
Differential Revision: D31209331
fbshipit-source-id: 43c2d2ab7caac110a1474856da09c119a5e72429
Summary:
EdenApiUploads: eliminate extra lookup if no stacks
In EdenApiUploads we filter heads first and then we filter the commits belonging to these stacks.
However, in some usecases users don't use stacks. If there is no any single stack, the second lookup would be redundant and it would be nice to avoid it completely.
We can pass a flag to the upload code saying that extra filtering is not needed.
For example, in configerator repo users usually don't do stacks.
Reviewed By: markbt
Differential Revision: D31203489
fbshipit-source-id: 0921a01198bfc377afc3af3f7319fd0c5fec04d7
Summary: Plus a minor refactoring to use the io::IsTty trait in edenfs_client::status instead of calling into libc directly.
Reviewed By: quark-zju
Differential Revision: D31156633
fbshipit-source-id: 218f06a4e64836be88b4afac98dcfa140373c730
Summary:
There is no need to read from the LocalStore twice, the tree is either present
in it, or not.
Reviewed By: chadaustin
Differential Revision: D31187972
fbshipit-source-id: 15bdeef9176b51e6ba3f62ed16550032b0024b94
Summary:
Some of EdenFS backing store requires EdenFS to cache objects locally to avoid
potentially expensive network fetches, while others already have some form of
local caching. In the past, all backing store fell in the first category, but
thanks to Mercurial's native backing store implementation the LocalStore
caching has become pure overhead for it. Previously, this was worked around by
configuring the LocalStore to not cache blobs locally, but this wasn't done for
trees. This config also conflicts with the need to cache blobs and trees
locally for backing stores in the first category (such as ReCas).
Since we know at construction time what backing store needs local caching, we
can simply wrap these in the newly introduced LocalStoreCachedBackingStore
store.
For now, since the Mercurial backing store always writes a proxy hash to the
LocalStore, bypassing the LocalStore for trees would be a regression due to the
added disk IO. Once proxy hashes are gone for Mercurial, we can remove the
LocalStoreCachedBackingStore wrapper.
Reviewed By: chadaustin
Differential Revision: D31118905
fbshipit-source-id: 4a2958eafeeb8144ee4421ec44dbd30cedceee29
Summary: Same as D30974102 (91c4748c5b) but for mercurial cs.
Reviewed By: StanislavGlebik
Differential Revision: D31145642
fbshipit-source-id: c1be7b49bf0cbe70b844f1a31de706215a51d1ae
Summary: Same as D30974102 (91c4748c5b) but for fastlog.
Reviewed By: ahornby
Differential Revision: D31142066
fbshipit-source-id: 44a79e8a9db180736324db734b018344a77c070a
Summary:
Same as D30974102 (91c4748c5b) but for deleted manifest.
Needed some changes regarding using `DerivationContext` instead of `BlobRepo`.
Reviewed By: StanislavGlebik
Differential Revision: D31121260
fbshipit-source-id: f37daac320173b0896f12c83bdd8a723d22ec876
Summary:
Same as D30974102 (91c4748c5b) but for fsnodes.
Needed some changes regarding using `DerivationContext` instead of `BlobRepo`.
Reviewed By: StanislavGlebik
Differential Revision: D31113044
fbshipit-source-id: 6e996135f59f26e76e52b0b24ea61917216d1e53
Summary:
Same as D30974102 (91c4748c5b) but for skeleton manifest.
Needed some changes regarding using `DerivationContext` instead of `DerivedDataManager`.
Reviewed By: StanislavGlebik
Differential Revision: D31111484
fbshipit-source-id: eacc1d3247dffac4537745ec2a2071ef0abcbd43
Summary:
Same as D30974102 (91c4748c5b) but for changeset info.
This turned out quite simple, as we already have the bonsai changeset, so there's no need to do any async stuff.
Reviewed By: StanislavGlebik
Differential Revision: D31110319
fbshipit-source-id: 952686ae5583b858361b7a2a67fe914bfe5239d6
Summary: Now that EdenAPI fetching is turned on everywhere, let's make it the default.
Reviewed By: quark-zju
Differential Revision: D31184213
fbshipit-source-id: 450c1167d42ee867b505a2a14b0c636bed81107d
Summary:
It can be surprising when a job suddenly is no longer able to run sudo, or no
longer run as root that all the tests are marked as being successful, despite
the fact that they no longer run. Let's recognize when we run on EdenFS to
allow tests to fail if they can no longer run EdenFS.
Reviewed By: zhengchaol
Differential Revision: D30357402
fbshipit-source-id: c3758d7a5a3c575dd68bd97062ae24abe4124874
Summary:
Now that we might have multiple kernel protocols per mount (i.e. both fuse and
nfs on macOS) let's include them in eden rage.
Reviewed By: xavierd
Differential Revision: D31154042
fbshipit-source-id: 38e7630829d70fe9dd6dbeabacc3b538ee798e0d
Summary: dyn Drop produces warning because everything is Drop
Reviewed By: quark-zju
Differential Revision: D31175376
fbshipit-source-id: 78f55a60c9bb6d51cde9433ab2815ec133b15ecc
