Summary:
This is fairly mechanical diff that finalizes split of Hash into ObjectId and Hash20.
More specifically this diff does two things:
* Replaces `Hash` with `Hash20`
* Removes alias `using Hash = Hash20`
Reviewed By: chadaustin
Differential Revision: D31324202
fbshipit-source-id: 780b6d2a422ddf6d0f3cfc91e3e70ad10ebaa8b4
Summary:
A spawner type is required for new thrift clients, specify the noop one for now.
This also requires regenerating the generated thrift libraries.
Reviewed By: yancouto
Differential Revision: D31338518
fbshipit-source-id: cbecf3ec6f9678918ca459c19f1cc160214fadfd
Summary:
We now use post-pull hook to mark commits as landed:
hooks.post-pull.marklanded=hg debugmarklanded
So there is no need to mark them inside the pull command. In fact, it seems
something is wrong (phases aren't invalidated properly?) so that the
pullcreatemarkers logic might actually hide commits during pull incorrectly:
commit 1e964a4302c03e5ae48e5b85b0fc0bf27f847b09
Author: metalog <metalog@example.com>
Date: Tue Sep 28 17:29:49 2021 +0000
pull
Parent: a4511a83cf862cc7216c15d83a3f4ff9d3b3241b
Transaction: pullcreatemarkers
RootId: 18d81a1531ecea65affe83c25804c790cac57c59
diff --git a/visibleheads b/visibleheads
index eb58137..da5f45a 100644
--- a/visibleheads
+++ b/visibleheads
@@ -1,16 +1,6 @@
v1
-e82de77adcc261cb306dafeb6cbe15f26f7de768
-91cf8c4b47e433acbe3f774e608eee42a3ad089d
8c071e5aa26d920f1b88c4b1cd10f6e946d4312a
536ff436cde4ec53d74c02ae2c5ed6f60609e01a
-e49b834a6ff9b61a95a743d22703dc6634f2918c
-c53b65542d4583ae82835144ac7f72dce7a6f335
1010312ed7ac683c5e97ad765cdbcb4927ddf62c
-a563a5f35d2df4c54ba1fe2401aa1cd929a218bf
-55c0c7483c6cd85114a9be587d11c87aaceaeff4
-74079567b677c5799b6c67855683ec346ebc3cce
33b5fd6055da2284cf5224f70e1bb9791ed87641
-ec221dff2393793f7b1e11ea9f9e9ea87b79da1f
-cc466614d43a6af24520b7a81653435f4a614fbb
2d7466be885e757d2d41630a3148fd31f5199ffa
-226c8136603dcbcc408133a2122e93fc045527fa
As we're here, document some other config options existed in the code.
Reviewed By: DurhamG
Differential Revision: D31295709
fbshipit-source-id: e26c728215a209ab5dfaee7a84daece8197a1cc4
Summary:
With the lookup processor now returning an ImmediateFuture, we can focus on its
caller, starting with getChildRecursive.
Reviewed By: chadaustin
Differential Revision: D31283396
fbshipit-source-id: 97abc57b9efe3540c5770aa952995c257e6eda4b
Summary:
While profiling glob queries sent by Buck, I noticed that EdenFS spends almost
as much time resolving the inode to perform the glob on as EdenFS takes to
actually perform the glob. Futures related overhead shows up as predominent,
thus let's convert these to ImmediateFuture to speed this up.
Reviewed By: chadaustin
Differential Revision: D31283395
fbshipit-source-id: 7355ddf7498f722ed8ec2989f010a28fb15c293f
Summary:
While std::variant is convenient, they are both slow to compile, and the
compiler cannot optimize it as well as a manually written tagged union. Since
ImmediateFuture is performance critical for EdenFS, let's use a tagged union
and speed them up by an additional 40%.
Reviewed By: chadaustin
Differential Revision: D31272296
fbshipit-source-id: e34be4489a596d3577b3bd900a1f20d6c7d8b693
Summary:
The max duration would cause UBSAN failures due to folly's SemiFuture code
multiplying the value which understandably cannot be represented. Splitting the
function is easy and avoids the problem entirely.
Reviewed By: genevievehelsel
Differential Revision: D31272297
fbshipit-source-id: c15ca70ad771c11b4f68bb9974422c0986d4928b
Summary:
When Buck is using the EdenFS globber, the searchRoot argument is set, thus
let's add a new argument to the benchmark to simulate a Buck workflow.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D31283399
fbshipit-source-id: 5e32b2aceb6090e26e88cf7f0d163448d56107d4
Summary:
The goal of this stack is to remove Proxy Hash type, but to achieve that we need first to address some tech debt in Eden codebase.
For the long time EdenFs had single Hash type that was used for many different use cases.
One of major uses for Hash type is identifies internal EdenFs objects such as blobs, trees, and others.
We seem to reach agreement that we need a different type for those identifiers, so we introduce separate ObjectId type in this diff to denote new identifier type and replace _some_ usage of Hash with ObjectId.
We still retain original Hash type for other use cases.
Roughly speaking, this is how this diff separates between Hash and ObjectId:
**ObjectId**:
* Everything that is stored in local store(blobs, trees, commits)
**Hash20**:
* Explicit hashes(Sha1 of the blob)
* Hg identifiers: manifest id and blob hg ig
For now, in this diff ObjectId has exactly same content as Hash, but this will change in the future diffs. Doing this way allows to keep diff size manageable, while migrating to new ObjectId right away would produce insanely large diff that would be both hard to make and review.
There are few more things that needs to be done before we can get to the meat of removing proxy hashes:
1) Replace include Hash.h with ObjectId.h where needed
2) Remove Hash type, explicitly rename rest of Hash usages to Hash20
3) Modify content of ObjectId to support new use cases
4) Modify serialized metadata and possibly other places that assume ObjectId size is fixed and equal to Hash20 size
Reviewed By: chadaustin
Differential Revision: D31316477
fbshipit-source-id: 0d5e4460a461bcaac6b9fd884517e129aeaf4baf
Summary:
https://pxl.cl/1Qh3j
This is the most called edenapi endpoint by far. If we sample logging of it, we can increase the retention of the scuba table.
if we wish, it's possible to not change retention for some "non-trivial" requests, but I haven't done that.
Reviewed By: liubov-dmitrieva
Differential Revision: D31277391
fbshipit-source-id: ee19e9daa4cd39c5d3eac1063e82aa40fc108bc7
Summary:
This is used to uniquely identify requests in gotham. It's logged to output, on errors, and on Scuba.
Problem: On scuba, this packs very badly, as it is a large string (36 chars), unique for all requests.
Solution: Let's get a prefix of it, it should reduce size used on scuba. Got a prefix of size 5.
This affects both LFS and EdenApi.
Pros:
- Reduces size
- Very easy fix
Cons:
- More chance of conflict. The space of this id is 16^5 = 10^6. There will surely be conflicts, but maybe that's not a huge deal?
Alternative: Using 8 digits, that's about 4bi ids, which will reduce conflicts significantly in exchange for more space.
Why not use an int id (example: u64), or using other characters in id (not only hex): This would reduce the size of data significantly, but has drawbacks:
- For int, would require a big refactoring, as everything assumes the id to be string. Specially since this goes through client-server, might be complicated.
- Not just getting a prefix means more processing on each request, and means we need to recalculate it everytime.
- Size reduction might not be that big, as scuba already packs stuff pretty well.
Reviewed By: krallin
Differential Revision: D31305547
fbshipit-source-id: 23f6b6cb7de5b7a090864db414d4d71cd68c4946
Summary: D31115820 (ae87b82eaf) updated quickcheck, but there's some stuff we need to fix forward. This diff fixes the remaining failures I could find.
Reviewed By: farnz
Differential Revision: D31305392
fbshipit-source-id: a6684d47833bc0fd933751c13cdd71392cb1833b
Summary:
use `hg cloud upload` command if usehttpupload enabled
For few users who opted out of cloud sync, we should use `hg cloud upload` command instead of `hg cloud backup` if usehttpupload enabled
Reviewed By: markbt
Differential Revision: D31205919
fbshipit-source-id: 7619e7b299e19a7626782e7b3c1a69e7cd7dbc1b
Summary:
Somehow before this fix I've seen us runing out of semaphores and deadlocking
because not freeing semaphores immediately after finishing running the
function requiring git repo.
Reviewed By: mojsarn
Differential Revision: D31310626
fbshipit-source-id: ba12b2d4918ecc30ca0aa6ff011176f7634badf9
Summary: Need this for cargo check/rust-analyzer to work.
Reviewed By: guswynn
Differential Revision: D31319911
fbshipit-source-id: ebd3fa72d8fc3667391a2067f95cab9e5f53301f
Summary: I'm parsing some deeply-nested JSON, and it's running into the limit. This feature enables a potential footgun, but even with the feature enabled you have to add code to reach for said footgun.
Reviewed By: jsgf
Differential Revision: D31284743
fbshipit-source-id: 00ea5d7d7db8bdeb878d48fe390831f39e007409
Summary:
When calling `changelog.idmap`, it takes a snapshot of the IdMap.
When flushing the changelog, the IdMap might change. Some Ids might get
re-assigned.
For code (phases?) that keeps the `nameset` for a while the idmap might become
out-dated after a changelog flush. That might be a cause of removing visible
heads incorrectly.
Let's change `_torev` and `_tonode` to properties so they always take the
"latest" idmap snapshot.
Reviewed By: DurhamG
Differential Revision: D31296057
fbshipit-source-id: 1b3fec5d21649eab772ab3a150a3182a18b94edf
Summary:
It was added on the client side in D30686450 (7eb11cb392) to handle octopus merges correctly,
let's add it on mononoke as well, otherwise new_streaming_clone fails to parse
a revlog.
Reviewed By: mitrandir77
Differential Revision: D31305651
fbshipit-source-id: 976d7fdb8775f859e4732fd8a68f9b28f04ce4f9
Summary: This implements unix socket support for mercurial's HTTPConnection so commitcloud can use it.
Reviewed By: ahornby
Differential Revision: D31229256
fbshipit-source-id: a610c3c34be608ac2d9b41f3a7b6b62b44227b94
Summary:
We want to reduce duplicated work. Since requests will be consistently hashed, each instance of service will receive some set of requests from multiple clients. By storing requests together with shared future derivation, any client can get the state of derivation.
In addition upon receiving requests we can clean up the map to remove already completed futures so the map will not grow indefinitely.
Reviewed By: StanislavGlebik
Differential Revision: D30776322
fbshipit-source-id: 961055f8b3328378451edd677506d7e716a9afd2
Summary:
Now the runlog for an hg command invocation includes any progress bar metadata. I repurposed the existing rust progress thread to also upate the runlog progress (only if it has changed).
To avoid race conditions with the main thread writing the final "exit" runlog entry, updating the runlog progress is a no-op if the runlog's exit code has already been set.
Reviewed By: quark-zju
Differential Revision: D31065260
fbshipit-source-id: 181661cb06ab2910d8a0e41f5aa767528eb234f5
Summary:
The runlog's purpose is to store live information for every hg invocations. Users/VSCode will access the runlog data to see details about active hg commands.
In this initial commit I've added basic start/end updates to the runlog. The only current storage option is JSON files written to ".hg/runlog/<random ID>". Cleanup of the files will be added later. In the future I may look at sqlite as an alternative.
Set runlog.enable=True to turn on the runlog.
Reviewed By: quark-zju
Differential Revision: D31065258
fbshipit-source-id: 3ff29e1b8473f7e0b6b0d02537d1f18c2c5026fb
Summary:
The old message was a little misleading as in some case EdenFS was running while it couldn't mount the repository. Mercurial will still tell the user that EdenFS is not running. It is not accurate.
The new message is trying to cover this case to avoid confusing people.
Reviewed By: zhengchaol
Differential Revision: D31278947
fbshipit-source-id: dd3e599654390269b6cf31d8842105970cb29cc0
Summary:
This updates the following crates to their latest versions:
- `rand`: 0.7 => 0.8
- `quickcheck`: 0.9 => 1.0
Both crates introduced some breaking changes, so affected clients had to be fixed accordingly. Most changes are rather mechanical and shouldn't change the existing logic. In addition, a few buggy property tests were uncovered, presumably due to `quicktest` becoming smarter with its choice of inputs in the newer version, and the fixes are included in this diff.
Reviewed By: yancouto
Differential Revision: D31115820
fbshipit-source-id: 60a61dfac3236fd93cd4f03b86506654d81d330f
Summary: This diff fixes some integration test errors after enabling the new edenfsctl.
Reviewed By: xavierd
Differential Revision: D30789741
fbshipit-source-id: 02d74defc41def4fb6ea0cc4694f944b4c0044e2
Summary:
Some detail polishing.
Incomplete commands are commented out. Help messages are now printed correctly. Fixed a small behavior divergence in `eden config` (`to_string_pretty` uses multi-line string instead of escaping characters).
Reviewed By: xavierd
Differential Revision: D30547011
fbshipit-source-id: 98d323744ce7a7fc989cbf79dd07ed8af3cee09d
Summary: This diff adds the Rust edenfsctl to our open source build.
Reviewed By: xavierd
Differential Revision: D30788685
fbshipit-source-id: 603caa933ecfc5af0ede7e22f6c7911462da3a65
Summary:
The lookup of content ids was not working as expected.
Reasons:
- If content id was provided, we never checked it was actually on the blobstore, and failed when building the metadata for it. This was happening since D30016963 (f64520a312)
This diff fixes that by explicitly checking it exists. I also added some comments to clarify.
Reviewed By: liubov-dmitrieva, StanislavGlebik
Differential Revision: D31268102
fbshipit-source-id: 9801a7f4ce1536e68f44ebe114087e53cf094d7a
Summary:
Functions that only takes boolean arguments are fairly confusing and error
prone. Here, since we only ever pass a single true value to it, we could simply
inline setting the right counter in the caller. This makes the code easier to
read, and less error prone.
Reviewed By: genevievehelsel
Differential Revision: D31188413
fbshipit-source-id: 64c019ff52b1ff5644e5bea11a361e586044403f
Summary:
Changes to edenfs-client seem to be breaking the hgbuild windows job https://www.internalfb.com/intern/sandcastle/job/27021598254894733/
Original commit changeset: 218f06a4e648
Reviewed By: DurhamG
Differential Revision: D31244893
fbshipit-source-id: e9ef7c2142d0a6afca342f84574d553b136b5fdb
Summary: I would like to use httpclient.HTTPConnection client because in the following diffs I am adding unix domain socket support to it and jplopezgu will add use that support for commitcloud.
Reviewed By: ahornby
Differential Revision: D31229252
fbshipit-source-id: 8999f27b68f9c7aa9f725d65c291f4d338d3b813
Summary:
One way to mitigate the skiplist inefficiencies is to just use segmented
changelog if we can.
Currently we can do it only for commits on master bookmarks for most repos but
upcoming defrag work from farnz would allow us to include release branches
there as well. That will cover most of the is_ancestor queries.
NOTE: This is not the end of diffs switching us to use segmented changelog. I'm planning to also do it for other places where we do ancestry checks and lower common ancestor operations.
Reviewed By: StanislavGlebik
Differential Revision: D31169338
fbshipit-source-id: 9d4b27d3fb22016b0239c52d71a9b2d9ae9a103b
Summary: This would allow us to benefit from segmented changelog server-side
Reviewed By: StanislavGlebik
Differential Revision: D31169337
fbshipit-source-id: 3c648ed2f144cee57de7c319692a37b04adf5705
Summary: Previously, all EdenAPI methods supported callback-based progress reporting. With the new HTTP progress bars, this old progress API is no longer used anywhere (except for a test program). Let's clean it up to get rid of the extra parameter for every method.
Reviewed By: andll
Differential Revision: D31184693
fbshipit-source-id: 996959e0d81dd7685fcfaca98f162e7267684306
Summary:
this admin command in D29734333 (3f8de3336a) started depending on innerRepo because it
needed access to ephemeral blobstore. It didn't need other parts of inner repo
so there's no need for that dependency.
Reviewed By: krallin
Differential Revision: D31210293
fbshipit-source-id: 004fb95d17e7e1d3095db0258f3c55dadaf5524c
Summary:
This mode rederives commits and compares that what was rederived is the same to
what has already been derived. It's useful to test any changes to derive data
logic and make sure these changes don't have any bugs
Reviewed By: markbt
Differential Revision: D31143741
fbshipit-source-id: 618dbf12ab444b5686d50f83a590314adc6c5dda
Summary: Remove some more path clones by changing within_restrictions to take Option<&MPath>
Reviewed By: StanislavGlebik
Differential Revision: D31175004
fbshipit-source-id: 92f0b4b594c4b3e30258acd019e7f42d9b3bc5fb
Summary: Remove a couple of clones of path by moving up ChangesetPathContentContext::new_with_fsnode_entry
Reviewed By: StanislavGlebik
Differential Revision: D31175005
fbshipit-source-id: fa686f69087e317877c2c9a9c0cffe05a6006775
Summary:
`self.map.contains_vertex_name` only checks the local `map` without triggering
remote protocols. That means with lazy changelog clone, if `master` points to
a lazy commit, the clone will fail. Fix it by switching to `self.contains_vertex_name`,
or even better, `self.vertex_id_batch` to do proper batching.
Reviewed By: StanislavGlebik
Differential Revision: D31228524
fbshipit-source-id: 229d8a92c5517ac5a1dbfa3f440df88a4ab8e3e6
Summary:
In advance of Thrift servers defaulting the queue timeout to 100 ms,
which is quite low for EdenFS's needs, explicitly set our queue
timeout to 5 seconds.
Reviewed By: zhengchaol
Differential Revision: D31218348
fbshipit-source-id: 35a109fb6848f7c81c4b58d70e2beae90557e1c8
Summary: we can just use getBackingStores like how is done for `startRecordingBackingStoreFetch` and only record non-empty fileAccesses. This will enable fetch logging for LocalCacheBackingStores which use an HgQueuedBackingStore under the hood
Reviewed By: zhengchaol
Differential Revision: D31215109
fbshipit-source-id: 443d28a57144fdcf078bd653ecf5726825f55740
Summary: fix the dynamic casting for getting a tracebus for the trace hg entrypoint. dynamic cast still makes sense at this point since `trace hg` should only be called on hg backed mounts
Reviewed By: chadaustin
Differential Revision: D31214737
fbshipit-source-id: 65e018e6658d934d8ecd3434bdfc3d72f6873d2b
Summary: instead of dynamic casting to find the repo name, all backing stores can return an optional reponame, and can check if the optional is set.
Reviewed By: zhengchaol
Differential Revision: D31214723
fbshipit-source-id: 9d10114ff6bde13254d3a3caaf2401f87d07ffd7
