Summary:
Since fsprobe itself requires a 'plan' to run, we need separate script to standartize list of plans we think are relevant
This scripts allows to generate fsprobe plans and run them
Reviewed By: DurhamG
Differential Revision: D30908892
fbshipit-source-id: eb722fe1f6d982e42b66614f08bc73345e04f9e6
Summary:
We got errors like:
error.IndexedLogError: "repo/.hg/store/lfs/pointers/meta": when reading LogMetadata
in log::OpenOptions::open(Filesystem("repo/.hg/store/lfs/pointers"))
Caused by 1 errors:
- failed to fill whole buffer
from Sandcastle. There seems no easy way to get a sample of the broken `meta`
file content. Let's include the file content to make progress on debugging.
Reviewed By: DurhamG
Differential Revision: D30939737
fbshipit-source-id: ccd77f6b67e4aaf75af2248118845fd5b3434ff1
Summary: This `allow` is no longer needed.
Reviewed By: yancouto
Differential Revision: D30859520
fbshipit-source-id: 36b810a72a28af25513404739bccf471e380cdf1
Summary: Update TreeStore to use CommonFetchState and update TreeStore and BackingStore to use the other utility types already in use for files (`StoreTree`, `FetchResults`, etc).
Reviewed By: andll
Differential Revision: D30739008
fbshipit-source-id: e210b8d76614c762ba127d5f2e26391681da004f
Summary: Adds a utility method for converting a `StoreTree` to a `manifest-tree::Entry`, which wraps an hg manifest blob and provides methods for parsing it as a tree manifest (and a `TryFrom` impl used to convert it to a pre-parsed `manifest::List`, which is used by BackingStore in the next change in this stack).
Reviewed By: andll
Differential Revision: D30859470
fbshipit-source-id: 411e80a14861e0739b0c398290055002b35e59d3
Summary: This change does not add aux data support, so for now the types are a bit useless.
Reviewed By: DurhamG
Differential Revision: D30313314
fbshipit-source-id: 11968199b12c4f870c58c7e939b5c8ed5cd9afea
Summary: More refactoring of scmstore `TreeStore`. Introducing a new `tree` submodule as we'll be adding tree-specific metrics, types, etc. soon (as currently exist for files).
Reviewed By: andll
Differential Revision: D30313460
fbshipit-source-id: f20d3ee62520b1d9ea34ad04eb1880ad9b5a00c3
Summary: Extract out `CommonFetchState` from `FileStore`'s `FetchState`. Currently, direct field access is still used for computing derivations and a few other places, but this will be changed in a later diff.
Reviewed By: DurhamG
Differential Revision: D30308289
fbshipit-source-id: 16d34904412572facc9f51cbd791e30413bfe634
Summary: Don't show progress bars for pending HTTP requests until they actually start running, so that the user always sees progress bars from active transfers.
Reviewed By: quark-zju
Differential Revision: D30914241
fbshipit-source-id: ca2f85af055dc9324123d0f9cc765f42d3b36ad4
Summary: Add a new `first_activity` event to the `Response` event listeners that fires the first time we detect nonzero progress for either uploading or downloading. This is useful for situations where requests are queued and we want to be notified when the request becomes active (e.g., to register progress bars).
Reviewed By: DurhamG
Differential Revision: D30914242
fbshipit-source-id: 83445724ed81e77ac25954b644e6bbafcbe5cadb
Summary: This adds inode number to NFS trace event so that we can use it in ActivityRecorder to show the filename of the FS request.
Reviewed By: xavierd
Differential Revision: D30849770
fbshipit-source-id: 580faf5fccb1a225399d9aec843e23eae1874e87
Summary:
We have an option on GlobFiles for listing hidden files, but we don't have a
cli option. We default to false in the cli. Let's pipe this option all the way through.
so that we can control this flag from the cli.
Reviewed By: xavierd
Differential Revision: D30915118
fbshipit-source-id: 28b91d4fd2dd4bdf9e342929f570f64db14e8ad0
Summary:
`eden prefetch` and `eden glob` return lists that despite being called
"maching files" actually contains both files and directories.
In some cases, we only want the list of files and it introduces unnessecary
overhead on our clients for them to have to stat all the files in the list to
filter out the dirs. Let's add an option to just list files.
Reviewed By: chadaustin
Differential Revision: D30816193
fbshipit-source-id: 6e264142162ce03e560c969a0c0dbbc2f418d7b9
Summary: The error message that currently exists here does not correspond to the command ran, its just missing the "redirect" part
Reviewed By: xavierd
Differential Revision: D30914616
fbshipit-source-id: 866ab7d494b728af13fbb3656edb8740a399755f
Summary:
There's no real equivalent of hg changeset of snapshot, so let's not derive it.
Closes task T97939172
Reviewed By: liubov-dmitrieva
Differential Revision: D30902073
fbshipit-source-id: 8128597c25e12e40e719cdd7800d4b9b792391c9
Summary:
`hg snapshot info` command will be used to get information about the snapshot (similar to `hg show` for commits)
It's still not easy to do this, as we want to have derived data for snapshots, which is still unimplemented.
For now, this makes the command only check if the snapshot exists or not. In the future more functionality will be added (and likely the edenapi endpoint we query will be different).
Reviewed By: liubov-dmitrieva
Differential Revision: D30900088
fbshipit-source-id: 4dc6915d74694a03496c756f03bc073d1a0819f2
Summary: This is a similar diff to D30915090, but for EdenFS.
Differential Revision: D30915126
fbshipit-source-id: 9a718e47237924ebe20176c522a1b1193224236c
Summary:
To eliminate the need for proxy hashes, we need variable-width object
IDs. Introduce an ObjectId type much like RootId.
Reviewed By: genevievehelsel
Differential Revision: D30819412
fbshipit-source-id: 07a185ba6b866b475c92f811e70aa00a8a9f895f
Summary: As a first step to moving the repo name inside the EdenAPI client itself, add it as a (currently unused) field to the config. Later diffs will use this instead of having each method take a `repo` argument.
Reviewed By: quark-zju
Differential Revision: D30746379
fbshipit-source-id: 07957e53e940ce72f84b2297f506b796117ec46a
Summary: We use it as an unique key for the detector
Reviewed By: ginfung
Differential Revision: D30703470
fbshipit-source-id: cb8e7dae5dc4192402530b2cfe564b86aa23c7c8
Summary:
Edenapi lookup (for file content, filenodes and trees): check all the multiplexed blobstores when we check is_present.
This will help us to avoid undesired behaviour for commit cloud blobs that haven't been replicated to all blobstores. Healer currently doesn't check commit cloud blobs.
Reviewed By: StanislavGlebik
Differential Revision: D30839608
fbshipit-source-id: d13cd4500f7b14731d8b75c763c14a698399ba02
Summary:
The new debugscmstorereplay command replays scmstore fetches given an activity log created previously via the scmstore.activity log config parameter.
Replaying activity logs may help to understand or reproduce performance issues related to file fetching. Currently the replay tool ignores all complications such as concurrent fetches or variable backends.
Differential Revision: D30288701
fbshipit-source-id: c6b24acdbd37b5a51ccba3e74e8f074062e880e5
Summary:
The new scmstore.activitylog config knob optionally specifies a file for scmstore to record fetch activity. Currently it only records file fetches, but it is intended to also record tree fetches once that is fully baked.
The purpose of the log is to record file access patterns to help debug command performance. The following commit will include a tool to replay scmstore activity from the log file.
Activity is stored in the log as newline delimited JSON objects. In addition to fetched keys, we also record the start time and duration of each fetch.
Differential Revision: D30288715
fbshipit-source-id: c40177e95b1f613ebed41e50a476cbf39e6d9364
Summary:
Make it more detailed, especially about corner cases. Avoid ambiguous words
like "valid" etc.
Reviewed By: farnz
Differential Revision: D30876339
fbshipit-source-id: a45ca643c6454645f7729053a7ea5dd78016fc68
Summary:
Same fix as D30874167 (9edb2cafe7), but for hg-server. This was broken in a recent
update.
Reviewed By: yancouto
Differential Revision: D30882520
fbshipit-source-id: 7e23556f619e3ead585e9e756456f30578ff7cab
Summary:
Some time ago (see D25910464 (fca761e153)) we've started using Background session class
while deriving data. This was done to avoid overloading blobstore sync queue - if Background
session class is set then multiplex blobstore waits for all blobstores to
finish instead of writing to the blobstore sync queue right away. However if any of the
blobstores fails then we start writing to the blobstore sync queue. In theory it should have avoided the problem of overloading blobstore sync queue while having the same multiplex reliability (i.e. if only a single blobstore fails the whole multiplex put doesn't fail)
Unfortunately there was a flaw - if blobstore put of a single blobstore wasn't
failing but was just too slow, then the whole multiplexed put operation becomes
too slow. This diff fixes this flaw by adding a timeout - if multiplexed put is
taking too long then we fallback to writing entries to the blobstore sync
queue.
Note that I added a new session class - BackgroundUnlessTooSlow -
because I figured that in some cases we are ok with waiting a long time but not
writing to the sync queue. Skiplist builder might be a good example of that -
since it's doing overwrites then we don't want to write to the blobstore sync
queue at all, because healer doesn't process overwrites correctly.
Reviewed By: farnz
Differential Revision: D30892377
fbshipit-source-id: 69ac1795002b124e11daac13d8bfe59895191168
Summary: When specifying `HGPLAIN`, only the hash is outputted, which is easier for automation.
Reviewed By: StanislavGlebik
Differential Revision: D30899254
fbshipit-source-id: 32457c6b92d14305c5b0bafb1217d574ec83a85c
Summary:
I added logging in D30805504 (d5e2624fbb), however it wasn't really logging anything,
because I forgot to pass scuba sample builder to CoreContext (facepalm).
This diff fixes it.
Reviewed By: HarveyHunt
Differential Revision: D30899642
fbshipit-source-id: 6e20f1e84fc96175be8ca7a6f91c0fc61caf8e49
Summary:
It looks like the comment is misleading (we don't really derive anything in
this block, just finding underived commits), and this CoreContext override
doesn't seem necessary anymore. Let's remove it
Reviewed By: farnz
Differential Revision: D30899641
fbshipit-source-id: 2850905891a9bd8b01f3f6fa9ef15c572fc2f07a
Summary:
`createremote` is a slightly inconsistent name.
The reasoning behind it was that this command creates the snapshot on server side only.
But since actually making the client snapshot-aware is pretty far away, I prefer to make it "create".
Reviewed By: StanislavGlebik
Differential Revision: D30871026
fbshipit-source-id: fde5d65e38249998f71e51b76ccb7d7b6b9bf24d
Summary:
This was a very well-hidden bug that I failed to notice in the integration tests.
Turns out the in serde enum unit variants are serialized to strings, while struct variants to a dictionary.
My code assumed it was always a dictionary. And because python is python, it worked, as the `in` operator works for strings and dicts. But `"Deletion" in fc` for strings means if Deletion is a substring of `fc`, which is also true for `UntrackedDeletion`, thus all untracked deletions were treated like normal deletions.
Reviewed By: StanislavGlebik
Differential Revision: D30868534
fbshipit-source-id: e574de6493bcd8e8d42d6e22da4dc482d083f22d
Summary: the rage summary is getting hard to quickly parse, so this underlines each section header, as well as unifies underline looks (with `eden stats`). This adopts the underline code from `eden du` and makes it a util function for shareability.
Differential Revision: D30857773
fbshipit-source-id: 66b5b06f5b0125304d45d3465a8bc2248693b791
Summary: I think it would be helpful to see the path of the inode that causes this check to fail
Reviewed By: kmancini
Differential Revision: D30880645
fbshipit-source-id: 08cad2277484568a6e325b1db7a89a9cf0fe1d3f
Summary: Refresh doesn't have any special meaning outside of packfiles, but in some contexts (like BackingStore) is used to trigger a flush. Previously I'd implemented refresh as a no-op. With this change, refresh will trigger a flush, and if a contentstore fallback is available will also forward the refresh call to contentstore for packfile-specific refresh behavior.
Reviewed By: andll
Differential Revision: D30851223
fbshipit-source-id: 893b4256fd8edc0f60612e61c177d885252cb85b
Summary: This test can fail for not having `USERNAME` environment variable when the test is run as root. Let's just skip the test when this happens because it doesn't make sense to drop priv as root.
Reviewed By: xavierd
Differential Revision: D30868518
fbshipit-source-id: 14ff6db218b1477f5905f2df3ad075a5ca186117
Summary:
Add an endpoint to provide repo configuration information, such as whether
segmented changelog is supported by the server or not. This helps the client
make decisions without hitting actual (expensive) endpoints and distinguishing
from unrelated server errors. It would allow us to remove error-prone
client-side config deciding whether to use segment clone.
Reviewed By: krallin
Differential Revision: D30831346
fbshipit-source-id: 872e20a32879e075c75481f622b2a49000059d04
Summary:
In a future diff, we want an endpoint to test if segmented changelog is
supported for a repo without doing any real computation using segmented
chagnelog. This would be useful for the client to decide whether it can
use segmented changelog clone or not, instead of relying on fragile
per-repo configuration.
Reviewed By: farnz
Differential Revision: D30825920
fbshipit-source-id: 16dc5bf762da2d2b9cd808c129e1830285023f3d
Summary: it would be helpful to see a user's or sandcastle job's eden config, especially in the case of a gated feature rollout / staged feature rollout.
Differential Revision: D30857763
fbshipit-source-id: ee2a311ee643fc9db5acef1b02017564c51d2362
Summary:
just a typo fix
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: fanzeyi
Differential Revision: D30849172
fbshipit-source-id: 9779832870c909d080548ec71ecf86aa53767dbc
Summary:
This logic (A Wire and Api object that are just the same object) is used in more places ad-hoc (for Vec<u8> and u32).
This diff makes it simpler by using the macro introduced before, and derives it for a lot of basic types (integers and bytes).
Reviewed By: kulshrax
Differential Revision: D30605781
fbshipit-source-id: 7520b529e52cfde0a5c5d17d91f5f85b0633fa7f
Summary:
It's nice to have these functions to open source and target repos.
Previously we always had to get repo id first, and then call
open_repo_internal_with_repo_id
Reviewed By: yancouto
Differential Revision: D30866314
fbshipit-source-id: dd74822da755de232f4701f8523088e0bb612cb9
Summary:
D30829928 (fd03bff2e2) updated some of Mononoke's integration tests to take into
account whitespace changes. However, it removed the globs from some parts of
the tests.
As the assigned port changes on each test run, the globs are required. Add them
back in again, as well as fix up some whitespace in a test.
Reviewed By: markbt
Differential Revision: D30866884
fbshipit-source-id: 1557eee2143a2459a6412b8649e7e3dce5a607a4
Summary:
It's nice to have that can quickly count and print stats about a commit. I'm
using it now to understand performance of derived data.
Reviewed By: ahornby
Differential Revision: D30865267
fbshipit-source-id: 26b91c3c05a1c417015b5be228796589348bf064
Summary:
`rust_include_srcs` is supported on `thrift_library` as a way of including other Rust code in the generated crate, generally used to implement other traits on the generated types.
Adding support for this in autocargo by copying these files into the output dir and making sure their option is specified to the thrift compiler
Reviewed By: ahornby
Differential Revision: D30789835
fbshipit-source-id: 325cb59fdf85324dccfff20a559802c11816769f
Summary:
The default Windows encoding can't handle some unicode characters
apparently, so let's use utf-8 by default.
Reviewed By: quark-zju
Differential Revision: D30850982
fbshipit-source-id: 51a7fdf5464d075549afe4f0bcd307c0f2eb7fa0
Summary:
Add impls for Layer for Box/Arc<L: Layer> and <dyn Layer>. Also a pile of other
updates in git which haven't been published to crates.io yet, including proper
level filtering of trace events being fed into log.
Reviewed By: dtolnay
Differential Revision: D30829927
fbshipit-source-id: c01c9369222df2af663e8f8bf59ea78ee12f7866
Summary:
Bump all the versions on crates.io to highest to make migration to github
versions in next diff work.
Reviewed By: dtolnay
Differential Revision: D30829928
fbshipit-source-id: 09567c26f275b3b1806bf8fd05417e91f04ba2ef
Summary:
We don't need to pass the bubble id to the server, it can find it from the changeset id.
This fixes a TODO I added previously, and should make the `restore` command complete.
Reviewed By: ahornby
Differential Revision: D30609423
fbshipit-source-id: d1c8eb0e0556069fa408520a0aea91a0f865fbe1
Summary:
Uses the endpoint added on previous diffs to download the snapshot files to the repo, and adds them correctly to the snapshot restore.
This should almost complete the `snapshot restore` command, missing is getting the bubble id from the snapshot hash.
{gif:sqc6yx6c}
Reviewed By: StanislavGlebik
Differential Revision: D30583038
fbshipit-source-id: 6549a52f767c50444c316b358d9704bc4a136934
Summary:
This adds the `downloadfiles` method to the python EdenApi wrapper.
It uses multiple calls to the endpoint added on previous diffs to download each file and place it somewhere on the repo. It also does deduplication of downloads.
Reviewed By: StanislavGlebik
Differential Revision: D30582638
fbshipit-source-id: 34e864d03c0e48a7605ee8e4e92376881dbb2de9
Summary:
When using hashbinary with a removed/moved file, hg throws with `TypeError: object supporting the buffer API required` this is because we are trying to `sha1(None)`.
This diff falls back to the `Binary file %s has changed` message when we have a removed file.
Reviewed By: quark-zju
Differential Revision: D30845897
fbshipit-source-id: a3d2b7d11d9c1ca3855140c9abd7550cf7076bbc
Summary: This adds the support for FS events logging for NFS. For context, each type of event is assigned a sampling group that determines its sampling rate. In TraceBus subscription callback, events are sent to `FsEventLogger` to be sampled and logged through `HiveLogger`.
Reviewed By: xavierd
Differential Revision: D30843863
fbshipit-source-id: 65394d31b1197efd69c7fd4c1b24562f5abd5785
Summary:
Previously, it was only possible to register event listeners for request completion on the `HttpClient` itself, rather than on individual `Request`s. This diff adds similar event listeners to `Request`s themselves, so that its possible to register a callback to fire when any request completes, regardless of whether it was sent via an `HttpClient` or as a one-off.
This is similar to `RequestCreationEventListeners`, which run for the creation of every request, whether or not the request is associated with a client.
Notably, to avoid circular references the new event listeners take a `RequestInfo` argument instead of a `RequestContext` (since the listeners are themselves stored inside the `RequestContext`). In practice, the `RequestInfo` should contain all of the information one might want to access about the request.
Reviewed By: quark-zju
Differential Revision: D30831840
fbshipit-source-id: 77ca9dc5fd9f8fc5ee60319baabd77171af70d45
Summary:
The content store repair binding ate the entire repair message, making
it hard to debug when it wasn't working.
Reviewed By: quark-zju
Differential Revision: D30824740
fbshipit-source-id: 52dbfe79f2dd1568285cda63fb54cacf532aa4a1
Summary:
Make `verify` check the lazy changelog properties:
- Universal id <-> name mappings are known locally.
- Segments are sane (ex. high-level segments are built from low-level and there
are no cycles)
With `--dag`, also check the graph with a newly cloned remote graph.
This just calls the verification logic added in Rust `dag` crate to do the
heavy lifting.
Differential Revision: D30820773
fbshipit-source-id: 8f62f41738c3c8e3fe88442860a83fdb4944f178
Summary:
In certain situations, users may cause EdenFS to falsely return a path not exist result while the path is available. Windows will cache that and causing subsequent access to that file to automatically return a file not exist error.
We currently only invalidate this negative cache during checkout and rebooting the machine as the cache is even kept during EdenFS restarts. In this diff, we starts to invalidate the negative path cache at startup so if the user ever had issues an `eden restart` would be sufficient to fix.
Reviewed By: xavierd
Differential Revision: D30814059
fbshipit-source-id: 53283f471702762b2eed0c5d0f6a9cc49f4db739
Summary:
This adds the plumbing to access download a file using the endpoint from the previous diff via the EdenApi trait, which does the actual http request.
It concats the stream into a Bytes object and returns it.
Reviewed By: StanislavGlebik
Differential Revision: D30582422
fbshipit-source-id: ed0fe5e34e3fecc6c1b26d2dceb322dfcf5f8e37
Summary:
This diff adds an endpoint `/download/file` that allows to download a file given an upload token.
This will be used for snapshots, as we need to download the snapshot changes, and there's no way to do that right now.
Other options, and why I didn't do them:
- Using the existing `/files` endpoint: Not possible, as it needs hg filenodes and we don't have those.
- Returning the file contents in the fetch_snapshot request: Might make the response too big
- Returning just a single Bytes instead of a stream: I thought streaming would be preferred, and more future proof. In the stack I still put everything in memory in the client, but maybe in the future it should be possible to stream it directly to the file. I'm happy to remove if preferred, though.
Reviewed By: StanislavGlebik
Differential Revision: D30582411
fbshipit-source-id: f9423bc42867402d380e831bc45d3ce3b3825434
Summary: This proved useful couple of times when folks experienced problems with the agent.
Reviewed By: ahornby
Differential Revision: D30837676
fbshipit-source-id: aec769f60a09ecb83857e6e60d49a5662b4ce0b2
Summary:
Add back the octopus merge support for revlog.
This recommits D30686451 (b13579fdf9) and D30686450 (7eb11cb392) as-is, with updates to test files.
Original commit changeset: 9f213766e7c4
Reviewed By: StanislavGlebik
Differential Revision: D30784681
fbshipit-source-id: ace0c317652ad8b657c8edd9a0130532dad53078
Summary:
As far as I could tell, this was legacy from some refactorings.
It was only used in one place, and it was easy to fix.
Also, if we really need it in the future, we can probably use `#[auto_impl]` instead of doing it manually.
Reviewed By: StanislavGlebik
Differential Revision: D30574803
fbshipit-source-id: 20715364713775818fe0e83844637f48b310d87f
Summary: createremote only worked from root of the repo. This fixes it, and tests that in the integration test
Reviewed By: StanislavGlebik
Differential Revision: D30546582
fbshipit-source-id: 84aa304d346e448b44e5d7fb9e9607d84a67da25
Summary:
This adds basic logic for `snapshot restore` command.
- It updates to the parent of the snapshot
- It loads the snapshot changes
For now I did not do changes/tracked changes, as it will need to download the file contents, which will need a new edenapi endpoint, so I'll leave it for a future diff. It just restores your deleted files for now.
Reviewed By: StanislavGlebik
Differential Revision: D30543507
fbshipit-source-id: 080588ceff0ecd595ce739044f0d4118fb8e1a3f
Summary:
log sync reason for `hg cloud sync`.
This will help us to investigate issues better and measure impact for new Eden Api Uploads case by case (after amend, rebase, etc) on different platforms.
Reviewed By: yancouto
Differential Revision: D30775519
fbshipit-source-id: 696e954ec8db19226fb67ad0952e23e2b67e9931
Summary:
Put code using the usage service behind an `EDEN_HAVE_USAGE_SERVICE` macro.
Previously the C++ code was simply guarded by a `__linux__` check, and the
CMake code did not have a guard at all. This caused builds from the GitHub
repository to fail on Linux, since the code attempted to use the usage service
client which was not available.
Reviewed By: xavierd
Differential Revision: D30797846
fbshipit-source-id: 32a0905d0e1d594c3cfb04a466aea456d0bd6ca1
Summary:
In the v1 sparse config arrangement, if all rules were excludes then we
would include a default "**" rule. This was always a little confusing and caused
some weird behavior. Let's remove it from the v2 world.
This actually bit us because the fbsource_exclude profile only has excludes,
which caused it to insert a ** include, which pulled in all of fbsource. We
could fix it to only check if a profile is excludes-only once all the transitive
profiles have been loaded, but I think the cleaner fix is to remove this logic
since it's confusing and never actually used in production.
Differential Revision: D30824082
fbshipit-source-id: adcf4c820cc9f7636f79759d03fc0b387b9f55fa
Summary:
Any error inside the decode stream was being propagated up as a decoder
error. This caused higher level code to not handle certain errors appropriately.
For instance, the lfs retry logic only retries for certain classes of curl
errors. So let's propagate up HttpClientErrors as is.
Reviewed By: kulshrax
Differential Revision: D30798108
fbshipit-source-id: 7316f6cdc47de090c202ff6a1f28d0fba60f7a15
Summary:
The previous version had two issues:
1. It's UB to cast uninit away as it may be actually uninitialized.
2. Because of the cast, the buffer was not actually written nor advanced after written to, causing the caller to think nothing was read.
https://docs.rs/tokio/1.11.0/tokio/io/struct.ReadBuf.html
Reviewed By: dtolnay
Differential Revision: D30823808
fbshipit-source-id: d5f67e4c03f1d63f2241421dd35082ee96b5afd8
Summary: For some reason it got broken, need to call `as_ref()` to properly cast type
Reviewed By: quark-zju
Differential Revision: D30740629
fbshipit-source-id: f49275caae9d360859e97c03709a720dabc22e9e
Summary:
LocalStore no longer special-cases Tree objects with kZeroHash
ids. Instead, unconditionally write into LocalStore with the Tree's
hash.
Reviewed By: xavierd
Differential Revision: D29155470
fbshipit-source-id: aee3840fe8dfd7aa46305b6db6f7950efb2e41d2
Summary:
In preparation for expanding to variable-width hashes, rename the
existing hash type to Hash20.
Reviewed By: genevievehelsel
Differential Revision: D28967365
fbshipit-source-id: 8ca8c39bf03bd97475628545c74cebf0deb8e62f
Summary:
Do not assume `changelog.parents` returns 2 items.
This changes the behavior for root commits. `parents()` used to return
`[repo[nullid]]`, now it returns `[]`.
Reviewed By: andll
Differential Revision: D30784684
fbshipit-source-id: 73f58c85457391fb74b96b88dc4dcb69a25e81ac
Summary:
In a future change, `ctx.parents()` returns `[]` instead of `[repo[nullid]]`
for root commits. Make the change to preserve absorb behavior.
Differential Revision: D30816385
fbshipit-source-id: afded91a6e72d4eb54faf87dcdfc52a81ea1d66f
Summary:
In a future change, `ctx.parents()` returns `[]` instead of `[repo[nullid]]`
for root commits. Make the change to preserve rebase behavior.
Differential Revision: D30816386
fbshipit-source-id: ca7c489991ae149c9640b7da0e6e54f76afbc250
Summary:
We're going to change parents() to return an empty list instead of `[nullctx]`
for roots. This change makes it more compatible with upcoming changes.
Reviewed By: andll
Differential Revision: D30787305
fbshipit-source-id: 1de523964faa64a6496a7bb0197af597e393d859
Summary: This will be used by the next change.
Reviewed By: andll
Differential Revision: D30784683
fbshipit-source-id: 59a37c5f428eaf5950584d8f17471d358bfefee7
Summary: Integrate http hash prefix lookup into the pull operation. One unfortunate change here is that if the prefix is ambiguous, we're only able to output possible full hashes as suggestions. Previously we'd also print commit log information. To retain that we'd need to add an error option to the response and have the server send back an error message with the log information or send another request to download the extra information.
Reviewed By: andll
Differential Revision: D30716050
fbshipit-source-id: 33f8bc38b0bfe7fce4ec11cd8def7feda3b3d3da
Summary:
As title, sampling group determines the sampling rate at which an FS event is logged. The higher the sampling group the more heavily its events are dropped, thus, more frequent events are assigned to the higher sampling groups.
I ran activity recorders on a few workflows, buck build, getdepts, and vscode editing and came up with the following assignment. Note that only a subset of events are assigned to a sampling group (so events not included will not be logged) as we just start to tune the sampling rates and these events should be good for a start.
```
Group1 (1/10)
FUSE_MKDIR
FUSE_RMDIR
FUSE_CREATE
FUSE_RENAME
Group2 (1/100)
FUSE_WRITE
FUSE_LISTXATTR
FUSE_SETATTR
Group3 (1/1000)
FUSE_GETXATTR
FUSE_GETATTR
FUSE_READ
FUSE_READDIR
Group4 (1/10000)
FUSE_LOOKUP
```
For reference, here are the counts of FS events of a cold buck build. The frequencies of other workflows are similar.
```
FUSE_LOOKUP 60.09 98733
FUSE_READ 12.80 21037
FUSE_GETXATTR 8.91 14645
FUSE_FORGET 8.01 13162
FUSE_GETATTR 5.55 9116
FUSE_READDIR 3.21 5270
FUSE_LISTXATTR 0.59 969
FUSE_READLINK 0.54 892
FUSE_STATFS 0.21 338
FUSE_WRITE 0.04 64
FUSE_CREATE 0.02 28
FUSE_RENAME 0.01 23
FUSE_SETATTR 0.01 13
FUSE_UNLINK 0.00 6
FUSE_RMDIR 0.00 1
FUSE_MKDIR 0.00 1
FUSE_MKNOD 0.00 1
```
Reviewed By: xavierd
Differential Revision: D30770533
fbshipit-source-id: 90be881ddbeba2113bbb190bdb1e300a68f500a0
Summary: The new `EdenApiHandler` framework for defining EdenAPI endpoints provides a common place where responses are encoded. This diff adds automatic content compression at this point, using the received `Accept-Encoding` header from the request to determine what compression, if any, should be used. As a result, all endpoints that implement `EdenApiHandler` will get compression for free.
Reviewed By: yancouto
Differential Revision: D30553242
fbshipit-source-id: 9eda54cbf81dd1e03abec769744c96b16fad64ea
Summary:
It can sometimes be difficult to work out from the logging which commit cloud
requests came from which client repo. Previously you could often infer it from
the client identities, however if the request is proxied, the originating hostname can be
lost, and it still doesn't handle the case where the host contains multiple
repos.
This diff adds a new `ClientInfo` struct, which is included by the client
on every `get_references` and `update_references` request. This is logged
by the service, allowing us to correlate which client it came from, and what
workspace version the client had at that time.
Reviewed By: StanislavGlebik
Differential Revision: D30697839
fbshipit-source-id: 8fe2e03f0be2f443f8ae1814f083c04ba5d1805e
Summary: It was not used and hard to implement retry with
Reviewed By: yancouto
Differential Revision: D30716647
fbshipit-source-id: a90b629f7758486c9e526d1eaf3fd29da305f2e7
Summary:
D30704344 (5704ad51f6) upgraded tokio for the buck build. We need to do the same for
the non-buck build. This unbreaks hgbuild.
Also clean up some compiler warnings while I'm here.
Reviewed By: fanzeyi
Differential Revision: D30798315
fbshipit-source-id: 47005c7674d87196aab42b3ddf2194acced3bb6c
Summary:
We have two mode of deriving data: the "normal" way and using backfilling.
Backfilling is different from "normal" mode in that it derives a few commits at
once, and saves them all to blobstore at once.
Backfilling mode seemed to have helped us when we need to derive a lot of data
(e.g. backfill the whole repo). But
a) We don't know how much it helps, and we don't know if it depends on the repo
b) We don't know if it helps when we derive data for newly landed commits (i.e.
we use "backfill" mode in derived data tailer to derive data for latest public
commits)
So this diff adds a bit of logging to a separate scuba table so that we can get
an idea about things like:
1) How long does it take to derive a stack of commits?
2) Where do we spend most of the time (e.g. deriving, saving the blobs, saving
the mapping).
Reviewed By: mitrandir77
Differential Revision: D30805504
fbshipit-source-id: d82c905cafa87459990d74769a0dddcc91fac174
Summary:
It allows us to do 3 things:
1) Remove derive function
2) Add support for backfill mode so that we can compare perf with and without
it
3) Use all derived data types, and not just 3 of them
Reviewed By: krallin
Differential Revision: D30804258
fbshipit-source-id: 604723a3d845a60cfd94b4e090a121f5b5191536
Summary: This command can be useful to split a large bonsai commit into a smaller one
Reviewed By: mitrandir77
Differential Revision: D30776789
fbshipit-source-id: dc56d7c51eb0e9e0988dcba868c6008ebf488927
Summary:
While we don't really need it, creation mononoke matches fail if they are not
present. Let's just enable it here - it's not a bad thing to initialize them
Reviewed By: mitrandir77
Differential Revision: D30780463
fbshipit-source-id: c4199c6711ae7bd9641e9f51643b94d020051dbd
Summary: The code for accessing config fields had a lot of repetitive boilerplate. Let's move that to a helper function.
Reviewed By: andll
Differential Revision: D30785932
fbshipit-source-id: fb4d47337a27bd6e75eeb38d5a9d1de5b1fac6ce
Summary: Implement serverside graph endpoint for fetching the mapping of commits to commit parents for the missing segment of a commit graph. This implementation uses the find_commits_to_send method from the get_bundle_response library. What may be missing from pull and the old bundle protocol now is mutation markers.
Reviewed By: yancouto
Differential Revision: D30485672
fbshipit-source-id: ba3a30d9e482d60831cbe7a8e89f20dab947d9a1
Summary:
Since the find_commits_to_send method was added, common is already a
hashset not a vector, so it doesn't needed to be converted to a hashset.
Reviewed By: quark-zju
Differential Revision: D30622028
fbshipit-source-id: e5d1b6c60115d13c906b25142043652ba9e89d70
Summary:
Not flushing the data to disk makes studying performance almost impossible due
to not being able to avoid fetching from the network. By forcing a flush to
disk, we can ensure that data will always be on disk, making performance
measurement easier. This will also prevent users from re-fetching the same data
multiple times.
Reviewed By: fanzeyi
Differential Revision: D30784399
fbshipit-source-id: 0250c209b5f49f95cf2f43873573cacc661a4989
Summary:
Since this method wasn't overriden, EdenFS would never periodically flush data
to disk.
Reviewed By: fanzeyi
Differential Revision: D30784400
fbshipit-source-id: d88e535250a476582868dd82e57137a0ac38f921
Summary: Previously, there were special variants for missing and invalid URLs (since the server URL is presently the only required config option). In order to support other required config options, let's simplify the enum to just have variants for missing and invalid config fields respectively.
Reviewed By: yancouto
Differential Revision: D30745971
fbshipit-source-id: e414ec2fadc5d04e9c788bf290a70f6cf52dbe58
Summary: Saw this when reading related code.
Reviewed By: kulshrax
Differential Revision: D30783665
fbshipit-source-id: f9b598b9301619346972bd0abf893f089d902022
Summary:
I'm not sure what this was for, but it doesn't seem necessary, and removing it simplifies the code a lot, enabling us to make other improvements later.
This is an alternate, less ambitious version of https://www.internalfb.com/diff/D30620443.
Reviewed By: DurhamG
Differential Revision: D30674016
fbshipit-source-id: 17dee50b82c78d31e45492dc23826d8c3fe838e5
Summary:
This test relies on Mononoke, so it fails for make local build/test,
which breaks hgbuild. Let's only enable it for buck tests.
Reviewed By: quark-zju
Differential Revision: D30782799
fbshipit-source-id: 4b543beeb248715702b9072d84cdb8211dcd4a9b
Summary:
Currently there are two things preventing us from running add_sync_target
on existing target:
* already existing bookmark
* already existing config
Both need to be deleted to create new target. This diff removes the second
one to simplify code and make it easier to recreate the target (it's easy to
forget about manually removing the config as they otherwise don't need
human interventions).
Reviewed By: StanislavGlebik
Differential Revision: D30767613
fbshipit-source-id: f951c0e1ef9bde69d805dc911331fcdb220123f2
Summary:
This logic scans all the ancestors of the working copy that are not
ancestor of the graft source and checks their extras. With lazy changelog this
is extremely expensive. Let's just drop this logic.
Reviewed By: quark-zju
Differential Revision: D30734017
fbshipit-source-id: ca5606cea08efe10f29847970379d6bff4eb4aee
Summary:
Update the `Filenodes` trait so that it doesn't require the repository id to be
passed in every method invocation. In practice a filenodes instance can only
be used for a single repo, so it is safer for the implementation to store the
repository id.
At the same time, update the trait to use new futures and async-trait.
Reviewed By: yancouto
Differential Revision: D30729630
fbshipit-source-id: a1f80a299d9b0a99ddb267d1f7093f27cf21f1af
Summary:
Make the derivation process for mercurial changesets and manifests not depend
on `BlobRepo`, but instead use the repo attribute (`RepoBlobstore`) directly.
This will allow us to migrate to using `DerivedDataManager` in preparation
for removing `BlobRepo` from derivation entirely.
Reviewed By: yancouto
Differential Revision: D30729629
fbshipit-source-id: cf478ffb97a919c78c7e6e574580218539eb0fdf
Summary:
Make the derivation process for blame and deleted files manifest not depend
on `BlobRepo`, but instead use the repo attribute (`RepoBlobstore`) directly.
This will allow us to migrate to using `DerivedDataManager` in preparation
for removing `BlobRepo` from derivation entirely.
A `BlobRepo` reference is still needed at the moment for derivation of the
unodes that these depend on. That will be removed when `DerivedDataManager`
takes over co-ordination of derivation.
Reviewed By: yancouto
Differential Revision: D30729628
fbshipit-source-id: 4504abbe63c9bf036d69cb4341c75b13061fae18
Summary:
Make the derivation process for fsnodes and skeleton manifests not depend on
`BlobRepo`, but instead take the `DerivedDataManager` from the `BlobRepo` and
use that instead. This is in preparation for removing `BlobRepo` from
derivation entirely.
Reviewed By: yancouto
Differential Revision: D30301855
fbshipit-source-id: a2ed1639526aad9ddbe8429988043f0499f7629c
Summary:
Make the derivation process for unodes not depend on `BlobRepo`, but instead
take the `DerivedDataManager` from the `BlobRepo` and use that instead. This
is in preparation for removing `BlobRepo` from derivation entirely.
Reviewed By: yancouto
Differential Revision: D30300408
fbshipit-source-id: c35e9e21366de74338f453aaf6be476e7305556d
Summary:
The derived data manager also has a reference to the repo blobstore. This must
also be overridden when we override the blobstore.
Reviewed By: yancouto
Differential Revision: D30738354
fbshipit-source-id: b0e16ef810c5244cd056a3c9e5b9ceaaddb5ecea
Summary:
modernise tests (removing disabling treemanifest)
treemanifest is the default now, so it shouldn't be disabled unless absolutely necessary.
Also, if we would like to switch some of the tests to Mononoke, it shouldn't be disabled.
Only two tests left with it in commitcloud. Those two a bit harder to fix.
```
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm/tests
[139] → rg 'disable treemanifest' | grep cloud
test-commitcloud-backup-bundlestore-short-hash.t: $ disable treemanifest
test-commitcloud-backup.t: $ disable treemanifest
```
Reviewed By: kulshrax
Differential Revision: D30278754
fbshipit-source-id: cf450084669c2b6b361cd34952bf986e913de1a8
Summary:
I want to use `ReplicaFirst` read connection type since `ReplicaOnly` is a bit too restrictive.
We've had 2 MySQL SEVs this year when all the replicas went down crashing our services despite the primary instance working normally.
There was also a case when I've deleted too much rows at once and all replicas went down due to replication lag (I know better now)
RFC
- Yay or Nay?
- Should I expand `ReadConnectionType` to mirror all options of `InstanceRequirement`?
- Perhaps it's worth moving it into the `common/rust/shed/sql` crate?
I kept cleaning up all the usages out of this diff to keep the changes minimal for RFC
Differential Revision: D30574326
fbshipit-source-id: 1462b238305d47557372afe7763096c53df55f10
Summary:
Segmented changelog seeder spends a significant chunk of time fetching
changesets. By saving them to file we can make reseeding significantly faster.
Reviewed By: farnz
Differential Revision: D30765374
fbshipit-source-id: 0e6adf12e334ad70486145173ae87c810880988a
Summary:
In backfill_derived_data we had a way to prefetch a lot of commits at once, so
that backfill_derived_data doesn't have to do it on every startup.
I'd like to use the same functionality in segmented changelog seeder, so let's
move it to the separate binary.
Reviewed By: mitrandir77, farnz
Differential Revision: D30765375
fbshipit-source-id: f6930965b270cbaae95c3ac4390b3d367eaab338
Summary: ContentDataStore is meant to be implemented local-only. Fetching remotely seems to cause the issue observed in https://fb.workplace.com/groups/scm/permalink/4192991577417097/ (though I'm not quite sure why yet)
Reviewed By: kmancini
Differential Revision: D30744817
fbshipit-source-id: 68875a4912905f9b8f88cf4be804c5d988c3905d
Summary: If the bind unmount fails in in the privhelper, theres a possibility of infinite recursion in this method. This adds a flag to indicate if we've tried the bind unmount before.
Differential Revision: D30732857
fbshipit-source-id: 6ee887d211977ee94c8e66531287f076a7e61a2c
Summary:
It sounds like macOS has a bug where an APFS subvolume may be falsely created.
Let's retry with the hope that the retry will succeed.
Differential Revision: D30657706
fbshipit-source-id: 60bc74f789a0d34b2be53073103b95474a9a18e6
Summary: This is regenerated rust lib using the latest compiler
Reviewed By: krallin
Differential Revision: D30720130
fbshipit-source-id: 3d3389ec8504568fc356dda1577e1f7801cb7e96
Summary:
~~Also enable the `derive` feature so it isn't necessary to separately
depend on `strum_macros`.~~
This turns out to break a lot.
Reviewed By: dtolnay
Differential Revision: D30709976
fbshipit-source-id: a9181070b8d7a8489eebc9e94fa24f334cd383d5
Summary: Move `edenapi::Client`'s internals to an `Arc<ClientInner>`. This makes the client `Clone`-able while sharing the same underlying state. This is particularly useful for scenarios where `Future`s or `Stream`s returned by the client need to hold a reference to the client itself (e.g., in order to issue subsequent HTTP requests).
Differential Revision: D30729803
fbshipit-source-id: c97e700c9e3702f818eb86ded1a46f920a55cfd1
Summary: The `Fetch<T>` type has basically turned into the canonical type EdenAPI for all EdenAPI responses. Originally, this type was merely an implementation detail (essentially just a named tuple returned by the `fetch()` method, hence the name), but given its prominence in the API, the name is confusing. As we add more functionality and usage to this type, it makes sense to give it a more suitable name.
Differential Revision: D30730573
fbshipit-source-id: 7acd2a86b55bdfc186bd9110f6a99333df9d29d3
Summary:
Some of the method names used internally by `edenapi::Client` are a bit terse.
This was OK back when there were only handful of private methods which were used by a small number of API methods that were doing more or less the same thing (sending concurrent POST requests for a set of keys).
Today, there are way more API methods, most of which set up requests in different ways. As such, it makes sense to give these older private methods more explicit and descriptive names so that their intended usage is clear.
Differential Revision: D30729802
fbshipit-source-id: 5adfd8e7ba153df8c036e4dbb312f95b9b1d7335
Summary: Allow repack to be called on treescmstore via the ContentStore shim like filescmstore is already supported.
Reviewed By: andll
Differential Revision: D30687145
fbshipit-source-id: 7559af08e98cfb22da6dbf45dc1746312b1e6d28
Summary:
Provide a basic implementation of the LegacyStore trait for TreeStore to allow repack calls to be forwarded to the fallback ContentStore for trees.
Repack will be removed entirely before contentstore is deleted, and the `unimplemented` methods are never called, so this should be safe.
Reviewed By: andll
Differential Revision: D30687136
fbshipit-source-id: d238d70fbf6be5c25c2e1c9610430a53d031bf3b
Summary: Looks like it was lost during the last refactoring, let's add it back.
Reviewed By: farnz
Differential Revision: D30728456
fbshipit-source-id: 20c638b3c5a8664f2367f871cd29a793fd897de3
Summary:
Some users have reported errors of the form:
```
error.HttpError: [65] Send failed since rewinding of the data stream failed (seek callback returned error 2)
```
These are caused by the fact that we're passing the HTTP request body directly to libcurl in memory rather than via a file, but we haven't implemented the `seek()` method necessary for libcurl to retransmit the data if needed. This diff implements the method.
Reviewed By: DurhamG
Differential Revision: D30654625
fbshipit-source-id: f21a067ad02ee540b86cf2e6eff2c6f08f45a3e4
Summary:
Like it says in the title, this updates us to use Daemonize 0.5, though from
Github and not Crates.io, because it hasn't been released to the latter yet.
The main motivation here is to pull in
https://github.com/knsd/daemonize/pull/39 to avoid leaking PID files to
children of the daemon.
This required some changes in `hphp/hack/src/facebook/hh_decl` and `xplat/rust/mobium` since the way to
run code after daemonization has changed (and became more flexible).
Reviewed By: ndmitchell
Differential Revision: D30694946
fbshipit-source-id: d99768febe449d7a079feec78ab8826d0e29f1ef
Summary:
At the moment when segmented changelog is updated and/or reseeded mononoke
servers can pick it up only once an hour (this is a current reload schedule)
or when mononoke server is restarted. However during production issues (see
attached task for an example) it would be great to have a way to force all
servers to reload segmented changelog.
This diff makes it possible to do so with a tunable. Once tunable changes its
value then monononoke servers almost immediately (subject to jitter) reload it.
This implementation adds a special loop that polls tunables value and reloads
if it changes. Note that in theory it could avoid polling and watch for configerator
changes instead, but it would be harder to implement and I decided that it's
not worth it.
Reviewed By: farnz
Differential Revision: D30725095
fbshipit-source-id: da90ea06715c4b763d0de61e5899dfda8ffe2067
Summary:
Previously, extremely large prefetch calls could cause an OOM if the requested files were all fetched remotely from EdenApi, in which case the memory would remain in use until the entire batch had been fetched.
With this change, at most 1000 EdenApi files will be held in memory at once (or 10GB of memory). This is a stop-gap solution, a better approach would be to avoid storing all EdenApi files in memory after a certain amount, or allow the batch fetching implementation to understand we're only prefetching, and thus avoid reading anything back from disk or storing EdenApi files in memory unnecessarily.
Reviewed By: andll
Differential Revision: D30686054
fbshipit-source-id: 022e353760c515961a8956f7958b43f429143971
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------
build-break (bot commits are not reviewed by a human)
Reviewed By: farnz
Differential Revision: D30717040
fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
Summary: Keep two versions of fbthrift_ext that one with tokio-0.2 and the other with tokio-1.x. This diff is just renaming.
Reviewed By: dtolnay
Differential Revision: D30558441
fbshipit-source-id: bfe7e96b95529f2745f635190df5118a0cb44014
Summary:
Having the same queue for all three makes the dequeue code overly complicated
as it needs to keep track of the kind of request that needs to be dequeued.
Incidently, the previous code had a bug where request in "putback" would be
requeued at the end of the queue, even though there were at the beginning of it
if they all had the same priorities.
This is theory should also improve the dequeue performance when the queue has a
mix of blobs/tree requests, but I haven't measured.
Reviewed By: genevievehelsel
Differential Revision: D30560490
fbshipit-source-id: b27e5429105c07e5f9eab482c12e5699ca3413f7
Summary:
Since the background condition is before the actual prefetching of files,
specifying the background option would just glob files but not prefetch them
which is equivalent to prefetching all the trees.
Reviewed By: genevievehelsel
Differential Revision: D30618753
fbshipit-source-id: 5533b1c78d614342ac3341ce033795be3850750a
Summary:
It looks like a few scmstore changes landed with warnings (probably fixed higher up in the tree unification stack).
This change fixes those warnings.
Differential Revision: D30686092
fbshipit-source-id: d80625dea64f35683f815b58c83a3e5bb7cbdfa8
Summary:
When we remove a file from a sparse profile and commit the profile, it
should delete the file on disk. There's a bug where it doesn't actually delete
the file. This fixes it by passing the correct commit parents to the refresh
function.
Reviewed By: andll
Differential Revision: D30683677
fbshipit-source-id: 7f012faa99975d8270209f2962e7f9236890daed
Summary:
Use `populate_missing_vertexes_for_add_heads` (added by D27630093 (f138b012e9)) to avoid
excessive lookups for non-master ids that remain in non-master. The function
was used in two other `flush` cases, but missed the id reassignment case. It
works basically by using the "discovery" logic to quickly rule out what's
missing and what's present (ex. if a root is missing in the server graph, then
all descendants of the root is missing).
Reviewed By: andll
Differential Revision: D30700451
fbshipit-source-id: 1f1cd88399dbffd4af75083fef1f3e363a5c60fe
Summary: During reassigning non-master ids, it might trigger too many remote lookups.
Reviewed By: StanislavGlebik
Differential Revision: D30700452
fbshipit-source-id: 2483335e466c3de8a362f7b6a15fc4ba9e2693be
Summary:
Support octopus merge defined in the following format:
- The revlog flag has `1 << 12` set.
- Extra `stepparents` is set to `hexnode1,hexnode2,...` format.
This is mainly used to support revlog from stream clone.
Reviewed By: StanislavGlebik
Differential Revision: D30686450
fbshipit-source-id: d5aa2f18a02f5f0d7aa033210fb4f79b729c0d26
Summary:
Extend the struct so we can support more than 2 parents.
The size of the sturct is now 16 bytes, from 8 bytes. This might have some
performance overhead.
Not using `Box<[u8]>` because that will make the struct 24 bytes.
Reviewed By: StanislavGlebik
Differential Revision: D30686451
fbshipit-source-id: c0f8d0472c7e578f34d771dacecffc91585650c3
Summary:
In `vertex_id_with_max_group(name, group)`, if `group` is master and the `name`
exists in the non-master group, then there is no need to lookup remotely because
a same name (vertex) cannot be present in both master and non-master group. In
that case, just return that the `name` does not exist in the master group.
Reviewed By: StanislavGlebik
Differential Revision: D30699215
fbshipit-source-id: 5170abe719aa7cc31533912e18bc0e21f133e1f4
Summary:
Added a test about excessive remote lookups when flush() reassigns vertexes
from non-master to master.
Reviewed By: StanislavGlebik
Differential Revision: D30699214
fbshipit-source-id: 0547707764855ab9a563178740612b54df4a5fc9
Summary: They are used to narrow down issues related to S242328.
Reviewed By: StanislavGlebik
Differential Revision: D30699216
fbshipit-source-id: 28f4f0bfadadb2dea5510878168c2d7b47a8641c
Summary: Split out the request ID, URL, and HTTP method from `RequestContext` into a new `RequestInfo` struct, which can be cheaply cloned and included in the response returned to the caller. This enables the caller to correlate requests and responses, which is useful when working with many concurrent requests.
Reviewed By: DurhamG
Differential Revision: D30650365
fbshipit-source-id: 68efedcf852c91387450443ebe46062809633f10
Summary:
Make it possible to call the CheckIntegrity APIs from Python such as:
In [1]: cl.inner.checkuniversalids()
Out[1]: []
In [2]: cl.inner.checksegments()
Out[2]: []
In [3]: cl.inner.checkisomorphicgraph(cl.inner, cl.dageval(lambda: heads(mastergroup())))
# take a while
Out[3]: []
Reviewed By: andll
Differential Revision: D30682536
fbshipit-source-id: 23f280bf261def3d20d5f7dc15a48c2fc2d79d77
Summary: This makes other crates easier to implement CheckIntegrity.
Reviewed By: andll
Differential Revision: D30682540
fbshipit-source-id: 4333f37fa7bafe55a8bee9f149b2f23a463c51af
Summary:
Makes the revlog index provides dummy graph integrity checks so it can
be used as a generic object in the Python bindings.
Reviewed By: andll
Differential Revision: D30682542
fbshipit-source-id: 25c6e8640de46188d7bf45a927e11e0779a8ad40
Summary: Make it possible to check a graph against a reference graph.
Reviewed By: andll
Differential Revision: D30682539
fbshipit-source-id: 57db952dcda5656ff6000e9961448c9b64afbaf0
Summary: Make it possible to check segment integrity.
Reviewed By: andll
Differential Revision: D30644243
fbshipit-source-id: 24bb0c8c8c9394d688e3e9320e59268bc2a4ed3f
Summary: Make it possible to check universal ids externally.
Reviewed By: andll
Differential Revision: D30644242
fbshipit-source-id: f312ff59dbdf68e57c5249d57c5d44da0b10e398
Summary: This will be used to verify graph integrity later.
Reviewed By: andll
Differential Revision: D30644244
fbshipit-source-id: 0d22b70121da37c411adf17200a6c752fefa80ad
Summary: This breaks all use of `hg sparse`, because `.hg*` cannot be matched.
Reviewed By: mitrandir77
Differential Revision: D30666349
fbshipit-source-id: c06d1b798a57490f2e5560f178a2839ae5425146
Summary: We've got multiple manifold parameters now, two of which are Option<i64>, so lets create a struct to name them
Reviewed By: HarveyHunt
Differential Revision: D30305462
fbshipit-source-id: 44eee00d478e4485d074a14fcccec2f0f9572ecd
Summary:
This allows to quickly identify the program that emitted the error.
Per user feedback: https://fb.workplace.com/groups/clifoundation/posts/433922134631466
Reviewed By: StanislavGlebik
Differential Revision: D30604611
fbshipit-source-id: 712bc9f466c5a7b5c97a1b83a10fbe277341a300
Summary:
The mockall crate's `automock` attribute previously created nondeterministic output, which leads to frequent random "Found possibly newer version of crate" failures in Buck builds that involve cache.
The affected trait in Conveyor is:
https://www.internalfb.com/code/fbsource/[4753807291f7275a061d67cead04ea12e7b38ae2]/fbcode/conveyor/common/just_knobs/src/lib.rs?lines=13-23
which has a method with two lifetime parameters. Mockall's generated code shuffled them in random order due to emitting the lifetimes in HashSet order. The generated code would randomly contain one of these two types:
`Box<dyn for<'b, 'a> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
`Box<dyn for<'a, 'b> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
Reviewed By: jsgf
Differential Revision: D30656936
fbshipit-source-id: c1a251774333d7a4001a7492c1995efd84ff22e5
Summary: Adds an option to print the path to the eden log file. Similar to `eden pid`, this can be used for shell one-liners.
Reviewed By: chadaustin
Differential Revision: D30558294
fbshipit-source-id: ca70addaef2093e10f0321bae0cff3b1bfc7dc75
Summary: `eden debug log --upload` fits in better with the format of the other cli tools (rather than `eden debug log upload`)
Differential Revision: D30557691
fbshipit-source-id: 32e47e1487703560f2adb5f0f79f1002d29eea93
Summary:
In a previous diff we made sparse matchers become union matchers, since
they are a collection of each individual sparse profiles matcher. In order to
maintain the performance benefits of having sparse computations run on
non-python matchers, we need to update the matcher extractor to support union
matchers.
Reviewed By: quark-zju
Differential Revision: D30588256
fbshipit-source-id: 15014be844e1d713e19ae8f2959d947516b4e3c7
Summary:
We were copy/pasting metadata.get("version", "1") everywhere. Let's
make it a helper function.
Differential Revision: D30586162
fbshipit-source-id: ff6a9706f1970f84ffeb7de0e1362c3ba507fc00
Summary:
Sparse profiles should be roughly scoped around the files needed to
work on a certain product. If an engineer needs to work on multiple products
they should be able to enable multiple profiles.
Previously, multiple v2 profiles would be combined into an ordered list of
include/exclude rules, which meant that profiles enabled later could exclude
files included by the earlier profiles.
To fix this, let's treat each profile separately and create a matcher for each.
We then combine these into a union matcher, which means we're guaranteed to have
all the files that each profile specifies.
Differential Revision: D30586161
fbshipit-source-id: 2e04cfdba670ffce381a7c041706f315775ad7b0
Summary:
In a future diff we'll process sparse profiles differently at the
matcher creation level. To do so we need to expose the profile object to that
layer. Let's do it by storing the profile instead of just the profile name.
Differential Revision: D30586163
fbshipit-source-id: d90343b4101c43fbd838512289362aca7c3f816a
Summary:
With lazy changelog, it is possible that `vertex_name` is unable to translate
an id to name in non-async context. Do not treat it as errors.
Differential Revision: D30615948
fbshipit-source-id: 4e7abd77c6eb116db00e25489685563b7cf78a9c
Summary: The segments is stored in the shared `.hg`, not in the local repo `.hg`.
Differential Revision: D30615949
fbshipit-source-id: 9d2b7c1ce245553a2df070b066429fbcead5d827
Summary:
Similarly to the previous diff, reducing the lock scope will improve
concurrency leading to higher performance in EdenFS.
Reviewed By: andll
Differential Revision: D30595787
fbshipit-source-id: 1d52e4a8d362f7e2e3e18c2a57a3ebb7628f549e
Summary:
Similarly to the previous diff, let's not hold any read/write locks when not
needed. This will improve concurrency of the code.
Reviewed By: andll
Differential Revision: D30595786
fbshipit-source-id: 6ea6c689e4deca713051a9f3611647334c528bc7
Summary:
In the context of EdenFS, during a prefetch operation, EdenFS will call into
the revisionstore (via the backingstore) from multiple threads and use the
ContentStore::prefetch method in each of them with batches of keys.
Unfortunately, the IndexedLogDatastore lock is write held both when serializing
fetched data and when writing to the underlying IndexedLog. The serialization
part can be fairly CPU intensive as it will compress data with LZ4, but doesn't
require the lock to be held. If a concurrent get_missing call to the
IndexedLogDatastore is made, that will thus wait for the compression to
complete, reducing overall concurrency and fetching speed.
Reviewed By: andll
Differential Revision: D30583566
fbshipit-source-id: 06f5f4988c1bc911ae155189317232b54915a5cf
Summary: Add a new `BackingStore` implementation based on `scmstore` instead of `ContentStore` and allow the backend to be selected with the `scmstore.backingstore` config.
Reviewed By: andll
Differential Revision: D29672974
fbshipit-source-id: dc6b8662903bcfc941b586544aad487de6b8c956
Summary: Introduce a new `StoreValue` trait, implemented for `StoreFile`, for use in `CommonFetchState`, which will be shared between `TreeStore` and `FileStore`.
Reviewed By: kulshrax
Differential Revision: D30295987
fbshipit-source-id: 862349980befa13cb3d420ee617e4fe5464d890a
Summary: Introduce a new `StoreAttrs` trait, implemented for `FileAttributes`, for use in `CommonFetchState`, which will be used by both `TreeStore` and `FileStore`.
Reviewed By: kulshrax
Differential Revision: D30295969
fbshipit-source-id: 603242fe08bea72035b1b1d0f37c46514e1fb92b
Summary: Refactoring in preparation for updating `TreeStore` to match `FileStore`
Reviewed By: andll
Differential Revision: D30295957
fbshipit-source-id: 0f1677311eb2578d8e0dae12c07a1599edc3b500
Summary: Making `FetchResults` generic in preparation for using it with trees too.
Reviewed By: andll
Differential Revision: D30286592
fbshipit-source-id: a48cf2dbcdcc2c4b8a102eaa02ac465c367c6793
Summary: I'll be adding metrics to the `TreeStore` soon, so I'm refactoring code that will be shared between files and trees. This change moves the non-file-specific metrics types up to the top-level `scmstore` module, rather than inside `scmstore::file`.
Reviewed By: andll
Differential Revision: D30284933
fbshipit-source-id: 31f48312bca8d75d4893220cd189b9735a37a5a0
Summary: Helps figure out what happens to metalog internally.
Differential Revision: D30563249
fbshipit-source-id: 10323d36d762edda93206dd01c88d1f0d8abdf8d
Summary:
The failpoint feature supports more complex injections, such as sleep, or fail
after a few times. There is no need to keep the adhoc faultinjection feature.
Differential Revision: D30495223
fbshipit-source-id: b5613811e489a5a52e9c0dd1ebf1096c848a402b
Summary: Currently, tree imports are queued regardless of whether they are in the `hgcache`. This adds unnecessary delay, especially if the queue is busy (importer takes a long time and causes queue to backlog). This diff adds the logic to check if the tree is in `hgcache` before enqueuing a tree import request.
Reviewed By: xavierd
Differential Revision: D30514871
fbshipit-source-id: eb23f64b7f059832571f957fb67d18c3821d2844
Summary:
this library is a more general version of the `panic_unpack` lib I
made in fbcode. I made this library, its mit-apache licensed
Reviewed By: dtolnay
Differential Revision: D30607308
fbshipit-source-id: ee4fad3924fdae021753055cd3fd88c99cb99512
Summary:
This allows us to insert FAILPOINTS in Python so we can use sleep, return error
etc.
Differential Revision: D30495224
fbshipit-source-id: aef56d03bc32eefb69573cfa586aa63a301edffc
Summary:
I abandoned D30603353 in favor of this one because
its_cleaner
We don't need repo name in every hook, just 2 of them.
This will allow us to have predefined, named hook configs that are repo-agnostic. E.g. when repositories are similar, they could share one set of hook configs in configerator.
There were two hooks that had repo_name in configerator hook config: verify_integrity and verify_reviewed_by
Reviewed By: StanislavGlebik
Differential Revision: D30605229
fbshipit-source-id: c310b16b564808d0dc0909d21cc3521a57e06fad
Summary: The result is too flakey now that we've introduced threading.
Reviewed By: andll
Differential Revision: D30607733
fbshipit-source-id: f8bfa2a57d427731fb4ac3011f4364190a83b771
Summary:
Some code in the HgDatapackStore is overly complicated due to the fact that
revHash returns a owned Hash and this forces the code to thus copy it onto a
temporary vector. By having a method that can directly return a slice to the
hash, this issue disappears, thus let's add it.
Reviewed By: chadaustin
Differential Revision: D30582458
fbshipit-source-id: dc102117bc82ab72378293c0abfe9acfd862e9e6
Summary: Cleaned up all remaining usages of this deprecated API in CTP codebase
Differential Revision: D30517771
fbshipit-source-id: 6b2c7fb6c569bf5a928a7eec60fdd890baad312f
Summary:
Handling of mutable renames was incorrect for two reasons:
1) We didn't add an entry to history graph, so only a single changeset before
rename was returned. That was easy to "fix" (just add a new entry to history
graph), but...
2) ...all history operations now have to use a different path (the source of
the rename path).
To fix it let's track not just the changeset id, but also the path for the
given changeset id. Since the path can potentially be large I wrapped it into
Arc to avoid expensive clones.
Differential Revision: D30576342
fbshipit-source-id: a99f6269c34b0a0c626104ec47c9392f984328fb
Summary:
This diff calls the `/:repo/snapshot` EdenApi endpoint added on D30514854 (ab17c4d181) from the `hg snapshot restore` command.
For now, it just prints the parent of the snapshot, but in next diffs it will update to it and restore the dirty changes.
Reviewed By: StanislavGlebik
Differential Revision: D30517984
fbshipit-source-id: e1381eaed561a7184ee02ab99d0282f11a1d944f
Summary: This diff adds the `fetch_snapshot` method to the EdenApi trait, implements it for talking with the EdenApi service, and also adds python bindings for ease of use from python.
Reviewed By: StanislavGlebik
Differential Revision: D30517973
fbshipit-source-id: 41c24ba25040b397b7d739c2885a47acfb9100d2
Summary:
I noticed that every time we fetch blob from hg, we calculate sha hash and put it into metadata table.
Both calculating sha1 of content and writing it to rocks is fairly expensive, and it would be nice if we can skip doing so in some cases.
In this diff I use inexpensive cache check to see if we already calculated metadata for given blob and skip recalculation
In terms of performance, it reduces blob access time in hot case from **0.62 ms to 0.22 ms**.
[still need to do some testing with buck, but I think this should not block the diff since it seem farily trivial]
This is short-medium term fix, the longer term solution will be keeping hashes in mercurial and fetching them via eden api, but this will take some time to implement
Reviewed By: chadaustin, xavierd
Differential Revision: D30587132
fbshipit-source-id: 3b24ec88fb02e1ea514568b4e2c8f9fd784a0f10
Summary: Similarly to the enqueue benchmark, let's have a dequeue benchmark.
Differential Revision: D30560489
fbshipit-source-id: ae18f7e283e4bab228aaa0f58bff2e6f2cfa3021
Summary:
In order to enqueue and find an element in a hash table, the key needs to be
hashed. Hashing a HgProxyHash relies on hashing a string which is significantly
more expensive than hashing a Hash directly. Note that they both represent the
same data and thus there shouldn't be more collisions.
Reviewed By: chadaustin
Differential Revision: D30520223
fbshipit-source-id: 036007c445c28686f777aa170d0344346e7348b0
Summary:
Allocations are expensive, especially when done under a lock as this increase
the critical section, reducing the potential concurrency. While this yields to
a 1.25x speedup, this is more of a sideway improvement as the allocation is now
done prior to enqueuing. This also means that de-duplicating requests is now
more expensive, as no allocation would be done before, but at the same time,
de-duplication is the non-common code path, so the tradeoff is worthwhile.
Reviewed By: chadaustin
Differential Revision: D30520228
fbshipit-source-id: 99dea65e828f9c896fdfca6b308106554c989282
Summary: The F14 hashmap are significantly faster than the std::unordered_map.
Reviewed By: chadaustin
Differential Revision: D30520225
fbshipit-source-id: d986908c5eac17f66ae2c7589f134c430a3c656e
Summary:
When turning on the native prefetch, EdenFS will enqueue tons of blob requests
to the import request queue. The expectation is then that the threads will
dequeue batch of requests and run them. What is being observed is however
vastly different: the dequeued batches are barely bigger than 10, far lower
than the batch capacity, leading to fetching inefficiencies. The reason for
this is that enqueuing is too costly.
The first step in making enqueuing less costly is to reduce the number of times
the lock needs to be acquired by moving the de-duplication inside the enqueue
function itself. On top of reducing the number of times the lock is held, it
also halves the number of allocation done under the lock.
Reviewed By: chadaustin
Differential Revision: D30520226
fbshipit-source-id: 52f6e3c1ec45caa5c47e3fd122b3a933b0448e7c
Summary:
It turns out that we do want to use a Future to make sure that the tracebus and
watches are completed on the producer and not on the consumer of the future. We
could use a `.via(inline executor)` but the code becomes less readable, so
let's just revert the diff.
Reviewed By: chadaustin
Differential Revision: D30545721
fbshipit-source-id: 524033ab4dbd16be0c377647f7f81f7cd57c206d
Summary:
I need to be able to run 'hg sparse files $PROFILE' in an Eden checkout
for a Sandcastle job that analyzes sparse profiles. We can't run 'hg sparse' at
all in Eden repos, so instead I run it in the backing repo. But that repo is
checked out to the null commit, so let's support --rev on 'hg sparse files' so I
can inspect an arbitrary rev.
Differential Revision: D30561529
fbshipit-source-id: 93b46caa9b63637bf4d63d5438bd23cbffb3983a
Summary:
This config option was used to slowly roll out LFS for a repo.
However, it is no longer used and can therefore be removed.
Reviewed By: StanislavGlebik
Differential Revision: D30511880
fbshipit-source-id: 59fe5925cc203aa609488fdf8ea29e9ff65ee862
Summary: Apparently OSX has different ls -R output than linux.
Reviewed By: singhsrb
Differential Revision: D30566611
fbshipit-source-id: 2f232b12d1971bea18c7131c1ec82244252527c7
Summary:
This adds a new endpoint to EdenApi: `/:repo/snapshot`.
It fetches information about a snapshot (for now, parents and files).
This will be used by the `hg snapshot restore` command.
- Parent will be used to do a normal "hg update" to it
- File changes will be used to apply the working state file changes on top of it.
Reviewed By: ahornby
Differential Revision: D30514854
fbshipit-source-id: b6c6930410cf15fe874eca1fce54314e5011512a
Summary:
Very basic shell for the `hg snapshot restore` command.
For now just reads the id and prints it to stdout. Next diffs will add functionality to it.
Reviewed By: StanislavGlebik
Differential Revision: D30449567
fbshipit-source-id: 9f4e0bb284dff7e6ffb4398942f3247a09511933
Summary:
This refactors the complete_trees endpoint to use the abstraction added on D30451710.
This one needed some extra refactoring, the reason why I separated it on its own diff:
- It was the only endpoint that also returned HTTP 400 errors, as well as 500.
- I made the handler trait return a custom error that may be 400 or 500 errors, but the default is 500 and like anyhow Error, errors automatically convert to that to make the happy path very simple
- Also added a type aliasing that makes the return type simpler to look at.
- Then moved this endpoint to use EdenApiHandler, while keeping the previous behaviour of return 400 error when the path is wrong.
Reviewed By: StanislavGlebik
Differential Revision: D30483501
fbshipit-source-id: 5cb4cd7f3fc2bbdafb9f0e6a78739086c829eee1
Summary:
This diff refactors most endpoints to use the new abstraction added on D30451710.
There are still a few endpoints that couldn't be refactored (see summary of previous diff).
Reviewed By: StanislavGlebik
Differential Revision: D30483500
fbshipit-source-id: ea2cc24afb119bf63a45a6f4bf48706a004743cc
Summary:
Uses the abstraction built on D30451710 to make more endpoints simpler. It needed to be made a little more generic by using streams instead of `Vec`.
Some endpoints can't be migrated as of now:
- hash_to_location and trees as they a custom CBOR stream. We might want to implement a similar abstraction that can deal with custom cbor stream, though since it's just two endpoints, maybe we should just move them to the same as the others, if possible.
- revlog_data as it doesn't use the Wire format. This seems like it's just legacy, and should probably be moved to use the wire format, though the transition is tricky.
- Some endpoints have no input, and this assumes input is a wire cbor struct.
- Some endpoints don't output a stream, but just a cbor object, that will need a different EdenApiHandler trait but it's doable.
The rest seem to be possible to migrate with current abstractions.
Reviewed By: StanislavGlebik
Differential Revision: D30453597
fbshipit-source-id: 0d1068021e83f3e1143f3bc9ee68d20e4c34cb50
Summary:
While I was creating EdenAPI endpoints, I noticed that there was quite a bit of duplicated code in it, for example, the logic to read things from inside a request and write a response.
It also needed several "manual" steps to create a new one, and though types made it hard to make a mistake, it was still possible, as nothing guaranteed you were using the same types everywhere. This diff aims to fix all of that.
Creating an endpoint, before:
1. Create handler function, that receives `State` and returns `TryIntoResponse`, by copying from other functions the initial/final handling, which includes adding some handler info, parsing path and query string extraction, creating a repo context and parsing the wire request.
2. Write a PathExtractor struct by copying the other structs, it must take a repo parameter.
3. Create a new variant in EdenApiMethod and update related methods.
2. From the handler function, convert stuff and call your inner logic function, that actually is different on all endpoints.
3. Write a wrapper for your wrapper, with can be done using `define_handler!`
4. Call `route.post(URL)`, don't forget to use the same Path/QueryStringExtractors.
Creating an endpoint, now:
1. Write a struct that implements `EdenApiHandler`, in which you specify the HTTP method, endpoint, and inner logic.
3. Create a new variant in EdenApiMethod and update related methods.
3. Call `Handlers::setup::<YourHandler>(route)`
This makes it easier to create a new endpoint and harder to make mistakes while doing so. It could be even easier with some procedural macro dark magic, but this is already pretty good and simple.
**On this diff, I only refactor one of the endpoints, to make it easier to review the new "refactor logic", on the next diffs I'll move more endpoints**
Reviewed By: StanislavGlebik
Differential Revision: D30451710
fbshipit-source-id: 9de4de483d4a755bfceafb6d52ec79bbf3f3b5e7
Summary:
Currently we have a problem in tunables - if a tunables was added, and then
removed, then value of tunable is not renewed until the binary is restarted.
This is not great - it makes reverts harder (e.g. if a diff added "tunable =>
true", then reverting the diff won't make any difference until the server is
restarted. So in order to revert the value of the tunable we'd need to write a
diff that does "tunable => false" or restart the binaries, neither of which is
great).
This is counter-intutivite, and this diff fixes it. From what I can tell, this
problem affected only strings/bools/ints - per repo variable weren't affected.
Reviewed By: Croohand
Differential Revision: D30544003
fbshipit-source-id: 353970a259a7faa0682866aae9a5699b8a783b22
Summary: It's convenient to be able to derive multiple derived data types at once
Reviewed By: mitrandir77
Differential Revision: D30545263
fbshipit-source-id: 79ba541981af1340a59145f44a2f7d5a54cc28e1
Summary:
fburl.com/botsdiffs is a random diff
fburl.com/botdiffs is the wiki
Differential Revision: D30547647
fbshipit-source-id: 337d6457cb6403f11fbbc9654f3d34f50d69b0e5
Summary:
Sparse profile change computation is quite slow, since globs mean it
has to iterate over a large portion of the tree even if the sparse profile
change is small. Let's make the BFS async and parallel.
Reviewed By: andll
Differential Revision: D30554961
fbshipit-source-id: bce461ad0b21d1dab1013cf3f501b5744b295c30
Summary:
Avoiding the GIL can speed up tree diff/iteration a lot. Let's extract
the pure Rust matcher when possible. We also convert a Python differencematcher
into a rust DifferenceMatcher when possible, since that is the core use case for
sparse profile change computation.
Reviewed By: andll
Differential Revision: D30553727
fbshipit-source-id: 41d2f13130da18a55f64f9f0f047825bd24144c6
Summary:
In a future diff we'll make tree BFS iteration async and parallel. To
do so we need the matchers to be Send/Sync. Let's update the types for now.
Reviewed By: andll
Differential Revision: D30553731
fbshipit-source-id: 82a441a0caa86b65ca7eaea7283bbbbbb79d5c88
Summary:
In a future diff we'll want to extract matchers into their pure Rust
forms when possible. Let's create a helper function for matcher extraction. In a
later diff we'll make this function smart.
Reviewed By: andll
Differential Revision: D30553730
fbshipit-source-id: 0612a3be54f0286308fc4c43d9a2e5e9aa431a16
Summary:
In a later diff we'll be making sparse profile change computation use a
pure Rust matcher. To do so, we need to be able to represent a difference
matcher in Rust.
Reviewed By: andll
Differential Revision: D30553729
fbshipit-source-id: df2194bbacaa7924bafbfc33e00f25e524df4613
Summary:
The diff algorithm kept track of left vs right stores, and applied the
left vs right tree lookups to the appropriate store. In reality we only ever
have one store, so we're just wasting cycles doing two sequential lookups. Let's
just get rid of the distinction.
Reviewed By: andll
Differential Revision: D30553726
fbshipit-source-id: 72de95d298dad71dc04728d5140b4d489073a33c
Summary:
When testing hg checkout with an empty cache, I noticed the tree
diff'ing phase would alternate between downloading data and doing cpu work. This
implied it was all synchronous. This diff speeds it up by doing the downloading
on a background thread, while the main thread does the actual diff.
Ideally we'd make the actual diff itself parallel as well, but the tree
structure is a tree of semi-mutable references, which would require a larger
refactoring.
Reviewed By: andll
Differential Revision: D30553728
fbshipit-source-id: 4ea953909827cf1ec4fd67ac297f63bfadd67483
Summary:
This change has the unintended effect of causing any Thrift calls to
potentially issue a recursive EdenFS call due to symlink resolution requiring
running `readlink` on the root of the repo itself.
Fixing this isn't really possible, thus let's revert the change altogether, we
can force clients to issue a realpath before issuing EdenFS Thrift calls.
Reviewed By: kmancini
Differential Revision: D30550796
fbshipit-source-id: 9494c8e08c8af2392eeb344879f156cb56f93ea6
Summary:
The documentation allows for not having to test in enqueue if the queue is
still running: if called in the destructor of the owner, no enqueue can
logically happen, and thus we do not need to protect against it.
Reviewed By: chadaustin
Differential Revision: D30520227
fbshipit-source-id: 9d6280ccd7fe875cd06b0746151a2897d1f98d61
Summary:
When trying to push thousands of requestst to the queue, the dequeue side only
manage to pull batches of ~10 requests at most. Let's measure the cost of
enqueue to optimize it.
Reviewed By: chadaustin
Differential Revision: D30503110
fbshipit-source-id: d06ae6741b13b831fa3711fb2dd0e38c3e54193c
Summary:
If hg sync job is pushing a commit, then we have no choice but to accept it,
even if it's too big. So let's not fail if too many commits are pushed.
Reviewed By: ahornby
Differential Revision: D30535025
fbshipit-source-id: eb607a8fbd691d6591ad990e0920411b1ad2f09c
