Summary:
Seem summary on the previous diff.
This functionality should be a super set of the previous logic.
Reviewed By: mitrandir77
Differential Revision: D45395320
fbshipit-source-id: 2ebf08c6b8c3480ea30cd2bf4635fc116ecd65b6
Summary:
This diff includes the following key changes:
- Added `generate_changeset_for_tag` method in the `GitUploader` contract for creating changesets for annotated tags pointing to commits
- Handled `BookmarkCategory` of tags and branches when creating the Mononoke bookmark for git branches or tags
- Introduced the `TagMetadata` type that captures relevant data from a git tag
- Implemented `generate_changeset_for_tag` for `DirectUploader` in `gitimport`
- Modified integration test to validate annotated tag handling during `gitimport`
Differential Revision: D45534203
fbshipit-source-id: 15d00b16831fab601226a85f86bd2c2b65968ae3
Summary:
Ensures that the doctor check for Windows version, done by running `cmd.exe /c ver`,
can handle non-utf-8 output and report it as a problem instead of terminating
with an unhandled exception.
Reviewed By: xavierd
Differential Revision: D45620254
fbshipit-source-id: c97642b3e48a76a2ff46ebc74d5257ad4c1fec18
Summary: VirtualInode included several things it never used. Remove them.
Reviewed By: genevievehelsel
Differential Revision: D45629434
fbshipit-source-id: db937ddba1886c0fa2461ec9f1e5842462275be9
Summary:
I need to use `std::optional<ObjectId>::operator==`, which uses
`ObjectId::operator==`.
I tried to make it hard to accidentally compare ObjectId, but that's
fighting a losing battle, so bring it back.
Reviewed By: genevievehelsel
Differential Revision: D45629405
fbshipit-source-id: 83c88afff0fb2954e40dd59204d0cfc76155fa70
Summary: Add `copytrace.max_rename_candidates` to limit the number of candidate files to check.
Reviewed By: quark-zju
Differential Revision: D45670741
fbshipit-source-id: 7512f91d31dfe6b58eab7b4fa777578791e3e422
Summary: Added config object to RenameFinder, so we can use config to control the number of candidate files to check; and also use config to control content similarity threshold.
Reviewed By: quark-zju
Differential Revision: D45606906
fbshipit-source-id: 67d580fdac517a9a2713378c1c53f9c334ba63ae
Summary: This include is no longer necessary, so we can remove it.
Reviewed By: genevievehelsel
Differential Revision: D45620895
fbshipit-source-id: d8d9ae00cc38ad1331f8f6fb1f1c55b9d4276095
Summary:
Skip certificate loading if the auth proxy is enabled. We don't use the certs in this case, so there is no point in loading them.
Also, when using an auth proxy (e.g. x2pagentd) the certs may not be present on disk, and we don't want to error out assuming they will be present.
Reviewed By: quark-zju
Differential Revision: D45486476
fbshipit-source-id: 3638299f72e7e5769eba48425be47aa90f96914e
Summary:
One of the main objective of the LocalStoreCachedBackingStore is the ability of
configuring its policy depending on the underlying backing store. For instance,
while some backing store want to cache everything in the LocalStore, others may
only want to cache BlobMetadata. Configuring the policy per-backingstore allows
the removal and special case inside the RocksDbLocalStore of blob caching: this
is now done entirely inside the LocalStoreCachedBackingStore.
Reviewed By: chadaustin
Differential Revision: D45550480
fbshipit-source-id: 386e384f0148e38fad8d4008b4e312f360edb676
Summary: The tunable is already flipped to true. We don't need the rollout gating anymore.
Reviewed By: Croohand
Differential Revision: D45405145
fbshipit-source-id: 4ae2314dc29eb650efc39fded5213c057fc0c562
Summary:
this will be needed for pycopytrace for converting Python object to
Rust dyn object
Reviewed By: quark-zju
Differential Revision: D45657741
fbshipit-source-id: 9ce9b720bc47a3513c561ced59bb7eb2f34f413d
Summary:
reduce buffer size for cbor stream from 1 MB to 8 KB by default
Currently, we don’t attempt to deserialise any data less than 1 MB on cbor
stream. The value is too high for most of our requests meaning the deserialization will be
performed at the end. Since the value was chosen arbitrarily in
https://www.internalfb.com/diff/D22370164, let's adjust it to 8K. This is more
reasonable default. For example, this is the frame size BufReader is using, and so
our zstd stream.
We will still move to 1MB buffers and large once we hit resize logic if we encounter an item that doesn't fit the smaller buffer. This may happen for files endpoint.
The current implememtation doesn't allow, for example, properly track number of incoming commits coming from commit graph stream since the entries
are very small and we are blocked for long on filling the buffer.
Reviewed By: quark-zju
Differential Revision: D45619933
fbshipit-source-id: b1e9d6e97aaa67aeaa718e3127baedcd615c82f7
Summary: The `git_extra_headers` field in a changeset contains the headers associated with the corresponding git commit which was translated into a Mononoke changeset. The `git_extra_headers` is a mapping of arbitrary bytes to bytes which might not necessarily be valid strings. When used in `scsc`, particularly for `scsc info`, we just want to display headers that make sense to a human (i.e. strings instead of bytes). This was already the case for the default output mode but when outputting in `json`, we use `serde` serialization which doesn't accept non-string keys when serializing `BTreeMaps`. A user [faced an error](https://fb.workplace.com/groups/2071217289839797/permalink/6054134597969443/) when trying to get the info for such a commit. This diff excludes non-string headers for the purpose of displaying output to the user.
Reviewed By: YousefSalama
Differential Revision: D45689738
fbshipit-source-id: 7be727650ad7637213ae59dc36e8a73464406f1a
Summary: This diff deletes `derived_root_unode` from our code base, then we're going to remove the blobs as the next step.
Reviewed By: RajivTS
Differential Revision: D45606240
fbshipit-source-id: 028309c66c55a33ba0f73cee9082d414d3d69b4c
Summary:
Outside of the HTTP endpoint specifically (which uses JSON), Scribe doesn't *actually* require the use of UTF-8 strings in its payloads.
To enable use cases that want to be able to offer non-UTF-8 messages into Scribe, the `ScribeClient` API has been changed to accept a more generic `&[u8]` instead of a strict `&str`.
Existing callsites that expected to only log UTF-8 strings now must enforce that at runtime or at their own API boundaries, which shouldn't be a problem in the scope of this codemod as that was (until now) enforced by the compiler.
Reviewed By: kuecks
Differential Revision: D45657053
fbshipit-source-id: bb08b02d3129e2cee895847c634fe9e809995f72
Summary:
Re-implement the xrepo lookup as a namespace to map xrepo commits to local commits, and an autopull predicate to pull the translated commit.
There are a couple subtle concerns:
1. We don't want to trigger any xrepo work for normal cases such autopulling a normal commit. To achieve that, made it possible to defer the generation of an autopull attempt. This way, the xrepo autopull can cancel itself if the revision in question was successfully pulled by a higher priority autopull predicate.
2. We don't to trigger an xrepo query in `repo["deadbeef"]` (i.e. in commitctx.__init__). This could have surprising performance implications. To achieve this, we only do the xrepo translation during the autopull step. Later, the namespace lookup will only use the cached translation result.
I also moved the behavior into an extension to separate it better.
Reviewed By: quark-zju
Differential Revision: D45537964
fbshipit-source-id: 602db5218c618132672c708edabf366615646600
Summary: This test was flaky since a long time ago we made some changes to `drawdag`, which this diff fixes
Reviewed By: zzl0
Differential Revision: D45442022
fbshipit-source-id: 5a36a70f985d81c42f822f9933e0dc43d87cb591
Summary: I use these commands often but they don't have autocompletion.
Reviewed By: zzl0
Differential Revision: D45659357
fbshipit-source-id: 6f55324116cda45e159426f463974df175f9b25e
Summary: There's nothing unix-specific about these tests, so enable them on Windows.
Reviewed By: genevievehelsel
Differential Revision: D45586168
fbshipit-source-id: 5acbcbe81c5c03b16e43fc561762dbafb47696b8
Summary:
If a user relies on an environment variable in their invocation of edenfsctl,
e.g. `--config-dir=$EDEN_DEV_STATE`, then if the user accidentally leaves the
variable unset they may inadvertently specify an empty string as the path to
their config dir.
When running `edenfsctl stop`, the effect of an empty config dir path is to
shut down your main eden daemon instead of your dev daemon, so this has proven
a bit of a footgun.
Reviewed By: xavierd
Differential Revision: D45591968
fbshipit-source-id: 6df6eb8c05d94dafcaee1d6976a52310454a42d4
Summary: Do not hard-fail when edenrc is corrupted
Reviewed By: xavierd
Differential Revision: D45550209
fbshipit-source-id: 88b5e7249a7da2ef4c6df69a4c65f08b7832d8d2
Summary:
After a takeover, the new socket was initialized with a "remote
EventBase", causing stopAccepting to be asynchronous, tripping an
assertion.
There was a latent bug here: it might have been possible for a new
connection to be accepted by the old process after takeover begun.
Reviewed By: xavierd
Differential Revision: D45584154
fbshipit-source-id: cdd8118d82100bc1e19f6f19676cc2fa412b8775
Summary: This is no longer used in favor of Rust locking.
Reviewed By: quark-zju
Differential Revision: D41998651
fbshipit-source-id: 8cadbe737ef064ac64b41bae1389b2123ef672b3
Summary: This is no longer used since we rely solely on Rust repo locks.
Reviewed By: quark-zju
Differential Revision: D41998654
fbshipit-source-id: 0f955821add1cf10f3220abf70238343fc5c9f8e
Summary: The force release behavior is broken, and may no longer be necessary. It is broken because it unlocks by deleting the legacy compatibility lock file, which is not load bearing for locking anymore. It could be updated to delete the right lock files, but I don't think that is necessary since locks on all platforms are released automatically when the locking process exits.
Reviewed By: quark-zju
Differential Revision: D41998655
fbshipit-source-id: f3833e07668d108b7a2464dd272a03c76379f32b
Summary: This was only used by the "debuglocks" command. Instead, make it get the lock information from the LockHeld exception. This is a better approach since it is "atomic" with regard to the corresponding lock contention (i.e. there is no race condition going to read the lock contents when they might have changed since the lock contention).
Reviewed By: quark-zju
Differential Revision: D41998653
fbshipit-source-id: 3c80b2321a97b052025ef1f316defce3177c60ef
Summary: Now that everything is using the rust_only locking mode, revert back to having a single "lock" class, but with the rustlock guts.
Reviewed By: quark-zju
Differential Revision: D41998650
fbshipit-source-id: f216df93e3758597c3df7e8c434bebcc8c64160d
Summary:
Hook things up so the Python repo.lock() and repo.wlock() functions delegate to the Rust repo's RepoLocker object. This way, repo locks that Python acquires are re-entrant across calls into the Rust workingcopy.
In particular, this fixes a deadlock when calling into the Rust status code since the status code needs to acquire the wlock to update the treestate.
I tweaked the out-of-order lock detection to allow the wlock to be released before the store lock if the wlock had been acquired out-of-order via a no-wait/non-blocking acquisition. The Python status code does `repo.wlock(wait=False)` in a best effort attempt to update the dirstate. The store lock may already be held, so in this case they locks are acquired out of order, and released out of order (but there is no risk of deadlock, so it is okay).
In general I've taken a conservative approach to maintain all the existing Python logic, such as acquire/release callbacks, warnings, shared wlock acquisition, etc. I considered a more aggressive change where we push more into Rust, but we can do that later.
Reviewed By: sggutier
Differential Revision: D45510005
fbshipit-source-id: 178b2213db7e6ec57c3e59c66ca3e21d27674cbf
Summary: I'm going to add another bit of data to this in the next commit.
Reviewed By: quark-zju
Differential Revision: D45579290
fbshipit-source-id: b882b06b110fd7b038166a592677acf8ac6410d7
Summary: This diff makes `dagcopytrace` choose different RenameFinders (Sapling or Git) based on storage format.
Reviewed By: quark-zju
Differential Revision: D45233571
fbshipit-source-id: 42da2bceff1e20f6679fe8960635a80486f766c4
Summary:
This diff is preparatory work to allow creation and upload of changeset corresponding to git annotated tags. In particular, this diff:
- Makes fields in `BonsaiAnnotatedTag` public so that they can be set via the caller
- Makes the `gitimport` function use a reference to `GitUploader` instead of owning the uploader, so that it can be reused at multiple call sites
- Modifies the `create_annotated_tag` method in `mononoke_api` to allow for empty tag `author` and `author_date`
- Rename the `generate_changeset` method to `generate_changeset_for_commit` so we can have a `generate_changeset_for_tag` method later on. Renamed the `decode_commit_message` method for similar reasons
Reviewed By: mitrandir77
Differential Revision: D45529031
fbshipit-source-id: b4d12cbb705592cbd0114c4a5c241e90273f03f8
Summary:
Staging an update to the latest fmt version triggered lots of build errors due to non-`const` methods on custom formatters. This fixes the `format()` methods to be `const` as they don't mutate any state anyway, as well as `parse()` methods that don't need to mutate internal state. This mitigates many future build errors.
Updates were identified and executed by using regular expression search/replacements such as:
`(constexpr auto parse\(ParseContext& [^)]*\)) \{` -> `$1 const {`
`(constexpr auto parse\(ParseContext& [^)]*\)) ->` -> `$1 const ->`
`(auto format\(.*, FormatContext& [^)]*\)) \{` -> `$1 const {`
`(auto format\(.*, FormatContext& [^)]*\)) ->` -> `$1 const ->`
Any changes to third-party code was then reverted. Some small changes detected from subsequent build errors were then applied.
Reviewed By: vitaut
Differential Revision: D45420948
fbshipit-source-id: 211bf7f3b59616eda3cd228caf4e93ff81d4fc2b
Summary: Newer versions of fmt require that you pass an iterator rather than a direct memory buffer as an output to the `format_to()` family of functions. This does that to unblock an upgrade to a newer fmt version.
Differential Revision: D45555419
fbshipit-source-id: a4e2217299ada0ab5e56942dacc9107153e4260d
Summary: Doesn't seem to be a live config setting anymore.
Reviewed By: quark-zju
Differential Revision: D45556674
fbshipit-source-id: 83b2a08410692757e68678b24b58f79343186665
Summary:
Adds an `eden doctor` check to report a problem if EdenFS is running in an
elevated process on Windows.
Reviewed By: xavierd
Differential Revision: D45169813
fbshipit-source-id: 41fd500adc1aed5641602d6dfe67cc6061d5b15d
Summary:
We don't use the Cython-based Thrift client because it never worked in
the open source build. Removing it fixes the tests when run with
nix2rpm Python binaries.
Reviewed By: xavierd
Differential Revision: D45508963
fbshipit-source-id: d6f0e162177082a78504a3a6014a8a4d32670ccc
Summary:
Expose the RepoLocker via the pyrepo bindings. Python will make use of this to allow sharing of locks between Python and Rust (i.e. allow locks to be re-entrant across Python -> Rust boundary).
I added a new RepoLockHandle.count() method to enable some sanity checks in the bindings. In particular, we make sure that when the Python code acquires a lock, it isn't already acquired. Python has its own logic for storing the lock on the repo, so it should only try to acquire once. In general Python assumes it is the top level locker, so I want to catch any craziness where Rust acquires a lock and then calls into Python and the lock is re-acquired.
Similarly, we check that when Python releases a lock, the lock is _actually_ released (i.e. there isn't some other Rust code still holding a reference to the lock handle).
Reviewed By: quark-zju
Differential Revision: D45510004
fbshipit-source-id: 8625da19b82b4cd92a052bca56e6e46affe1bc08
Summary:
Add RepoLocker try_lock_working_copy() and try_lock_store() methods that only make a single, non-blocking attempt to acquire the lock.
This will be useful to integrate Rust repo lock w/ Python land.
Reviewed By: quark-zju
Differential Revision: D45510003
fbshipit-source-id: a9b2e44f6fe52e6d942ce910a1e145b63e80c045
Summary:
In lock_working_copy(), only make sure store lock isn't held if we don't already hold the wlock. This is so we don't error out on locking orders like wlock() -> lock() -> wlock().
Similarly, when releasing locks, defer check of store lock until we actually drop the wlock. This avoids an error when unlocking an order like wlock() -> lock() -> wlock().
Reviewed By: quark-zju
Differential Revision: D45510002
fbshipit-source-id: f2c65d0bb10aac196f2bc152d8b4fff49f9c598f
Summary:
mzr did migrate one of the rpms in D45309155 but we need to migrate all to
unbreak contbuild.
Reviewed By: YousefSalama
Differential Revision: D45502892
fbshipit-source-id: b9e659f9ae79ed2166927b649c127b71b32cde47
Summary: Mark's stack D45216982 added support to prepare derived data before creating stack of commits using scs, but it wasn't complete, since mononoke_api didn't allow any other types except Fsnodes. This diffs adds skeleton_manifests, so configerator rsync tailer can use it.
Reviewed By: mitrandir77
Differential Revision: D45568306
fbshipit-source-id: 23e1eef2485cc81d6dfb2ca426999a5d39c53aea
Summary: This diff renames fetch_blame_compact to fetch_blame_v2, because we have already removed CompactBlame in our last diff
Differential Revision: D45566569
fbshipit-source-id: 466af2310c702699fb6463e0d41cbd1edef81fee
Summary: Now that `ContentMetadataV2` is rolled out completely, we can get rid of everything that references or depends on the old `ContentMetadata`. I will be relying on rust and integration tests to tell me if I broke something.`
Differential Revision: D43906528
fbshipit-source-id: 2169e48e70a4fc50592603ae8d47b14579677b24
Summary:
simplify code, keep the original ordering from ancestors_difference
we don't need any stable ordering anymore (see D45368943), so let's get rid of
any sorting here and also let's maintain the order returned from the
ancestors_difference (topo sorted from children to parents)
Reviewed By: mitrandir77
Differential Revision: D45394327
fbshipit-source-id: 586b1df057e6281aa75e2baaa9f959cddc082b2c
Summary:
I had to bump them when testing stack editing features on my own stack.
It seems we're fine with 70-ish commits and more than 2MB files. So let's bump
the limits.
Reviewed By: muirdm
Differential Revision: D45540481
fbshipit-source-id: f9ae73271604afaadc2195cd26fa718b6313594b
Summary:
MountProtocol and FsChannel::getName() are the same information, so
just log the latter in this rare EDEN_BUG situation.
Reviewed By: genevievehelsel
Differential Revision: D45159212
fbshipit-source-id: 4bf4bce7889ffa704a25d7dbebcf7e13520abe1b
Summary:
A primary advantage of existentials over sum types is that they allow
they breaking dependencies. Now that channel_ is an FsChannelPtr, we
no longer need to include PrjfsChannel.h in EdenMount.h.
Reviewed By: kmancini
Differential Revision: D45109931
fbshipit-source-id: 7774804fe4056ef5b897de294707496ff722db28
Summary:
On Windows, macOS, and Linux, EdenMount::channel_ is an
FsChannelPtr. Code that wants to branch on a specific implementation
can use getFuseChannel or getNfsChannel or getPrjfsChannel, though
those will be removed later.
Reviewed By: kmancini
Differential Revision: D45109907
fbshipit-source-id: d4b3b46bd78b9487909bf1cb7c1c754e48b6e9a9
Summary:
Our build signals for the Python CLI are currently fragile because other code
pulls main_lib into complex test cases, when it really only wants the CLI's
return code constants. This decouples main_lib from these external tests.
Reviewed By: genevievehelsel
Differential Revision: D45404900
fbshipit-source-id: 4adb18030ac0ca934c6ceab7cbcd50ebd20c707d
Summary:
Fixes `buck2 run edenfsctl` by ensuring daemon.py knows how to use the
`__EDENFSCTL_RUST` environment variable, which we already plumb through our run
target, to lookup the path to the CLI. This is needed so that `edenfsctl
start` can perform mounts at startup on macOS, for example.
Reviewed By: chadaustin
Differential Revision: D45533101
fbshipit-source-id: c9df700e506409f734ca98a4fb205f962fd8ed78
Summary:
The shell scripts and awk fu did not always work correctly, so use
Python to parse the Linux mount table.
Reviewed By: mshroyer
Differential Revision: D45238375
fbshipit-source-id: 6d8918b8ecbdfe37e8ceba2b0962e5f821794140
Summary: importstack is for automation so let's skip the messages for human.
Reviewed By: muirdm
Differential Revision: D45530213
fbshipit-source-id: b9fde9e24b5d53a77af29aa03c0c685db791d9d1
Summary: Wrong indentation. `cleanupnodes` should be called once for N commits.
Reviewed By: muirdm
Differential Revision: D45530212
fbshipit-source-id: 6cd21a6b0de3e91bc9c7a2c64333540f2b886cf3
Summary: With all the building blocks in place, this diff implements chunking support in the core sharding framework. Because the framework level contract is being modified, it affects all the services and jobs using sharding. In usual cases the changes are minimal but in x-repo and chunked jobs, there are some additional modifications. Note that these changes are not backward compatible so I will be stopping the existing chunked jobs (`AliasVerifyProd` and `AliasVerifyBackup`) before rolling out these changes. I plan to test these changes on the test sharding job.
Reviewed By: mitrandir77
Differential Revision: D45479019
fbshipit-source-id: 6e2290de356642a2dc33e35b56c9ab25c2571769
Summary:
This test have been broken because it relied on deprecated openssl binary from
platform 009.
Differential Revision: D45505489
fbshipit-source-id: b3c52c5b22fb7f77fd0812ec4a2d4d6337b71b51
Summary:
As a follow-up to a code review comment in a previous diff, move
acceptStopped into the EventBaseState to make it clear it's only
accessed on the EventBase thread.
Reviewed By: kmancini
Differential Revision: D45106980
fbshipit-source-id: 5bd201a0652fa2b539e18a7584fae48fd3abd54f
Summary:
FuseChannel already had an asynchronous initialize(). Lift that to
FsChannel and unify initialization of Prjfs and NFS.
Reviewed By: kmancini
Differential Revision: D45106889
fbshipit-source-id: 632ddfa275c732fc30efc1984bda43f61c37fd5e
Summary:
EdenMount shouldn't know about the vagaries of FsChannel teardown. In
fact, FuseChannel already has its own requirements, hidden behind a
unique_ptr deleter. Use the same mechanism for NFS, allowing us to
remove a .via() special-case.
Reviewed By: kmancini
Differential Revision: D44810223
fbshipit-source-id: db2f88bc3843b3842de60c5a993dfd9890dc3c15
Summary:
mshroyer suggested renaming run_with_fault in D45135531 as the previous name
was confusing and he expected IO to fail, not to be blocked.
Reviewed By: fanzeyi
Differential Revision: D45459918
fbshipit-source-id: c39e5d414a6c0fda4eeff8314e6a266fa8eee185
Summary:
When unmount is called, the mount is kept alive until all the pending
ProjectedFS callbacks have completed (D45135531) to avoid use-after-free.
However, the notification callback is handled entirely in the background and
would thus not prevent the mount from being unmounted, and not being freed.
In order to fix this, we need to keep the PrjfsChannelInner alive for the
duration of the notifcation handling, which as a side effect will keep the
mount alive.
Reviewed By: mshroyer
Differential Revision: D45251808
fbshipit-source-id: e78afbc388cdf75f6d8d4e774bced8fdf841db5e
Summary: Will look at seeing if the script exists in the repo to determine whether to use absolute path to run a script.
Reviewed By: muirdm
Differential Revision: D45106495
fbshipit-source-id: 88393b6563470e273218849cb71d41ab05a27df1
Summary: As we need to add some functionality to `mononoke newadmin` that is not ready to be shared with `open-source`, extend the `subcommands` macro and adapt the `mononoke newadmin`'s `main` function to allow loading subcommands that are private to `facebook`.
Reviewed By: RajivTS
Differential Revision: D45398027
fbshipit-source-id: 493b24920c5310d67417274ab96e434a2bc0fbac
Summary:
I propose that instead of logging from pushredirection code we log from
backsyncer code. In that case it won't matter how the commits made it to
large repo, they'll be logged in small repo regardless.
Differential Revision: D45401754
fbshipit-source-id: 471cf6c3a4afd3da2f6a01729b7443f506e0d092
Summary: This code will be useful for anybody doing the logging.
Differential Revision: D45401755
fbshipit-source-id: 0d1e4f933e241cce9d8078ded8963ee0f74080a3
Summary:
I lied in the previous diff. Without some extra scribe configuration the extra
logs wouldn't make it to fake scribe output in the test
Differential Revision: D45401756
fbshipit-source-id: f582dc6cb8692b0fd4ff2fec5da95f17b59366a4
Summary:
I needed to use a bit different repo object than blobrepo and it's been frustating that I can't use it here.
This is not a complete deblobrepoification yet but it's definitely step in the right direction.
Reviewed By: YousefSalama
Differential Revision: D45356780
fbshipit-source-id: f8a9698e3afbcf4f08c6b3db485466ab19fc154d
Summary:
I needed to use a bit different repo object than blobrepo and it's been
frustating that I can't use it here.
Reviewed By: YousefSalama
Differential Revision: D45356781
fbshipit-source-id: b7effca4d547975bf085e58d704247f48dae9593
Summary: Phabricator uses the `commit_file_diffs` method to get metadata diff which it then uses to render information on the diff view page. With `ContentMetadataV2`, there are opportunities to avoid loading the files in the commit in memory cause we already have the necessary metadata in hand. However, this optimization will only come in play when the file is non-textual (i.e. binary or non-UTF8). This diff adds logging changes to identify such requests from clients.
Reviewed By: YousefSalama
Differential Revision: D45395092
fbshipit-source-id: 19d2e7370adafff6e626387e1480249349da9be1
Summary: We know there are some `ContentMetdataV2` blobs that haven't been generated with the correct format and when `ContentMetadataV2` gets generated directly on user request path, it doesn't generate the necessary aliases. This diff adds the necessary support for that use-case.
Reviewed By: Croohand
Differential Revision: D44197817
fbshipit-source-id: d8dd654228618232ed1e24d6755c673322fc18db
Summary: Now that `ContentMetadataV2` is exposed via `FileStore` and all the relevant metadata methods, we no longer need to validate or scrub `ContentMetadata` blobs. Once we remove support from walker, we can then go ahead and delete the unused `ContentMetadata` blobs.
Reviewed By: mitrandir77, yancouto
Differential Revision: D43905796
fbshipit-source-id: 20c98a65192244546b7212b82eb24612abc92c4a
Summary: We were displaying a line's current line number instead of the original line number when that line was first introduced.
Reviewed By: quark-zju
Differential Revision: D45458176
fbshipit-source-id: 75c6a43f0d015bdea6585058ebec74526104dbf6
Summary:
Now if the user requests a blame including working copy changes, we still use EdenAPI as long as the file is clean.
For a real fix we can use the server data as a base and then augment it as required. That is non-trivial, so let's do that later.
Note that I noticed we are displaying the wrong line number in blame output. Will fix in following diff.
Reviewed By: quark-zju
Differential Revision: D45415546
fbshipit-source-id: 47f412c835d20107ad945dc030cb8b3eb541bf6e
Summary: Before we do a slower search for all matching paths, do an optimistic "get()" for the exact path. If it exists and is EXIST_NEXT, we can confidently return it early. I think this is the normal case in practice because case insensitive filesystems are case preserving, so things normally match up between dirstate and filesystem.
Reviewed By: quark-zju
Differential Revision: D45378856
fbshipit-source-id: 7fb5fdadd417d95a39bb559d703dc792e336bca0
Summary: I don't think we need directory results from watchman, and in particular they were triggering use of the GitIgnorematcher because we were stupidly considering them as untracked files.
Reviewed By: quark-zju
Differential Revision: D45378855
fbshipit-source-id: c3dce4212801749ac25cc4f7ccd5c86cf8d8ee76
Summary:
We were using an ExactMatcher to walk the treestate and discover tracked files that werent returned by watchman after fresh instance (and hence must have been deleted). This was slow, so replace with a simple HashSet check.
With this change, however, we lose some case-insensitive correctness (although I have discovered things didn't work that well before anyway). Improve things by ignoring file deletions if they case insentitively match other files we got from watchman. This handles two cases:
1. File "foo" is renamed to "FOO" and watchman sends us both "foo deleted" and "FOO updated" (now we ignore the "foo" deletion). This issue was pre-existing.
2. Watchman restarts and "foo" has been changed to "FOO". Watchman only reports "FOO", and we need to ignore the "foo" deletion we infer when walking the treestate. This issue was introduced by this commit's perf optimization.
Reviewed By: quark-zju
Differential Revision: D45378862
fbshipit-source-id: 3dd56ef2ff3baf6115333326d4c8386c91715ab5
Summary: Break down the "status" work into more spans so I can tell what is going on in traceprof output for slow commands.
Reviewed By: quark-zju
Differential Revision: D45378863
fbshipit-source-id: a9604d888b3c9ca37e284a023c33105404df8707
Summary:
I noticed the parallel file detector was slower when there aren't many dirty files that need content checks. Now that we use the watchman file metadata, we don't have to "stat" clean files at all, so the parallel overhead outweighs the the light bookkeeping work.
Address by combining the non-parallel detector and parallel detectors. The metadata comparison is done without parallelism since we always have the metadata in-hand. Any content comparisons are still done in parallel since that work is IO heavy.
Reviewed By: quark-zju
Differential Revision: D45378865
fbshipit-source-id: 31d0890e88183b09b7703d66f69be86f86a9b0e8
Summary:
Per Microsoft (Christian Allred), calling any Prj functions after
`PrjStopVirtualizing` will lead to use-after-free due to the internal
virtualization context being freed on `PrjStopVirtualizing`.
In EdenFS, there is a subtle cases where this may happen, in particular, if a
callback is ongoing and detached to a background thread, while
`PrjStopVirtualizing` is called as a result of `PrjfsChannel::stop`. In that case,
the callback will attempt to use a freed virtualization context and crash
EdenFS!
To prevent this, we need to make sure that `PrjStopVirtualizing` is only called
when no in-flight callbacks exits. This can be achieved by simply moving the
`PrjStopVirtualizing` into the `PrjfsChannelInner` destructor: a `shared_ptr` of
`PrjfsChannelInner` is held alive for the duration of callbacks.
After `PrjfsChannel::stop` is called, no new callbacks will be accepted due to
the `PrjfsChannelInner` field being `nullptr` thus failing IO early ensuring
that the lifetime of the mount won't be extended indefinitively.
Reviewed By: mshroyer
Differential Revision: D45135531
fbshipit-source-id: 15b25efc121ab1577c4355e07357148d122cc39f
Summary:
The issue is that revset parser raised an error for `.^::`, the user's
expectation was something like `(.^)::`, but the current parser thinks `^` is
an infix in `.^::` [2], and then it tried to parse `::` as a prefix, and
eventually got the "not a prefix: end" error. I would agree with the user and
think `.^::` should be parsed as `(.^)::`.
I asked Yuya this question, the current parser can't recover from
the error while parsing. He suggested to backport a workaround from
https://www.mercurial-scm.org/repo/hg/rev/beb667c9880f.
This diff is to introduce that change to Sapling. Another potential fix might be
resolving this ambiguity in an ealier stage by checking token type after ^, so
we don't need the `_fixops` function to rewrite the tree, but that would need
more code.
Related issue: https://github.com/facebook/sapling/issues/612
Reviewed By: muirdm
Differential Revision: D45443282
fbshipit-source-id: 8eba6fbba76bdde5051342abc1d1c739eced2661
Summary:
Loop vars have been out of favor since 3.7. in 3.10 they are not acceptable in most places.
#buildmore
Reviewed By: itamaro
Differential Revision: D45072252
fbshipit-source-id: 667af4b1d44e709cec84e9f5e4ef173063b0cb9e
Summary:
Previously we prefetched the trees for all relevant commits so that fetching the fctx via ctx[path] was fast. Fetching the trees was still slow, though, since it takes O(path depth) round trips. However, I've realized we don't actually need the file context since all the info used to display the blame output comes from the change context.
Tweak things so that annotate can provide either the change context and path, or the file context.
This gets rid of the slowest remaining part of edenapi assisted blame. This reduces "hg blame" latency of edenscm/merge.py from 8s to 1.5s.
Reviewed By: quark-zju
Differential Revision: D45405272
fbshipit-source-id: b65813f958645009baf41240cae15f0f60e2bed5
Summary: We're seeing even more users that are crashing due to win32 errors. It seems like these sorts of E-Menu issues shouldn't crash Eden entirely. A non-functional E-Menu is better than crashing Eden entirely. Therefore, let's log these as errors for now.
Reviewed By: xavierd
Differential Revision: D45365471
fbshipit-source-id: f4092a9d44c1c5e6749e52d7e656c5de339551d3
Summary:
join expensive remote bookmarks fetching into a single fetch in cloud sync
This can be done because streaming has been implemented for public pulls!
And if we join it will be faster overall to pull.
Although, this is very rare, I've seen a workspace with oculus-10, oculus-12, and portal-pie branches together.
Reviewed By: mitrandir77
Differential Revision: D45394191
fbshipit-source-id: 901370f4be8b846b1124894554f17cf8b0067d1e
Summary: Directs RangeNodeStream usages to use commit_graph().range_stream instead (conditional on a tunable (enable_new_commit_graph_range_stream) for critical ones).
Differential Revision: D45356044
fbshipit-source-id: 379f214e8a6da1dc69d8744950e0e313abe8d813
Summary: Migrating from using the Changesets facet to the CommitGraph facet would otherwise require deblobrepoifying too much stuff.
Differential Revision: D45319163
fbshipit-source-id: f93c091befc5b3419a7d1364107cc717c9d19303
Summary:
The previous diff introduced `RepoShard` as the struct containing the parsed information from the passed in string shard-id. This diff adds relevant test cases to ensure that the logic for parsing the string based ID is accurate.
This diff also modifies the original parsing logic in response to the errors discovered during unit testing.
Reviewed By: mitrandir77
Differential Revision: D45355079
fbshipit-source-id: 78f28fa80bf4a3186ab66b08b29f20d39230094d
Summary: make mutation tracing message similar to other requests
Reviewed By: mitrandir77
Differential Revision: D45392969
fbshipit-source-id: 9ba180962fbde4607b777273837d41aeefe30263
Summary: clean up left over legacy code
Reviewed By: quark-zju
Differential Revision: D45368971
fbshipit-source-id: 2cbba53b571f1cfc004ec59e44871ea3e8343b2c
Summary:
This will allow tests to easily block some methods to perform concurrent
operations to them. For instance, this will allow that unmount concurrently to
read will not crash EdenFS.
Reviewed By: mshroyer
Differential Revision: D45186931
fbshipit-source-id: 3bd1fa9191d6e18cec985159914c0ee03a22a107
Summary:
Background runs of eden rage are making tracing pastes. We should not make these
pastes in dry-run automation runs.
I should have caught this in code review, my bad.
This uses the same fix as D28541898. Perhaps the paste method should not work in automation.
Reviewed By: xavierd
Differential Revision: D45362138
fbshipit-source-id: c28a18bc739921b3dc6e5b906c2491427f501644
Summary:
EdenFS doesn't care about the min/max definition from Windows.h, thus define
NOMINMAX. This allows removal of a lone `#define NOMINMAX` in FaultInjector.h.
Reviewed By: janondrusek
Differential Revision: D45359806
fbshipit-source-id: 7f24bda46e9720fdac471c24ff05a32c504d3eff
Summary: Add GitRenameFinder, this diff only consider the files that are exactly the same as source file as a rename. Will add content similarity check later, I am considering using [similar](https://github.com/mitsuhiko/similar) crate for this, since it has a nice API: https://docs.rs/similar/latest/similar/fn.get_diff_ratio.html
Reviewed By: quark-zju
Differential Revision: D45232985
fbshipit-source-id: d462063aff74103f8332c838b137df9dabc01501
Summary:
streamimg support for public pulls
If all heads in the request are public, use the streaming API, so that we are not
limited to any keep alive timeouts on HTTP side, since we can return things more gradually.
This is to be used for hg pulls in not segmented changelog repos, pulling of
expensive branches, etc
Commit Cloud would still use the API where we calculate the whole graph before returning it but this is fine because:
* we need to derive filenodes and calculate phases in parallel
* that function is a little bit more optimised since better batching of hg <=> bonsai conversions, for example.
Reviewed By: mitrandir77
Differential Revision: D45352368
fbshipit-source-id: 7549396d249a50adc163b5de1586d8789776b0df
Summary:
eliminate lifetimes from ancestors_difference_stream method since they
create unnecessary complexity and some consumers require a static stream.
self (arc under the hood) and ctx is cheap to clone there
Reviewed By: markbt
Differential Revision: D45352367
fbshipit-source-id: 6eca62838d1e31ff046ce987c52afced84227a6c
Summary: This diff introduces the `RepoShard` type which will serve as the holder of shard-level state information. Currently, we keep the information encoded in the shard_id (e.g. `fbsource_TO_ovrsource_CHUNK_1-11_SIZE_1000`) and expose it directly to the clients who then have to parse it themselves to get the relevant pieces of information. To avoid this duplication and to centralize the parsing and error handling logic, we will use the `RepoShard` struct. I have added it in `sharding_ext` crate so the type itself can be used independently of the sharding framework (for lightweight dependencies).
Reviewed By: markbt
Differential Revision: D45277195
fbshipit-source-id: 6ae9717fdcf312f0546e7b75d530e8dec8f69d32
Summary:
The `sharding_lib_ext` crate was created to contain the add-on logic that is used by `ShardedProcessManager` but also by other components (e.g. encoding, decoding repo-name for shard-aware routing). I plan to use this crate further for adding chunking support. Before I do that, I decided to clean things up a bit. Particularly,
- Move `sharding_lib_ext` crate under `cmdlib` instead of keeping it under `mononoke` since its a library
- Renamed the crate to `sharding_ext` removing the `lib` since all crates under `cmdlib` are libraries anyway
Reviewed By: markbt
Differential Revision: D45272976
fbshipit-source-id: 90be4f1f4659a03a591a8acbb04fdcbf06234a74
Summary: Similar to direct `gitimport`, this diff adds support in `remote-gitimport` to upload raw git annotated tags when pushed to the remote git server. Note that this only uploads annotated tags since basic tags get directly converted into branches while annotated tags need a corresponding changeset to be represented in the Mononoke data model. This diff doesn't delete existing tags when deletions are pushed to the server and modifications of the raw git tag are prevented by the no-update model in Mononoke blobstores.
Reviewed By: markbt
Differential Revision: D45271703
fbshipit-source-id: f7cdbed7fb665c9d32b08e26d7e9f404175b6eb9
Summary:
The `criterion` crate is a heavyweight crate that shouldn't be part of `types`
dependency tree.
Reviewed By: muirdm
Differential Revision: D45314783
fbshipit-source-id: 6476539c8d95927a6e6fa821e312632c6fffcced
Summary:
xdiff has logic to "cleanup" (force "unmatch" lines that are unique on one
side) with the intention to speed things up. However, in the `edit_cost`
case this might actually slow things down. So let's skip the cleanup logic
accordingly.
Using the test case of `test_edit_cost_large_diff()`, the `edit_cost(100)`
now completes in ~0.6s, down from ~1.5s.
In [7]: %time bindings.xdiff.edit_cost(a,b,100)
CPU times: user 253 ms, sys: 347 ms, total: 599 ms
Wall time: 597 ms
Out[7]: 100
In [6]: %time bindings.xdiff.edit_cost(a,b,1000)
CPU times: user 282 ms, sys: 351 ms, total: 633 ms
Wall time: 632 ms
Out[6]: 1000
Reviewed By: muirdm
Differential Revision: D45305089
fbshipit-source-id: 20f49bce742f7ef02f17fc5a0d0a2008cb1eecbd
Summary: This makes it possible to use `edit_cost` in Python.
Reviewed By: muirdm
Differential Revision: D45303297
fbshipit-source-id: f948f317f907f9e808aec66f5e388abe632c768a
Summary:
`edit_cost` returns the edit cost with a maximum cap, without producing actual
diffs.
This can be useful for similarity check without calculating the full diff.
See `test_edit_cost_large_diff` for performance comparsion.
Reviewed By: muirdm
Differential Revision: D45303299
fbshipit-source-id: c33bb6c186c2a5281910530988397ec00e37769d
Summary:
Update the bindgen logic so it skips C stdlib items and impls Default.
This makes the next change smaller.
Reviewed By: muirdm
Differential Revision: D45305448
fbshipit-source-id: dc3462c3ff037bafc3ebc867350c28aeda81c3a3
Summary:
In this code:
ps1 = msmall.ptr + msmall.size - 1
In case an empty slice is passed from Rust,
- `msmall.ptr` is 0x1, since Rust uses pointer 0x1 for empty slices.
- `msmall.size - 1` is `-1` (overflow).
- ps1 will be 1 - 1 = NULL, which makes UBSAN unhappy.
Resolve it by skipping the code block for `size == 0` cases.
This issue wasn't found before, because previously `xdiff` tests do not use empty slices.
Reviewed By: muirdm
Differential Revision: D45305368
fbshipit-source-id: 2b9ca3dd8dd48e9a3c23e41c556f63af0a59fc6c
Summary:
xdiff has internal logic to optimized out lines unique on one side, since they
never match the other side. We want to track how many lines are optimized out
for bounded similarity check.
The value will be used by the next change.
Reviewed By: muirdm
Differential Revision: D45303298
fbshipit-source-id: a9c58197664800013473f59dead76852a10c0e4a
Summary: Replace the C xdiff extension with the Rust one.
Reviewed By: muirdm
Differential Revision: D45303295
fbshipit-source-id: a278a0c7a26d6d7f73cc1aaf70c1ccfcf5e10f3b
Summary: This will replace the C Python `xdiff` extension.
Reviewed By: muirdm
Differential Revision: D45303296
fbshipit-source-id: 29a16b8c75f2109fa3b5bb5f4406a34ff880c160
Summary:
Implement the `blocks` API in xdiff Rust wrapper so it can serve mdiff.py
use-cases.
Reviewed By: muirdm
Differential Revision: D45303294
fbshipit-source-id: 8eae63a52fa2e7ec825d55b140f598a06e068c0a
Summary: These no longer apply. This change should have been part of D44715340.
Reviewed By: sggutier
Differential Revision: D45339827
fbshipit-source-id: f112625dfae011c70dfb17c3f9c13501b96428e7
Summary:
We've seen at least 1 user that crashed due to:
```
E0417 09:41:18.412118 20840 WindowsNotifier.cpp:68] TrackPopupMenuEx failed: Error (0x5a6) Popup menu already active.
F0417 09:41:18.434692 20840 WindowsNotifier.cpp:454] Exception occurred in E-Menu WndProc: TrackPopupMenuEx failed: Error (0x5a6) Popup menu already active.
```
Failures like this really shouldn't cause the entirety of Eden to crash, so let's ignore it for now.
Reviewed By: chadaustin
Differential Revision: D45130458
fbshipit-source-id: f906a17ac359f2a3c48a852d0668f70014ccd63d
Summary:
Add a new commit resolution step that checks if unknown hashes are from another repo, and if so translates to corresponding hash of current repo.
This behavior is enabled by configuring the other repo name under the megarepo.transparent-lookup config.
Reviewed By: sggutier
Differential Revision: D45092192
fbshipit-source-id: 9434a21d258cdfda153eb81e2d1eb0149f8a063e
Summary: Allow specifying repo name for the http edenapi client. This makes it easier to do edenapi queries for other repos.
Reviewed By: quark-zju
Differential Revision: D45092190
fbshipit-source-id: 77d09eba0cb4f1cbf169d24bdf424cf1e1519844
Summary:
Respect the optional "from" and/or "to" repo names to convert the commit ids from and/or to different repos. Only applies for repos that are megarepo'd together.
I added a get_repo method to the EdenApiContext object to facilitate the xrepo behavior.
I thought about disallowing requests where both "from" and "to" are different from the "subject" repo in the path, but it seems harmless.
Reviewed By: quark-zju
Differential Revision: D45019173
fbshipit-source-id: 8a25f5506521765296142553963b19f47570e865
Summary: Previously any outer error returned by an EdenApiHandler got turned into a 500. With this change, we no longer allow the generic `Into<anyhow::Error>` propagation. This allows for specialized HttpError propagation that retains response code.
Reviewed By: quark-zju
Differential Revision: D45296090
fbshipit-source-id: bb21518e5f8c77f578eec6d42a2582c474b5d1e9
Summary:
Use the CommitId directly as HashMap key to get rid of some per-type bookkeeping.
This minor change allows my next diff for xrepo translation to be significantly simpler.
Reviewed By: markbt
Differential Revision: D45019175
fbshipit-source-id: eb369b9e2d51345482590bbbbd9a1758b140b309
Summary:
I want to enable EdenApiHandlers to fetch other repo objects, but that requires various other objects not currently available. I don't want to complicate the handler signature with multiple additional args, so introduce a context object that can contain the dependent objects (i.e. RequestContext and ServerContext).
I tucked the path and query extractors into the context since they are barely used by any handlers (in fact, the path extractor is not used at all).
Reviewed By: markbt
Differential Revision: D45019176
fbshipit-source-id: ec4427d2f134277e89a8c48018f29c5f56cdc60d
Summary: Implements a stream that yields all changesets that are simulatenously a descendant of one changeset and a descendant of another, in topological order.
Reviewed By: markbt
Differential Revision: D45156176
fbshipit-source-id: 6c70d40252b5e2402ec1718e6cbe4c35ec8e9c1e
Summary: Make this help text more helpful. Also record that `--bookmark-info` and `--path` are mutually exclusive.
Reviewed By: MichaelCuevas
Differential Revision: D45315761
fbshipit-source-id: 150c80e1e1369c3f0c76d6708e9453fb64e3e7e0
Summary: `boost/regex.hpp` probably includes `windows.h` at some point, this causes a `min()` / `max()` macro conflict
Reviewed By: ispeters
Differential Revision: D45318842
fbshipit-source-id: 19fc29b03ca7dbf75fb6bfaef1787d8e73c329c2
Summary:
Context: https://fb.workplace.com/groups/learningrust/permalink/3405570553032635/
The previous diff makes use of `//third-party/rust:blake3` (or `//third-party/blake3:blake3-rust`) in `revisionstore` for computing the `seeded_blake3` hash of raw file content in `LazyFile`. `revisionstore` is a dependency for `edenfs` which also depends on the C++ library `eden/fs/digest:blake3_hash`.
When building `edenfs`, the linker comes across multiple definitions of the same symbol coming from the C++ and the Rust Blake3 library. Minimum repro in D45176271. Since there is no known solution for this, this diff uses the `blake3_c_ffi` crate which is a FFI wrapper (instead of native rust implementation) over the same C blake3 code as used by `eden/fs/digest:blake3_hash`. This solves the build problem while giving the same behavior in code. Note that the OSS version of the code continues to use the open-source Blake3 crate since the `duplicate symbols` error is not observed in OSS builds.
Reviewed By: mitrandir77
Differential Revision: D45310838
fbshipit-source-id: 9273bcc64871d5a86d07e23cc0200e305be7f4fd
Summary:
The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which were modified in the previous diff to support the new hash and are used here.
NOTE: This diff was already landed (D44010967) but then had to be reverted (D45156142) since it was causing build issues for EdenFS on ARM64 MacOS. More context [here](https://fb.workplace.com/groups/learningrust/permalink/3405570553032635/)
Reviewed By: quark-zju
Differential Revision: D45226440
fbshipit-source-id: 051dee9898d2441d6a03c10809988c6b99f36ab4
Summary:
We want `py2` to be a feature that returns False instead of crashing the tests
complaining about unknown feature.
Reviewed By: sggutier
Differential Revision: D45321821
fbshipit-source-id: 80a2a0077a50b29284a8db06f16def5e16081f14
Summary:
Fix issue:
AttributeError: module 'edenscm.ext.extlib.pywatchman.compat' has no attribute 'PYTHON3'
in various tests caused by D44715340.
Reviewed By: sggutier
Differential Revision: D45321822
fbshipit-source-id: 49025b54b04b819804be58ee0265d2530b67de66
Summary:
prefetch expensive bookmarks separately
this is to avoid timeouts until we have streaming support and ublock rollout
for AOSP repo
also, this provides better ux since we notify users that slow fetch is being executed
I also noticed that the commitcloud.remotebookmarkssync config option wasn't mentioned in the help. So, add it to avoid confusion.
Reviewed By: quark-zju, zzl0
Differential Revision: D44998087
fbshipit-source-id: f24bd959f02380ff0d7ff4f30dbe7e1647403329
Summary:
The goal is to avoid any serial fetching per commit/path in the blame history.
To get there, we now prefetch:
- Relevant commits and their parents. Parent hashes are used to verify commit hashes in repo.changelog.revision().
- Trees for all relevant paths for all relevant commits. This makes use of the newly improved manifest BFS iter. Note this is still the slowest part since it has a round trip for each directory as it walks the tree. I want to make the trees endpoint to support deep fetches to avoid these round trips.
Reviewed By: quark-zju
Differential Revision: D45273889
fbshipit-source-id: 7935874da178d9f39d9aab3cc9e026503db919ab
Summary:
This extension is not used and depends on server side features Mononoke doesn't implement.
Server assisted blame is instead achieved via an EdenAPI blame endpoint that fetches precomputed blame data.
Reviewed By: sggutier
Differential Revision: D45273888
fbshipit-source-id: 4efd269ad16f52b3df98e558b1fded7135c87419
Summary: Tweak the manifest BFS iter to take a list of initial nodes. I'm going to use this to allow prefetching trees for multiple commits at once.
Reviewed By: quark-zju
Differential Revision: D45273890
fbshipit-source-id: d3487a1a2ee71f602e535b8b7f673aa5d6a91aea
Summary:
First, kill the "depth" and "path" args since neither were set by any callers.
Then, use the bfs_iter to walk the tree instead of the naive level-by-level walk. This should be much faster, but more importantly is a step towards making prefetch support fetching multiple root trees/paths at once.
Reviewed By: quark-zju
Differential Revision: D45273891
fbshipit-source-id: 8b93ac1f5a327555d80c22410c7c95ab56af9984
Summary: removed python 2 specific references along with sys.version.info and dependencies.
Reviewed By: sggutier
Differential Revision: D44715340
fbshipit-source-id: 0dae8cf1bd9cb21a8ca336d22efecf796930bccc
Summary: I will reuse this `SearchDirection` in GitRenameFinder in the following diff.
Reviewed By: sggutier
Differential Revision: D45233573
fbshipit-source-id: 4885fc5df8beda729d23843471d63aa8e870c8f4
Summary:
Code42 was known to crawl the repository without taking
into account that files are virtual. It seems that this has been corrected in the new CrashPlan 11.0.1 agent, but we will land this diff as a safety precaution to not allow them to read the repository
Reviewed By: xavierd
Differential Revision: D45282206
fbshipit-source-id: 32f77aa96745bccb36e43d4df8ff47755a7670bc
Summary:
This data may be useful for analysing the blame history of the file.
* `approx_commit_count` is the total number of commits in the blame history of the file. It is approximate as some merged in commits may be omitted.
* `distinct_range_count` is the number of distinct ranges in the blame, which may indicate the file's complexity.
Reviewed By: RajivTS
Differential Revision: D45218574
fbshipit-source-id: c0c399f525f8c047a0415cf9d6f6e3483c033700
Summary: bunch of unit tests that relied on the specific error formatting weren't caught by CI and are failing. this fixes them by changing their error formatting to use `{:#}`.
Reviewed By: markbt
Differential Revision: D45245281
fbshipit-source-id: 6b0c9c5f3a3ecb594292d2a658ecce457f007e7e
Summary: Directs DifferenceOfUnionsOfAncestorsNodeStream callsites to use commit_graph().ancestors_difference_stream instead conditional on a tunable (enable_new_commit_graph_ancestors_difference_stream) being set to true.
Reviewed By: markbt
Differential Revision: D45042733
fbshipit-source-id: ca508d9b1a2e9efd7b45d1ee834f7587a7d886c3
Summary: Implements a streamable variant of ancestors_difference to be used as a replacement for DifferenceOfUnionsOfAncestorsStream
Reviewed By: Croohand
Differential Revision: D45042732
fbshipit-source-id: aa7377512866038172899545ee9c43a2d6db39f7
Summary: Use ChangesetsCommitGraphCompat when constructing Changesets in TestRepoFactory to enable double writing to the new commit graph
Reviewed By: mitrandir77, singhsrb
Differential Revision: D45227317
fbshipit-source-id: 6ce1809a2e30d930f8edef1efda0f9bef53b2a7c
Summary: Replaced by using TestRepoFactory directly and adding the extra config to the default config.
Reviewed By: RajivTS
Differential Revision: D45227320
fbshipit-source-id: bd93d1a2fd3a10398ca1cfd4b647a32627b83179
Summary: Has the same behaviour as the contruction of warm bookmarks cache in Repo::new_test_common. To keep the same behaviour this required making TestRepoFactory into an async builder and modifying all of its callsites.
Reviewed By: RajivTS
Differential Revision: D45227318
fbshipit-source-id: 51303d48bebc0c4b915a359a7d0942bf485e8db5
Summary: Adds a sync method that awaits the completion of all ongoing updates. Mostly useful for tests to avoid adding artificial sleeps.
Reviewed By: RajivTS
Differential Revision: D45227319
fbshipit-source-id: 9f3a9c7d063ad065b913db521a27624041b2400a
Summary:
fix 'repo changed while backing up'
the current logic is still incorrect for non besteffort (regular) mode
we introduced a high level transaction in D44664729, so now it is safe to drop
the check for non besteffort (regular) mode entirely
Why is it incorrect?
This is because the check is inside the retry loop (have lost the race),
inside the loop we apply commit cloud changes from the currect sync,
so basically if we haven't synced yet at this stage we will always fail the check.
We are loosing our retry logic that we should keep for the regular mode.
After a high level transaction has been added, we can drop the check and
resuscitate our retry logic.
The only possibility for another transaction to change the repo is that the transaction belongs to the previous iteration of the loop in the current cloud sync.
Reviewed By: quark-zju
Differential Revision: D45271716
fbshipit-source-id: 78de00c512f6ac737b107053563bd74b43080d32
Summary:
✨(clone): set publicheads to remote HEAD when cloning
c8b66c6899 added support for cloning the remote git repo's HEAD (in this
case `develop`), instead of main/master by default.
However, this doesn't mark `remote/develop` and its ancestors as public
commits.
This commit will also set remote git repo's HEAD as a public commit
Closes https://github.com/facebook/sapling/issues/600
Pull Request resolved: https://github.com/facebook/sapling/issues/607
Pull Request resolved: https://github.com/facebook/sapling/pull/607
Test Plan: - Added test `test-git-clone-sets-publicheads.t`
Reviewed By: sggutier
Differential Revision: D45249747
Pulled By: zzl0
fbshipit-source-id: 8c3a36bd21a25a856d13cc3b2c78f9cbdd9a94c5
Summary:
When creating a stack, make it possible to request that a derived data type is prepared as part of creating the stack. Since the data for the stack is in local cache, this should be faster than it would otherwise be.
This is a simple initial implementation. It requires that the data is already derived for the parents of the stack, and does not support parallel derivation of the different types, nor types that include interdependencies. In fact, we only support fsnodes and skeleton manifests right now.
Differential Revision: D45216982
fbshipit-source-id: c6ca7b265b1e6219a1193c51d0a984794aa34965
Summary:
Add support for skeleton manifests as a type of derived data that can be prepared.
The current enum for derived data types does not follow best practices and uses zero. Fix this now by adding a new variant for `FSNODE`. Once server support is available, we will remove `FSNODE_NEW` and switch to that value for `FSNODE`.
Differential Revision: D45216983
fbshipit-source-id: b0714ef6f9121139cc8d4602a5a8d1beaac00a3f
Summary: Move the preparation of derived data into a `mononoke_api` method in preparation for re-use.
Differential Revision: D45216984
fbshipit-source-id: ee99379685a202018b4c32b4b56e2930755d723a
Summary:
After my previous refactoring, the memory safety bug in Nfsd3 no
longer manifests, and we can remove the unnecessary defer call.
Reviewed By: kmancini
Differential Revision: D44988749
fbshipit-source-id: 2ebd5fbf96d05ce6850fc248a642409f827ffeec
Summary:
All of the comments in RpcServer are helpful. I found a couple
opportunities to improve their clarity.
It also helps to decouple RpcStopReason from the connection status.
Reviewed By: kmancini
Differential Revision: D44988608
fbshipit-source-id: 967f35ba09533a9c8c953ef33cd595b51e7c3111
Summary:
I'm carefully reading the takeover code and noticed a few things to
improve.
Reviewed By: kmancini
Differential Revision: D44946740
fbshipit-source-id: 558441a47425b15c19ca2a6816ae2e8bf3d68019
Summary: When tests are run in parallel, it can take more than a couple of seconds for the server update to become visible. Increase the wait to up to to 10 seconds.
Reviewed By: RajivTS
Differential Revision: D45229687
fbshipit-source-id: 24bc9607c285a4098c35a4105b8deac7916a7591
Summary:
add `debugcloudstatus` command
this command is to be used by automation, for example, on demand to get a name
of commit cloud workspace
currently they have to parse the name from the text output of the `hg cloud status` but providing json output is less error prune.
Reviewed By: quark-zju
Differential Revision: D45227570
fbshipit-source-id: 9e1a67edcefa7a2f7f11c7a80e99d061e5ca4bad
Summary:
Fix a typo. The word capabilities hurts my eyes when I look at it too
long, but it's the correct spelling, so sed it into place.
Reviewed By: kmancini
Differential Revision: D44945525
fbshipit-source-id: 05e82c058c3015b0590b07d6458840b4c56730b6
Summary:
Out of curiosity, I'd removed a .via(getEventBase()) from
TakeoverServer, and it manifested as hard-to-understand segmentation
faults in the integration tests. The issue is that FutureUnixSocket
has EventBase affinity and must only be accessed from it.
To make this safer and more obvious, put FutureUnixSocket and the
other takeover connection state in EventBaseState.
Reviewed By: kmancini
Differential Revision: D44945469
fbshipit-source-id: add3af84a35d451385a73afb622f9b63c7cdb217
Summary:
I want to use EventBaseState in TakeoverServer, so move it from
RpcServer to eden/fs/utils.
Reviewed By: kmancini
Differential Revision: D44945418
fbshipit-source-id: b9738162c84ca20da800d3662be6271f4b492acc
Summary:
Some dead code, a judicious use of makeFutureWith, and replacement of
thenTryInline+makeAsyncTask with via+thenTry makes the implementation
of TakeoverServer easier to follow.
Reviewed By: kmancini
Differential Revision: D44945165
fbshipit-source-id: 67c7b68b892c5112620bdc1ab2a0feb7df4d1864
Summary:
For small handle types, passing by value is the correct choice,
because it indicates unconditional ownership transfer. (It also
generates slightly smaller code and is less typing.)
Reviewed By: xavierd
Differential Revision: D44927835
fbshipit-source-id: 5fefee7889e8216e46548f4801e83f51cde8b8c4
Summary:
Simplify ServerState a bit by returning a const reference to the
member shared_ptr.
Reviewed By: xavierd
Differential Revision: D44924418
fbshipit-source-id: cd5f3a17d92b677ec7be43d9a07ed38dca1a3c24
Summary:
No need to have an intermediate object to forward callbacks from
ReadCallback to RpcConnectionHandler. This simplifies both memory
management and data flow.
Reviewed By: kmancini
Differential Revision: D44860344
fbshipit-source-id: af02f7d7f331d6ff3a8c4f49abb5ba1b95db5f85
Summary:
Adding the below cli outputs to eden rage
`eden trace inode --retroactive` (this one is empty though because we don't publish events right now)
`eden trace thrift --retroactive`
`eden trace hg --retroactive`
Reviewed By: kmancini
Differential Revision: D45165616
fbshipit-source-id: f4a89377fdc489e10bd78dc4175bdf89e2992cbf
Summary: Backing out the revert of D44173515 to unblock RE with their development. I will revert my own diff instead.
Reviewed By: genevievehelsel
Differential Revision: D45190063
fbshipit-source-id: 2b14fdc2fa118719aed0f3215393d172a162639f
Summary:
Based on statiscal data, most renames have either same basename or
same directory, so we will use this information when checking potential
renamed-to files.
Reviewed By: quark-zju
Differential Revision: D45184854
fbshipit-source-id: 7b4ed45d27781725c372beb33d24a0960d442816
Summary:
add file_path_similarity function, we will use this to reduce the
number of files to compare for Git use case; and also use it for sorting
the potential files for Sapling use case.
Reviewed By: quark-zju
Differential Revision: D45157537
fbshipit-source-id: 476c9fa07f2409a2cf0d1a6a72c57069a42ad347
Summary:
UDS don't have port numbers, so trying to register their port with the
portmapper is going to fail. This can cause eden to fail to start when we
enable uds. The current portmapper version we use can't support
this, so let's just skip attempting to register uds servers with the
portmapper.
Reviewed By: xavierd
Differential Revision: D45196379
fbshipit-source-id: 584e6e04129d1ddbcc756ed66c55a03c3955705a
Summary:
The case that I met was like the following DAG, and I'd like to skip those `Landed` commits when running `hg ab` since they already landed.
```
╷ o zzz Apr 21 at 18:18 zhaolong
╷ │ commit zzz
╷ │
╷ o yyy Apr 21 at 17:18 zhaolong
╷ │ commit yyy
╷ │
╷ x ccc [Landed as ddd] Apr 21 at 9:00 zhaolong
╷ ╷ commit bbb
╷ ╷
╷ x aaa [Landed as bbb] Apr 12 at 08:00 zhaolong
╭─╯ commit aaa
```
Reviewed By: sggutier
Differential Revision: D45184140
fbshipit-source-id: a539468843032d41ab4d61e5c461b9a1de53cb40
Summary: Like in the previous diffs, this diff uploads the raw git tags during repo import through `gitimport`.
Differential Revision: D45118571
fbshipit-source-id: f5a6bb9b5805235e9f66f97ab3a58bd165ef475c
Summary: The previous diff uploaded raw git commit data while the commit gets imported in Mononoke through gitimport or remote-gitimport. This diff ensures that the git `tree` associated with the commit also gets uploaded in Mononoke.
Differential Revision: D45085959
fbshipit-source-id: cf4f4a5ebd0d077972f75fe9c6a8d1dfc5067180
Summary:
With the `upload_object` method in place, this diff utilizes the new uploader method to upload the raw git commit in Mononoke store before creating its corresponding Changeset at Mononoke end.
This diff has the following changes:
- Modified the `get_object` method in `GitReader` to return the raw git content (i.e. header + body bytes of git object) along with the parsed `GitObject`
- Added the `read_raw_commit` method in the `gitimport_objects` library to read the raw commit bytes
- Modified the `ExtractedCommit` type to contain the raw bytes of the commit as well
- Modified all modes of `gitimport` execution to upload the raw git commit before creating the corresponding changeset at Mononoke end
- Made changes in integration test to validate the commit upload behavior
Differential Revision: D45082135
fbshipit-source-id: f339b2851646996be7f43f02458ea5afabb9ca48
Summary:
This diff includes the following changes:
- Introduces the `upload_object` method in the `GitUploader` trait for uploading raw git-objects (e.g. commit, tree, tag) in the mercurial mirror of the repo
- Implements the `upload_object` method in `Direct` and `Remote` implementations of the `GitUploader` trait allowing the objects to uploaded from both `gitimport` and `remote-gitimport`
This diff just introduces this method, follow-up diffs will use it for uploading different commit objects.
Differential Revision: D45046811
fbshipit-source-id: 9b4e6a295ec22e06294e410735e67877e6d0ef10
Summary:
The existing `mononoke_api` git methods were defined under `RepoContext` which relies on a specific type of `Repo` facet container as defined in `mononoke_api` crate. These methods can easily be used through SCS which serves as an external entry point into the `mononoke_api` crate. `remote-gitimport` uses SCS and thus can leverage the necessary endpoints.
However, `gitimport` (or `direct-gitimport`) directly references the logic behind these methods. To ensure that we don't repeat ourselves, there are two alternatives.
- Make `gitimport` use the same `Repo` facet container as the one in `mononoke_api` crate
- Decouple the git methods from `RepoContext` and define them with explicit `facet` dependencies
I chose the later since using the same facet container would have made `gitimport` use a much bulkier `Repo` definition than is necessary. I also made some changes in the `create_annotated_tag` method which are as follows:
- Removed the `start_write()` statement which was essentially an authorization check to validate if writing to the repo is allowed. This check makes sense when validating an external caller (e.g. SCS) but direct references of the library should be allowed
- Removed `CreateChangeset` permission check since we are not actually creating a changeset that corresponds to a commit. Additionally, any auth checks need to be performed at the call-site (e.g. SCS) and not in the API
- Removed the `allowed_no_parents` check since any changeset created for representing a `Tag` object will not have a parent. Infact the concept of parent is invalid for a tag.
- Replaced the returned `MononokeError` into `GitError` to maintain consistency with other methods
- Replaced the instance `save_changeset` method with the struct-level `save_changeset` method. The new method doesn't log to the scribe category thus preventing users tailing it from receiving an invalid commit creation notification. Additionally, the new method doesn't consider any bubble changes but they are anyway irrelevant in the context of these git methods.
Differential Revision: D45046639
fbshipit-source-id: 7ff02cde3fd356eb7a18f5ca12e6c478e23ee315
Summary:
Previously, the nfs.allow-apple-double was tentatively rolled out but had to be
rolled back as XCode disliked when an AppleDouble file was present on disk but
a new one couldn't be created. To avoid this issue, let's prevent AppleDouble
from being loaded from the Overlay and re-write the Overlay for directories
that contains one at the same time.
As a slight behavior change, the nfs.allow-apple-double is no longer read on
the fly but only read at startup time to avoid cases where a TreeInode is
already loaded with an AppleDouble and the config is flipped which would cause
XCode to fail to save files.
Reviewed By: chadaustin
Differential Revision: D44854105
fbshipit-source-id: 2c263411719d19eb1d457e26661620e8d6c905ce
Summary: In preparation for deleting blame v1, make blame v2 the default.
Differential Revision: D45044338
fbshipit-source-id: c9c71a148b42f4a3301d6c4fd0de5ad7ed321a3e
Summary:
Allow the walker to walk the blame v2 graph.
Since the walker is the main user of blame in the integration tests, we can make v2 the default for the integration tests, too.
Differential Revision: D45044294
fbshipit-source-id: 52c991dd81be78053ffc0a6953caf8a068ef59e3
Summary: This is a benchmark, so put it with the other benchmarks.
Differential Revision: D45114045
fbshipit-source-id: c524c76671d4183f6baa71ca9cd4c3cdfc402dac
Summary:
We don't need folly::Synchronized for the RpcServer state
machine. It's only touched from the EventBase, so make it
EventBaseState.
Reviewed By: xavierd
Differential Revision: D44860240
fbshipit-source-id: bcbc3d02e7cc5f2dc377718065c7f95eed30202d
Summary:
There's nothing TCP-specific about RpcTcpHandler, so give it a more
accurate name.
Reviewed By: xavierd
Differential Revision: D44860198
fbshipit-source-id: 55b9dc95b2caa9d87179a17435313b91bacfee49
Summary:
RpcServer can handle the accept callbacks directly. There's no need
for an intermediate object.
Reviewed By: xavierd
Differential Revision: D44860163
fbshipit-source-id: fd2d96735810a52cd287c357534c8f5bae9eee90
Summary:
I plan to use StateWrapper elsewhere so rename it to EventBaseState
and move it to the top level.
Reviewed By: xavierd
Differential Revision: D44860040
fbshipit-source-id: fb03b3b726df50099bcce144e343be0caeeaf1be
Summary:
There is no need for an intermediate object: RpcTcpHandler can handle
the write callback methods directly.
Reviewed By: xavierd
Differential Revision: D44860027
fbshipit-source-id: fe1f44b83a371291a21a3b1b26422bef5cbbf2a0
Summary:
To simplify RpcServer further, assert that removal of the accept
callback is a synchronous operation. Previously, it wasn't, because
specifying any EventBase in addAcceptCallback created a RemoteAcceptor
which scheduled operations on the other EventBase.
Reviewed By: xavierd
Differential Revision: D44853408
fbshipit-source-id: 9369cdcb0bd8450d42e195c15738362d4a53d2e7
Summary:
I'm trying to track down a subtle lifetime rule violation in
EdenMount. While reading RpcServer, I noticed a few possible
simplifications.
Reviewed By: xavierd
Differential Revision: D44849773
fbshipit-source-id: 4c27c47a7e2c211dcd040acce955e9b2f617b55b
Summary:
This removes an ifdef _WIN32 in mount shutdown by standardizing the
takeover logic on all three platforms.
Projected FS doesn't support takeover, but removing ifdefs allows
simplifying the control flow and unblocking NFS or other FsChannel
implementations on Windows.
Reviewed By: kmancini
Differential Revision: D44737273
fbshipit-source-id: fc6b718e01698ef3700b5c198750c0e3b1f62063
Summary:
By moving NFS's StopData behind FsStopData, the std::variant and
runtime checks during takeover can be unified into one path. The next
diff will do the same for Windows.
Reviewed By: kmancini
Differential Revision: D44737160
fbshipit-source-id: 19ca623f90b367d25385f177cafa44d7bf7ebb62
Summary:
Adds a buck target to drop us into a Python repl, to ease interactive
development of Python commands in the CLI.
Reviewed By: genevievehelsel
Differential Revision: D45159857
fbshipit-source-id: 31ff61ea29ab080cc1131905c6cf363368f3c617
Summary:
Original commit changeset: a755f0750ed3
Original Phabricator Diff: D44010967
need to revert to fix our release
https://www.internalfb.com/intern/sandcastle/job/36028797946731051/insights
Reviewed By: xavierd
Differential Revision: D45156142
fbshipit-source-id: 6e5362cf609f69cdbfeb7da30981227f93d42b65
Summary: The original `find_renames` finds all the renames of the commit and checks if it contains the path that we are interested in. This diff optimizes that logic to make it return early when found the target rename. In order to do that we split `find_renames` into two functions `find_rename_forward` and `find_rename_backward`, since they will have different optimization logics.
Reviewed By: quark-zju
Differential Revision: D45149506
fbshipit-source-id: b77b3b9ef4700581e6d044644d40bf405d66d263
Summary: fix a typo in the doc of read_file_contents function
Reviewed By: quark-zju
Differential Revision: D45126147
fbshipit-source-id: 6f541acb41c5de71f56c5573a5f56b536b669959
Summary:
Practically the graph log and the find_renames can use different impls
independently and form different combinations. This will be used for suppoting
Git use case in following diffs.
Reviewed By: quark-zju
Differential Revision: D45125851
fbshipit-source-id: 785800a0e4b1c74c48b97cf47132b45b268dc0bb
Summary: This tool was used for the original import of globalrevs. We are now the source of truth for globalrevs, so it is no longer needed.
Differential Revision: D45112829
fbshipit-source-id: 8ff106c66de74a79e6cd963ca48bd1875d6ee44c
Summary: This tool was used to rechunk the filestore. The filestore is always chunked now, so this tool is no longer needed.
Differential Revision: D45112830
fbshipit-source-id: cd03dd06f7dc55a6b47b8fc338accf9d300977d2
Summary: This tool was used for the initial population of the XDB blobstore. It has been superseded by the walker, and can now be deleted.
Differential Revision: D45112828
fbshipit-source-id: 134e2c2b761fd27a9b096df4d64404bc14c89c43
Summary: This tool lints toml-format configs which we don't use in production any more, and as such the tool is unused.
Differential Revision: D45112827
fbshipit-source-id: a4f7783144ab782adba5f0a0fb346e8cb542ad97
Summary: This tool was used early on to compute commit stats, but is no longer needed as it has been superseded by other data.
Differential Revision: D45112824
fbshipit-source-id: e88282647e8e3ed85a95ff2a9b15d9212851b8ff
Summary: This tool was introduced to fix up the problem with linknodes pointing at commit cloud commits, however that has long since been resolved, so the tool is no longer needed.
Differential Revision: D45112822
fbshipit-source-id: 350b770e9a3499a8bad8aca29badc3e20a02f168
