Summary:
Add EdenAPI fallover scuba logging to `HgBackingStore`.
The reason that this is being added to `HgBackingStore` instead of `HgQueuedBackingStore` is that we don't have good access to the actual point where trees and blobs are imported from HgImporter in there. As a result we have to get StructuredLogger in there and send events directly from there.
Reviewed By: xavierd
Differential Revision: D34217906
fbshipit-source-id: 795b61ef6a2ea52e8b67cd08633ec017e5bdbb06
Summary:
On Windows, the default asyncio event loop would throw an exception on tear down.
This diff fixes that by opting to the older event loop. The older event loop doesn't support subprocess but it doesn't look like we use it anyway.
https://stackoverflow.com/a/62413786/694963
Reviewed By: xavierd
Differential Revision: D34486221
fbshipit-source-id: 765056f1e82c9a53d89482527725154445e59494
Summary: Version 2 sparse profiles get unioned at the top level (e.g. if you do "hg sparse enable A" and "hg sparse enable B" and they are v2 profiles, you get "matches(A) OR matches(B)"). However, debugsparsematch was always applying the union logic to the given profile, so for example "hg debugsparsematch -s A ..." would union profiles included in A, which does not match the actual behavior when you enable A.
Reviewed By: quark-zju
Differential Revision: D34463674
fbshipit-source-id: b5e5dfcdf2bd36e0e00c6a1b814fafbc6a6d59cc
Summary:
My recent change to updating from the null commit changed the output
here on case insensitive filesystems. Let's update the test.
Reviewed By: singhsrb
Differential Revision: D34524860
fbshipit-source-id: cad48971d17da06abfaa5c12922f66b30ce84faf
Summary: This was undocumented, but you needed to understand it to use `caching_ext`. Document it
Reviewed By: yancouto
Differential Revision: D34519857
fbshipit-source-id: 56bf089ac0997d6310290985eaa470994368d9f9
Summary:
All clients should be using files2 now, so let's delete the files1 code
path.
Reviewed By: quark-zju
Differential Revision: D34346264
fbshipit-source-id: 86f32237f84ad5521d2f660f10e1000e588187e5
Summary: A large number of things got deprecated, let's follow the recommendations.
Reviewed By: zertosh
Differential Revision: D34467979
fbshipit-source-id: fba2a95c1e4a0de73100cf7f97541ac604b1cdb1
Summary:
On Windows, /proc doesn't exist, let's use sysinfo to collect all processes and
test if they are running.
Reviewed By: zertosh
Differential Revision: D34467978
fbshipit-source-id: d90e8bca7770d2226891bedb02313cd76092d4f4
Summary:
We can directly use the thrift fetch context instead of building a null
context.
Reviewed By: chadaustin
Differential Revision: D34487575
fbshipit-source-id: 15c2fe9167b392d436638e882f4f8aaabebe4662
Summary:
Users frequently get into states where a clone fails and leaves a bunch
of pending changes. We have some logic to detect this and resume the checkout,
but they can still hit issues due to these files and end up having to hg purge
and start again. I just hit this when my machine rebooted and many of the files
that had been written ended up as empty, and therefore considered a conflict.
Let's just make clone checkouts be force checkouts. This not only prevents the
user from getting blocked, but also skips a ton of "check unknown" and "check
conflict" logic, which also bypasses the need to run status and wait for
watchman.
Differential Revision: D34323008
fbshipit-source-id: 5d49a8c645282d17694dbb726c70e8a8d7c66a58
Summary:
The nativecheckout resume feature was failing when a symlink had been
written because the "is it up-to-date" check compared the expected link path
with the contents of the file at the link target.
Let's change vfs::read() to return the symlink link instead of the content, as
that is what Mercurial wants. I can't think of any case where Mercurial ever
wants to read through to the symlink target.
This does add the overhead of a stat call to each read.
Differential Revision: D34314987
fbshipit-source-id: 676647a16f65fa516d6ef97e21d137a1ae137b3f
Summary:
When resuming a checkout we need to go over all the actions and check
if they match the contents on disk. In the old code we did this twice, once when
checking unknown files and once when doing the actual checkout. When the number
of files to check was large, like when resuming a large clone, this was
extremely slow (i.e. many minutes).
Let's only do it once.
Differential Revision: D34314688
fbshipit-source-id: 848dc4e76cb72f173741a9061bd8c71d7120a4bc
Summary:
Previously the "Querying watchman" progress bar covered a bunch of
stuff. I noticed it was hanging a lot when resuming clones. Let's break it into
smaller progress bars.
Differential Revision: D34290594
fbshipit-source-id: 69995f2637c4062062f352e73fdecf30599f9f11
Summary:
LFS and edenapi have different retry logic. Right now LFS doesn't retry for TLS errors or for a number of Http errors. Let's update LFS to match edenapi's more generous retry philosophy.
This should address TLS RecvError's I was seeing during large clones.
Differential Revision: D34290595
fbshipit-source-id: bcd071eee9cd2496e03380d686fcda4947f02a19
Summary:
When resuming a clone, watchman may not be enabled since there is no
.watchmanconfig. Therefore status calls may use the default Python file system
walking code. This currently has no progress bar, so it appears like a hang.
Let's add a progress bar.
Reviewed By: quark-zju
Differential Revision: D34278706
fbshipit-source-id: 425a1c1dadd9ffa68a868766e51c5877b9345562
Summary:
The old native checkout check_unknown logic would iterate over every
new file action and check if the file exists. This can be expensive because 1)
it stats every path, even though most will not exist, and 2) the stat goes
through the path auditor, which stats each parent directory as well.
For initial clones where there are no files on disk already, this isn't too bad,
since the top level directories aren't present so the path auditor aborts early.
When resuming a partially completed checkout however, this can be quite slow
(multiple minutes) since some directories already exist and therefore the
path auditing becomes much more expensive.
We have to run status anyway for the checkout, so let's go ahead and run it
earlier and use its results to only check the actual unknown files. This makes
the check O(unknown files) instead of O(total files to be written).
Additionally, the list of unknown files should be reduced since a resumed
checkout will remove already-written files from the list of files to be checked.
So really this is O(unknown files that differ from what's expected).
Differential Revision: D34278256
fbshipit-source-id: 39ad20387ad2622695864608c7033aefa1eb0df1
Summary:
All errors coming from scmstore fetches should be associated with the
input keys. Let's get rid of the use of other_error in the edenapi fetch path.
Differential Revision: D34257932
fbshipit-source-id: 176ee1fd767df76c8ce6140ddf849a64b3cb3621
Summary:
LFS fetch errors are reported via sha256, and in theory we should have
a Key for every sha256, so we can log keyed rrors all the time. Let's annotate
the non-keyed errors as invalid.
Eventually we'll come back and remove other errors entirely, once we can confirm
this is never hit.
Differential Revision: D34256994
fbshipit-source-id: b00db226d8ef1f222a345f0388c3cb2a5773e008
Summary:
Every error from an scmstore fetch should be associated with an input
key. As part of doing that, let's change top level lfs failures (i.e. '?' calls
inside the lfs fetch logic) to be associated with all keys being fetched.
Differential Revision: D34256261
fbshipit-source-id: b0df6ee4e653e264a4ee1954828a841bccdbb2ee
Summary:
We are seeing fetch failures with empty error lists ("[]"). This should
never happen. Every error should be attributable to an edenapi or a lfs server
fetch failure. There is a clear code path in scmstore that adds an empty error
list if we don't have an error. Let's make this explicitly call out that it's an
unknown error, so we can track these errors down in the future.
Reviewed By: quark-zju
Differential Revision: D34254835
fbshipit-source-id: baa46d641d9b951b1563ac643517dcef15efa4bc
Summary:
This diff fixes some incorrect logic in the physicalfilesystem pendingchanges Rust code that leads to duplicate entries in the "lookup" list (files that need to have their contents checked to know whether they're dirty). We need to fix this, otherwise we'll do 2x the number of (potentially expensive) checks, and we'll need to deduplicate later anyway.
I believe this bug was originally introduced in D25697099 (02606da6c5) -- the change to this region of the code changes the behavior. I believe this was just an oversight when the code was refactored, since the change doesn't have anything to do with the purpose of the diff.
Reviewed By: DurhamG
Differential Revision: D34403900
fbshipit-source-id: 4c38e6a0ba46dfefc8f7ba327b943cb7ccee7fc5
Summary:
Some of EdenFS's integration test functionality can build on macOS, so
implement the required tweaks.
Reviewed By: kmancini
Differential Revision: D34291419
fbshipit-source-id: 296f077f4a3311d7fc8f04ed705fd06019e34212
Summary: Also check for new /files2 endpoint.
Reviewed By: quark-zju
Differential Revision: D34423002
fbshipit-source-id: 409435633e8f776729d25ebf05342020f423b828
Summary: If you give us paths of interest, and ask us to follow mutable rename information, let's do it.
Reviewed By: yancouto
Differential Revision: D34302823
fbshipit-source-id: 647fa2d798436f7a0beb1741570cb280aae2814b
Summary: Tweak "grep lfs" pattern to not match an "lfs" that shows up in tempdir name.
Reviewed By: quark-zju
Differential Revision: D34399625
fbshipit-source-id: 1122d0be4e11f32ed9cf33c046ccf34a99f5cdc3
Summary: Build is failing right now because of deprecation warning.
Reviewed By: mitrandir77, mzr
Differential Revision: D34449857
fbshipit-source-id: 70b7d871ef68995dd3d90a49c1f4af27613550b4
Summary:
Right now, when we create a snapshot on top of a commit that's not on commit cloud, we output things like `queue 0 files for upload` even when we are on plain mode, in which we'd like to only output the hash of the snapshot.
So I made it so the ui is quiet when on plain mode, which makes it not print that.
Reviewed By: quark-zju
Differential Revision: D34378357
fbshipit-source-id: 69b17147b8e72c6d72176f6069d729689130fdc9
Summary: The TOP here feels unecessary, let's remove it.
Reviewed By: fanzeyi
Differential Revision: D34434160
fbshipit-source-id: b8fa901d649694742db7f0b61c3923e9bce3274d
Summary:
Once minitop gains more and more functionality and becomes more interactive,
we'll want an easy way to quit, use 'q' for that. Since we're also in raw mode,
also add back the ability to quit with ctrl+c
Reviewed By: fanzeyi
Differential Revision: D34322721
fbshipit-source-id: 9f92d283d98a8d9d974f773156a0962936983f15
Summary:
With the terminal in raw mode, we need to use curor movement instead of \n for
new lines.
Reviewed By: genevievehelsel
Differential Revision: D34349787
fbshipit-source-id: a6e81100ba5fa8e568ddf89d3f72c524c7c0b103
Summary:
Raw mode allows for capturing input without it being echoed. It however
unfortunately means that \n no longer works correctly and thus printing will
need to be done line by line.
Reviewed By: genevievehelsel
Differential Revision: D34349788
fbshipit-source-id: 0f9362121ee2a40134d711ce56002347c95d0430
Summary:
Once minitop returns, we need to make sure that the terminal state is restored
to its old state. Use an RAII object to make sure that we always restore the
state that was set even in the error cases.
Reviewed By: genevievehelsel
Differential Revision: D34326731
fbshipit-source-id: b5d09988c92b7f469a8f311b106dd28f63ba69f1
Summary:
The interactive mode will redraw on top of the previous update, instead of
scrolling.
Reviewed By: kmancini
Differential Revision: D34326732
fbshipit-source-id: 9c2afd207d51ca4e8bb70c6943e6d620fa8fc1d3
Summary:
This has been turned on everywhere, time to set it to the default, so we can
then clean up the code.
Reviewed By: kmancini
Differential Revision: D34422426
fbshipit-source-id: 1d72a9d1bdbb162a4e1c9b69dbbe5aa4855f4f26
Summary:
The test tries to pushrebase the same commit onto two different bookmarks that are pointing to the same commit. In pushrebase by default we override `author_date` with current timestamp, so the test can [fail](https://fburl.com/sandcastle/5255yczk) if those pushrebases happen in different seconds (result hashes will differ).
Let's fix this by passing `rewritedates: false` pushrebase flag.
Differential Revision: D34416859
fbshipit-source-id: 59a258dc7536833ade8c64c1745e364ee94f6a57
Summary:
Traffic replay is gone. Now we can delete code that dumped wireproto traffic.
The logging that's left could be still somewhat useful: https://fburl.com/scuba/mononoke_test_perf/uismnrv9
Reviewed By: HarveyHunt
Differential Revision: D33898167
fbshipit-source-id: f5f12f3626c578ef90db99a45e5749fe8a94049f
Summary:
This diff includes the following changes:
1. Added a new admin command in commands.rs.
2. Added skiplist.rs for the primary skiplist command in newadmin and build.rs and read.rs for providing building and reading functionalities as subcommands.
3. Added integration test for newadmin skiplist.
4. Updated AdminRepo to include ChangesetFetcher and Changesets.
5. Updated test-backfill-derived-data-tail.t integration test to use newadmin skiplist instead of oldadmin skiplist.
6. Updated TARGETS file to pull the necessary dependencies.
Reviewed By: yancouto
Differential Revision: D34381835
fbshipit-source-id: 24a4f4ee27025f398f446114d9f2d92051c24e06
Summary: This a typo in the macro, which is trying to reference itself but fails. We haven't seen this before because we never used it with more than one argument, but when I tried it failed.
Reviewed By: Croohand
Differential Revision: D34391157
fbshipit-source-id: eb2f2704deac39b4d5f8d537fae75785b77eee5f
Summary:
Test multiple lazy changelog endpoints - pull, commit->location,
location->commit when the client and server have diveraged "main branch" config.
Reviewed By: farnz
Differential Revision: D34276252
fbshipit-source-id: 5ac323387fab509e8832cb5448fa80bb74fd8ca5
Summary:
markbt noticed some commit cloud commits weren't being logged to the scribe category. This is because the EdenApi path to create changesets doesn't do the logging.
This diff makes sure we also do the logging in that case.
Ideally, we'd have an unified place to create changesets, be it from hg changesets, git, or from raw data, but that needs a lot more refactoring and it's riskier.
Reviewed By: markbt
Differential Revision: D34146757
fbshipit-source-id: e82f14e1be6c598f89722b68c20cd6fc572633e0
Summary:
This does some changes to scribe logging logic:
Move it from `bookmarks_movement` to `blobrepo`. Being in `bookmarks_movement` was wrong, as it doesn't only happen in that case. `blobrepo` is not ideal but it's better, as it's close to where we save changesets, which is more or less when this happens.
This is done so the next diff can reuse more of the logging logic without having to duplicate code.
Reviewed By: markbt
Differential Revision: D34145742
fbshipit-source-id: 1dfb94775ea60e4b83412ae80f5c6d3f5e65d952
Summary:
This makes it so HashMaps are serialized in a consistent way in tests (on this crate), which is good because it prevents flakiness.
It also does so at no cost to non-test execution.
Reviewed By: markbt
Differential Revision: D34339601
fbshipit-source-id: e9ec1cb74308f83fb1a346c68f646dfa12021098
Summary: Cleaning up after verify_integrity rollout that switched from Python to Rust client. The rollout switch just appended "_rust" to the binary path.
Reviewed By: yancouto
Differential Revision: D34110979
fbshipit-source-id: 2d7c7d2b00b57626fc3c4f8d23859f520afabdf0
Summary:
Removed the line where we say Mononoke doesn't build, as it does!
Added links to the few .md docs that Mononoke has in repo
Reviewed By: mzr
Differential Revision: D34373614
fbshipit-source-id: 1bec3f4cf4ed158e4698b02a871d1b41d236cd1f
Summary:
While user report of Problem works just fine, telemetry report it as just a
Problem, and it's impossible to figure out which problem it refers to. Let's
force every usage of Problem to instantiate their own class.
Reviewed By: chadaustin
Differential Revision: D34352742
fbshipit-source-id: 30713d64ed5f72bc098e5531075a2f1aef504654
Summary:
Prints the output of `sample $edenfs_process_id` in `eden rage`.
On linux we can use `ps f` to see the process hierarchy we want to have similar
information on macOS. We would also like to have info on the state of the
threads, since we can not log into their machine and look for our selves this
would be a good thing to have in the rage. This give us visibility into eden to
see where the problem might be when there is a rage particularly if/where eden
is hanging.
Reviewed By: xavierd
Differential Revision: D16745028
fbshipit-source-id: 8dcbc1d80b45c3ebe4d065f445f56800104dbd25
Summary:
AppSettings::DisableHelpFlags has been renamed AppSettings::DisableHelpFlag,
and the compiler warns about it.
Reviewed By: fanzeyi
Differential Revision: D34322930
fbshipit-source-id: e376b3ec71fb05a4001b7abe870e16d5e2e55203
Summary: All of these are now available on Windows, thus we do not need to exclude them.
Reviewed By: kmancini
Differential Revision: D34259777
fbshipit-source-id: d7978665cce54e0b8e86424a82cfa13ce51fd947
Summary:
All of these are called during `eden rage`, and since we want `eden rage` to
never hang if EdenFS is hanging, let's add small timeouts for these.
Reviewed By: fanzeyi
Differential Revision: D34317837
fbshipit-source-id: a0537fe6edf912f5e0f946ce233ecef9e146c96c
Summary:
The TSocket.setTimeout in EdenFS is always called before calling open. What
this means is that settimeout is never called on the WindowsSocketHandle
object. In TSocket.open, settimeout is called right before trying the
connection, we should be doing that too.
Reviewed By: fanzeyi
Differential Revision: D34317838
fbshipit-source-id: b89dafdf84d418ec5f7be6acec25ea0e94a0ad52
Summary:
The latter is deprecated, and since this file doesn't import socket, this code
simply doesn't run.
Reviewed By: fanzeyi
Differential Revision: D34317839
fbshipit-source-id: 91baf5750153f7251385ec676edcd82dd5c331be
Summary:
The crossterm documentation states that queue! is more efficient due to needing
less system calls. Let's use it.
Reviewed By: fanzeyi
Differential Revision: D34322722
fbshipit-source-id: b158f86bfc58fd7efac118c27b297bc37f9fd36c
Summary:
In forgetStaleInodes we are checking for unlinked unloaded inodes and
de-referencing them. The dereferencing is happening inline with where we
are iterating through unloaded inodes.
This is causing us to invalidate an iterator that we are using. Which causes
a use after free.
let's collect the inode number of inodes to de-reference and de-reference them
after we finish iterating.
Reviewed By: xavierd
Differential Revision: D34315130
fbshipit-source-id: b390959bd26edf1545e900f0e5f72fc1885efa3a
Summary:
Since MacOS now has multiple architectures, and those architectures
can have different versions of things deployed, this might be pertinent in
investigating issues.
let's include which architecture we are running on in rage.
Reviewed By: fanzeyi
Differential Revision: D34319322
fbshipit-source-id: d5e3308b74f982b79e124400b85340951d626bce
Summary:
The shared futures stored within the mononoke API contexts are not cheap to
instantiate, and since they are not always used, we are paying an unnecessary
overhead. Migrate them to `LazyShared` so that they are initialized lazily.
Reviewed By: yancouto
Differential Revision: D34198428
fbshipit-source-id: bfb491437edfc76a707d523905cd984933f037ed
Summary:
Implement a new method pair for the Source Control Service:
`commit_path_last_changed` and `commit_multiple_path_last_changed`.
These methods return the last commit that changed a path for one or more paths
in a commit. If something exists at that path, this will be the last commit to
modify a file at that path or a file in a directory at that path. If nothing
exists at that path now, but something previously existed, then this will be
the commit that deleted the file or directory at that path.
Similar to the other `commit_multiple_path` method,
`commit_multiple_path_last_changed` does not return entries for paths that never
existed. This is done because efficient manifest traversal prunes these paths
before they are reached.
Reviewed By: yancouto
Differential Revision: D34047425
fbshipit-source-id: 4290e3193aa7834e6d44c90eb4b2626d580b43ae
Summary: This fixes a bug when snapshots with symlinks don't restore well when doing the operation from outside of the repo root. This happened because we were careful about relative paths on the normal file create codepath, but not on the snapshot codepath.
Differential Revision: D34302718
fbshipit-source-id: a8b06fb8a52e53c99735840fdbf5f3c74512a0c9
Summary: This simplifies the code a bit and makes future changes easier.
Differential Revision: D33865779
fbshipit-source-id: 52c284c9f61a24210465104f6d96a9f0bcb8b513
Summary:
The `calculate_definitely_unassigned_vertexes` function is an important
optimization to avoid excessive remote lookups when adding new commits
to the lazy graph. It contains the a "discovery" algorithm that uses a
bisect-ish way to find out what vertexes are known or unknown remotely.
That "discovery" algorithm can also be useful in other places, including:
- Figure out what vertexes need to be "reassigned" from the non-master
group to the master group.
- With proper Python bindings, replace the Python `setdiscovery.py`
entirely.
This diff makes the above possible by moving the algorithm to a separate
function.
Differential Revision: D33865775
fbshipit-source-id: 3b54de8929dd69ea80bdd2d8667b1791fe60ed50
Summary:
Make it a bit clearer that the callsite needs extra normalization if they
depend on the order.
Differential Revision: D33831518
fbshipit-source-id: 382a62d31e58137844ae954d38af6574d5bd5fb1
Summary:
The indexedlog iddag is written to disk. In the future "remove_non_master" might
be replaced by "strip". Add a test to ensure "remove_non_master" entry is still
supported.
Differential Revision: D33827176
fbshipit-source-id: a2dc76ae2a99badf8186ddb4721653c032378c5e
Summary:
The method does locking and flushing. In a future diff we want the
"flush_cached_idmap" behavior but we alredy have obtained a lock.
Let's split the logic so we can use the "flushed_cached_idmap"
logic without its locking and reloading overhead.
Differential Revision: D33822899
fbshipit-source-id: 672d165b2136f0aa7bd616836080b080135bc3d8
Summary: Don't trigger a background cloud sync for empty transactions. We had a recent issue where errant "hg log" commands kept trying to autopull an invalid commit hash, each time triggering a cloud sync.
Reviewed By: DurhamG
Differential Revision: D34289175
fbshipit-source-id: 35a6c0ef2ef30e0a384b554a8984340cbe6c8c2c
Summary:
This can be used for stripping set and its descendants.
See D33779403 (a8c32280cc) for high level goals.
Differential Revision: D33821384
fbshipit-source-id: 5dd6a550a166bb3a4eee847894afa4aaa2baa55c
Summary:
Creating bookmarks is a frequent use-case. Let's make drawdag support it so it's
possible to reduce the number of `hg bookmark` commands.
Reviewed By: DurhamG
Differential Revision: D34259550
fbshipit-source-id: b0cc750aaa3bd35f72ae0b92e15ec7b9e700dec6
Summary:
This is the recommended practise because the `node` name is often used as local
variables.
Reviewed By: DurhamG
Differential Revision: D34259549
fbshipit-source-id: d0101eb5da813ede865ac0d6685938c9a39e85ae
Summary:
Previously there are 3 places doing lock and transaction for drawdag.
Unify them to just 1 place.
Reviewed By: DurhamG
Differential Revision: D34259552
fbshipit-source-id: bb5773e63537dcd2c8fa07081178ea3622b73f5f
Summary:
We don't use obsmarkers. Some comments are outdated or confusing.
Update them without changing the drawdag feature.
Reviewed By: DurhamG
Differential Revision: D34259551
fbshipit-source-id: da92801dccae825d32eb71dd8da70de2e0cb4ef6
Summary:
**Context**: See T110025458
This diff is the final piece to solve T110025458. It uses the `filestore::copy` method added on D33193941 in conjunction with `copy` added on D33193941 to have fast copying.
Testing locally, the improvement is a bit disappointing. For a 1G file, it went from 18s to 12s. I expected much, much less. That said, this was a prod build vs a non-opt build with a lot of print-debugging.
Reviewed By: farnz
Differential Revision: D33193939
fbshipit-source-id: 3c976f259f66d27da1f19c9d129ffdbb64745a8e
Summary:
**Context**: See T110025458
This diff adds a method to filestore to copy a file somewhere else, using the BlobCopier interface. On the next diff we will use this to optimise snapshots.
- The order of operations is the same as in finalize, for the same reasons.
- We're using BlobCopier interface and not a blobstore, because e.g. in snapshots, we actually want to mess around with prefixes before doing the actual copying, this doesn't make sense in a single blobstore.
Reviewed By: farnz
Differential Revision: D33193941
fbshipit-source-id: ede51b0149173a8f0c1e4f0d516255cdfc88b8eb
Summary:
**Context**: See T110025458
This diff adds the `copy` method to the blobstore trait. It is a very basic method that copies one key to another. The default implementation simply loads the value into memory and stores it again, but faster implementations can exist.
I moved the implementations from `BlobstoreWithLink`, and renamed that to `BlobstoreUnlinkOps`, as we don't need this on the default Blobstore trait (for now).
Reviewed By: farnz
Differential Revision: D33193940
fbshipit-source-id: b10beecbd827396d1685f8196bab290e81ba4039
Summary:
This makes use of the new Rust repo initialization added in previous commits into the `clone` command in Python by overriding the repo initialization done by creating a new `localrepo` object (and setting `create` to `True`). There are few things of note here:
- New repo initialization was only done by the `init` and `clone` commands
- This was the only way new repos were initialized when calling the `clone` command.
- Repo initialization via the `init` command is already being done by the Rust repo initialization and this override should not affect it.
Reviewed By: DurhamG
Differential Revision: D34094589
fbshipit-source-id: 079d641b8001b51854838a4089ba4ef37942fc69
Summary:
Currently the source control service will accept bookmark moves for repos that
are the small-repo in a push-redirected megarepo config. This means it's
possible for users to get bookmarks out of sync with a bookmark move, which
requires manual intervention to fix.
Prevent this from happening by adding a check in `bookmarks_movement`. If a
repo has push-redirection enabled, then bookmarks of that kind are not allowed
to be moved by the source control service. Users should move the bookmark in
the source-of-truth repo and allow the backsyncer to move it in the
push-redirected repo.
Reviewed By: mitrandir77
Differential Revision: D34222495
fbshipit-source-id: 64a0fa929af53c9991c2a0f8bb84a21e4a1de9fc
Summary:
Use `InnerRepo` when calling bookmarks movement code. This will allow us to
extend to include a dependency on `RepoCrossRepo`, even though this is not part
of `BlobRepo`.
Reviewed By: mitrandir77
Differential Revision: D34222494
fbshipit-source-id: 9efbd501a9435949abaae4a7445d6b1808a491d5
Summary:
When processing post-resolve actions, use facet traits rather than `&BlobRepo`
to specify the repo. This will allow us to add a dependency on `RepoCrossRepo`,
which is not part of `BlobRepo`.
Reviewed By: mitrandir77
Differential Revision: D34222498
fbshipit-source-id: 6443378c2922e15a435965bc469c5c8a24fe6e19
Summary: Use a facet container for `repo_import`. This will allow us to continute to call `bookmarks_movement` code when we add dependencies on `RepoCrossRepo`, which is not part of `BlobRepo`.
Reviewed By: mitrandir77
Differential Revision: D34222497
fbshipit-source-id: 8f321f4c11348782738336ae4ff157c0c6c658cf
Summary: Add `RepoCrossRepo`, a new repo attribute facet that encapsulates the cross-repo sync state. This will mean cross-repo sync information will be available to code that uses repo attributes (e.g. bookmark movement).
Reviewed By: mitrandir77
Differential Revision: D34210452
fbshipit-source-id: 5cc314958d82abe5bd30c56171f975bd948daa43
Summary:
Convert the bookmarks_movement crate to use repo attributes (i.e. facets), rather than `BlobRepo`.
Since pushrebase still uses `BlobRepo`, we rely on `AsBlobRepo` to let us get back to the blobrepo for now.
Reviewed By: yancouto
Differential Revision: D34203293
fbshipit-source-id: 9d2f82697255fb78551a592ebf5f3609c9dc1da2
Summary:
This diff passes the `workingcopy.rustwalkerthreads` config value along to the physical filesystem `pending_changes` function, allowing it to use a multi-threaded walker when `rustwalkerthreads > 0`.
We also need to use `extract_matcher` to get a Rust Matcher from a Python one for use in the multi-threaded walker. Using `UnsafePythonMatcher` causes the walker to deadlock (presumably due to the GIL -- see P480409358 for an example backtrace of a hung thread). This was the last usage of `UnsafePythonMatcher`, so I removed it as well.
Reviewed By: DurhamG
Differential Revision: D34251644
fbshipit-source-id: 1f11f2038ebcc6e098432132b9e5726b25c18911
Summary:
This diff adds a new `Walker` struct, which can contain either a `SingleWalker` or a `MultiWalker`. The concrete Walker implementations are hidden and are chosen by the `num_threads` parameter -- single-threaded walker if the thread count is 0, and multi-threaded walker otherwise. This maintains the logic in `pyworkingcopy`.
This change has the downside of adding additional trait bounds (`Send + Sync + 'static`) on the `Matcher` used, even if you only need a single-threaded walker. However, the next diff in this stack would require that anyway in the only place that doesn't already have that bound. Going forward, it seems like future uses of Matcher would also like to dynamically select between single and multi-threaded walkers, and thus would also need those trait bounds anyway.
Reviewed By: DurhamG
Differential Revision: D34251645
fbshipit-source-id: 1273a3f2da59dfe3f9fd165b2e0ab35477c37180
Summary:
When a bubble is opened, `RepoContext` has two `BlobRepos`: the one inside the
context, which has the bubble opened, and the one inside the `InnerRepo`, which
does not, and is shared with the main repo.
This could be a problem in the future when we start using `InnerRepo` instead
of `BlobRepo`. Change it so there is only one `BlobRepo`: the one inside the
`InnerRepo`.
Reviewed By: yancouto
Differential Revision: D34222496
fbshipit-source-id: e576fd72b8f8c9187ed9a518ce59d30e189afae2
Summary:
To let us convert from `BlobRepo` to new facet-based container repos, introduce
a new trait `AsBlobRepo`, that will let us convert back to `BlobRepo` during
the migration.
Reviewed By: yancouto
Differential Revision: D34203296
fbshipit-source-id: 9092bd6b5145a417de99e1db0546e492e61c443e
Summary:
Use the `ArcChangesetFetcher` type (which is `Arc<dyn ChangesetFetcher + Send + Sync>`)
so that it is compatible with the facet type.
Reviewed By: Croohand
Differential Revision: D34203295
fbshipit-source-id: 6be4d6014bde8fb5461c9ce775b597ae2184b31d
Summary:
Use the `quickcheck_arbitrary_derive` crate to auto-derive the implementation of the `quickcheck::Arbitrary` trait for amenable types under `fbcode/eden`.
An implementation is amenable to auto-derive if the macro-generated code is functionally equivalent with the replaced code.
Reviewed By: yancouto
Differential Revision: D34191759
fbshipit-source-id: 994904cbd1970318a9bfc2462b801aa84eda6861
Summary:
Use the `quickcheck_arbitrary_derive` crate to derive the `Arbitrary` trait for all the types for which the auto-derived impl would match the existing impl.
note: I did not commit the updated snapshots from the `buck test ... -- --env INSTA_UPDATE=1` execution because I still don't know if they are correct or not. :) yelp! :D
Reviewed By: yancouto
Differential Revision: D34161995
fbshipit-source-id: 567c6c43b1fb666abd460c4a7ac62bf095220600
Summary:
This code hasn't been used in a long time, remove the code and config
parsing logic. Further, remove the commit cloud bookmarks filler binary.
Reviewed By: mzr
Differential Revision: D34169819
fbshipit-source-id: b2a87b0b79f0626b26e4bcfcb932b1b4aa7a209c
Summary:
The autocargo was complaining about collision between clap and clap-old
when we tried to migrate target that shared Cargo.toml file with unmigrated
target. This should be enough to alleviate problem.
NOTE: I've also had to make a copy of value_t macro from clap as it was
referring the crate by name.
Reviewed By: markbt
Differential Revision: D34219408
fbshipit-source-id: b0bb1328d8808e7387944d8df14dc374e569ece2
Summary:
A Mononoke repo can be marked as read only in config, or be read only
depending on a database value. The only place that this used to be checked was
at the start of the unbundle code.
In order to allow sync jobs and commit cloud to write bundles to Mononoke, a
`BYPASS_READONLY` pushvar was checked in the unbundle code. Both of these
clients hard code the bypass push var.
After some refactoring, we now check whether a repo is read only both at the
start of the unbundle path and also when a bookmark is moved.
This diff removes the read only check at the start of the unbundle path. The
motivation is so that usage of the `BYPASS_READONLY` pushvar can be restricted
to just sync jobs. Once this diff lands, I'll be able to remove the
`BYPASS_READONLY` pushvar from the commit cloud code path.
Before this change, a client trying to unbundle to a locked repo will be given
an error before they start to upload content. After this change, they will only
discover the repo is locked after attempting to move a bookmark.
Reviewed By: markbt
Differential Revision: D34108102
fbshipit-source-id: 8b5cf5ddded6c699686592141bc7664c1ed15924
Summary:
The LFS server included separate fields in its config for object
popularity category and threshold. However, they have been replaced by a dedicated
`ObjectPopularity` struct that also includes a `window` field.
Remove the old configuration from the code, as it's unused and confusing.
Reviewed By: mzr
Differential Revision: D34214336
fbshipit-source-id: 7c3ecd5aa8d7e882783c8ff8630edf48999428e9
Summary:
We can use _replace_fields instead of copying all the filed on an old
CheckoutConfig to a new CheckoutConfig. This means less copy pasting,
less chance for a silly mistake, and reads a little easier.
Reviewed By: xavierd
Differential Revision: D34229260
fbshipit-source-id: a0511d9397d455ef11b9178c745dd74c82bf9c90
Summary:
checkunknown is a large source of random slowdowns for users, since if
they have a lot of untracked files in their working copy, it could result in
downloading a lot of data, reading the files off disk, and comparing it. This
causes a lot of pain, and in contrast the checkunknown functionality likely
saves users time very rarely (vs just rewriting the files on disk), so let's
remove it.
Reviewed By: quark-zju
Differential Revision: D34113766
fbshipit-source-id: 442b347e5484422c5b27154763bd878b2bb8803e
Summary:
Previously filesystem._compareondisk relied on ctx.cmp(). Which
inspected the size, read the content out of the store, and compared the store
contents with what's on disk.
This is expensive for a few reasons.
1) Fetching LFS file size from the store actually fetches the entire lfs blob,
due to some unfortunate API design. This means we fetch the entire lfs blob,
then look at the lfs pointer to get the size and completely ignore the blob.
Then, if the size matches we later fetch the lfs blob *again* to compare it with
what's on disk.
We can fix the API later. For now let's circumvent ctx.cmp() and go straight to
filelog.cmp() to skip the size check, since we already did a size check earlier
in the status computation.
2) remotefilelog.cmp() requires that we have the store contents, which ends up
requiring an expensive prefetch step.
In reality we can do the cmp by computing the p1+p2+text hash of the contents on
disk and comparing it with the filenode in the manifest. So let's change
remotefilelog.cmp to use this method instead. This does require a history
prefetch (so we have p1 and p2), but that's much cheaper than a content
prefetch.
Reviewed By: quark-zju
Differential Revision: D34093036
fbshipit-source-id: 4a4386e59bca178ae58b10897c5cf68d3aa15dda
Summary:
We have seen some users with very long startup times due to very large WAL
files on disk. Let's reduce the WAL size to make sure EdenFS startup is always
fast.
Reviewed By: chadaustin
Differential Revision: D34260876
fbshipit-source-id: dc4d94484c7afdc8350b3cbb862cfb8b4b8aa60b
Summary: When cloning large repo's, sometimes the tls connection fails or is flaky. Let's retry for that case.
Reviewed By: quark-zju
Differential Revision: D34257302
fbshipit-source-id: 540df4fcad6db5d2d9e11683cd3e91d4093f8681
Summary:
The `--no-write-fetch-head` flag was added in git 2.29. It does not exist in
all git used in production yet.
Reviewed By: DurhamG
Differential Revision: D34214948
fbshipit-source-id: 97ab132205f48c92213cf54b5d79cb8d83f95873
Summary:
See D34095156 (035e8b61a5) for context. `util.system` is easy to misuse.
Rename it to a more scary name and document that `ui.system` is preferred.
Reviewed By: sggutier
Differential Revision: D34095594
fbshipit-source-id: 61a77b552e51de729a579edb34a6470d26681c7b
Summary:
Monkey patching `repo.newreporequirements` won't be supported by Rust repo
initialization. Let's replace the mokey patch with an explicit requirements
update in the clone code path.
Reviewed By: sggutier
Differential Revision: D34254583
fbshipit-source-id: 903fb7925687d50eb9035f3a063ef98cf2551c31
Summary:
LFS batch fetching would just silently eat any server errors it
encountered. Let's route those errors up to scmstore so they get reported
appropriately.
Note, this doesn't apply to standard http connection errors, put does apply to
the server's response to batch request queries. These errors happen if the
server doesn't know about the blob being asked for, etc.
Differential Revision: D34191914
fbshipit-source-id: 432aefb9871ddc91e5e9dc98f64182e8ef4ea471
Summary:
I introduced this "fb" feature to try to fix conditional compilation of fbwhoami into the hostname shed crate, but that wasn't the right way to do it (and it didn't work anyway).
The correct way:
1. The OSS build should use the public_autocargo version of hostname which doesn't reference fbwhoami in Cargo.toml.
2. The OSS build has fbcode_build=false which the hostname source uses to not reference fbwhoami.
Reviewed By: quark-zju
Differential Revision: D34010737
fbshipit-source-id: 3ea3b12aaf5c89fb9af8f34c4a2a3192f4fd93d7
Summary:
Replace the Python drawdag parsing logic with the Rust implementation.
This allows drawdag to parse horizontal graphs, and ranges.
The `o` special case to support `log -G` output is now gone and related
tests are migrated to not use `o`.
Differential Revision: D34163476
fbshipit-source-id: c37e336b60aa3fc06f00dfcccd21ecba11e9e074
Summary:
See https://matklad.github.io/2021/09/04/fast-rust-builds.html
It seems all users of `parse` just passes in `&str`. So let's drop the `impl`.
The `commit` function is less used so it's unchanged.
Differential Revision: D34163477
fbshipit-source-id: 71cb6333a0261f216366d4e1bda8fcec55baef96
Summary:
The Python drawdag implementation allows revset expressions as names.
To replace the Python drawdag with the Rust drawdag, let's support
the special chars used in revset.
Differential Revision: D34163104
fbshipit-source-id: 4f140858cb760d40f80326af337884d1f77c6319
Summary:
We want to exercise "large-ish" commit graph, to check if certain operations
might cause too many network round-trips. It wasn't easy to do that using
drawdag.
This diff adds `:` and `.` special edges for ranges, so
one can create a linear range quickly, like:
A..Z
or:
A001..A999
or vertically:
A999
:
A001
Differential Revision: D34163018
fbshipit-source-id: a043b2b9b3dc73b20e4ee37598b9d182a6d65c29
Summary:
This will be used in the next change to create a range of linear commits
easily, like A..=Z, A01..=A20, etc.
The implementation is inspired by the `String::succ` method in the Ruby
programming language, except for the non-alphanumeric case, because that's
a case not intersted by drawdag, and finding the next valid utf-8 code point
in Rust seems non-trivial.
Differential Revision: D34163017
fbshipit-source-id: 9d1036cabab7a7f33e93a00c3e7c90e40534c994
Summary:
We use Rust verify_integrity client in prod which always does checks remotely while tests use Python client with local checks. Let's change tests to match prod while also unblocking removal of the old client.
See D34076245 for more details.
Reviewed By: yancouto
Differential Revision: D34109250
fbshipit-source-id: eb01ce1b61a2dd30a216fca7637ec721eb1f9f07
Summary:
As described in D32638849 (ec370e31b7), there is a footgun in `MemWritesBlobstore::persist`.
For the duration of the persist, the items being persisted are temporarily
invisible from outside.
Previous attempts to fix this were made by trying to stop people from accessing
the blobstore while the persist was ongoing. Instead, let's just extend
`MemWritesBlobstore` so it doesn't have the hole in the first place. While we
are persisting the items, we keep a reference to the items being persisted in
the cache. Any key lookups that fail for the live cache can also be checked
against the flushing cache. Once the flush has completed, we can clear those
items out.
A downside here is that we must clone the keys and values: there need to be two
copies, one left in the flushing portion of the cache, and one that is passed
by value to `Blobstore::put`. However, since this is a `String` and a
`BlobstoreBytes` (which reference-counts the bytes), cloning should be
relatively cheap.
Reviewed By: mitrandir77
Differential Revision: D34074710
fbshipit-source-id: 4562833266b80bf95ed21a01b4f871e341b2f357
Summary:
The various mappings are inconsistent in how they work. The bonsai-git
mapping is mostly where we want it to be, however the builder name is wrong,
and the implmentation is mixed into the same module as the trait.
Clean this up by spliting the implementation out to its own module, and
renaming the builder to `...Builder`.
Reviewed By: yancouto
Differential Revision: D34196512
fbshipit-source-id: e8de38d4d495ac911a6fa1e1c65c10337c5bf281
Summary: Combined with the existing ability to dump starting from a known point, this gives us all the pieces to maintain a set of files in Manifold that we can download and update in a SEV
Reviewed By: yancouto
Differential Revision: D34219264
fbshipit-source-id: 543c31bb151637be07b253e65e12675dd2d8f953
Summary:
The various mappings are inconsistent in how they work. The bonsai-svnrev
mapping has a separate struct to hold the repo_id, and must be constructed and
used in a different way to the others. Continue to make them consistent and
clean up this gap, this time with the bonsai-svnrev mapping.
Reviewed By: yancouto
Differential Revision: D34140241
fbshipit-source-id: 432f1394b05431625349781b95430b37a0478c6c
Summary:
There are currently multiple ways to map between bonsai changesets and
globalrev: using the bonsai-globalrev mapping repo attribute, or using the
method on blobrepo that is very similar (but takes `ctx` by value).
Standardize on using the repo attribute, and convert all callers.
Reviewed By: yancouto
Differential Revision: D34093301
fbshipit-source-id: b1e0858a933d2fa6bcae0c04cdb10da2f438ddc6
Summary:
The various mappings are inconsistent in how they work. Some, like the
bonsai-globalrev mapping, require the client to pass the repo_id on every
request (and behaviour is undefined if the client gets this wrong). Continue to
make them consistent and clean up this gap, this time with the bonsai-globalrev
mapping.
Reviewed By: yancouto
Differential Revision: D34093084
fbshipit-source-id: de67d4da54039a35baa4db65d208d71f1195975c
Summary:
Lets log what generation is being marked.
Also update the terminology in log messages. In mark/sweep GC, the sweep is the deletion of unreferenced blobs, so this process does marking not sweeping
Reviewed By: farnz
Differential Revision: D34211813
fbshipit-source-id: 478dfda23d46938e575fd48372f74165b5fdf3f2
Summary:
macFUSE support on apple is going away soon. If macFUSE breaks for folks,
they will need to migrate to NFS. This makes the migration a one command step.
In case folks see any issues with NFS during our migration we also have
an option to migrate back to fuse. This will temporarily unblock them
while we fix any bugs that come up.
Reviewed By: xavierd
Differential Revision: D34081417
fbshipit-source-id: ea89345b5ccc019cbfe6387a1dfa366ae9dd6345
Summary:
Failing to use the fast pull path is not fatal. We can still use the generic
pull path.
Reviewed By: sggutier
Differential Revision: D34099015
fbshipit-source-id: 45a6e94c4192bda7257e97c42f544c0a4654204a
Summary:
For pull, the id map includes both "low"s and "high"s of flat segments, not
just "parents". Include them in the clone hints so the hints might work for
pull use-cases.
Reviewed By: farnz
Differential Revision: D34216070
fbshipit-source-id: 614266fedc2f767e25478859c8a7b59ff6640f74
Summary:
`ui.system` handles more things properly, including suspending progress bars,
and telling chg to use the original terminal so ctty is available.
Differential Revision: D34095156
fbshipit-source-id: 20909fbfe7925a18993f0db05a01fbe9ea8c6afc
Summary:
Status has two real phases 1) Gets the best effort results from
Watchman, and 2) Looking at the files on disk that may or may not have changes.
Today we show one "Querying watchman" progress bar for this entire operation,
even though most of the time is spent in phase 2, after watchman has returned.
Even worse, the current progress bar is just a spinner and gives no sense of
progress, even though we know how much work happens in phase 2.
This diff splits the two progress bars.
Reviewed By: quark-zju
Differential Revision: D34093035
fbshipit-source-id: 44fb1a7c705e0a8b9e1bf8451931aa345876962f
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary: This diff addresses the issue of the `--json` flag ignoring the `-q` flag (short date).
Reviewed By: markbt
Differential Revision: D34208590
fbshipit-source-id: 5ab39095ccdff09116e85e0e9c2a62c0efb121ab
Summary:
We want to go down the mutable history route on request in path history. Do that.
A related change will be needed for diff history
Reviewed By: mitrandir77
Differential Revision: D34142424
fbshipit-source-id: ae253958f93d2debf13ac1bc5d3b9e86d6ef7db3
Summary: Change read_from to use an iterative approach. We hit some stack overflows in production we think were caused by this recursion.
Reviewed By: quark-zju
Differential Revision: D34188307
fbshipit-source-id: d62d559096519b7c8dbce89f07a93fc8cf671595
Summary: Number of visits to file content can vary depending on path taken through graph
Reviewed By: yancouto
Differential Revision: D34212574
fbshipit-source-id: b43ed6bb363e42556ecb9428957277af3238c551
Summary:
All the separation between rows and column use a lot of space, making it IMHO
hard to read. Let's use a nicer UTF8 formatted table with no separators.
Reviewed By: genevievehelsel
Differential Revision: D34154329
fbshipit-source-id: 973fc925195b22598c5e5b5081e97bb4fa280ef8
Summary:
In some cases, even in environment where configerator isn't present it tries to
initialize itself, leading to log spew. This environment variable allows to
disable this code, let's allow it to be passed through.
Reviewed By: genevievehelsel
Differential Revision: D34190070
fbshipit-source-id: 190496ce6155843d166be2d77db1a3ff0926787c
Summary:
When the master bookmark moved backwards, clients may have old location of the bookmark pulled locally. It was failing due to the fact that server couldn't find that bookmark location in it's IdMap.
Let's allow SC to build idMap in memory up to the client heads so it can correctly respond to the pull request.
We will only check some number of parent commits from the client head to check if they are ancesstors of master to avoid expensive SC rebuilds. And we still need to check for the merge commits with large number of commits not in the SC. (D33654925 (78c19c7a79) already reduce the possibility of this)
Reviewed By: farnz
Differential Revision: D34041066
fbshipit-source-id: aa7274f97c675e32e87a791aca336061dc75d645
Summary:
There are currently multiple ways to map between bonsai and hg changesets: using the bonsai-hg mapping repo attribute, or using the method on blobrepo that is very similar (but takes `ctx` by value). Standardize on using the repo attribute, and convert all callers.
There is one exception to this: when callers wish to trigger derivation, they
continue to call `BlobRepoHg::get_hg_bonsai_mapping`. Despite its name, this
method triggers derivation in the bonsai-to-hg direction only. We will
separate out this functionality in the future.
Reviewed By: mitrandir77
Differential Revision: D34071889
fbshipit-source-id: f0c4ce7987376702dc36eb27683d1fcb8a22df60
Summary:
The various mappings are inconsistent in how they work. Some, like the
bonsai-hg mapping, require the client to pass the repo_id on every request (and
behaviour is undefined if the client gets this wrong). Start to make them
consistent and clean up this gap, starting with the bonsai-hg mapping.
Reviewed By: mitrandir77
Differential Revision: D34059165
fbshipit-source-id: 12c62e0e5f08685929218088bad11a55305795df
Summary:
`status` should not show changes in submodules. That's already working with the
vanilla `status` implementation but the fsmonitor one needs some updates.
Watching nested roots seems to cause timeouts. Therefore fsmonitor is disabled
for submodule repos.
Differential Revision: D34025075
fbshipit-source-id: 45f97fa62f634e44f66dec75acad055e819b964c
Summary:
This fixes annotate on git repos, which do not have filelogs but can use
pathhistory for file traversal.
Renames are not yet supported.
Differential Revision: D34017426
fbshipit-source-id: 817b0d635a31e0f052c82defb4615c87909668f5
Summary:
The file history traversal is based on hg filelog, which is considered as a
techdebt and is not present in git. Move it to a separate function so it's
cleaner to implement other kinds of file history traversal.
Differential Revision: D34017424
fbshipit-source-id: 50b9e9a818250dbb56d89460533e8859a3abc36b
Summary:
The main algorithm of "annotate" is relatively abstracted.
Move it to a separate module.
Type annotations are added to demostrate that it is abstract.
Differential Revision: D34017428
fbshipit-source-id: cf9817de807ede05621ab40432b73f146463a7e7
Summary: Expose the `NameSet::to_parents` API so it can be used in Python.
Differential Revision: D34017427
fbshipit-source-id: 9a602031eb9f709f43d89a0cd23448d9b8285021
Summary:
A nameset is a "flat" set without graph (parents) informatin.
In certain cases (ex. graph log, annotate) the parents information
is needed. Add an API for that.
This might potentially replace some logic in Python `graphmod.py`.
Differential Revision: D34017425
fbshipit-source-id: 38d0bccb5c455cd9ec2d60113c18207a90645de9
Summary:
When a checkout is in progress, get_snapshot will throw, let's make sure we
catch it and do something sensible about it.
Reviewed By: chadaustin
Differential Revision: D34183262
fbshipit-source-id: 8579c4f57b22306b08434d095a48209eb5b16831
Summary:
Providing the command makes it easier for users to resolve the issue themself
and should decrease the number of potential posts.
Differential Revision: D34183263
fbshipit-source-id: 93d2a85947e7640e0941070e5276fd07cb7d2cfd
Summary:
When doing a clone, the nativecheckout action plan included a remove
action for every file not in the sparse checkout. This is because it didn't
account for the file not existing in the old manifest and just blindly queued
the remove.
On Linux this is pretty fast, but on Windows this is extremely slow to do for
every path. It's probably slow because of NTFS, but also because we do some path
auditing logic for Windows and Mac.
This fixes it to not do remove for a file that wasn't in the old manifest.
Reviewed By: sggutier
Differential Revision: D34188271
fbshipit-source-id: 714b2dcf98cfa9ebec287f00d7cd5397126f693e
Summary:
The old tracing info was noisy and hard to follow. The new one is much
more concise and provides useful insight into what is being fetched, at what
stage, and what errors happen at each stage.
Reviewed By: quark-zju
Differential Revision: D34159138
fbshipit-source-id: e1de87de9f9f64deb7e33ef76e6441bb6721f9b9
Summary:
These produce a ton of hard to read trace output, with enter/exit's
everywhere. Let's delete them. In the next diff we'll add more human readable
explanations of what's going on in scmstore.
Reviewed By: quark-zju
Differential Revision: D34159139
fbshipit-source-id: 48d1175edd20d3de952f12dc8fcf7a6661469d64
Summary:
When ran in the destructor, the class inheriting from RequestContext would have
been destroyed, and thus the vtable for the RequestContext would no longer from
the child class. In practice that meant that the PID of all requests was always
a nullopt, making `eden minitop` and others completely useless.
To solve this, we can simply create a shared_ptr with a custom deleter that
calls finishRequest first before calling the destructor.
Reviewed By: genevievehelsel
Differential Revision: D34160296
fbshipit-source-id: 7f6d9bfa36bc3db256cfe0f61478c22a44028001
Summary:
scmstore was writing lfs pointers via the contentstore. This meant that
when we disabled the contentstore fallback, this path failed. This path is used
during unbundle (like in 'jf get') when the bundle contains an lfs pointer.
Reviewed By: quark-zju
Differential Revision: D34127065
fbshipit-source-id: d50245b78dd87e6c71d8775123fe954084289e2b
Summary:
When initializing a new repo in the old new repo initialization, directories would be created recursively if the target path did not exist. This diff fixes the new repo initialization not doing that.
The unit test for directory creation was also reworked.
Reviewed By: quark-zju
Differential Revision: D34093210
fbshipit-source-id: 3aa0aade9bd3034448615678a005966a7da6ef5c
Summary: Storing errors instead of their string representation should be much more useful for showing errors. This diff does that for `InitError`.
Reviewed By: quark-zju
Differential Revision: D34161428
fbshipit-source-id: 4f77428ef379db6b42b492bb5424a8fdd04b954a
Summary:
Add some counters to track lfs transient/fatal error response codes:
- "lfs.transient_error.{method}.{code}" counts error codes we got but ended up
succeeding after retries
- "lfs.fatal_error.{method}.{code}" counts errors we got and then ended up not
succeeding
- "lfs.success.{method}" counts successful queries
Reviewed By: quark-zju
Differential Revision: D33962157
fbshipit-source-id: 901625eb72cac2eba57f4b2424a778f4f8ac1779
Summary:
`test-snapshot-update.t` was identified as flaky.
I modified `assert_on_base_snapshot` to print more information on failure, and stress tested it with:
```
> buck2 test eden/mononoke/tests/integration/facebook:snapshot -- --stress-runs 50 test-snapshot-update.t
```
On 2 of those 50 runs, it failed because `hg status` didn't realise the `modified_file` was modified. Notice that we also `cat` that file and check its contents, and the content was indeed correctly modified, so this is likely a problem in `hg status` somehow being out of sync.
Since checking `hg status` was not that important in this case, I just removed it to make the test less flaky, and we still should get good signal.
Reviewed By: HarveyHunt
Differential Revision: D34110526
fbshipit-source-id: fe726e0652db202b7668ff666e30df3bff453aab
Summary:
Rename `ArgExtension` to `AppExtension` and extend the mechanism to allow the args for the extension to be parsed once, and then obtained from the app, using the type of the extension as the key.
This leads to a fairly ergonomic usage:
```
let builder = MononokeAppBuilder::new().with_app_extension(Extension::new());
let app = builder.build();
let extension_args = app.extension_args::<Extension>();
```
Reviewed By: mitrandir77
Differential Revision: D33928951
fbshipit-source-id: 31def0b59e3a0d6f900f503c1db22c50f20134d3
Summary: Lets document that the way to build OSS eden_scm, edenFS and Mononoke is via getdeps.py
Reviewed By: chadaustin
Differential Revision: D34148404
fbshipit-source-id: 89a53d9b7263bdeb56af4e2cff6873aae5337038
Summary:
This feature is only used for e-discovery, and we don't need to build a
new e-discovery package from eden/scm anymore. Let's remove this feature since
the monkeypatch interferes with updating the hg status paths.
Reviewed By: quark-zju
Differential Revision: D34124434
fbshipit-source-id: b41382bc1a3cfeda82e1dd2af511244ad16a9fb8
Summary:
Convert type comments to type annotations.
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden
```
from a parent directory
Reviewed By: shannonzhu
Differential Revision: D34110680
fbshipit-source-id: 23081a8867b93cd349271075428176af514efa2b
Summary: For repo imports, we want to be able to build SC with an extra set of heads to cover the newly imported commits. Make it possible to do that by allowing you to add heads on the CLI and also bring in the imported heads.
Reviewed By: mitrandir77
Differential Revision: D34142590
fbshipit-source-id: e188a4e5f5e87fa406f63a1a9f17f33fe5f0ba5f
Summary:
We've been seeing some errors on ASIC [1] that say `bubble X does not exist, or has expired`. Since we don't expire bubbles yet, and we only call this endpoint after having actually created the bubble, I conclude this is likely caused due to replication lag.
To fix that, and at the same time avoid always querying the master SQL replica, which is undesirable, let's first try a normal read, and only if it yields no results, then we query master. This should only add an overhead when replication lag happens, which is quite rarely.
Also, we don't use this `open_bubble` function to check if a bubble exists, we always call it when we know it already exists, so this should not add any undesirable overheads in that case.
Reviewed By: mrkmndz
Differential Revision: D34045137
fbshipit-source-id: 5928f7d457d5104505df847d91c83eee0582e343
Summary:
Some of our integration tests investigate scuba logs. However, there
is now extra information included in these logs. Strip it out so that the tests
pass across all environments.
Reviewed By: mzr
Differential Revision: D34105595
fbshipit-source-id: adeecaf9b8fcc72da5c681f5831502953f7cea52
Summary:
To encourage the migration away from BlobRepo, introduce `AdminRepo`: a facet
container for the things `admin` needs.
We will use a single type for most of the subcommands so that monomorphisation
will resolve everything to `&AdminRepo`, and we won't end up duplicating code.
Commands that have special requirements can still use their own Repo types at
the expense of monomorphisation potentially duplicating things.
Reviewed By: mitrandir77
Differential Revision: D34050210
fbshipit-source-id: bcea2bc1de0ba1316e00548a0061d111251caa09
Summary:
`eden trace hg` currently fails with:
```
Traceback (most recent call last):
File "C:\Python38\Lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python38\Lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\tools\eden\bin\edenfsctl.real.exe\__main__.py", line 3, in <module>
File "C:\tools\eden\bin\edenfsctl.real.exe\eden\fs\cli\main.py", line 2283, in zipapp_main
File "C:\tools\eden\bin\edenfsctl.real.exe\eden\fs\cli\main.py", line 2266, in main
File "C:\Python38\Lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python38\Lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\tools\eden\bin\edenfsctl.real.exe\eden\fs\cli\main.py", line 2241, in async_main
File "C:\tools\eden\bin\edenfsctl.real.exe\eden\fs\cli\trace_cmd.py", line 48, in run
AttributeError: 'Namespace' object has no attribute 'reads'
```
Reviewed By: xavierd
Differential Revision: D34125938
fbshipit-source-id: 4d7504feda88123f828f4bfb395f98df35b0ac36
Summary:
Convert type comments in eden/hg-server/edenscm/hgext
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden/hg-server/edenscm/hgext
```
from a parent directory.
Reviewed By: grievejia
Differential Revision: D34110679
fbshipit-source-id: 9692dda348b66e830de68a3cd04f6de9e2b82cc4
Summary:
Convert type comments in eden/hg-server/edenscm/mercurial
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden/hg-server/edenscm/mercurial
```
from a parent directory.
Reviewed By: grievejia
Differential Revision: D34110683
fbshipit-source-id: 6b962de8d3c4eadae260c336673898835f959a39
Summary:
Convert type comments to type annotations in eden/scm/edenscm/hgext
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden/scm/edenscm/hgext
```
from a parent directory.
Reviewed By: grievejia
Differential Revision: D34110682
fbshipit-source-id: acf8d32290411ca8673904e4f80ddd846ebceef0
Summary:
Convert type comments to type annotations in eden/scm/edenscm/mercurial
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden/scm/edenscm/mercurial
```
from a parent directory.
Reviewed By: grievejia
Differential Revision: D34110681
fbshipit-source-id: 3f21aa39de9384cdf7d5b09a232d9605eadc1481
Summary: Support the case of a snapshot modifying a symlink by removing the symlink destination before trying to create the symlink.
Reviewed By: yancouto
Differential Revision: D34097684
fbshipit-source-id: 48d4a05f61a69e686e5bd530a7ed150326a32765
Summary:
This case is getting hit quite a lot on NFS mounts on M1. This is filling up
the logs.
We need to investigate why this is getting hit so often since its a bit unexpected
how often this case is hit, but to reduce the load of the logs for now we can
turn off this log in production.
Reviewed By: fanzeyi
Differential Revision: D34112646
fbshipit-source-id: 96cc8e096c0cc3388f14ff2e667565f3bc3911b8
Summary:
We occasionally want to move master bookmark backwards.
Currently pull with segmented changelog fails after that.
This diff adds the integration test for that. I'm going to fix segmented changelog in the next diff, so this test would verify the fix.
Reviewed By: yancouto
Differential Revision: D33820377
fbshipit-source-id: e1a13b46de52b9abaea64a0211f3ee003ee40249
Summary: Unlike the python repo lock files, the rust lock files are not deleted when the lock is released. That means if the locks are created as root, they will break future commands run as other users. Try to mitigate that eventuality by chmoding the lock files to 0o666 on unix.
Reviewed By: quark-zju
Differential Revision: D34016719
fbshipit-source-id: 2bd1ec9d5da0e27a12eb95da5c2a4cb3f6286137
Summary:
Note: Admittedly fb303 args could still use some work, I'm not happy with
having to contruct the fb303args struct twice.
Reviewed By: markbt
Differential Revision: D33843863
fbshipit-source-id: 0694fe555f9dc8b167d89c6128c14ebd1aa00831
Summary:
The published `terminfo` crate has a WTFPL license which is forbidden from being used by Meta.
{F687594009}
However the WTFPL allows forking and relicensing, so I've gone ahead and relicensed the same code as dual MIT OR Apache 2.0 for fbsource to use. We can patch the properly licensed code in place of all transitive dependencies on the `terminfo` crate, as they have the same public Rust API.
The relevant changes: 9b4e1c6aae
Reviewed By: metajack
Differential Revision: D33190944
fbshipit-source-id: 0a9002b8799b5ceeaca3c20a211f15beec5a407f
Summary: Adds trace bus and other tracing components to PrjfsChannel. This will be used to enable tracing in the next diff.
Reviewed By: xavierd
Differential Revision: D33845993
fbshipit-source-id: a4666ebff3e535773786b220371e6cf2a0a18ea9
Summary:
While pending import matters, understanding how many requests are actually
being fetched is another very useful metric to have. Let's thus print it
alongside the pending in minitop.
Reviewed By: kmancini
Differential Revision: D34062527
fbshipit-source-id: b6a3938b8fb2b514f2f0a98cdce91d626912b0e3
Summary: Sometimes we can get errors, debugging info, or advices in headers returned from services and proxies. We know some headers that we can parse, but we don't have a comprehensive list. To improve debugging we might as well just print all headers of a response on failure. As of now we don't handle that in one place (like D33918785), because technically HTTP 5xx is a valid response from the POV of http library. It's the caller's logic that need to decide if it's a failure or not. In the future we could perhaps have such debugging logic in a trait like `ResponseExt` that could have `fn debug_failed_req` to be called by multiple callers if they decide a particular request was indeed an unrecoverable failure.
Reviewed By: quark-zju
Differential Revision: D34035084
fbshipit-source-id: 8247462ed9875b6530f215fb161c311325ea83fe
Summary: This was a breaking change, making it the default fixes the breakage.
Reviewed By: miaoyipu
Differential Revision: D34080461
fbshipit-source-id: 9d98391b0c8813896618bb2d833706bd5aa37b02
Summary: Added bubble id to FetchSnapshotResponse. Also updated hg client to store latest bubble information when updating snapshot.
Reviewed By: yancouto
Differential Revision: D34042287
fbshipit-source-id: 3c206caba4fa7b833cf1f8e23123b5b6006e2890
Summary: This was used to replay wireproto traffic. We hardly serve any wireproto traffic, and we don't log it anymore.
Reviewed By: farnz
Differential Revision: D34073952
fbshipit-source-id: 3ce5a6abb19d6079035ac81360928f73b2168374
Summary: This basically reverts D33791829 (6866997a01) as we don't need to retry on streaming clone anymore as the retry is done on the SQL crate since D33894164.
Reviewed By: farnz
Differential Revision: D33894165
fbshipit-source-id: 2974e359db077bad1c84f445e3444a070acadfd4
Summary:
Sometimes hg might recognise an untracked file as being added, but by the time it tries to load it, the file is gone, causing a crash.
For snapshots, it should be fine in this case to simply ignore this file. That is what this diff does. It specifically ignores `FileNotFoundError`, meaning other files still get classified as errors and we're not swallowing anything we shouldn't be.
Reviewed By: mitrandir77
Differential Revision: D34043083
fbshipit-source-id: cec7d528652c1704f3a3db707eafc0e43f24ac74
Summary:
This just adds a `with_context` call to when we're loading files from disk, so it prints which file failed to load.
I've seen several `[Errno 2] No such file or directory (os error 2)` and they're really annoying.
To do this, I had to refactor the lambda into a function, because try blocks are not stable yet :(.
Reviewed By: markbt
Differential Revision: D34043084
fbshipit-source-id: e0fd4142d75d6b392440f7763217a9b23ee8c8a4
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/101
public_autocargo already existed for edenfs, but needed to move up a level so to make edenfs thrift definition visible via autocargo.
Reviewed By: markbt
Differential Revision: D32693522
fbshipit-source-id: 4c070cfa2d6294a28c17697338f0309dcab1576f
Summary: Used the dirstate object to pull the current rebase related commits. If there exists a merge conflict, the commits are included in the error message (otherwise no change [e.g. non-conflicting interrupted rebase])
Reviewed By: quark-zju
Differential Revision: D34022102
fbshipit-source-id: 51bfacaf79cac5d4df4cdb99f8c92a493972ffd1
Summary:
This is meant to replace two modules:
* [the generic one from cmdlib](https://fburl.com/code/ij5fvr88)
* [the specialized one from mononoke/server](https://fburl.com/code/eyiaeij6)
I've been thinking whether I should put fb303 arg in the args/ module
but I'd rather have them closer to where they're used. I've reexportem them
there for convenience.
Reviewed By: markbt
Differential Revision: D33843864
fbshipit-source-id: 026201937456f75fcc869e9462dc72b17e7fca5e
Summary: This hook will allow any type of args to wrap the logging.
Reviewed By: markbt
Differential Revision: D33843866
fbshipit-source-id: ef7a7654f25714145cdc0cd2212aa0e3c7dd47b0
Summary:
I'm planning to add other hook points for extension so let's rename this one to
singnify the fact that it allows mutating environment.
Reviewed By: markbt
Differential Revision: D33843865
fbshipit-source-id: 62fd11e1e4b05570dc5b6ce7490ec1c9bd49f23b
Summary:
This restructures cmdlib_logging library to have args and all the function
using those args in the same file. This feels like better organization to me.
Reviewed By: kris1319
Differential Revision: D33842985
fbshipit-source-id: 686ce3b8bdacbdf7b422163c6539df8a3dd88f4a
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/103
Automate maintenance of the edenscm_* github actions yamls
Add job file and name options and support for the Rust install section
Reviewed By: fanzeyi
Differential Revision: D34044422
fbshipit-source-id: 7d5f07d37bab1eff5de30a88e710dbf7479ca192
Summary:
This is a straight replacement for the existing admin convert, with support for
svnrev and globalrev identity schemes added.
Reviewed By: yancouto
Differential Revision: D34044496
fbshipit-source-id: 7530e26c3cf4527430022d6796318c0578e89a44
Summary: Add `"extra"` as a way of specifying commit extras.
Reviewed By: yancouto
Differential Revision: D34044497
fbshipit-source-id: 9e3f71ddfeacbb20e17ce1c461d78f00d0102eec
Summary:
Now both IdDag stores have remove_flat_segment implemented, let's test the
resize feature based on remove_flat_segment.
Reviewed By: DurhamG
Differential Revision: D33821387
fbshipit-source-id: 5b92adadf8cc6989298b2637e3fb7141898c4503
Summary:
The `index_parent_key` function can derive `group` from its `child_id`
parameter. So let's drop the redundant parameter.
Reviewed By: DurhamG
Differential Revision: D33821382
fbshipit-source-id: 3a31c3fa5d4049ec36974421c21347cbdc6c473c
Summary:
The `StoreId` is an implementation detail of `InProcessStore`. It does
not apply to other kind of stores. Move it to a better place.
Reviewed By: yancouto
Differential Revision: D33821388
fbshipit-source-id: d7d20d9a61ebeea58e22f3f779fc9651097857df
Summary: Previously the assertions run per entry. Make them only run once.
Reviewed By: yancouto
Differential Revision: D33821389
fbshipit-source-id: 6a7380071a3f014d2034dfbc5760f94bd6d99dc2
Summary:
See previous diff for context. Implements segment deletion for
`InProcessStore`. Extra care needs to be taken for serialization.
Reviewed By: DurhamG
Differential Revision: D33821381
fbshipit-source-id: ace59345425988c2dae30b9bd55388679c36ae63
Summary:
See D33779403 for context. This is a step forward to implement "strip".
This diff defines the API that can be used to remove or "shrink"
a flat segment and adds some sanity checks.
The actual implementation is in upcoming diffs.
Testing will focus on "remove" first, "resize" tests will be added later.
Differential Revision: D33821386
fbshipit-source-id: 61848593e698fe7f9358b25a4cb72788637c50a5
Summary:
See D31146158 (39f61996c7) for reasons why SegmentWithWrongHead exists.
The tl;dr is the parent->child index in the indexedlog implementation (key:
(parent, child), value: segments) can point to old segments that were replaced.
Let's ignore the "segments" from the index and just do an extra lookup
using "child" to find the up-to-date (replaced) segment. This simplifies
upcoming changes and provides better error messages in some cases.
Differential Revision: D33821383
fbshipit-source-id: 3f63ac0e3a3953d0ddcccebe2be4de2a3a3e0589
Summary:
Upcoming changes need the ability to query ranged parent->child for the entire
graph, not just the master group. Update the range query API to do that.
The only user of `iter_master_flat_segments_with_parent_span` is
`IdDag::to_first_ancestor_nth_known_universally_with_errors` and it is fine with
the change. Even better, by returning non-master segments we can answer the
`to_first_ancestor_nth` query with non-master ids. That makes the server more
flexible, as shown by the added `test_lazy_hash_on_non_master_group` test.
Differential Revision: D33821379
fbshipit-source-id: 330bd84f8535f857f6306f9ce791066688a39d48
Summary: See the previous diff for context. Add `IdMapWrite::remove_range` to remove range of entries in an IdMap.
Differential Revision: D33779404
fbshipit-source-id: 24ede5949ad3c1cb0b93c3c64ca692c04b354457
Summary:
By changing the index key it allows us to delete specific parent->child
entries in the index without affecting parent->other_child indexes.
Long story:
This is the first step to implement in-place "strip".
Having a proper way to "strip" unblocks:
- Repalce costly "rebuild_non_master" with more precisely removals without
rebuilding anything.
- Remove problematic commits in the master group to work better with stripped
heads server-side.
- Remove broken commits in the non-master group which lost corrosponding
data (ex. commit message, or trees).
Previously there were a few blockers to "strip":
- IndexedLog is append-only.
- Segments must be continuous.
Now the blockers are "mostly" clear:
- IndexedLog's Index supports removal by key.
- Discontinuous segments supports punching "hole"s.
Once problem about the index removal is the parent index. Previously it
is a single key => multiple entries index. For example, parent 6 might
have children [7, 8, 9]. This makes it harder to just remove child 8
without affecting children 7 or 9. By making the index key contain
child. The index will look like (6, 7) => [7], (6, 8 ) => [8], (6. 9)
=> [9] and it's easier to remove just (6, 8 ) without affecting others.
Note the values in indexedlog is a linked list so it's in theory possible
to make indexedlog support removing just part of the values. But at this
point I'd like to implement it without changing indexedlog. We can always
go back and revisit the indexedlog fancier index removal later.
Differential Revision: D33779403
fbshipit-source-id: 4d73e7bdc66d3a8ec1482fd25f8ed5b9cc301aec
Summary:
With clap 3.0 being out, the structopt functionality has been integrated, and
structopt will no longer receive new features. Let's thus migrate to clap.
Reviewed By: fanzeyi
Differential Revision: D34050818
fbshipit-source-id: 5ca4fec5397b7f47f8fb7cd73ea9a8d5adc85161
Summary:
This makes use of the new Rust repo initialization added in previous commits into the `init` command in Python by replacing the initialization that was done mostly by the `localrepo` object. The codepaths for `git` and `bundle` remain active.
Another smaller addition was the option of removing `narrowheads` from the list of requirements if `experimental.narrow-heads` was explicitly disabled in the configurations.
Reviewed By: quark-zju
Differential Revision: D33901725
fbshipit-source-id: d9d976a1b844e1dd6a5e065f8ec8bf1d04fc4c03
Summary:
'x~-1' should be equivalent to 'children(X)' when there is one visible
child. Today, ~-1 doesn't take visibility into account, so if there are
invisible descendants it will error out.
This diff makes ~-1 only return visible results, which matches the children
behavior.
Reviewed By: quark-zju
Differential Revision: D33900217
fbshipit-source-id: 0b4bb015c03fa3498746e6a754a522693d0c2edf
Summary:
We've seen cases in CI where the length of the path to the socket is just over
108 characters, breaking tests. Let's shave 10 characters to be under the
limit.
Reviewed By: fanzeyi
Differential Revision: D34046658
fbshipit-source-id: 249b003db2a1f18f53c28702d81577ff55dcb237
Summary: These could be part of a path traversal attack. Filter them at the edge.
Reviewed By: mitrandir77
Differential Revision: D33794130
fbshipit-source-id: 729def7d166e24be1343b867afd4857646ebfd46
Summary: The walker is being migrated onto the new clap lib.
Reviewed By: markbt
Differential Revision: D33924390
fbshipit-source-id: 4d1a8b6eae9e1f577dc531cd47644875362f1177
Summary:
Prefetch code paths execute the prefetch as one huge batch, and it
turns out the lfs fetcher only fetches one file at a time. Let's increase the
parallelism
Note, checkout is less affected by this, since nativecheckout issues prefetches in smaller batches and in parallel, so we get some parallelism at a higher layer, but I do see a 15% speed up in a large file heavy repository.
Reviewed By: quark-zju
Differential Revision: D34020837
fbshipit-source-id: fb11c1aebf175b49563f48cb20f8b7c527fa3019
Summary:
Since watches are always added at the end of the list, and time is
monotonically increasing, requests are implicitely ordered by their running
time in the list, with the longest running one at the front. Thus in order to
compute the max duration, we can simply look at the front of the list.
Reviewed By: kmancini
Differential Revision: D33994895
fbshipit-source-id: 3c4f10da60b7554126a08aad7042cc4c712699c3
Summary:
We don't have metrics for how long trees/blobs are fetched, let's add some
metrics.
Reviewed By: kmancini
Differential Revision: D33994268
fbshipit-source-id: 7f6931e11c5b226bb2d9c39f94a6f762338ee7ae
Summary:
On macOS, the crash dumps can also be in the home directory, let's collect
them.
Reviewed By: MichaelCuevas
Differential Revision: D34019181
fbshipit-source-id: 5d119ad87c051f0fd04cd7aa3454f44aa5b3e1eb
Summary: It moved from facebookexperimental to facebook sometime ago
Reviewed By: fanzeyi
Differential Revision: D34009265
fbshipit-source-id: c11ca0e92d9574aa8eccd31d578b9f61a411833c
Summary: clear the decks, much of this likely to rebase away
Reviewed By: jsgf
Differential Revision: D34006468
fbshipit-source-id: aeda14f2c77f9517b7cdff612702301dc6fb16e4
Summary:
Unfortunately, Mercurial changed a lot since then and is unable to understand
the data stored in it. Let's simply remove it, the 2021 snapshot is still
working and we should keep it as such.
Reviewed By: fanzeyi
Differential Revision: D34010250
fbshipit-source-id: 3c890a232d7c87576276933b1430adfe5e3a45f6
Summary:
The original had to be reverted due to causing checkout to recursively call
into EdenFS which was making EdenFS deadlock. These issues have now been solved
and even if EdenFS were to recursively cause a callback to be triggered, this
wouldn't lead to a deadlock.
To make sure this is true, let's have the fast code behind a config so we can
roll it slowly to user and roll it back immediately if needed.
Reviewed By: genevievehelsel
Differential Revision: D33950740
fbshipit-source-id: 2a91e4aabc7466bb710ec7c65d3c9a10c972842b
Summary: It was only printing local bookmarks, this makes the test clearer.
Reviewed By: mzr
Differential Revision: D33917181
fbshipit-source-id: f3b93e20bcd2f0404cb517a1a22605a57e80f77a
Summary: Refactor manual random retry logic to use common library for exponential retry
Reviewed By: yancouto
Differential Revision: D33987857
fbshipit-source-id: e5d1ce73f3eae08640cbd5b15c29e32cf3a92736
Summary:
As part of the effort of mononoke team to start using buck2, this diff makes it so the incremental test run also works with buck2.
It should be less necessary with buck2, as it doesn't waste time on startup, as buck1 did, but this can still be useful, when e.g. you change code in a common part of codebase (like blobstore), but are only testing this in the mononoke server, so you can just build mononoke again instead of everything that depends on blobstore.
Since the paths get a little complicated, and change between buck1 and buck2, I made helper scripts to do the incremental runs, which frees the coder from needing to know where those files are, and I added instructions to the README on how to use them. Basically just:
```
> ./incremental_integration_setup.sh group (once)
> buck2 build what_you changed
> ./incremental_integration_run.sh group test.t (repeat)
```
Reviewed By: mzr
Differential Revision: D33846052
fbshipit-source-id: 88f11800c7f29d9cd37d4e431ed710981603368a
Summary: Metadata doesn't depend on ssh at all. Let's move it out of there.
Differential Revision: D33893834
fbshipit-source-id: 09bf87ec3733cf8ea3b365d2f43c5d364b3bdfc7
Summary:
Certain git bundles might contain references like "BUNDLE_HEAD" that would be
ignored by usual paths. Normalize them to visibleheads refs.
Differential Revision: D33968793
fbshipit-source-id: a3d6fc1a7d73328cddf0ebddfe7b7a021d929528
