Summary:
See the previous diff for context. This allows the code to run from non-main
thread.
Reviewed By: singhsrb
Differential Revision: D23759712
fbshipit-source-id: 044193a9d7193488c700d769da9ad68987356d69
Summary:
The idea is to extend D22703916 (61712e381c)'s way of calling functions from just edenapi to
the entire command for better Ctrl+C handling. Some code paths (ex. pager,
crecord) use `signal.signal` and `signal.signal` does not work from non-main
thread.
To workaround the `signal.signal` limitation, we pre-register all signals we care
about in the main thread to a special handler. The special handler reads a
global variable to decide what to do. Other threads can modify that global
variable to affect what the special signal handler does, therefore indirectly
"register" their handles.
Reviewed By: kulshrax
Differential Revision: D23759711
fbshipit-source-id: 8ba389072433e68a36360db6a1b17638e40faefa
Summary:
Before this change, for a long-running function wrapped by 'threaded',
it might:
background thread> start
main thread> receive SIGINT, raise KeyboardInterrupt
main thread> raise at 'thread.join(1)'
main thread> exiting, but wait for threads to complete (Py_Finalize)
background thread> did not receive KeyboardInterrupt, continue running
main thread> continue waiting for background thread
Teach `thread.join(1)` to forward the `KeyboardInterrupt` (or its subclass
`error.SignalInterrupt`) to the background thread, so the background thread
_might_ stop. Besides, label the background thread as daemon so it won't
be waited upon exit.
Reviewed By: kulshrax
Differential Revision: D23759713
fbshipit-source-id: 91893d034f1ad256007ab09b7a8b974325157ea5
Summary:
Move the wrapper to util.py. It'll be used in dispatch.py to make the entire
command Ctrl+C friendly.
Reviewed By: singhsrb
Differential Revision: D23759715
fbshipit-source-id: fa2098362413dcfd0b68e05455aad543a6980907
Summary: This will be used to test Ctrl+C handling with native code.
Reviewed By: kulshrax
Differential Revision: D23759714
fbshipit-source-id: 50da40d475b80da26b7dbc654e010d77cb0ad2d1
Summary: This makes it easier to test the API via debugshell.
Reviewed By: kulshrax
Differential Revision: D23750677
fbshipit-source-id: e29284395f03c9848cf90dd2df187e437890c56e
Summary: It is handy to test edenapi methods directly.
Reviewed By: kulshrax
Differential Revision: D23750709
fbshipit-source-id: 33c15cecaa0372ba9e4688502e7d8f3fdda7c3b8
Summary:
Add a command to rebuild the changelog without recloning other parts of the
repo. This can be used as a way to recover from corrupted changelog. It
currently uses revlog because revlog is still the only supported format during
streamclone.
In the future this can be used for defragmentation.
Reviewed By: DurhamG
Differential Revision: D23720215
fbshipit-source-id: 6db0453d18dbf553660d55d528f990a4029d9da4
Summary:
This commit moves a compile-time template parameter
to be a runtime boolean parameter.
There's a bit of fan-out that, while I don't think it is
super awesome, isn't super terrible either.
The case sensitivity value is read from the checkout config
added in the prior diff in this stack.
Reviewed By: xavierd
Differential Revision: D23751192
fbshipit-source-id: 46f6fe25bfa6666305096ad9c416b510cd3aac8f
Summary:
This diff teaches the CheckoutConfig how to determine
whether a given checkout should be case-sensitive (the default)
or case-insensitive-case-preserving.
This option is passed through to the fuse channel initialization,
so that the kernel will respect it, however, our DirEntry layer
doesn't yet know that it should respect this.
There's currently no UI to set this option. My game plan
is to suggest the following steps to folks that want to try
this out:
```
$ eden stop
$ vim ~/local/.eden/clients/ovrsource/config.toml
```
and then add this line to the `[repository]` section:
```
case-sensitive = false
```
and finally:
```
$ eden start
```
Reviewed By: xavierd
Differential Revision: D23751184
fbshipit-source-id: 6facb23c460cfff6e37d0091b51b97ab06f62c91
Summary:
Improve help to reflect that the system is also meant for managing backups
add missing commands
reshuffle a bit
Reviewed By: markbt
Differential Revision: D23782794
fbshipit-source-id: d7fd3fa06ca7acd649cef557f3fe020295259e3d
Summary:
Compressed responses from LFS are slower than they should right now. Normally,
we'd expect something along the lines of normal response time + compression
time, but right now it's a lot more than this.
The reason for this is that our compressed streams are eager, i.e. they will
consume and compress as much of the underlying stream as possible before
sending off the data. This is problematic for LFS, because we try very hard to
serve everything out of RAM directly (and very often succeed), so that means
we compress the whole stream before sending it off.
This means we might spend e.g. 500ms compressing (this is how long it takes
zstd to compress the object I was testing on, which is a ~80MiB binary that
compresses down to 33% of that), and _then_ we'll spend some time transferring
the compressed data, when we could have started transferring immediately while
we were compressing.
To achieve this, let's simply tell our compressed stream to stop waiting for
more data once in a while (every 4 MiB, which seems very frequent but actually
really isn't).
Reviewed By: StanislavGlebik
Differential Revision: D23782756
fbshipit-source-id: a0d523d84f92e215eb366f551063383fc835fdd6
Summary:
I saw this throw the LFS server into an infinite loop when I tested it. We're
not using this right now, so I'm not investing time into root-causing the
issue, and instead let's just take this out.
Reviewed By: StanislavGlebik
Differential Revision: D23782757
fbshipit-source-id: f320fc72c3ff279042c2fe9fcb9c4904e9e1bfdf
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/51
This diff extends capabilities of CargoBuilder in getdeps so that individual manifests can be build even without workspaces. Thanks to that a build for edenapi/tools can be made and its artifacts can be used in mononoke integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D23574887
fbshipit-source-id: 8a974a6b5235d36a44fe082aad55cd380d84dd09
Summary: The command will be provided as hint if username changes has been detected in configuration.
Reviewed By: markbt
Differential Revision: D23769942
fbshipit-source-id: 3e84ecef6dd68267022b92bf10f5e68dfc07f270
Summary:
This makes deletion commits a bit less confusing, but it also have another
benefit.
Without the sort some directories might have been changed multiple times in
deletion commits e.g. if a directory had 5 files, and these files were deleted
in 5 different deletion commits then the directory would be changed 5 times.
This was not good, because it made some data derivation slower (in particular,
fastlog), because it had to regenerate the same data over and over again.
Reviewed By: ikostia
Differential Revision: D23780066
fbshipit-source-id: d5c52b13f58dcaf2012d9c12bf77398561cf10ef
Summary:
Spotted a TODO in fsnode get_fsnode_id. There was only one user of the function, which didn't really need to call it as it had the blob already.
As well as being a bit tidier this also saves a clone of the fsnode.
Reviewed By: StanislavGlebik
Differential Revision: D23758689
fbshipit-source-id: e0a8c124c929fda3af4c96a76d441a79e5bfbd5b
Summary:
Save memory in walker state tracking by not memoing hash values. For large repos this is significant.
I was expecting a small slowdown from this, but so far looks pretty much the same walk rate. Speculation: this may be due to the num cpus lock sharding fix in dashmap 3.11.10 which means there are many more shards than when the memo was tested with 3.11.9, so saving time inside locks is less significant.
Reviewed By: StanislavGlebik
Differential Revision: D23680550
fbshipit-source-id: 351b5ec39885fc30996207c7dccc22c749e30321
Summary:
The `gotham_ext::response` module was getting a bit large, so this diff moves `ContentMeta`, `ContentStream`, and `CompressedContentStream` into a new submodule, alongside the contents of the old `content_encoding` module. This way, the `response` module remains entirely centered around the `TryIntoResponse` trait (and the various body structs that implement that trait).
Later diffs in this stack will be adding an additional layer between the content streams and the body structs, at which point it probably doesn't make sense to have these right next to each other. Splitting them out now will allow for better code organization going forward.
Reviewed By: krallin
Differential Revision: D23777492
fbshipit-source-id: 86e598dcb37578d3b22217a2a65f1bde84d72215
Summary:
`scratch` provided by `fb-scratch` was replaced by `mkscratch` provided by
the Mercurial package. See linked task for details.
Reviewed By: quark-zju
Differential Revision: D23773840
fbshipit-source-id: de0582069ce1a09c3cd9fc6b02d2d149f70d0d78
Summary:
Computing all successorsets is exponential with the number of splits
that have happened. This can slow things down tremendously.
The obsoletenodes path only needs to know "is there a visible successor" in
order to determine if a draft commit is obsolete. Let's use allsuccessors
instead of successorset.
Reviewed By: quark-zju
Differential Revision: D23771025
fbshipit-source-id: 666875e681c2e3306fc301357c95f1ab5bb40a87
Summary:
`hg cloud join --merge` doesn't really solve rename problem because it doesn't
preserve:
1. old heads
2. history
I added a proper API in Commit Cloud Service for renaming workspaces and now we
can use it to provide a rename command and 'rehost' command which is a version
of renaming to bind the current workspace to the current devserver.
Rehost command is meant to be used after dev server migration. I am plannig to
add this to the dev server migration wiki.
Next diff will cover how we can use the rename command to fix a username in workspaces names after username has been changed.
Reviewed By: markbt
Differential Revision: D23757722
fbshipit-source-id: dc11cb226eb76d347cdab70b3c72566448dcd098
Summary:
The Rust contentstore has no way to flush the shared stores, except
when the object is destructed. In treemanifest, the lifetime of the shared store
seems to be different from with files and we're not seeing them flushes
appropriately during certain commands. Let's make the flush api also flush the
shared stores.
Reviewed By: quark-zju
Differential Revision: D23662976
fbshipit-source-id: a542c3e45d5b489fcb5faf2726854cb49df16f4c
Summary:
Now that treemanifests can use Rust stores, we need to update the
Python repack code to support that.
Reviewed By: quark-zju
Differential Revision: D23662361
fbshipit-source-id: c802852c476425eef74181ead04f70b11ff9a27c
Summary:
This makes Rust contentstore prefetch route through the remotetreestore
prefetch logic to reach the lower level tree fetching, and makes the higher
level Python fetching route through the Rust contentstore to do prefetching. The
consequence of this is that there's a relatively unified code path for both
Python and Rust, and hopefully we can delete the janky Python bits once we're
completely migrated to Rust.
The way this diff works is pretty hacky. The code comment explains it, but the
tl;dr is that Rust prefetch works by providing references to the mutable stores,
while Python prefetch assumes they are stored and accessible on the repository.
Inorder for the old python tree fetching logic to work with both models, we
monkey patch the Rust mutable store references we receive into the function that
will later be called to request the repositories mutable stores. This is awful.
A cleaner fix might be to thread the mutable stores all the way through the
python fetching logic, then move the Python accessing of the repositories
mutable stores to the higher layer, near where Rust would provide it. That's a
lot of code churn though, so I'd like to do that in a later diff once we stop
using the non-rust logic entirely.
Reviewed By: quark-zju
Differential Revision: D23662351
fbshipit-source-id: 76007b6089ddf0e558581cd179a112311f8b58e3
Summary:
As part of moving treemanifest to use the Rust tree store, we need to
move prefetch to be able to be initiated from Rust. Rust requires a certain
signature for the prefetch function which accepts multiple keys.
In preparation for this requirement, let's refactor the current remotetreestore
fetching path to have a separate function. In a later diff we'll route Rust
prefetch requests through this function so the python and rust code shares the
same base tree discovery logic.
Reviewed By: quark-zju
Differential Revision: D23662196
fbshipit-source-id: 127045c279dc22914f7e1f3a619f6620586010ba
Summary:
Python 3 reports exceptions in except clauses by showing the original
exception, then saying another exception happened during the original exception
and hiding the second exception stack trace.
To make update exceptions more debuggable, let's move the handling outside the
except clause.
Reviewed By: quark-zju
Differential Revision: D23761667
fbshipit-source-id: bec758a3c7c0b88a5a569f794730058bf6f1eaad
Summary:
Rather than switch all of Eden at once to the thrift-py3 client,
rename get_thrift_client to get_thrift_client_legacy so uses of the
new client can be introduced piecemeal.
(I did try migrating everything at once but it's been quite painful.)
Reviewed By: fanzeyi
Differential Revision: D22423399
fbshipit-source-id: 9e6d938b90fff9fc3266ba20bc77e880e7f5b1aa
Summary:
This is the initial step to track username when the workspace has been created and provide users an appropriate advice how to fix their workspace names if the username in configuration has been changed
in another diff I will provide the advice itself
I will build rename workspace command based on D23703790
Reviewed By: markbt
Differential Revision: D23730312
fbshipit-source-id: a49dabba7ec4acf35f6ff99ed23cff5d6f46e2e4
Summary:
`experimental.template-new-builtin = true` has been rolled out to 100%
and seems to work fine. Therefore, remove code that supports
`template-new-builtin = false`.
Reviewed By: singhsrb
Differential Revision: D23745353
fbshipit-source-id: 178af269381c9d3e20522ba4484d63051589342b
Summary:
Some tests run `hg init` right inside the test directory, turning the
entire $TESTTMP into a repo. In future diffs we'll start to rely more on hgcache
being present during tests, which creates a directory in $TESTTMP. Let's make
sure all repos are created as sub-directories of $TESTTMP.
Reviewed By: kulshrax
Differential Revision: D23662077
fbshipit-source-id: 2b2b974ebfd1bd19ad6acd1ebe3e68dd03a09869
Summary:
Adds the initial condition and creation logic for creating a Rust
treemanifest store. Fetching and some other code paths don't work just yet, but
subsequent diffs enable more and more functionality.
Reviewed By: quark-zju
Differential Revision: D23662052
fbshipit-source-id: a0e7090c9a3bf27a7738bf093f2d4eb6098b1ed6
Summary: The old logic would just double pack some bits. Let's prevent that.
Reviewed By: xavierd
Differential Revision: D23661933
fbshipit-source-id: 155291fa08ec2c060619329bd1cb6040769feb63
Summary:
The rust pack stores currently have logic to refresh their list of
packs if there's a key miss and if it's been a while since we last loaded the
list of packs. In some cases we want to manually trigger this refresh, like if
we're in the middle of a histedit and it invokes an external command that
produces pack files that the histedit should later consume (like an external
amend, that histedit then needs to work on top of).
Python pack stores solve this by allowing callers to mark the store for a
refresh. Let's add the same logic for rust stores. Once pack files are gone we
can delete this.
This will be useful for the upcoming migration of treemanifest to Rust
contentstore. Filelog usage of the Rust contentstore avoided this issue by
recreating the entire contentstore object in certain situations, but refresh
seems useful and less expensive.
Reviewed By: quark-zju
Differential Revision: D23657036
fbshipit-source-id: 7c6438024c3d642bd22256a8e58961a6ee4bc867
Summary:
Instants do not represent actual time and can only be compared against
each other. When we subtracted arbitrary Durations from them, we run the risk of
overflowing the underlying storage, since the Instant may be represented by a
low number (such as the age of the process).
This caused crashes in test_refresh (in the next diff) on Windows.
Let's instead represent the "must rescan" state as a None last_scanned time, and avoid any arbitrary subtraction. It's generally much cleaner too.
Reviewed By: quark-zju
Differential Revision: D23752511
fbshipit-source-id: db89b14a701f238e1c549e497a5d751447115fb2
Summary: Previously, we used the call sign of the repo we import when checking any of the if the commits are parsed by Phabricator. However, we also use this callsign for other repos when checking Phabricator, which is an incorrect implementation. E.g. if fbsource back-syncs to ovsource, we would have used FBS callsign when checking Phabricator for both fbsource and ovrsource, but we should use OVRSOURCE callsign for repo ovrsource. This diff corrects this implementation by saving the callsigns of the small repos in their SmallRepoBackSyncVars.
Reviewed By: StanislavGlebik
Differential Revision: D23758355
fbshipit-source-id: b322acb2ec589eabed5362bfd6b963e2dd1d6ea9
Summary:
I had originally make the logic around ECHILD very strict,
thinking it impossible to have a situation where it may arise,
but it turns out that our daemonization makes this happen all
the time.
This commit treats an ECHILD return from waitpid as equivalent
to a success waitpid result and success child process termination.
Reviewed By: chadaustin
Differential Revision: D23683107
fbshipit-source-id: 7867d636afd8ee79b9f100454f84e7ef480109d8
