Summary: Since this is only used in the manifest target, fold it into it.
Reviewed By: DurhamG
Differential Revision: D24062629
fbshipit-source-id: c3241b53bde7abba8a80a2945661d1a24b7e3034
Summary: We now get progress bar output when fetching from memcache!
Reviewed By: kulshrax
Differential Revision: D24060663
fbshipit-source-id: ff5efa08bced2dac12f1e16c4a55fbc37fbc0837
Summary: These aren't included anywhere, we can remove them.
Reviewed By: DurhamG
Differential Revision: D24062627
fbshipit-source-id: 9ff101eb44965ac3502ada3265ffcc8acc09d2e5
Summary:
This is no longer about datapack, but only about parsing manifest entries, thus
renaming.
Reviewed By: DurhamG
Differential Revision: D24062634
fbshipit-source-id: 5c52b784d20437e87012dd4bc6cb13d879da9cb9
Summary: The code doesn't use anything from libmpatch, we don't need to depend on it.
Reviewed By: DurhamG
Differential Revision: D24055084
fbshipit-source-id: 0f7bac73f1b711da4395e25619577a0a2e0ca959
Summary: These are unused, no need to keep the code around.
Reviewed By: DurhamG
Differential Revision: D24055085
fbshipit-source-id: 6246d746983a575c051ddcb51ae02582a764a814
Summary: This is unused, no need to keep it around.
Reviewed By: DurhamG
Differential Revision: D24054164
fbshipit-source-id: 161b294eb952c6b4584aa0d49d8ff46cd63ee30f
Summary:
This code is effectively unused. The only bit still relevant is that EdenFS
still depends on the Manifest class to parse a manifest.
Reviewed By: DurhamG
Differential Revision: D24037723
fbshipit-source-id: 901ae2ffc8960a95ec655a2e14d79afb8d32dcab
Summary: This is unused, let's remove it.
Reviewed By: DurhamG
Differential Revision: D24037722
fbshipit-source-id: bc8a272809cb1f20f54d651a39ee42ff57169534
Summary:
The 'diff()' template needs to be able to output non-utf8 patches.
To enable this, let's change the templater to output bytes instead of strings.
Note, the templater supports json output, which requires utf8 encoded values. In
that situation we escape any non-utf8 binary and clients would have to be aware
and unescape them.
Reviewed By: quark-zju
Differential Revision: D24029105
fbshipit-source-id: aa011696c0d97281b91c6871c116037d76c178e0
Summary:
This is unused code. It broke a test while I was doing python 3 work,
so let's just delete this.
Reviewed By: quark-zju
Differential Revision: D24063953
fbshipit-source-id: 323b323ff2b9717c9c16eed54adac467a321e2ae
Summary:
From just scanning through the code I identified a couple more spots
that use "" when they should use b"". I also noticed that
_parsecompressedrevision was producing memoryviews instead of bytes, so it's "u"
check would always fail.
Reviewed By: quark-zju
Differential Revision: D24037437
fbshipit-source-id: b4c589e30b5d35a1bcee16e0d6aa37d04f30129d
Summary:
The eden extension was hard coding empty string as the path when
reading from the store. This works fine when the data is available in a local
store (since most data stores don't actually use the path component), but
doesn't work when it needs to go to the server.
This wasn't an issue historically because the above fetch did the fetch for it.
A recent code change caused the above fetch to throw an error though, which was
then eaten, which caused the get code path to try to perform the fetch with an
empty path.
Reviewed By: krallin
Differential Revision: D24020467
fbshipit-source-id: cad717618e947fbdca978b45ce2181d46422d456
Summary:
Check a few backup files for `hg cloud sl` command information.
I added a comment explaining why we would need this.
At the moment some commands like for example `hg cloud switch` or the `hg pull -r` goes through mononoke backend and they will update a different backup cache file comparing with the `hg cloud sync` command that goes through the mercurial backend (write path).
As a result, `hg sl` command displays that some commits haven't been backed up.
Reviewed By: markbt
Differential Revision: D24046449
fbshipit-source-id: 56d924bf641833231cda6e6701e67f63c8453894
Summary:
All the current workspaces should be already populated because the release has
been running for a while. Remove the migration code and also fix a bug with
string to boolean conversion.
Reviewed By: mitrandir77
Differential Revision: D24045272
fbshipit-source-id: 347f0f46d343a13fc1d9f762f912af364813a66f
Summary: Disable CLANGTIDY checks for several places at the code.
Reviewed By: zertosh, benoitsteiner
Differential Revision: D24018176
fbshipit-source-id: b2d294f9efd64b2e2c72b11b18d8033f9928e826
Summary:
This would have been easier if we can upgrade tokio (D24011447).
For now, let's just solve it by using a channel so the mutex is not held for long.
The implementation has some side effects, though:
- panic message is not preserved.
- 'static lifetime is required on Future.
The `'static` lifetime is incompatible with some existing code. The old function
is preserved as `block_on_exclusive` and is used in places where a future does
not have `'static` lifetime.
Reviewed By: sfilipco
Differential Revision: D24033134
fbshipit-source-id: 7b35d1ff636d2a289db9b04e60419c31bdea9453
Summary: Now that we progress bars in Rust, add them to the EdenAPI client bindings and remove any existing progress bars around callsites in the Python code.
Reviewed By: quark-zju
Differential Revision: D24037797
fbshipit-source-id: eb26ccaae35ab23eb76f6f2b2be575a22e1f1e53
Summary: Larges files are chunked and the initial buffer should be bytes.
Reviewed By: quark-zju
Differential Revision: D24034645
fbshipit-source-id: 98156b1901182b345baaeb82c71faeb3cc57b131
Summary: Log changelog backend types so we can filter commands by type.
Reviewed By: DurhamG
Differential Revision: D24022284
fbshipit-source-id: c402aea0ce3bd20d0f310fea167f24cb1b7a3ae6
Summary:
On Windows, some Mercurial commands may create files in `.hg` directory even if
EdenFS is not running. As a result, the command itself will fail while the file
still left in the directory. Users typically will then start EdenFS and repeat
the same command.
However, due to the current design of EdenFS, it will not be able to recognize
these files created when it is stopped and return errors for any attempts to
write/remove the file, generate another error. Users then need to stop EdenFS
and manually remove the file in `.hg` directory to recover the repository. This
creates a very bad user experience.
The correct way to fix this is to teach EdenFS to track the modifications
happened when it is not running, however this will take a few weeks to get
there. For a temporary measurement, we teach Mercurial to abort when EdenFS
isn't running to avoid trickier recoveries.
Reviewed By: quark-zju
Differential Revision: D24001090
fbshipit-source-id: abc1ebcdae3819756fe64b5321f52a6e62c0c360
Summary:
Add a null progress bar implementation that just keeps track of state, similar to the `progress.nullbar` in hg's Python code.
A benefit of this is that code that optionally shows progress can unconditionally update the progress bar rather than wrapping it in an `Option` and checking for presence each time.
Reviewed By: markbt
Differential Revision: D23982318
fbshipit-source-id: ffd762b59cc0c9bd2ad0c67c3ca785350db4850f
Summary: Add Python bindings to the Rust progress wrappers. This may seem pointless since the Rust code just calls right back into Python, but this is a useful step to get the Rust and Python code to use a common interface for progress. (Which, in turn, will allow switching to a Rust progress implementation down the line.)
Reviewed By: markbt
Differential Revision: D23999816
fbshipit-source-id: 9bca0f23170d3ca474a1cb5d547840e63572ec71
Summary: Add Rust wrappers around Mercurial's Python `progress` module, allowing Rust code to create and use Python progress bars. The wrapper types implement the traits from the `progress` crate, so they can be passed to pure Rust crates in `scm/lib`. In typical usage, the Rust bindings will create a `PyProgressFactory`, which will be passed to pure Rust code as a trait object or via generics.
Reviewed By: markbt
Differential Revision: D23982317
fbshipit-source-id: 4c0fde0b2423b6449c7c5155fdfd98f5da042b0d
Summary:
This diff introduces a new `progress` crate that provides an abstract interface for progress bars in Rust code:
- The `ProgressFactory` trait can be used to create new progress bars.
- The `ProgressBar` trait allows Rust code to interact with the progress bar.
- The `ProgressSpinner` trait is similar, but for spinner-type progress indicators.
These traits are intended to be used as trait objects, allowing pure Rust code to accept an opaque `ProgressFactory` and use it to report progress. This kind of abstraction, while not common in idiomatic Rust code, allows the progress implementation to be completed decoupled from the pure Rust code, which is important given that Mercurial's progress bars are currently implemented in Python.
Part of the goal of this crate is to allow a smooth transition to pure Rust progress bars (once we eventually implement them). As long as the Rust progress bars implement the above traits, the can be used as drop-in replacements for Python progress bars everywhere.
Reviewed By: markbt
Differential Revision: D23982319
fbshipit-source-id: 9ccf167f18d9518bb0ed66e1606a5b8188d98428
Summary:
The protocol for getpack is length-prefixed. However, we currently advertise
the number of characters in filenames instead of their byte length. So, the
lengths we send don't necessarily correspond to the amount of data we send.
Indeed, if a filename contains multibyte characters, we'll advertise a lower
byte count than what we actually end up sending. This results in the last
byte(s) of the filename being interpreted by Mononoke as the start of another
piece of data, and eventually causes Mononoke to hang as it waits for more data
that the client will never send.
This fixes that bug in reading, and also fixes an identical instance of the bug
on the server side. I also double checked the gettreepack code, which AFAICT
doesn't have this bug.
Reviewed By: ahornby
Differential Revision: D24013599
fbshipit-source-id: af716f2bf9c02d312c0c8d2f449988e8f8858ab8
Summary:
The `hg cloud switch` command could be nicely used to debug other people workspaces by the Source Control Team.
Sometimes broken workspaces. This option would allow us to switch back from a broken workspace to our original workspace.
Mostly useful for the Source Control Team
Reviewed By: markbt
Differential Revision: D24014167
fbshipit-source-id: f2116cc13897149c8ac79790a31ebcce1f18a260
Summary:
With the new tracing-core (0.1.10 -> 0.1.16), it's no longer effective
to set EDENSCM_TRACE_LEVEL to more verbose within the test.
Let's set it in run-tests.py which spawns the test process instead.
Reviewed By: kulshrax
Differential Revision: D24004327
fbshipit-source-id: db2cadc7334eb59b25a1e0517e3d1a513e31e0fe
Summary:
As EdenFS depends on a few bits of Mercurial code, these needs to be able to
compile with Buck.
Reviewed By: chadaustin
Differential Revision: D24000881
fbshipit-source-id: 078a2a958039a63db1b716785f872b4bbde3bab6
Summary:
This parameter is gone. Let's stop using it. For now we can achieve the
same result by setting the treemanifest config value. In the long term we'll
probably get rid of depth in favor of smarter algorithms, like bfs traversals.
Reviewed By: genevievehelsel
Differential Revision: D23971898
fbshipit-source-id: cabcf0c088c95557edfe07ae85ce7d07e55a3082
Summary: Make `parents`, `data`, and `metadata` optional, and introduce `WireTreeAttributesRequest` for selecting which attributes to request on the wire.
Reviewed By: kulshrax
Differential Revision: D23406763
fbshipit-source-id: 5edd674d9ba5d37c23b12ab4d7b54bbf6c9ff990
Summary:
Adds a `WireTreeQuery` enum for query method, with a single `ByKeys(WireTreeKeyQuery)` available currently, to request a specific set of keys.
Leave the API struct alone for now.
Reviewed By: kulshrax
Differential Revision: D23402366
fbshipit-source-id: 19cd8066afd9f14c7e5f718f7583d1e2b9ffac02
Summary: The size can change with zstd upgrades. Do not test them.
Reviewed By: sfilipco
Differential Revision: D23976933
fbshipit-source-id: d560061b6e4fefc3bb89513bdb12c770ea0bd881
Summary:
This makes it so commit hashes are serialized to bytes instead of tuples in Python:
In [1]: s,f=api.commitdata(repo.name, list(repo.nodes('master')))
In [2]: list(s)
Out[3]: [{'hgid': '...', ...}]
Some `Vec<HgId>`s cannot be changed using this way. It'd be nice if we can change
the default `HgId` serialization to bytes.
Reviewed By: kulshrax
Differential Revision: D23966989
fbshipit-source-id: 4d013525419741d3c5c23621be16e70441bab3c4
Summary:
`HgId` currently serializes into a tuple of 20 items. This is suboptimal in
CBOR, because the items are untyped. A byte might be serialized into one or two
bytes:
In [2]: cbor.dumps([1,1,1,1])
Out[2]: b'\x84\x01\x01\x01\x01'
In [3]: cbor.dumps([255,255,255,255])
Out[3]: b'\x84\x18\xff\x18\xff\x18\xff\x18\xff'
CBOR supports "bytes" type to efficiently encode a `[u8]`:
In [5]: cbor.dumps(b"\x01\x01\x01\x01")
Out[5]: b'D\x01\x01\x01\x01'
In [6]: cbor.dumps(b"\xff\xff\xff\xff")
Out[6]: b'D\xff\xff\xff\xff'
Add `serde_with` with 3 flavors: `bytes`, `tuple`, `hex` to satisfy different
needs. Check the added docstring for details.
Reviewed By: kulshrax
Differential Revision: D23966992
fbshipit-source-id: 704132648f9e50b952ffde0e96ee2106f2f2fbcf
Summary: This just adds some explanation of `-r` to the `hg sparse list` docstring.
Reviewed By: markbt
Differential Revision: D23961027
fbshipit-source-id: 64ab406b07fe5d66fd53d4e520935aad3b0b351b
Summary:
Dynamicconfig can generate configs two ways, 1) via `hg
debugdynamicconfig` and 2) synchronously in-process in an hg command when it
detects that the dynamicconfig is completely missing or has the wrong version
number.
In the first case, dynamicconfig gets the repo name from the standard config
object loaded by the hg dispatch. In the second case, the standard config
object isn't even loaded yet, so dynamicconfig does a mini-load of the user and
repo hgrcs so it can get the repo name and user name (needed for dynamic
conditions).
Unfortunately the second code path computed the wrong path (it had two .hg/'s)
which meant the reponame and user name were always none. This meant that the
dynamicconfig on disk could randomly be either computed with or without a
reponame.
Let's fix the path computation, and add a test. We may want to make
dynamicconfig fail if no repo name is passed, but I'm not sure if we'll want to
support no-repo configuration at some point.
This didn't cause a problem for most people, since it would only happen during a
hg version number change, and 15 minutes later the background 'hg
debugdynamiconfig' process would fix it up. It did affect sandcastle though,
since it often creates new repositories and acts on them immediately.
Reviewed By: quark-zju
Differential Revision: D23955628
fbshipit-source-id: c922f4b523d19df9223aa28c97700b7011fc03eb
Summary:
The old code tried to express 4GB by using ^ to do an exponent. That
operator is actually the bitwise xor, so this was producing a limit closer to 4
bytes. It doesn't seem to have mattered much since a later diff overrode the
default via dynamicconfig. But let's fix this anyway.
Reviewed By: krallin
Differential Revision: D23955629
fbshipit-source-id: 6abebcb7e84b7a47f70ac501fa11b0dc60dfda7b
Summary: Now that the `async_runtime` crate exists, use Mercurial's global `tokio::Runtime` instead of creating one for each EdenAPI store.
Reviewed By: quark-zju
Differential Revision: D23945569
fbshipit-source-id: 7d7ef6efbb554ca80131daeeb2467e57bbda6e72
Summary:
`arc install-certificate` is going away, so let's reference the replacement
Created from Diffusion's 'Open in Editor' feature.
Differential Revision: D23948479
fbshipit-source-id: e629458ed2be1e1c09d9674935e3bcc890b05ad9
Summary:
Now that public phases are separated from draft, hidden behind the
ispublic() api, and no consumers require the entire set anymore, let's make them
lazy by testing commits against a lazy ancestor collection.
Reviewed By: quark-zju
Differential Revision: D23921326
fbshipit-source-id: 7c53345f41ff89d2d0ec11b1c09bbb2f89529c21
Summary:
Previously, _genrevancestor applied it's cutfunc function (which is
usually used to bound an ancestor traversal from processing public nodes) when
producing the parents of the current rev. In the walkrevtree function this means
we have to invoke cutfunc before we enqueue new parents onto the heap. This can
mean we have to process very old commits, even though walkrevtree might never
get to them.
Let's move the cutfunc to be at the deque stage instead of enque. When we make
public phases lazy, this will let us avoid looking up really old commits and
speed things up a lot.
Reviewed By: quark-zju
Differential Revision: D23921324
fbshipit-source-id: 8b520e9184441b9bd7581cdb71414f406cf9b544
Summary:
Unfortunately the current add operator implementation for nameset sorts
the result, which causes lazy namesets to be evaluated. Let's change the phase
revset to avoid that operator for now.
In a future diff we'll make publicrevs lazy, which will require this change to
be performant.
Reviewed By: quark-zju
Differential Revision: D23921325
fbshipit-source-id: c3477c30c1c5061b641c14b8aa73d07da6f7f398
Summary:
In a future diff we'll be changing the way public phase computation
works. Let's start by moving the draft phase computation out to belong to
draftrevs(), so then we can change publicrevs() without worrying about draft. In
the process, let's simplify draftrevs to just be an only() query.
Reviewed By: quark-zju
Differential Revision: D23921327
fbshipit-source-id: 255eab4b796bf7bc4467544cf8231b3ed98c08c8
Summary:
In an upcoming diff we'll be splitting draft and public calculations
apart when in headbased repositories. To make this cleaner, let's stop using
loadphaserevs and start using draftrevs and publicrevs accessors.
Reviewed By: quark-zju
Differential Revision: D23921328
fbshipit-source-id: 26a2a6fdc1201416bf61c7d32e3ede62d9579546
Summary:
These tests rely on sleep to simulate race conditions. They are the
largest source of hgbuild breakages. Let's just disable them. Most of this code
is going away in a couple months anyway when Mononoke takes over and revlogs go
away.
Also fixes a failure in test-bookmarks-strip.t that showed up.
Reviewed By: singhsrb
Differential Revision: D23922660
fbshipit-source-id: 35cacc19de03c031c84c1ebf2124884f4b4248e6
Summary: Now that the EdenAPI server is using the `LoadMiddleware` from `gotham_ext`, each response will contain an `X-Load` header that contains the number of active requests that the server is currently handling.
Reviewed By: quark-zju
Differential Revision: D23922809
fbshipit-source-id: 973143de5ddccf074d28aa3ef38d73f9fc1501b6
Summary:
Network speeds are usually reported in megabits per second (Mb/s), whereas file sizes are usually reported in [mebibytes](https://en.wikipedia.org/wiki/Binary_prefix) per second (MiB/s). Previously, the HTTP client reported neither of those and instead reported megabytes per second (MB/s).
This diff changes the latter to the former so that the numbers are more immediately useful. As a bonus, the speeds are now directly comparable to those reported by `hg debugnetwork`.
Reviewed By: quark-zju
Differential Revision: D23920367
fbshipit-source-id: 46500a42681ab83fc7c4ead82980e8ed620a4d5a
Summary: Now that stats are logged to `tracing` by the `HttpClient` directly, we no longer need to log them here. This commit backs out D23858077 (613fbc858f) which added the logging.
Reviewed By: quark-zju
Differential Revision: D23919308
fbshipit-source-id: 23d3a12c5307bc4b84dd9ffd25bd376718e3cc91
Summary:
Improve the log output of the HTTP client to avoid spewing redundant debug messages.
As part of this change, logging now uses the `tracing` crate instead of the `log` crate for better integration with the rest of Mercurial's logging infrastructure. Right now, `tracing` is just being used as a drop-in replacement for `log`, but now that it's in use we can start using its full capabilities (such as defining tracing spans) in later diffs.
Reviewed By: quark-zju
Differential Revision: D23919310
fbshipit-source-id: 95555ad083ead805ceece39c6e30aaf879bdf2bc
Summary:
Fix the `hg mv` and `hg cp` commands to wait for the lock if the repository is
locked, instead of immediately failing.
The current behavior is quite annoying as `hg mv` can easily fail due to
commitcloud sync commands running in the background and holding the lock.
I can't see any good reason why this command should not wait for the lock to
be held. The current behavior of failing when the lock is held dates back to
2006, in rFBSf5b6e7d2f591, when locking was first added to this command.
Reviewed By: DurhamG
Differential Revision: D23896302
fbshipit-source-id: 29bfeb8fd356abd31b6e1eb30c8f211a31fb044b
Summary:
The remotefilelog.indexedlog{data,history}store configs are no longer in use,
so we shouldn't use them as a way to test whether the indexedlog are present.
Reviewed By: quark-zju
Differential Revision: D23918133
fbshipit-source-id: 902a369f5e1774b8e2fa1e4b3280fbb7cc377775
Summary:
We were using the timeout parameter on `Multi::wait` (equivalent to `curl_multi_wait` in C) incorrectly. Previously, we were passing in the timeout provided by `curl_multi_timeout`.
This is incorrect usage because the value returned by `curl_multi_timeout` is the current value of libcurl's internal timeout (based on the state of the transfers), which will always be respected. The actual intention of the timeout parameter is to allow the caller to specify a hard cap on curl's internal timeout, so we should just pass some reasonable default value here. ([See explanation here.](https://github.com/curl/curl/issues/2996))
The purpose of `curl_multi_timeout` is to allow libcurl to tell the application what its desired timeout is in situations where the application itself is waiting for socket activity (using something like `epoll`), which is not the case when using `curl_multi_wait`.
Reviewed By: DurhamG
Differential Revision: D23914093
fbshipit-source-id: 76a25d7c59a4b08437c8d7be3d24708fb37b9172
Summary: Add appropriate EdenAPI calls to allow for HTTP tree fetching in `treemanifest`. Enabling `remotefilelog.http` essentially reroutes `_prefetch` and `getdesignatednodes` to their HTTP equivalents.
Reviewed By: DurhamG
Differential Revision: D23877319
fbshipit-source-id: 8a71934b47d07d2655fa46c103a14fb99e2f7b1f
Summary: Use the functionality from D23910534 (721f5af278) to set a timeout for EdenAPI requests, configured via the `edenapi.timeout` option.
Reviewed By: DurhamG
Differential Revision: D23911552
fbshipit-source-id: 4a6e3de1094d0faa1daaf6fe4b9b7aafb37a25a8
Summary:
Cast `names` and `nodes` to lists. The reason that they are sets is deduplication, but otherwise the code doesn't rely on them being sets (and in fact casts them to lists at multiple points later).
The main motivation for this is to allow these to be passed to Rust code later. The Rust bindings make a distinction between sequence types and unordered types, so passing a set in place of a list would result in a type error.
Reviewed By: DurhamG
Differential Revision: D23893108
fbshipit-source-id: 9ce2addb824867bcb2d24ba14c589b8791a156e8
Summary: Add the ability to set a timeout on HTTP requests. Equivalent to [`CURLOPT_TIMEOUT_MS`](https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT_MS.html).
Reviewed By: DurhamG
Differential Revision: D23910534
fbshipit-source-id: a7aec792ec3c122a01aa44fcfe2e2df6e3a111fc
Summary: These concatenated strings and bytes and needed to be fixed.
Reviewed By: kulshrax
Differential Revision: D23907301
fbshipit-source-id: 0008d7d54469266ecbae8ddaaa7625820f62cb7e
Summary:
There are several places in the HTTP client where we log and discard errors. (Typically, these are "this should never happen" type situations.)
Previously, these were logged at the `trace` log level, meaning that in practice no one would ever know if we did hit these errors.
Let's upgrade them to `error` so that they'll be printed out. (In theory, users should never see these error messages unless something has gone horribly wrong.)
Reviewed By: DurhamG
Differential Revision: D23888268
fbshipit-source-id: 9007205f946ebb0127238c76812cf62524878047
Summary:
This test relies on precise timing of racing processes. This is flakey
in our automated tests. Since this is mainly about revlog based repos, and we
only have that on the servers now, and we're going to delete them soon, let's
delete this test to make our tests more stable.
Reviewed By: kulshrax
Differential Revision: D23908316
fbshipit-source-id: 3fd110a8267d3bc16bbcb4545b9ff921274f7588
Summary: Linkrevs are on their way out. Let's switch to linknodes.
Reviewed By: quark-zju
Differential Revision: D23765176
fbshipit-source-id: 0dc1e0db11d732ce1edd24d863f32f08a5a5ce42
Summary:
The rust contentstore doesn't allow runtime manipulation of the list of
stores, which is required in order to insert the bundle store into the store.
Let's continue using the old python union store in these cases. This still let's
us delete the python pack code later, but we'll have to keep around the python
union store until we come up with a better solution.
Reviewed By: quark-zju
Differential Revision: D23689630
fbshipit-source-id: 0c60e6f268e81804149afa24409f64b5303e1e34
Summary:
The remaining test failures are mostly around bundle support, which
I'll fix in a later diff.
Reviewed By: quark-zju
Differential Revision: D23664037
fbshipit-source-id: 2bdde3cb4fcded6e0cf3afdc23269662544821df
Summary:
The high level prefetch trees API had a depth parameter, but the Rust
prefetch path doesn't support that. In the long run we probably want to get rid
of the depth parameter (or make it more useful), so for now let's get rid of it
from the function signature. You can still set the depth via config, and the few
places that needed depth are changed to use the config.
Reviewed By: quark-zju
Differential Revision: D23772384
fbshipit-source-id: a037d7207d4076a47368366ef7fd2dc1cfbf5cfb
Summary:
Treemanifest needs to be able to write to the shared stores from paths
other than just prefetch (like when it receives certain trees via a standard
pull). To make this possible we need to expose the Rust shared mutable stores.
This will also make just general integration with Python cleaner.
In the future we can get rid of the non-prefetch download paths and remove this.
Reviewed By: quark-zju
Differential Revision: D23772385
fbshipit-source-id: c1e67e3d21b354b85895dba8d82a7a9f0ffc5d73
Summary:
Introduce separate wire types to allow protocol evolution and client API changes to happen independently.
* Duplicate `*Request`, `*Entry`, `Key`, `Parents`, `RepoPathBuf`, `HgId`, and `revisionstore_types::Metadata` types into the `wire` module. The versions in the `wire` module are required to have proper `serde` annotations, `Serialize` / `Deserialize` implementations, etc. These have been removed from the original structs.
* Introduce infallible conversions from "API types" to "wire types" with the `ToWire` trait and fallible conversions from "wire types" to "API types" with the `ToApi`. API -> wire conversions should never fail in a binary that builds succesfully, but wire -> API conversions can fail in the case that the server and client are using different versions of the library. This will cause, for instance, a newly-introduced enum variant used by the client to be deserialized into the catch-all `Unknown` variant on the server, which won't generally have a corresponding representation in the API type.
* Cleanup: remove `*Response` types, which are no longer used anywhere.
* Introduce a `map` method on `Fetch` struct which allows a fallible conversion function to be used to convert a `Fetch<T>` to a `Fetch<U>`. This function is used in the edenapi client implementation to convert from wire types to API types.
* Modify `edenapi_server` to convert from API types to wire types.
* Modify `edenapi_cli` to convert back to wire types before serializing responses to disk.
* Modify `make_req` to use `ToWire` for converting API structs from the `json` module to wire structs.
* Modify `read_res` to use `ToApi` to convert deserialized wire types to API types with the necessary methods for investigating the contents (`.data()`, primarily). It will print an error message to stderr if it encounters a wire type which cannot be converted into the corresponding API type.
* Add some documentation about protocol conventions to the root of the `wire` module.
Reviewed By: kulshrax
Differential Revision: D23224705
fbshipit-source-id: 88f8addc403f3a8da3cde2aeee765899a826446d
Summary: Add log messages for debugging using the `tracing` crate, which allows them to be enabled via `env_logger`.
Reviewed By: quark-zju
Differential Revision: D23858076
fbshipit-source-id: a8ef1afac6c9ecbfb5d6d78232aa0d03a2fe2054
Summary: Log HTTP stats to stderr to assist with ad-hoc debugging. Will not be printed unless `RUST_LOG` is set appropriately.
Reviewed By: quark-zju
Differential Revision: D23858077
fbshipit-source-id: 39acf3de3fd0ca4403a986eb5373a6a79f1d004a
Summary: Similar to D23819023 (c96de76ac0) but works on Python 2, too.
Reviewed By: DurhamG
Differential Revision: D23858273
fbshipit-source-id: b15be07c8657bc8cb37960b631f2b31e4a78892b
Summary:
test-commitcloud-sync.t is a new change and just needs to be made cross
platform.
I have no idea how test-common-commands-fb.t ever worked. When HGRCPATH is set,
I expect the system hgrc to not be loaded, and therefore we can't run hg-clone.
Let's just unset it, since this is meant to test if the new Mercurial can
execute a clone. Ideally we'd redirect the system hgrc to the in-repo
staticfiles, but that's more effort.
Reviewed By: singhsrb
Differential Revision: D23869645
fbshipit-source-id: 66669d9fd9c3a23b01bc43b365723185b7b2ed33
Summary:
Move some commit cloud operations under infinitepush read path:
those are:
* `hg cloud check` command
* `hg cloud sync` command when the local repo is clean
* `hg cloud switch` command will normally use the read path for the dest workspace because we clean up the repo before performing the switch
* `hg cloud rejoin` command we use in fbclone will normally go through the read path as it runs in a fresh repo
If something is broken, there is always a way to rerun any of these command with '--dest' flag pointing it to the write path.
```
./hg cloud check -r 0c9596fd1 --remote --dest infinitepush-write
./hg cloud sync --dest infinitepush-write
./hg cloud switch -w other --dest infinitepush-write
```
Those use cases are limited and the lag of forward filler shouldn't be noticeable for them but we will be able to collect more signal how Mononoke performs with Commit Cloud.
Sitevar to control the routing of read traffic:
https://www.internalfb.com/intern/sv/HG_SSH_WRAPPER_MONONOKE_ROLLOUT/#revisions_list
Reviewed By: mitrandir77
Differential Revision: D23840914
fbshipit-source-id: 40fbe2e72756e7a4cf8bc5be6a0b94f6cf4906b4
Summary:
With segmented changelog backend, the revs can be changed, even if len(repo)
didn't change. Caching revs might not get invalidated properly. Let's cache
head nodes instead.
Reviewed By: DurhamG
Differential Revision: D23856176
fbshipit-source-id: c5154c536298c348b847a12de8c4f582f877f96e
Summary:
On Ubuntu the output is a bit different:
```
$ hg cloud sync --use-bgssh
commitcloud: synchronizing 'server' with 'user/test/default'
- remote: /bin/sh: trashssh: command not found
- abort: no suitable response from remote hg!
+ remote: /bin/sh: 1: trashssh: not found
+ abort: no suitable response from remote hg: '[Errno 32] Broken pipe'!
```
Glob them out to make the test pass.
Reviewed By: DurhamG
Differential Revision: D23824735
fbshipit-source-id: 7f96149ee16daff31fd0a1c68975b5edfa27cc46
Summary:
It seems OSX python2 has SIGINT handler set to SIG_IGN by default when running
inside tests. Detect that and reset SIGINT handler to raise KeyboardInterrupt.
This fixes test-ctrl-c.t on OSX.
As we're here, improve test-ctrl-c.t so it checks a bit more things and run
quicker.
Reviewed By: DurhamG
Differential Revision: D23853455
fbshipit-source-id: 05c47650bc80f9880f724828d307c32786265e2c
Summary:
Phabstatus for smartlog uses `PeekeaheadList` rather than `PeekaheadRevsetIterator` as
all of the commits are known ahead of time, and we don't need to collect together
batches as we iterate across the revset.
However, we should still batch up requests to Phabricator, as users with very high
numbers of commits in their smartlog may hit timeouts.
Add a batching mechanism to `PeekaheadList` that splits the list into chunks to
return with each peekahead.
Reviewed By: liubov-dmitrieva
Differential Revision: D23840071
fbshipit-source-id: 68596c7eb4f7404ce6109e69914f328565e34582
Summary:
This provides a way to fix the local cache of backed up heads if it is in an
invalid state.
The most important, it will allow early dogfooding of write traffic from Mononoke
without the reverse filler in place for developers or for the team.
You could just run `hg cloud backup -f` assuming the repo is backfilled to fix
any inconsistency when switch between the two backends
Reviewed By: markbt
Differential Revision: D23840162
fbshipit-source-id: bbd331162d65ba193c4774e37324f15ed0635f82
Summary:
For Python 3 we must ensure that the displayer messages have all been converted
to unicode before providing them to the Rust graph renderer.
The is because the Python 3 version of `encoding.unifromlocal` is a no-op, so
the result may still be `bytes` that need to be converted to `str`.
Reviewed By: quark-zju
Differential Revision: D23827233
fbshipit-source-id: 8f2b707ceceb210c0a2b5b589b99d4016452c61c
Summary:
D23759711 (be51116cf4) changed the way signal handlers work, which apparently causes
this test to fail. The SIGCHLD signal of the child changing state is received
during os.waitpid, which apparently counts as a signal during a system call,
which throws an OSError.
I'm not sure what the real fix should be. Sleeping gets us past the issue, since
presumably the signal is handled before the system call.
Reviewed By: quark-zju
Differential Revision: D23832606
fbshipit-source-id: 70fca19e419da55bbf546b8530406c9b3a9a6d77
Summary:
This simplifies the code a bit, and avoids creating tokio Runtime multiple
times.
Reviewed By: kulshrax
Differential Revision: D23799642
fbshipit-source-id: 21cee6124ef6f9ab6e165891d9ee87b2feb553ac
Summary:
Exercises the PyStream type from cpython-async.
`hg dbsh`:
In [1]: s,f=api._rustclient.commitdata('fbsource', list(repo.nodes('master^^::master')))
In [2]: s
Out[2]: <stream at 0x7ff2db700690>
In [3]: it=iter(s)
In [4]: next(it)
Out[4]: ('6\xf9\x18\xe4\x1c\x05\xfc\xb0\xd3\xb2\xe9\xec\x18E\xec\x0f\x1a:\xb7\xcd', ...)
In [5]: next(it)
Out[5]: ('}\x1f(\xe1o\xf1a\x9b\x81\xb9\x83}\x1b\xbbt\xd2e\xb1\xedb',...)
In [6]: next(it)
Out[6]: ('\xf1\xf0f\x97<\xf3\xdd\xe41w>\x92\xd1\xc0\x9ah\xdd\x87~^',...)
In [7]: next(it)
StopIteration:
In [8]: f.wait()
Out[8]: <bindings.edenapi.stats at 0x7ff2e006a3d8>
In [9]: str(Out[8])
Out[9]: '2.42 kB downloaded in 165 ms over 1 request (0.01 MB/s; latency: 165 ms)'
In [10]: iter(s)
ValueError: stream was consumed
Reviewed By: kulshrax
Differential Revision: D23799645
fbshipit-source-id: 732a5da4ccdee4646386b6080408c0d8958dd67f
Summary:
Exercises the PyFuture type from cpython-async.
`hg dbsh`:
In [1]: api._rustclient.commitdata('fbsource', list(repo.nodes('master^^::master')))
Out[1]:
([...], <future at 0x7f7b65d05060>)
In [2]: f=Out[1][-1]
In [3]: f.wait()
Out[3]: <bindings.edenapi.stats at 0x7f7b665e8228>
In [4]: f.wait()
ValueError: future was awaited
In [5]: str(Out[3])
Out[5]: '2.42 kB downloaded in 172 ms over 1 request (0.01 MB/s; latency: 171 ms)'
Reviewed By: kulshrax
Differential Revision: D23799643
fbshipit-source-id: d4fcef7dca58bc4902bb0809adc065493bb94bd3
Summary:
Add a `PyFuture<F>` type that can be used as return type in binding function.
It converts Rust Future to a Python object with an `await` method so Python
can access the value stored in the future.
Unlike `TStream`, it's currently only designed to support Rust->Python one
way conversion so it looks simpler.
Reviewed By: kulshrax
Differential Revision: D23799644
fbshipit-source-id: da4a322527ad9bb4c2dbaa1c302147b784d1ee41
Summary:
The exposed type can be used as a Python iterator:
for value in stream:
...
The Python type can be used as input and output parameters in binding functions:
# Rust
type S = TStream<anyhow::Result<X>>;
def f1() -> PyResult<S> { ... }
def f2(x: S) -> PyResult<S> { Ok(x.stream().map_ok(...).into()) }
# Python
stream1 = f1()
stream2 = f2(stream1)
This crate is similar to `cpython-ext`: it does not define actual business
logic exposed by `bindings` module. So it's put in `lib`, not
`bindings/modules`.
Reviewed By: markbt
Differential Revision: D23799641
fbshipit-source-id: c13b0c788a6465679b562976728f0002fd872bee
Summary:
See the previous diff for context. Move the error handling and ipdb logic to
the background thread so it can show proper traceback.
Reviewed By: kulshrax
Differential Revision: D23819022
fbshipit-source-id: 8ddae019ab939d8fb2c89afca2a7769094ebe26a
Summary:
With D23759710 (34d8dca79a), the main command was moved to a background thread, but the
error handling isn't. That can cause less useful traceback like:
Traceback (most recent call last):
File "dispatch.py", line 698, in _callcatch
return scmutil.callcatch(ui, func)
File "scmutil.py", line 147, in callcatch
return func()
File "util.py", line 4358, in wrapped
raise value
Set `e.__traceback__` so `raise e` preserves the traceback information.
This only works on Python 3. On Python 2 it is possible to use
`raise exctype, excvalue, tb`. But that's invalid Python 3 code. I'm
going to fix Python 2 traceback differently.
Reviewed By: kulshrax
Differential Revision: D23819023
fbshipit-source-id: 953ac8bd6108f4c0dae193607bee3f931c2bd13e
Summary:
The parameter `mtimethreshold` should be used instead of a constant of 14 days.
This fixes an issue where sigtrace output takes a lot of space in hg rage
output.
Reviewed By: DurhamG
Differential Revision: D23819021
fbshipit-source-id: e639b01d729463a4822fa93604ce3a038fbd4a9a
Summary:
filter returns a filter object, so the second time we iterate, it is empty
This is only in Python3 I believe, so migration to py3 broke it.
Reviewed By: markbt
Differential Revision: D23815206
fbshipit-source-id: 1a6503b2bbfd44959307c189d17dec9b5d5ff991
Summary:
During an hg update we first prefetch all the data, then write all the
data to disk. There are cases where the prefetched data is not available during
the writing phase, in which case we fall back to fetching the files one-by-one.
This has truly atrocious performance.
Let's allow the worker threads to check for missing data then do bulk fetching
of it. In the case where the cache was completely lost for some reason, this
would reduce the number of serial fetches by 100x.
Note, the background workers already spawn their own ssh connection's, so
they're already getting some level of parallelism even when they're doing 1-by-1
fetching. That's why we aren't seeing a 100x improvement in performance.
Reviewed By: xavierd
Differential Revision: D23766424
fbshipit-source-id: d88a1e55b1c21e9cea7e50fc6dbfd8a27bd97bb0
Summary:
Automigration gets messed up with `hg cloud rejoin` command in fbclone code because it triggered by the pull command.
As a result fbclone ends up to join a hostname workspace instead of the default for some cases.
* make sure that the migration never runs if background commit cloud operations are disabled
* also, add skip the migration in the pull command in fbclone
Once of those would be enough to fix the issue but I prefer to make both
changes.
Reviewed By: markbt
Differential Revision: D23813184
fbshipit-source-id: 3b49a3f079e889634e3c4f98b51557ca0679090b
Summary:
The children revset iterated over everything in the subset, which in
many cases was the entire repo. This can take hundreds of milliseconds. Let's
use the new _makerangeset to only iterate over descendants of the parentset.
Reviewed By: quark-zju
Differential Revision: D23794344
fbshipit-source-id: 9ac9bc014d56a95b5ac65534769389167b0f4508
Summary:
Now that Mercurial itself can properly handle SIGINT, there isn't a need for a Python wrapper around the Rust EdenAPI client (since the main purpose of the wrapper was to ensure proper SIGINT handling--something that could only be done in Python).
Note that while this does remove some code that prints out certificate warnings, that code was actually broken after the big refactor of the Rust bindings. (The exception types referenced no longer exist, so the code would simple result in a `NameError` if it actually tried to catch an exception from the Rust client.)
Reviewed By: singhsrb
Differential Revision: D23801363
fbshipit-source-id: 3359c181fd05dbec24d77fa1b7d9c8bd821b49a6
Summary:
This extends the Ctrl+C special handling from edenapi to the entire Python
command so Ctrl+C should be able to exit the program even if it's running
some blocking Rust functions.
`edenapi` no longer needs to spawn threads for fetching.
Reviewed By: singhsrb
Differential Revision: D23759710
fbshipit-source-id: cbaaa8e5f93d8d74a8692117a00d9de20646d232
Summary:
Move bunch of code into a separate file (scm daemon related options). Move them
out of cloud sync.
Also introduce additional check that the `hg cloud sync` command scm daemon
runs is intended for the current connected workspace
In theory when we switch a subscription, the SCM daemon gets notified but races possible and it is better to have this additional check, so SCM daemon triggers cloud sync where it is supposed to.
Reviewed By: markbt
Differential Revision: D23783616
fbshipit-source-id: b91a8b79189b7810538c15f8e61080b41abde386
Summary:
The config is not actually used any more (with rust-commits, it is forced on, without rust-commits,
there is no point to keep it on). Therefore removed.
Reviewed By: singhsrb
Differential Revision: D23771570
fbshipit-source-id: ad3e89619ac5e193ef552c25fc064ca9eddba0c6
Summary:
See the previous diff for context. This allows the code to run from non-main
thread.
Reviewed By: singhsrb
Differential Revision: D23759712
fbshipit-source-id: 044193a9d7193488c700d769da9ad68987356d69
Summary:
The idea is to extend D22703916 (61712e381c)'s way of calling functions from just edenapi to
the entire command for better Ctrl+C handling. Some code paths (ex. pager,
crecord) use `signal.signal` and `signal.signal` does not work from non-main
thread.
To workaround the `signal.signal` limitation, we pre-register all signals we care
about in the main thread to a special handler. The special handler reads a
global variable to decide what to do. Other threads can modify that global
variable to affect what the special signal handler does, therefore indirectly
"register" their handles.
Reviewed By: kulshrax
Differential Revision: D23759711
fbshipit-source-id: 8ba389072433e68a36360db6a1b17638e40faefa
Summary:
Before this change, for a long-running function wrapped by 'threaded',
it might:
background thread> start
main thread> receive SIGINT, raise KeyboardInterrupt
main thread> raise at 'thread.join(1)'
main thread> exiting, but wait for threads to complete (Py_Finalize)
background thread> did not receive KeyboardInterrupt, continue running
main thread> continue waiting for background thread
Teach `thread.join(1)` to forward the `KeyboardInterrupt` (or its subclass
`error.SignalInterrupt`) to the background thread, so the background thread
_might_ stop. Besides, label the background thread as daemon so it won't
be waited upon exit.
Reviewed By: kulshrax
Differential Revision: D23759713
fbshipit-source-id: 91893d034f1ad256007ab09b7a8b974325157ea5
Summary:
Move the wrapper to util.py. It'll be used in dispatch.py to make the entire
command Ctrl+C friendly.
Reviewed By: singhsrb
Differential Revision: D23759715
fbshipit-source-id: fa2098362413dcfd0b68e05455aad543a6980907
Summary: This will be used to test Ctrl+C handling with native code.
Reviewed By: kulshrax
Differential Revision: D23759714
fbshipit-source-id: 50da40d475b80da26b7dbc654e010d77cb0ad2d1
Summary: This makes it easier to test the API via debugshell.
Reviewed By: kulshrax
Differential Revision: D23750677
fbshipit-source-id: e29284395f03c9848cf90dd2df187e437890c56e
Summary: It is handy to test edenapi methods directly.
Reviewed By: kulshrax
Differential Revision: D23750709
fbshipit-source-id: 33c15cecaa0372ba9e4688502e7d8f3fdda7c3b8
Summary:
Add a command to rebuild the changelog without recloning other parts of the
repo. This can be used as a way to recover from corrupted changelog. It
currently uses revlog because revlog is still the only supported format during
streamclone.
In the future this can be used for defragmentation.
Reviewed By: DurhamG
Differential Revision: D23720215
fbshipit-source-id: 6db0453d18dbf553660d55d528f990a4029d9da4
Summary:
Improve help to reflect that the system is also meant for managing backups
add missing commands
reshuffle a bit
Reviewed By: markbt
Differential Revision: D23782794
fbshipit-source-id: d7fd3fa06ca7acd649cef557f3fe020295259e3d
Summary: The command will be provided as hint if username changes has been detected in configuration.
Reviewed By: markbt
Differential Revision: D23769942
fbshipit-source-id: 3e84ecef6dd68267022b92bf10f5e68dfc07f270
Summary:
`scratch` provided by `fb-scratch` was replaced by `mkscratch` provided by
the Mercurial package. See linked task for details.
Reviewed By: quark-zju
Differential Revision: D23773840
fbshipit-source-id: de0582069ce1a09c3cd9fc6b02d2d149f70d0d78
Summary:
Computing all successorsets is exponential with the number of splits
that have happened. This can slow things down tremendously.
The obsoletenodes path only needs to know "is there a visible successor" in
order to determine if a draft commit is obsolete. Let's use allsuccessors
instead of successorset.
Reviewed By: quark-zju
Differential Revision: D23771025
fbshipit-source-id: 666875e681c2e3306fc301357c95f1ab5bb40a87
Summary:
`hg cloud join --merge` doesn't really solve rename problem because it doesn't
preserve:
1. old heads
2. history
I added a proper API in Commit Cloud Service for renaming workspaces and now we
can use it to provide a rename command and 'rehost' command which is a version
of renaming to bind the current workspace to the current devserver.
Rehost command is meant to be used after dev server migration. I am plannig to
add this to the dev server migration wiki.
Next diff will cover how we can use the rename command to fix a username in workspaces names after username has been changed.
Reviewed By: markbt
Differential Revision: D23757722
fbshipit-source-id: dc11cb226eb76d347cdab70b3c72566448dcd098
Summary:
The Rust contentstore has no way to flush the shared stores, except
when the object is destructed. In treemanifest, the lifetime of the shared store
seems to be different from with files and we're not seeing them flushes
appropriately during certain commands. Let's make the flush api also flush the
shared stores.
Reviewed By: quark-zju
Differential Revision: D23662976
fbshipit-source-id: a542c3e45d5b489fcb5faf2726854cb49df16f4c
Summary:
Now that treemanifests can use Rust stores, we need to update the
Python repack code to support that.
Reviewed By: quark-zju
Differential Revision: D23662361
fbshipit-source-id: c802852c476425eef74181ead04f70b11ff9a27c
Summary:
This makes Rust contentstore prefetch route through the remotetreestore
prefetch logic to reach the lower level tree fetching, and makes the higher
level Python fetching route through the Rust contentstore to do prefetching. The
consequence of this is that there's a relatively unified code path for both
Python and Rust, and hopefully we can delete the janky Python bits once we're
completely migrated to Rust.
The way this diff works is pretty hacky. The code comment explains it, but the
tl;dr is that Rust prefetch works by providing references to the mutable stores,
while Python prefetch assumes they are stored and accessible on the repository.
Inorder for the old python tree fetching logic to work with both models, we
monkey patch the Rust mutable store references we receive into the function that
will later be called to request the repositories mutable stores. This is awful.
A cleaner fix might be to thread the mutable stores all the way through the
python fetching logic, then move the Python accessing of the repositories
mutable stores to the higher layer, near where Rust would provide it. That's a
lot of code churn though, so I'd like to do that in a later diff once we stop
using the non-rust logic entirely.
Reviewed By: quark-zju
Differential Revision: D23662351
fbshipit-source-id: 76007b6089ddf0e558581cd179a112311f8b58e3
Summary:
As part of moving treemanifest to use the Rust tree store, we need to
move prefetch to be able to be initiated from Rust. Rust requires a certain
signature for the prefetch function which accepts multiple keys.
In preparation for this requirement, let's refactor the current remotetreestore
fetching path to have a separate function. In a later diff we'll route Rust
prefetch requests through this function so the python and rust code shares the
same base tree discovery logic.
Reviewed By: quark-zju
Differential Revision: D23662196
fbshipit-source-id: 127045c279dc22914f7e1f3a619f6620586010ba
Summary:
Python 3 reports exceptions in except clauses by showing the original
exception, then saying another exception happened during the original exception
and hiding the second exception stack trace.
To make update exceptions more debuggable, let's move the handling outside the
except clause.
Reviewed By: quark-zju
Differential Revision: D23761667
fbshipit-source-id: bec758a3c7c0b88a5a569f794730058bf6f1eaad
Summary:
This is the initial step to track username when the workspace has been created and provide users an appropriate advice how to fix their workspace names if the username in configuration has been changed
in another diff I will provide the advice itself
I will build rename workspace command based on D23703790
Reviewed By: markbt
Differential Revision: D23730312
fbshipit-source-id: a49dabba7ec4acf35f6ff99ed23cff5d6f46e2e4
Summary:
`experimental.template-new-builtin = true` has been rolled out to 100%
and seems to work fine. Therefore, remove code that supports
`template-new-builtin = false`.
Reviewed By: singhsrb
Differential Revision: D23745353
fbshipit-source-id: 178af269381c9d3e20522ba4484d63051589342b
Summary:
Some tests run `hg init` right inside the test directory, turning the
entire $TESTTMP into a repo. In future diffs we'll start to rely more on hgcache
being present during tests, which creates a directory in $TESTTMP. Let's make
sure all repos are created as sub-directories of $TESTTMP.
Reviewed By: kulshrax
Differential Revision: D23662077
fbshipit-source-id: 2b2b974ebfd1bd19ad6acd1ebe3e68dd03a09869
Summary:
Adds the initial condition and creation logic for creating a Rust
treemanifest store. Fetching and some other code paths don't work just yet, but
subsequent diffs enable more and more functionality.
Reviewed By: quark-zju
Differential Revision: D23662052
fbshipit-source-id: a0e7090c9a3bf27a7738bf093f2d4eb6098b1ed6
Summary: The old logic would just double pack some bits. Let's prevent that.
Reviewed By: xavierd
Differential Revision: D23661933
fbshipit-source-id: 155291fa08ec2c060619329bd1cb6040769feb63
Summary:
The rust pack stores currently have logic to refresh their list of
packs if there's a key miss and if it's been a while since we last loaded the
list of packs. In some cases we want to manually trigger this refresh, like if
we're in the middle of a histedit and it invokes an external command that
produces pack files that the histedit should later consume (like an external
amend, that histedit then needs to work on top of).
Python pack stores solve this by allowing callers to mark the store for a
refresh. Let's add the same logic for rust stores. Once pack files are gone we
can delete this.
This will be useful for the upcoming migration of treemanifest to Rust
contentstore. Filelog usage of the Rust contentstore avoided this issue by
recreating the entire contentstore object in certain situations, but refresh
seems useful and less expensive.
Reviewed By: quark-zju
Differential Revision: D23657036
fbshipit-source-id: 7c6438024c3d642bd22256a8e58961a6ee4bc867
Summary:
Instants do not represent actual time and can only be compared against
each other. When we subtracted arbitrary Durations from them, we run the risk of
overflowing the underlying storage, since the Instant may be represented by a
low number (such as the age of the process).
This caused crashes in test_refresh (in the next diff) on Windows.
Let's instead represent the "must rescan" state as a None last_scanned time, and avoid any arbitrary subtraction. It's generally much cleaner too.
Reviewed By: quark-zju
Differential Revision: D23752511
fbshipit-source-id: db89b14a701f238e1c549e497a5d751447115fb2
Summary:
When sending trees and files we try to avoid sending trees that are
available from the main server. To do so, we currently check to see if the
tree/file is from the local store (i.e. .hg/store instead of $HGCACHE).
In a future diff we'll be moving trees to use the Rust store, which doesn't
expose the difference between shared and local stores. So we need to stop
depending on logic to test the local store.
Instead we can test if the commit is public or not, and only send the tree/file
is the commit is not public. This is technically a revert of the 2018 D7992502 (5e95b0e32e)
diff, which stopped depending on phases because we'd receive public commits from
svn there were not public on the server yet. Since svn is gone, I think it's
safe go back to that way.
This code was usually to help when the client was further ahead than another
client and in some commit cloud edge cases, but 1) we don't do much/any p2p
exchange anymore, and 2) we did some work this year to ensure clients have more
up-to-date remote bookmarks during exchange (as a way of making phases and
discovery more reliable), so hopefully we can rely on phases more now.
Reviewed By: quark-zju
Differential Revision: D23639017
fbshipit-source-id: 34c13aa2b5ef728ea53ffe692081ef443e7e57b8
Summary:
Previously the MetadataStore would always construct a mutable pack, even
if the operation was readonly. This meant all read commands required write
access. It also means that random .tmp files get scattered all over the place
when the rust structures are not properly destructed (like if python doesn't
bother doing the final gc to call destructors for the Rust types).
Let's just only create mutable packs when we actually need them.
Reviewed By: quark-zju
Differential Revision: D23219961
fbshipit-source-id: a47f3d94f70adac1f2ee763f3170ed582ef01a14
Summary:
Previously the ContentStore would always construct a mutable pack, even
if the operation was readonly. This meant all read commands required write
access. It also means that random .tmp files get scattered all over the place
when the rust structures are not properly destructed (like if python doesn't
bother doing the final gc to call destructors for the Rust types).
Let's just only create mutable packs when we actually need them.
Reviewed By: quark-zju
Differential Revision: D23219962
fbshipit-source-id: 573844f81966d36ad324df03eecec3711c14eafe
Summary:
Some tools, like ShipIt, close stdin before they launch the subprocess.
This causes sys.stdin to be None, which breaks our pycompat buffer read. Let's
handle that.
Reviewed By: quark-zju
Differential Revision: D23734233
fbshipit-source-id: 0adc23cd5a8040716321f6ede0157bc8362d56e0
Summary:
Turns out crecord had a help screen. It was broken in Python 3. This
fixes it.
Reviewed By: singhsrb
Differential Revision: D23720798
fbshipit-source-id: 4aade9abb88355c19ee4445de116fdb40d5366bd
Summary: filter returns a generator in Python 3, but we need a list.
Reviewed By: singhsrb
Differential Revision: D23720661
fbshipit-source-id: 8de3f5844bfe8b85b37c44423733fd2a09967397
Summary: This was horribly broken, and we have no tests.
Reviewed By: singhsrb
Differential Revision: D23720984
fbshipit-source-id: 4ad47c767b0d18f700c855a7bb43f38f5c5ef317
Summary:
When I added the surrogateescape patch for the email parser decoder
used during patches, I incorrectly added a corresponding encoder on the other
end when we get the data out of the parser. It turns out the parser is
smart/dumb. When using get_payload() it attempts a few different decodings of
the data and ends up replacing all the non-ascii characters with replacement
bits (question marks). Instead we should use get_payload(decode=True), which
bizarrely actually encodes the data into bytes, correctly detecting the presence
of surrogates and using the correct ascii+surrogateescape encoding.
Reviewed By: singhsrb
Differential Revision: D23720111
fbshipit-source-id: ed40a15056c39730c91067b830f194fbe41e5788
Summary:
This test is flaky due to `hg up` not always reading data from the stores, and
thus not always failing to reading the LFS blob. A better way to force read
from the store is to simply use `hg log -p` to read from the stores.
Reviewed By: DurhamG, singhsrb
Differential Revision: D23718823
fbshipit-source-id: 98bc37a76e93a67d031ba7bfa124b1db816983a1
Summary: The files use Python 3 only syntax and is not really used. Skip them so Python 2 build won't hit invalid syntax issues.
Reviewed By: chadaustin
Differential Revision: D23717662
fbshipit-source-id: f911a83937be9ccc40194f321e3b41625a68e703
Summary:
Running `setup.py` with Python 3 for Python 2 build will cause issues as
`setup.py` writes `.pyc` files in Python 3 format.
Reviewed By: chadaustin
Differential Revision: D23717661
fbshipit-source-id: 38cfabdfdf20424a21f8a5bdaf826e74da2304ac
Summary:
In preparation of moving away from SSH as an intermediate entry point for
Mononoke, let Mononoke work with newly introduced Metadata. This removes any
assumptions we now make about how certain data is presented to us, making the
current "ssh preamble" no longer central.
Metadata is primarily based around identities and provides some
backwards-compatible entry points to make sure we can satisfy downstream
consumers of commits like hooks and logs.
Simarly we now do our own reverse DNS resolving instead of relying on what's
been provided by the client. This is done in an async matter and we don't rely
on the result, so Mononoke can keep functioning in case DNS is offline.
Reviewed By: farnz
Differential Revision: D23596262
fbshipit-source-id: 3a4e97a429b13bae76ae1cdf428de0246e684a27
Summary:
As it says in the title, this adds support for receiving compressed responses
in the revisionstore LFS client. This is controlled by a flag, which I'll
roll out through dynamicconfig.
The hope is that this should greatly improve our throughput to corp, where
our bandwidth is fairly scarce.
Reviewed By: StanislavGlebik
Differential Revision: D23652306
fbshipit-source-id: 53bf86d194657564bc3bd532e1a62208d39666df
Summary:
This imports the async-compression crate. We have an equivalent-ish in
common/rust, but it targets Tokio 0.1, whereas this community-supported crate
targets Tokio 0.2 (it offers a richer API, notably in the sense that we
can use it for Streams, whereas the async-compression crate we have is only for
AsyncWrite).
In the immediate term, I'd like to use this for transfer compression in
Mononoke's LFS Server. In the future, we might also use it in Mononoke where we
currently use our own async compression crate when all that stuff moves to
Tokio 0.2.
Finally, this also updates zstd: the version we link to from tp2 is actually
zstd 1.4.5, so it's a good idea to just get the same version of the zstd crate.
The zstd crate doesn't keep a great changelog, so it's hard to tell what has changed.
At a glance, it looks like the answer is not much, but I'm going to look to Sandcastle
to root out potential issues here.
Reviewed By: StanislavGlebik
Differential Revision: D23652335
fbshipit-source-id: e250cef7a52d640bbbcccd72448fd2d4f548a48a
Summary: That might be used to pass more data to the server
Reviewed By: markbt
Differential Revision: D23704722
fbshipit-source-id: a6e41d615f6548f2f8fd036814c59573a45f93bc
Summary:
EdenFS is adding a Python 3 Thrift client intended for use by other
projects, and the Mercurial Python 2 build doesn't understand Python 3
syntax files, so switch the default getdeps build to Python 3.
Reviewed By: quark-zju
Differential Revision: D23587932
fbshipit-source-id: 6f47f1605987f9b37f888d29b49a848370d2eb0e
Summary:
We've often had cases where we need to nuke peoples caches for various
reasons. It's a hug pain since we haven't a way to communicate with all hg
clients. Now that we have configerator dynamicconfigs, we can use that to reach
all clients.
This diff adds support for configs like:
```
[hgcache-purge]
foo=2020-08-20
```
The key, 'foo' in this case, is an identifier used to only run this purge once.
The value is a date after which this purge will no longer run. This is useful
for bounding the damager from forgetting about a purge and having it delete caches
over and over in the future for new repos or repos where the run once marker
file is deleted for some reason.
Reviewed By: quark-zju
Differential Revision: D23044205
fbshipit-source-id: 8394fcf9ba6df09f391b5317bad134f369e9b416
Summary:
`hg cloud rejoin` is used in fbclone
By providing a bit more information about the workspaces available we can improve user
experience and try to eliminate the confusion multiple workspaces cause.
Reviewed By: mitrandir77
Differential Revision: D23623063
fbshipit-source-id: 7598c1b58597032c9cfcef0b44b0ec1b00510ffa
Summary:
The corpus rev that biggrep has indexed may not be available in the
local client. Later on in the function it will pull that revision, but earlier
in the function the new logic I added a few weeks ago is just crashing.
That logic was trying to diff against the earlier revision, but that's pretty
arbitrary. Let's just diff against one of the revs at random
(deterministically) and get rid of the need for the hash to exist in the repo
early in the command.
Reviewed By: sfilipco
Differential Revision: D23635801
fbshipit-source-id: 1c284d710b8df9539a696e900183bc10d5d71869
Summary:
Fixes a few issues with Mononoke tests in Python 3.
1. We need to use different APIs to account for the unicode vs bytes difference
for path hash encoding.
2. We need to set the language environment for tests that create utf8 file
paths.
3. We need the redaction message and marker to be bytes. Oddly this test still
fails with jq CLI errors, but it makes it past the original error.
Reviewed By: quark-zju
Differential Revision: D23582976
fbshipit-source-id: 44959903aedc5dc9c492ec09a17b9c8e3bdf9457
Summary:
For repositories that have the old-style LFS extension enabled, the pointers
are stored in packfiles/indexedlog alongside with a flag that signify to the
upper layers that the blob is externally stored. With the new way of doing LFS,
pointers are stored separately.
When both are enabled, we are observing some interesting behavior where
different get and get_meta calls may return different blobs/metadata for the
same filenode. This may happen if a filenode is stored in both a packfile as an
LFS pointers, and in the LFS store. Guaranteeing that the revisionstore code is
deterministic in this situation is unfortunately way too costly (a get_meta
call would for instance have to fully validate the sha256 of the blob, and this
wouldn't guarantee that it wouldn't become corrupted on disk before calling
get).
The solution take here is to simply ignore all the lfs pointers from
packfiles/indexedlog when remotefilelog.lfs is enabled. This way, there is no
risk of reading the metadata from the packfiles, and the blob from the
LFSStore. This brings however another complication for the user created blobs:
these are stored in packfiles and would thus become unreadable, the solution is
to simply perform a one-time full repack of the local store to make sure that
all the pointers are moved from the packfiles to to LFSStore.
In the code, the Python bindings are using ExtStoredPolicy::Ignore directly as
these are only used in the treemanifest code where no LFS pointers should be
present, the repack code uses ExtStoredPolicy::Use to be able to read the
pointers, it wouldn't be able to otherwise.
Reviewed By: DurhamG
Differential Revision: D22951598
fbshipit-source-id: 0e929708ba5a3bb2a02c0891fd62dae1ccf18204
Summary:
hg-http's built client should provide integration with Mercurial's stats
collection mechanisms.
Reviewed By: kulshrax
Differential Revision: D23577867
fbshipit-source-id: 93c777021bc347511322269d678d6879710eed3e
Summary:
Add `with_stats_reporting` to HttpClient. It takes a closure that will be
called with all `Stats` objects generated. We then use this function in
the hg-http crate to integrate with the metrics backend used in Mercurial.
Reviewed By: kulshrax
Differential Revision: D23577869
fbshipit-source-id: 5ac23f00183f3c3d956627a869393cd4b27610d4
Summary: Rust based metrics so that even Rust libraries can write metrics.
Reviewed By: quark-zju
Differential Revision: D23577870
fbshipit-source-id: b19904968d9372c8ce19775fb37c7af53a370ea5
Summary:
We start off simple here. Python only really has counters so we only implement
counters. There are a lot of options on how to improve this and things get
slightly complicated when we look at the how ecosystem and fb303. Anyway,
simple start.
Reviewed By: quark-zju
Differential Revision: D23577874
fbshipit-source-id: d50f5b2ba302d900b254200308bff7446121ae1d
Summary:
Slash is probably the standard metric delimiter nowadays. Since we don't have
that many metrics I think that it makes sense to look at slash as the
standard metric delimiter going forward.
This diff updates parsing of metric names to treat both '_' and '/' as
delimiters.
Reviewed By: quark-zju
Differential Revision: D23577876
fbshipit-source-id: 03997b1285df9c52d6e2837b5af5372deb69b133
Summary:
The command is easier to use than `hg cloud join --switch`.
Also highlight the workspace name in the output of `hg cloud status`
Reviewed By: mitrandir77
Differential Revision: D23601507
fbshipit-source-id: 74eb17c9366a9dbe96881c8e3e0705619fadb3d6
Summary:
Streaming clone implementation did not check that received files have the corrects. This change addresses it.
Before this change if connection was interrupted for whatever reason client would treat fetch of changeset as successful and proceed with cloning operations, but later checks would report corruption of internal state of hg data. This is based on user [report](https://fb.workplace.com/groups/scm/permalink/3177150312334567/)
Reviewed By: quark-zju, krallin
Differential Revision: D23572058
fbshipit-source-id: d740b45ca217cd6db0a65e01aabc2ba9a4835221
Summary: The Mercurial codebase uses hyphens in crate names rather than underscores. This is similar to the convention favored by the larger Rust community, though it is different from Mononoke, which uses underscores. While we'll probably need to eventually settle on a consistent convention for all of projects in the Eden SCM repo, for now, `http_client` should be made consistent with the adjacent crates.
Reviewed By: sfilipco
Differential Revision: D23585721
fbshipit-source-id: d2e690d86815be02d7b8d645198bcd28e8cbd6e0
Summary: No more tokio-core! More `async/await`.
Reviewed By: kulshrax
Differential Revision: D23586509
fbshipit-source-id: b2e766ddb7575bc96963432f0c8582b4370b19aa
Summary:
This diff adds a `SocketTransport` implementation that no longer uses legacy `tokio-core` based futures but `tokio-tower` and `tower-service` for processing Thrift requests.
The old implementation is renamed to `SocketTransportLegacy` for better transitioning.
Reviewed By: dtolnay
Differential Revision: D20019196
fbshipit-source-id: 3bee684e9254bf1a81669ef0d2c2262a55e75daa
Summary:
In order to keep the hgcache size bounded we need to keep track of pack
file size even during normal operations and delete excess packs.
This has the negative side effect of deleting necessary data if the operation is
legitimately huge, but we'd rather have extra downloading time than fill up the
entire disk.
Reviewed By: quark-zju
Differential Revision: D23486922
fbshipit-source-id: d21be095a8671d2bfc794c85918f796358dc4834
Summary:
In a future diff we'll add logic to delete old pack files. We'll want
to use this pack iteration code, so let's move it to a function.
Reviewed By: quark-zju
Differential Revision: D23486920
fbshipit-source-id: 5f872e946ffe816289c925dd2e03c292e29da5af
Summary:
As the repository grows the opportunity for large downloads increases.
Today all writes to data packs get sent straight to disk, but we have no way to
prevent this from eating all the disk.
Let's automatically flush datapacks when they reach a certain size (default
4GB). In a future diff this will let us automatically garbage collect data packs
to bound the maximum size of packs.
Rotatelog already have this behavior.
Reviewed By: quark-zju
Differential Revision: D23478780
fbshipit-source-id: 14f9f707e8bffc59260c2d04c18b1e4f6bdb2f90
Summary:
See D23538897 for context. This adds a killswitch so we can rollout client
certs gradually through dynamicconfig.
Reviewed By: StanislavGlebik
Differential Revision: D23563905
fbshipit-source-id: 52141365d89c3892ad749800db36af08b79c3d0c
Summary:
Like it says in the title, this updates remotefilelog to present client
certificates when connecting to LFS (this was historically the case in the
previous LFs extension). This has a few upsides:
- It lets us understand who is connecting, which makes debugging easier;
- It lets us enforce ACLs.
- It lets us apply different rate limits to different use cases.
Config-wise, those certs were historically set up for Ovrsource, and the auth
mechanism will ignore them if not found, so this should be safe. That said, I'd
like to a killswitch for this nonetheless. I'll reach out to Durham to see if I
can use dynamic config for that
Also, while I was in there, I cleaned up few functions that were taking
ownership of things but didn't need it.
Reviewed By: DurhamG
Differential Revision: D23538897
fbshipit-source-id: 5658e7ae9f74d385fb134b88d40add0531b6fd10
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: zertosh
Differential Revision: D23568779
fbshipit-source-id: 477200f35b280a4f6471d8e574e37e5f57917baf
Summary:
This makes it easy for `metaedit` to be used by automation. Provided
with a simple JSON file with hash->{user, message} mapping metaedit will
do all of its work without any prompts.
Reviewed By: quark-zju
Differential Revision: D23545527
fbshipit-source-id: 18763ecacff9143b9ad492faf654b176b0f86d1f
Summary:
The "meaningfulparents" concept is coupled with rev numbers.
Remove it. This changes default templates to not show parents, and `{parents}`
template to show parents.
Reviewed By: DurhamG
Differential Revision: D23408970
fbshipit-source-id: f1a8060122ee6655d9f64147b35a321af839266e
Summary:
Now that the Rust revisionstore records undesired filename fetches,
let's log those results to Scuba in Python.
Reviewed By: StanislavGlebik
Differential Revision: D23462572
fbshipit-source-id: b55f2290e30e3a5c3b67d9f612b24bc3aad403a8
Summary:
We want to be able to record when fetches to certain paths happen.
Let's add recording infrastructure to the new ReportingRemoteDataStore.
A future diff will make the seen accessible from Python for scuba logging.
Reviewed By: xavierd
Differential Revision: D23462574
fbshipit-source-id: 5d749f2429e26e8e7fe4fb5adc29140b4309eac9
Summary:
We want to monitor what paths are fetched from our remote servers.
Since all of our remote stores are hidden behind the RemoteDataStore interface,
let's create a wrapper around that. A future diff will insert the actual
monitoring and reporting.
Reviewed By: quark-zju
Differential Revision: D23462571
fbshipit-source-id: e6031f19db23f7d1b09767efb9613d7528fb457d
Summary: This hopefully makes it more obvious so it looks less like an hg crash.
Reviewed By: kulshrax
Differential Revision: D23509569
fbshipit-source-id: 7174780bc7e9841e3f89a482280c49427b62fb74
Summary:
The revs can change after flush. For example, during pushrebase, some ctx might
initially have a non-master Id assigned, and later got assigned an Id in the
master group:
```
ipdb> p self.__dict__
{'_repo': <edenscm.hgext.fastannotate.protocol.localreposetup.<locals>.fastannotaterepo object at 0x7f2415b3f8e0>, '_rev': 72057594038527478, '_node': b'\xb6\x12\xcd\x81b#\xa3\x01\xe2pP\x84\x05{\xd2He\xbe\xcc\xf0'}
ipdb> p self._node
b'\xb6\x12\xcd\x81b#\xa3\x01\xe2pP\x84\x05{\xd2He\xbe\xcc\xf0'
ipdb> p self._repo.changelog.rev(self._node)
7198913
ipdb> p self._rev
72057594038527478
```
Note that `self._rev` becomes inconsistent with `changelog.rev(self._node)`.
The error looks like:
$ hg push -r . --to master --debug --trace --traceback --verbose
...
pushing rev 556400239977 to destination ...
...
1 commits found
list of changesets:
556400239977b9ed523eae5ad28773784c975f7f
sending unbundle command
...
added 79 commits with 0 changes to 0 files
moving remote bookmark 'remote/master' to 84829e9242e4
...
using eden update code path
Traceback (most recent call last):
...
File "/opt/fb/mercurial/edenscm/mercurial/merge.py", line 2220, in update
return eden_update.update(
File "/opt/fb/mercurial/edenscm/mercurial/eden_update.py", line 126, in update
stats, actions = _handle_update_conflicts(
...
File "/opt/fb/mercurial/edenscm/mercurial/context.py", line 503, in _changeset
return self._repo.changelog.changelogrevision(self.rev())
# self = <changectx 84829e9242e4>
File "/opt/fb/mercurial/edenscm/mercurial/changelog2.py", line 312, in changelogrevision
return changelogrevision(self.revision(nodeorrev))
# nodeorrev = 72057594038527521
File "/opt/fb/mercurial/edenscm/mercurial/changelog2.py", line 365, in revision
node = self.node(nodeorrev)
# nodeorrev = 72057594038527521
File "/opt/fb/mercurial/edenscm/mercurial/changelog2.py", line 280, in node
raise IndexError("revlog index out of range")
Traceback (most recent call last):
File "/opt/fb/mercurial/edenscm/mercurial/changelog2.py", line 278, in node
return self.idmap.id2node(rev)
error.CommitLookupError: 'N599585 cannot be found'
Change `context` object to not memorizing revs.
Reviewed By: DurhamG
Differential Revision: D23468702
fbshipit-source-id: b623bcec99b09d61169371e08c69fc6d6f38935c
Summary:
This is based on fbsource data, building level 5 proves to be not useful.
This would save 300ms in the write path.
Reviewed By: sfilipco
Differential Revision: D23494505
fbshipit-source-id: ca795b4900af40dbfdaa463d36f3169413bf6a62
Summary:
Previously the IdMap's "Name -> Id" index simply ignores the "reassign
non-master" request. It turns out stale entries in that index can cause
issues as demonstrated by the previous diff.
Update IdMap to actually remove both indexes of non-master group on
remove_non_master so it cannot have stale entries.
To optimize the index, the format of IdMap is changed from:
[ 8 bytes Id (Big Endian) ] [ Name ]
to:
[ 8 bytes Id (Big Endian) ] [ 1 byte Group ] [ Name ]
So the index can use reference to the slice, instead of embedding the bytes, to
reduce index size.
The filesystem directory name for IdMap used by NameDag is bumped to `idmap2`
so it won't read the incompatible old `idmap` data.
Reviewed By: sfilipco
Differential Revision: D23494508
fbshipit-source-id: 3cb7782577750ba5bd13515b370f787519ed3894
Summary: Some vertexes can disappear from the graph!
Reviewed By: sfilipco
Differential Revision: D23494506
fbshipit-source-id: ecbf2a4169e5fc82596e89a4bfe4c442a82e9cd2
Summary: The TestDag struct will be used to do some more complicated tests.
Reviewed By: sfilipco
Differential Revision: D23494507
fbshipit-source-id: 11350f9e448725ae49f50a7b6f19efc57ad84448
Summary:
A few things here:
- The heads must be bytes.
- The arguments to wireproto must be strings (we used to encode / decode them,
but we shouldn't).
- The bookmark must be a string (otherwise it gets serialized as `"b\"foo\""`
and then it deserializes to that instead of `foo`).
Reviewed By: StanislavGlebik
Differential Revision: D23499846
fbshipit-source-id: c8a657f24c161080c2d829eb214d17bc1c3d13ef
Summary:
Replacing places where the tokio runtime is instantiated inside the edenapi
client crate.
Reviewed By: quark-zju
Differential Revision: D23468596
fbshipit-source-id: ef68718c7d5b89b6477a2946daaa51618b53d06a
Summary:
At open time, it's pointless to attempt to create new levels. So let's just
read the existing max_level and do not try to build max_level + 1.
This turns out to save 300ms in profiling result.
Reviewed By: sfilipco
Differential Revision: D23494509
fbshipit-source-id: 4ea326a3cc21792790ea0b87e5bf608a94ae382b
Summary:
With MultiLog, per-log meta was previously entirely ignored. However, they can
be useful for updated indexes. For example, application defines a new index,
and opens a Log via MultiLog. The application would expect the new index is
built only once. Without MultiLog, per-log meta is updated at open time in
place. With MultiLog, the updated index meta is not written back to the
multimeta so the new index would be rebuilt multiple times undesirably.
Update MultiLog to reuse the per-log meta if it's compatible so it can pick up
new indexes.
Reviewed By: sfilipco
Differential Revision: D23488212
fbshipit-source-id: c8b3e6b5589dbda2e76a143d15085862a93dae22
Summary:
The poisoned meta makes investigation harder. ex. `debugdumpindexlog` won't
work on those logs.
Reviewed By: sfilipco
Differential Revision: D23488213
fbshipit-source-id: b33894d8c605694b6adf5afdaed45707fbd7357e
Summary:
Change dag_ops benchmarks to use different IdDagStores. An example run shows:
benchmarking dag::iddagstore::indexedlog_store::IndexedLogStore
building segments (old) 856.803 ms
building segments (new) 127.831 ms
ancestors 54.288 ms
children (spans) 619.966 ms
children (1 id) 12.596 ms
common_ancestors (spans) 3.050 s
descendants (small subset) 35.652 ms
gca_one (2 ids) 164.296 ms
gca_one (spans) 3.132 s
gca_all (2 ids) 270.542 ms
gca_all (spans) 2.817 s
heads 247.504 ms
heads_ancestors 40.106 ms
is_ancestor 108.719 ms
parents 243.317 ms
parent_ids 10.752 ms
range (2 ids) 7.370 ms
range (spans) 23.933 ms
roots 620.150 ms
benchmarking dag::iddagstore::in_process_store::InProcessStore
building segments (old) 790.429 ms
building segments (new) 55.007 ms
ancestors 8.618 ms
children (spans) 196.562 ms
children (1 id) 2.488 ms
common_ancestors (spans) 545.344 ms
descendants (small subset) 8.093 ms
gca_one (2 ids) 24.569 ms
gca_one (spans) 529.080 ms
gca_all (2 ids) 38.462 ms
gca_all (spans) 540.486 ms
heads 103.930 ms
heads_ancestors 6.763 ms
is_ancestor 16.208 ms
parents 103.889 ms
parent_ids 0.822 ms
range (2 ids) 1.748 ms
range (spans) 6.157 ms
roots 197.924 ms
benchmarking dag::iddagstore::bytes_store::BytesStore
building segments (old) 724.467 ms
building segments (new) 90.207 ms
ancestors 23.812 ms
children (spans) 348.237 ms
children (1 id) 4.609 ms
common_ancestors (spans) 1.315 s
descendants (small subset) 20.819 ms
gca_one (2 ids) 72.423 ms
gca_one (spans) 1.346 s
gca_all (2 ids) 116.025 ms
gca_all (spans) 1.470 s
heads 155.667 ms
heads_ancestors 19.486 ms
is_ancestor 51.529 ms
parents 157.285 ms
parent_ids 5.427 ms
range (2 ids) 4.448 ms
range (spans) 13.874 ms
roots 365.568 ms
Overall, InProcessStore > BytesStore > IndexedLogStore. The InProcessStore
uses `Vec<BTreeMap<Id, StoreId>>` for the level-head index, which is more
efficient on the "Level" lookup (Vec), and more cache efficient (BTree).
BytesStore outperforms IndexedLogStore because it does not need to verify
checksum on every read access - the checksum was verified at store creation
(IdDag::from_bytes).
Note: The `BytesStore` is something optimized for serialization, and hasn't been sent.
Reviewed By: sfilipco
Differential Revision: D23438174
fbshipit-source-id: 6e5f15188e3b935659ccde25fac573e9b963b78f
Summary: This allows them to use the SyncableIdDag APIs.
Reviewed By: sfilipco
Differential Revision: D23438170
fbshipit-source-id: 7ec7288cfb8186b88f85f0212a913cb0dffe7345
Summary: Other IdDagStores can also use the API. This will be used in benchmarks.
Reviewed By: sfilipco
Differential Revision: D23438180
fbshipit-source-id: 565552b66372dcfbb268c397883f627491d6e154
Summary:
Similar to `IdDagStore::sync` -> `GetLock::persist`, `reload` is more related
to filesystem/internal state exchange, and should be protected by a lock. So
let's move the API there, and requires a lock.
Reviewed By: sfilipco
Differential Revision: D23438169
fbshipit-source-id: 4228106b7739a1a758677adfddd213ad54aa4b6a
Summary:
`NameDag::reload` is used in `flush` to get a "fresh" NameDag.
In a future diff the `IdDag::reload` API gets changed, so let's
remove NameDag's use of it.
Instead, let's just re-`open` the path again to get a fresh NameDag.
It's a bit more expensive but probably okay, and easier to understand.
`get_new_segment_size()` was added as an internal API to preserve tests.
This also solves an issue where `NameDag` cannot recover properly if its
`flush` fails, because the old `NameDag` state is not lost.
After removing `NameDag::reload`, `idMap::reload` is no longer used publicly
and was made private.
Reviewed By: sfilipco
Differential Revision: D23438179
fbshipit-source-id: 0a32556a2cd786919c233d7efcae1cb9cbc5fb09
Summary:
The word "sync" is bi-directional: flush + reload. It was indexedlog::Log's
behavior. However, in the IdDag context "sync" is confusing - it is actually
only used to write data out, with protection from lock. Rename to `persist`
to clarify it's memory -> disk. Besides, requires a reference to a lock object
as a lightweight prove that some lock is held.
Reviewed By: sfilipco
Differential Revision: D23438175
fbshipit-source-id: 3d9ccd7431691d1c4e2ee74f3c80d95f5e7243b5
Summary:
This removes the need of cloning `IdMap`.
SyncableIdMap is a bit tricky. I added some comments to clarify things.
Reviewed By: sfilipco
Differential Revision: D23438176
fbshipit-source-id: fe66071da07067ed6c53a6437790af1d81b28586
Summary:
Make the test cover IndexedLogIdDagStore. The only change is the parent index
returns children in a different order.
Reviewed By: sfilipco
Differential Revision: D23438173
fbshipit-source-id: bcfabcd329e45bbc5e7e773103fa42307c23c35d
Summary:
There aren't too many thigs that we can do with the responses that we get back
from the server. Thigs are somewhat application specific for this endpoint.
One option that is not available right now and might make sense to add is
limiting the number of entries that are printed for a given location.
Reviewed By: kulshrax
Differential Revision: D23456220
fbshipit-source-id: eb24602c3dea39b568859b82fc27b7f6acc77600
Summary:
To reduce the size over the wire on cases where we would be traversing the
changelog on the client, we want to allow the endpoint to return a whole parent
chain with their hashes.
Reviewed By: kulshrax
Differential Revision: D23456216
fbshipit-source-id: d048462fa8415d0466dd8e814144347df7a3452a
Summary:
Renaming all the LocationToHash related structures to CommitLocationToHash.
This is done for consistency. I realized the issue when the command for reading
the request from cbor was not what I was expecting it to be. The reason was that
the commit prefix was used inconsistently for LocationToHash.
Reviewed By: kulshrax
Differential Revision: D23456221
fbshipit-source-id: 0181dcaf81368b978902d8ca79c5405838e4b184
Summary:
The default archive behavior archives the entire working copy. That is
undesirable and easy to accidentally trigger in a large repository. Let's
prevent it and require users to specify what they want archived.
Reviewed By: quark-zju
Differential Revision: D23464818
fbshipit-source-id: c39a631d618c2007e442e691cda542400cf8f4c3
Summary:
Replacing uses of the custom Runtime in lfs with the global runtime in the
`async-runtime` crate.
Reviewed By: xavierd
Differential Revision: D23468347
fbshipit-source-id: 61d2858634a37eb2d7d807104702d24889ec047a
Summary:
debugstacktrace is broken right now on Python 3: it wants to write to stderr,
which expects `bytes`, but it tries to write a `str`. This fixes it.
Reviewed By: DurhamG
Differential Revision: D23447984
fbshipit-source-id: 5896ae858f6022276fa47e08636c700159a2a678
Summary: Make it possible to test other IdDagStores.
Reviewed By: sfilipco
Differential Revision: D23438178
fbshipit-source-id: e5fc1b20833c71dd7569c77c31c76a26a6e357fe
Summary:
Now SpanSet can easily support `push_front`, we can just use SpanSet
efficiently without SpanSetAsc.
Reviewed By: sfilipco
Differential Revision: D23385246
fbshipit-source-id: b2e0086f014977fa990d5142e6eee844293e7ca5
Summary: To remove SpanSetAsc, its API needs to be implemented on SpanSet.
Reviewed By: sfilipco
Differential Revision: D23385250
fbshipit-source-id: ebd9d537287b5c1cde6e2c52ffb6da57dbd71852
Summary: This will make it possible to `push_front` and remove SpanSetAsc special case.
Reviewed By: sfilipco
Differential Revision: D23385249
fbshipit-source-id: 63ac67e9bce7cb281236399b3fb86eba23bbf8a0
Summary:
This makes it easier to replace Vec<Span> with VecDeque<Span> in SpanSet for
efficient push_front and deprecates SpanSetAsc (which uses Id in a bit hacky
way - they are not real Ids).
Reviewed By: sfilipco
Differential Revision: D23385245
fbshipit-source-id: b612cd816223a301e2705084057bd24865beccf0
Summary:
One user reports very very slow rebase (tens of minutes and running). The
commit is not very large. Python 2 can complete the rebase in 6 seconds.
I tracked it down to this code path. Making the change makes Python 3
rebase fast too (< 10 seconds). I haven't tracked down exactly why Python
3 is slow yet (maybe N^2 a += b)?
Some numbers about the slow merge:
ipdb> p len(m3.atext)
17984924
ipdb> p len(m3.btext)
17948110
ipdb> p len(m3.a)
613353
ipdb> p len(m3.b)
612129
ipdb> p len(m3.base)
612135
Reviewed By: singhsrb
Differential Revision: D23441221
fbshipit-source-id: 14b725439f4ecd3352edca512cdde32958b2ce29
Summary:
Previously the `is_valid()` function only checks about ordering.
Make it also check "no mergeable adjacent spans" and `span.low<=span.high`.
To provide better debug messages, the function does assertions
directly without returning a bool.
Reviewed By: sfilipco
Differential Revision: D23385247
fbshipit-source-id: 84829e9242e47e68dc2a4b2a6775b13331eba959
Summary:
Previously, `SpanSet::from_sorted_spans` allows having adjacent spans like
`[1..=2, 3..=4]`, while `SpanSet::from_spans` would merge them into `[1..=4]`.
Change it so `SpanSet::from_sorted_spans` merges them too. This simplifies
the `contains` logic and could make some Sets more efficient.
Reviewed By: sfilipco
Differential Revision: D23385248
fbshipit-source-id: 85b5ba9533f15034779e93255085a4fa09c6328a
Summary:
See the test change. Partially successful auto restack should have bookmarks
moved.
Reviewed By: DurhamG
Differential Revision: D23441932
fbshipit-source-id: 07e509a70bcc5cf81f702d40ec1b8dc4a5a781ff
Summary:
Those commands are broken right now: they try to write bytes but don't use
`writebytes`.
Reviewed By: DurhamG
Differential Revision: D23450968
fbshipit-source-id: 5d554771459f81718d90e5bad9a4c439cbb05d97
Summary:
When Python 3 wants to upload a file-like object, it does something a bit
awkward: it sets the `Transfer-Encoding` to `chunked`, but doesn't actually
chunk the data. Also, for some reason ,it still sets the `Content-Length`. I'm
not sure where that is coming from.
The thing is, when you set `Transfer-Encoding` to `chunked`, you do need to
chunk, or the other end is going to get very confused.
Unfortunately, this is not what happens here (note that the "send" logs are
from enabling http tracing in Python here, and those logs are basically one
line before `.send()` into a socket, so the chunking doesn't appear to happen
elsewhere):
```
[torozco@devbig051]~/opsfiles_bin % echo "aaaa" | ~/fbcode/buck-out/gen/eden/scm/__hg-py3__/hg-py3.sh debuglfssend https://mononoke-lfs.internal.tfbnw.net/opsfiles_bin
send: b'PUT /opsfiles_bin/upload/11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772/5 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-length: 5\r\nx-client-correlator: tQT3yBfFEzhVtqI5\r\naccept: application/mercurial-0.1\r\ncontent-type: application/x-www-form-urlencoded\r\nhost: mononoke-lfs.internal.tfbnw.net\r\ntransfer-encoding: chunked\r\nuser-agent: mercurial/4.4.2_dev git/2.15.1\r\n\r\n'
sendIng a read()able
send: b'aaaa\n'
reply: 'HTTP/1.1 400 Bad request\r\n'
header: Content-Type: text/html; charset=utf-8
header: Access-Control-Allow-Origin: *
header: proxy-status: client_read_error; e_upip="AcLKajO63Vab0hC4kzGZQsqck3P_YOu7HsBzshC-NCbuo31tlWWqCiVw5xVLh44LYYe7qioCPqYSb8-1cBpdvFDZb_t5oYRP1Q"; e_proxy="AcJjRKHG02qo6Bv6fEPCUVF7DpCyrq3rmSnXhRLWakKWREEvVpk4jc-tzDyG6l9jvn3vNo8PYPG_5hLtC3L1"
header: Date: Tue, 01 Sep 2020 13:10:35 GMT
header: Connection: close
header: Content-Length: 2959
```
What's a bit confusing to me here is where this Content-length header comes
from. Indeed, normally Python 3 will:
- Not infer a content-length for file-like objects (which is what we have)
https://fburl.com/ms94eq31
- Set Transfer-Encoding if no Content-Length is present:
https://fburl.com/f81g8v2j
So, it's a bit unexpected that a) we have a Content-Length (we shouldn't), and
that we b) also have a Transfer-Encoding header. That said, setting the
Content-Length does fix the problem, so that's what this diff does.
Reviewed By: DurhamG
Differential Revision: D23450969
fbshipit-source-id: e1f535ff3d0b49c0c914130593d9aebe89ba18ca
Summary:
As a follow up to the previous diff, let's also warn if dirstate includes
marker files that should not be included in any sparse profiles.
Reviewed By: DurhamG
Differential Revision: D23414361
fbshipit-source-id: 3d171328bf0ba5754e5bacde85f09abb4fed8603
Summary: There seems to be no need to use a shell.
Reviewed By: DurhamG
Differential Revision: D23124756
fbshipit-source-id: 7de1c23e2325fe88dc4c6a2c90563d06f109ed2f
Summary:
The Rust process utility avoids issues with interaction with Python and can do file
redirection on Windows.
Reviewed By: DurhamG
Differential Revision: D23124755
fbshipit-source-id: f72b88bafd19b3b41e53afbf6a4095d0d6bcb93a
Summary:
The Rust bindings handle the cross-platform differences and avoids issues
with Python / Rust interaction. Use it.
As we're here, extend the API to support cwd and env.
Reviewed By: DurhamG
Differential Revision: D23124171
fbshipit-source-id: fdc13f6eaeb25c05b53d385eb220af33dad984e1
Summary:
Spawning processes turns out to be tricky.
Python 2:
- "fork & exec" in plain Python is potentially dangerous. See D22855986 (c35b8088ef).
Disabling GC might have solved it, but still seems fragile.
- "close_fds=True" works on Windows if there is no redirection.
- Does not work well with `disable_standard_handle_inheritability` from `hgmain`.
We patched it. See `contrib/python2-winbuild/0002-windows-make-subprocess-work-with-non-inheritable-st.patch`.
Python 3:
- "subprocess" uses native code for "fork & exec". It's safer.
- (>= 3.8) "close_fds=True" works on Windows even with redirection.
- "subprocess" exposes options to tweak low-level details on Windows.
Rust:
- No "close_fds=True" support for both Windows and Unix.
- Does not have the `disable_standard_handle_inheritability` issue on Windows.
- Impossible to cleanly support "close_fds=True" on Windows with existing stdlib.
https://github.com/rust-lang/rust/pull/75551 attempts to add that to stdlib.
D23124167 provides a short-term solution that can have corner cases.
Mercurial:
- `win32.spawndetached` uses raw Win32 APIs to spawn processes, bypassing
the `subprocess` Python stdlib.
- Its use of `CreateProcessA` is undesirable. We probably want `CreateProcessW`
(unless `CreateProcessA` speaks utf-8 natively).
We are still on Python 2 on Windows, and we'd need to spawn processes correctly
from Rust anyway, and D23124167 kind of fills the missing feature of `close_fds=True`
from Python. So let's expose the Rust APIs.
The binding APIs closely match the Rust API. So when we migrate from Python to
Rust, the translation is more straightforward.
Reviewed By: DurhamG
Differential Revision: D23124168
fbshipit-source-id: 94a404f19326e9b4cca7661da07a4b4c55bcc395
Summary:
The Rust upstream took the "set F_CLOEXEC on every opened file" approach and
provided no support for closing fds at spawn time to make spawn lightweight [1].
However, that does not play well in our case:
- On Windows:
- stdin/stdout/stderr are not created by Rust, and inheritable by
default (other process like `cargo`, or `dotslash` might leak them too).
- a few other handles like "Null", "Afd" are inheritable. It's
unclear how they get created, though.
- Fortunately, files opened by Python or C in edenscm (ex. packfiles) seem to
be not inheritable and do not require special handling.
- On Linux:
- Files opened by Python or C are likely lack of F_CLOEXEC and need special
handling.
Implement logic to close file handlers (or set F_CLOEXEC) explicitly.
[1]: https://github.com/rust-lang/rust/issues/12148
Reviewed By: DurhamG
Differential Revision: D23124167
fbshipit-source-id: 32f3a1b9e3ae3a9475609df282151c9d6c4badd4
Summary:
It uses `sys.argv`, which might be rewritten by `debugshell`. Capture
`sys.argv` to make hgcmd more reliable.
Reviewed By: DurhamG
Differential Revision: D22993215
fbshipit-source-id: 5fa319e8023b656c6cdf96cb3229ea9f2c9b9b99
Summary: This allows us to run commands after changes were made to the repo.
Reviewed By: DurhamG
Differential Revision: D22993218
fbshipit-source-id: d9943dcda94da42970fb9107f48f4caa14b6a9d4
Summary:
Some code paths (ex. metalog.commit) use `util.timer()` as a way to get
seconds since epoch, and get 0 for tests. Other use-cases of `util.timer()`
are ad-hoc time measure for displaying speed / progress. They do not need high
precision or strong guarantee that the clock does not go backwards. Drop the
`time.perf_counter()` to meet the first use-case's expectation.
Reviewed By: singhsrb
Differential Revision: D23431253
fbshipit-source-id: 8bf2d1ed32e284e17285742e1d0fd7178f181fb3
Summary:
With segments backend, the revision numbers will be longer than commit hashes
and are confusing.
Reviewed By: DurhamG
Differential Revision: D23408971
fbshipit-source-id: e2057fa644fc7b6be4291f879eee3235bb4e687b
Summary:
Pulling from older repos (ex. years ago) could require GBs of commit text data.
Flush commit data if they exceed certain size.
This is for revlog compatibility.
In the future we probably just make commit text lazy to avoid this kind of issues.
Reviewed By: DurhamG
Differential Revision: D23408834
fbshipit-source-id: 273384f5a05be07877bb1c9871c17b53ba436233
Summary: This would be used to avoid excessive memory usage during pull.
Reviewed By: DurhamG
Differential Revision: D23408833
fbshipit-source-id: 8edd95ab8201697074f65cc118d14755a230567d
Summary:
`addcommits` is designed to be more efficiently if called with a batch of
commits. So let's buffer the commits to add then only call it once.
This avoids some N^2 behaviors, for example, the NameDag internally will
prepare "snapshot" of itself which involves coping the pending Rust vecs
about the segments and id <-> hash map.
The change makes `pull` usable from unusably slow:
Original Python Revlog backend:
```
In [1]: %trace repo.pull(bookmarknames=['master'],quiet=False)
5191 +466 | Apply Changegroup edenscm.mercurial.bundle2 line 516
| - Commits = 125 :
| - Range = a1d1b3ade136:2e3fe78af189 :
5191 +466 | changegroup.cg1unpacker.apply edenscm.mercurial.changegroup line 313
5192 +416 | Progress Bar: commits (progressbar)
5192 +415 | changelog.changelog.addgroup edenscm.mercurial.changelog line 536
5192 +409 | revlog.revlog.addgroup edenscm.mercurial.revlog line 2116
5215 +371 | changelog.changelog._addrevision (125 times) edenscm.mercurial.changelog line 558
```
DoubleWrite (Segments + Revlog) backend, Before:
```
In [2]: %trace repo.pull(bookmarknames=['master'],quiet=False)
2396 +154059 | Apply Changegroup edenscm.mercurial.bundle2 line 516
| - Commits = 323 :
| - Range = cb0b100180ba:5fb57c74f72e :
2396 +154059 | changegroup.cg1unpacker.apply edenscm.mercurial.changegroup line 313
2397 +151433 \ Progress Bar: commits (progressbar)
2397 +151433 | changelog2.changelog.addgroup edenscm.mercurial.changelog2 line 334
```
DoubleWrite (Segments + Revlog) backend, After:
```
In [2]: %trace repo.pull(bookmarknames=['master'],quiet=False)
4629 +512 | Apply Changegroup edenscm.mercurial.bundle2 line 516
| - Commits = 45 :
| - Range = cf23c6972934:1ff0c5f0e7ad :
4629 +512 | changegroup.cg1unpacker.apply edenscm.mercurial.changegroup line 313
4630 +494 | changelog2.changelog.addgroup edenscm.mercurial.changelog2 line 334
```
Reviewed By: DurhamG
Differential Revision: D23390435
fbshipit-source-id: dd97a5008dedd844d4134b87bfef190fa739a80b
Summary:
The users of addrevisoncb are gone.
This also removes the "alwayscache" parameter of "_addrevision".
Reviewed By: DurhamG
Differential Revision: D23390437
fbshipit-source-id: 7edd9dd0b93d4cb9d4f35d088a1aef719b450ec1
Summary: It is about legacy revlog formats that are no longer relevant.
Reviewed By: DurhamG
Differential Revision: D23390436
fbshipit-source-id: 58c2c432804181bcc6517d6c988777b843fc9ba4
Summary:
We have a few safeguards against creating full checkouts. However we have
sparse profiles that are not full, but that include very large directories
which normally should not be included.
This diff adds a logic that checks if a new sparse profile has any of the "marker"
files i.e. some files from a folder that should not be included. Operation
aborts if that the case, however there's always a way to workaround that.
Reviewed By: DurhamG
Differential Revision: D23414200
fbshipit-source-id: 626f392319eb1be8b35f39cadafb61f3c1dfefe3
Summary:
"hg diff" has --sparse option which diffs only files inside a sparse checkout.
The problem is that it doesn't work on eden checkouts because eden repo doesn't
have sparsematch() function.
This diff makes it so that if sparsematch() function doesn't exist then
--sparse option is just ignored.
The motivation for this change is
https://fb.workplace.com/groups/corehg/?post_id=687768245151742. There are some
diff calls that are triggered by arc lint that race with "hg update" and might download
loads of data on people's laptops. This diff doesn't fix the race, but it:
1) Makes sure we don't download too much data that are not in sparse profiles.
2) arc lint doesn't care about files outside of sparse profiles anyway, so
running --sparse make sense.
Reviewed By: DurhamG
Differential Revision: D23396918
fbshipit-source-id: 2a386fdbeab85187e2c2acab69cb86b74124d46f
Summary:
This is practically just 0 in our production setup during `pull`s. In the
future when the commit data become lazy, it's no longer possible to read the
files locally. So let's just don't scan the commits.
Reviewed By: DurhamG
Differential Revision: D23390438
fbshipit-source-id: 4c54c4aac5fd840205296ab86955ec1b8ab76607
Summary:
Mergedrivers can call dirstate.add directly and are adding paths with
"." and "..". Let's block those paths.
Reviewed By: quark-zju
Differential Revision: D23375469
fbshipit-source-id: 64e9f20169cfd50325ecd8ebcc1dd3be7a5cb202
Summary:
extdiff uses shutil.rmtree which calls os.rmdir with new python 3
options. Since we pathc os.rmdir, we need to support those options.
Reviewed By: quark-zju
Differential Revision: D23350968
fbshipit-source-id: 081d179dcd67b51ffdeb6b85899adf4e574a8d0f
Summary: Similar to D18528858 so module names do not need to be spelled twice.
Reviewed By: markbt
Differential Revision: D23091380
fbshipit-source-id: a2a261abc9c78c8805cea62b38498ba65398796d
Summary: This crate would fail to build without the "fb" feature because `serde_json` was listed as an optional dependency (but is used in a way that isn't conditional on the `fb` feature). This diff makes the dependency non-optional, and also silences several dead code warnings that are emitted when building without the "fb" feature.
Reviewed By: quark-zju
Differential Revision: D23386786
fbshipit-source-id: b00a8b0b8b0b978c1cfab2838629fcb388a076e9
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D23124169
fbshipit-source-id: 438296002eed14db599d6ec225183bf824096940
Summary:
A warning means that every tree fetched will be printed in the edenfs log,
which is way too much. Let's decrease this to a debug message.
Reviewed By: genevievehelsel
Differential Revision: D23385778
fbshipit-source-id: d77f1cac3efb945d4b95750822f2f12f48c75ffe
Summary: `len(repo)` can no longer predicate the next rev number. Use nodes instead.
Reviewed By: DurhamG
Differential Revision: D23307791
fbshipit-source-id: cc20e53f039eee2a714748352e8e98aab253095a
Summary:
Some functions might be called very frequently. For example,
`phases.phasecache.loadphaserevs` might be called 100k+ times.
That makes the tracing data harder to process.
Limit the count of spans to 1k by default so the data is cheaper to process,
and some highly repetitive cases can now be reasoned about. Note the limit
is only put on static Span Ids. If a span uses dynamic metadata or ask for
different Span Ids each time, they will not be limited.
In debugshell,
td = %trace repo.revs('smartlog()')
len(td.serialize())
dropped from 6MB to 0.87MB.
It's also possible to reason about:
td = %trace len(repo.revs('ancestors(.)'))
in debugshell (taking 30s, 98KB serialized, vs 21s without tracing), while
previously the result would be too large to show (`%trace` just hangs).
Reviewed By: DurhamG
Differential Revision: D23307793
fbshipit-source-id: 3c1e9885ce7a275c2abd8935a4e4539a4f14ce83
Summary: Set a default limit so the output won't be too long.
Reviewed By: DurhamG
Differential Revision: D23307792
fbshipit-source-id: 7e2ed99e96bbde06436a034e78f899fc2e3e03f8
Summary:
The debugshell command can be long running and contains uninteresting stuff.
Do not profile it.
Practically this hides showing the background statprof thread when using `%trace`.
Reviewed By: DurhamG
Differential Revision: D23278597
fbshipit-source-id: bad97de22e1be2be8b866bee705ea3a6755aa54b
Summary:
This allows entering ipdb for code like: `ipdb` or `ipdb()`. It can be handy to
debug something.
Reviewed By: DurhamG
Differential Revision: D23278599
fbshipit-source-id: 4355dd1944617aeb795450935789f01f66f094eb
Summary: This makes it possible to get tracing results, or run hg commands directly.
Reviewed By: DurhamG
Differential Revision: D23278601
fbshipit-source-id: e7dc92080d2881cb4155a481df5ca93f324828fc
Summary:
The `--trace` flag enables tracing Python modules.
For compatibility reasons, it also enables `--traceback`.
It can be used with debugshell to make `%trace` more useful.
Reviewed By: sfilipco
Differential Revision: D23278600
fbshipit-source-id: d6d0b34bd5c48111f8cd33d7df115f349b0e95b6
Summary:
I found this when I aborted an rebase Dxxx and trying rebasing again and it
complained about "nothing to rebase". It was caused by Dxxx resolving into
a hidden commit.
Reviewed By: sfilipco
Differential Revision: D23307794
fbshipit-source-id: f7a956b5300240089b6a4648f28cf4a152ee2433
Summary:
We shouldn't delete from a dictionary while iterating over it, instead we should iterate over a copy and then delete from the original.
`.items()` returns a view of the dict, while wrapping it in `list` makes a deep copy.
Reviewed By: DurhamG
Differential Revision: D23283668
fbshipit-source-id: a168eef1ed2a1ce02fe71b3f6e3aed090965d2a4
Summary:
Mononoke throws an error if we request the nullid. In the long term we
want to get rid of the concept of the nullid entirely, so let's just add some
Python level blocks to prevent us from attempting to fetch it. This way we can
start to limit how much Rust has to know about these concepts.
Reviewed By: sfilipco
Differential Revision: D23332359
fbshipit-source-id: 8a67703ba1197ead00d4984411f7ae0325612605
Summary:
Corp has a different concept of tier than prod. Let's load the corp
tier into our tier set as well.
Reviewed By: quark-zju
Differential Revision: D23354056
fbshipit-source-id: c9543b8253f042c7b1224578e0687b4bdf21738e
Summary:
The Python 3 email library internally stores the message as text, even
though our input and requested output is bytes. Let's make our own wrapper
around the parser to use ascii surrogateescape encoding so we can get the
actual bytes out later and not get universal newlines.
Based off the upstream 7b12a2d2eedc995405187cdf9a35736a14d60706,
which is basically a copy of the BytesParser implementation (https://github.com/python/cpython/blob/3.8/Lib/email/parser.py) with
newline=chr(10) added.
Reviewed By: quark-zju
Differential Revision: D23363965
fbshipit-source-id: 880f0642cce96edfdd22da5908c0b573887bed12
Summary:
`hg cloud rejoin` command is used in fbclone and it is supposed to print a
message on RegistrationError but this has been broken recently.
Reviewed By: markbt
Differential Revision: D23342773
fbshipit-source-id: 4f3318848953656dea65a2b5d4d832694f6b353c
Summary:
There are users who prefer run `hg cloud leave` if they notice they are
connected to commit cloud sync.
Proving more information and add a prompt might help them to change their mind.
For some users who left new fbclone will connect them back. So on next leave they can learn more information about Commit Cloud Workspaces.
Reviewed By: markbt
Differential Revision: D23346091
fbshipit-source-id: 72f170f7133cd64b772ec75ae29a85dc8809e351
Summary:
When updating to the null commit, the logic that computes the update
distance was broken. The null commit is pre-resolved to -1, which when passed to
a revset raw gets resolved as the tip commit. In large repositories this can
take a long time and use a lot of memory, since it's computing the difference
between tip and null.
Let's fix it to not pass the raw rev number, and also to handle the case of a 0
distance update.
Reviewed By: quark-zju
Differential Revision: D23358402
fbshipit-source-id: 3b0a1fe1bbcb07effba4d0ab2c092e66bdc02e67
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/46
See https://github.com/facebookexperimental/eden/runs/1034006668:
error: unused import: `env::set_var`
--> src/lfs.rs:1539:15
|
1539 | use std::{env::set_var, str::FromStr};
| ^^^^^^^^^^^^
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(unused_imports)]` implied by `#[deny(warnings)]`
error: unnecessary braces around method argument
--> src/lfs.rs:2439:36
|
2439 | remote.batch_upload(&objs, { move |sha256| local_lfs.blobs.get(&sha256) })?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these braces
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(unused_braces)]` implied by `#[deny(warnings)]`
error: aborting due to 2 previous errors
error: could not compile `revisionstore`.
I dropped `#![deny(warnings)]` as I don't think warnings like the above ones
should break the build. (denying specific warnings that we care about explicitly
might be a better approach)
Reviewed By: singhsrb
Differential Revision: D23362178
fbshipit-source-id: 02258f57727edfac9818cd29dda5e451c7ca80a7
Summary: Now that it is possible to control which features are enabled on manually-managed dependencies, we can reenable autocargo for `edenapi`. See D23216925, D23327844, and D23329351 (840e6dd6f6) for context.
Reviewed By: dtolnay
Differential Revision: D23335122
fbshipit-source-id: 8ce250c3a106d2a02f457f7ed531623dd866232f
Summary: The command does not crash but `-` lines are ignored.
Reviewed By: DurhamG
Differential Revision: D23357655
fbshipit-source-id: f48568bc193f947503bc19f3e192b33346c317e1
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/45
Fix referring to 'version' without proper codegen by making 'version' compile
without codegen. This fixes configparser test when version/src/lib.rs was not
generated.
Make unneeded deps without 'fb' feature optional.
This would hopefully fix the "EdenSCM Rust Libraries" GitHub workflow.
Reviewed By: DurhamG
Differential Revision: D23269864
fbshipit-source-id: f9e691fe0a75159c4530177b8a96dad47d2494a9
Summary: This makes the code simpler.
Reviewed By: sfilipco
Differential Revision: D23269858
fbshipit-source-id: bb9ac0bd1696f7429ca1856e6c63e04fabc2757a
Summary: This makes the code simpler.
Reviewed By: sfilipco
Differential Revision: D23269866
fbshipit-source-id: 30c9e9d218378c0d6df8b822b2a81df2b38f5b01
Summary: Will be used to simplify code.
Reviewed By: sfilipco
Differential Revision: D23269859
fbshipit-source-id: bed0c4dca075ff60900025642af1d84bdd03452d
Summary:
`impl<T> Trait for T` in the current Rust makes it impossible to have
`impl<Q> Trait for Q`. Avoid using it for IdConvert and PrefixLookup.
Reviewed By: sfilipco
Differential Revision: D23269861
fbshipit-source-id: a837f3984ff4e1bd5a3983dd1642b9f064f51a36
Summary:
`impl<T> Trait for T` in the current Rust makes it impossible to have
`impl<Q> Trait for Q`. Avoid using it for DagAlgorithm.
Reviewed By: sfilipco
Differential Revision: D23269860
fbshipit-source-id: 031e75e9bf1f1eec2b9e8f36220ef8b817a143a5
Summary: LowLevelAccess is a subset of NameDagStorage. Use the latter instead.
Reviewed By: sfilipco
Differential Revision: D23269865
fbshipit-source-id: 81ebb1e986d8b02c968a9a237ad9a97d4afd54bf
Summary:
If there are too many heads, the current `descendants` algorithm would visit
all "old" heads. For example, with this graph:
head9999 (N9999)
/
Z (master)
:
: (many heads)
:/
: head2 (N2)
:/
C head1 (N1)
|/
B head0 (N0)
|/
A
`A::head9999` or `Z::head9999` will visit N0, N1, ..., N9999, because
`descendands_up_to` is provided with `max_id = N9999` and Z as a vertex in the
master group, is before N0 in non-master. The current algorithm also means
`descendands_up_to` gets linearly slower as the user uses the repo more, which
is quite undesirable.
This diff changes `descendands_up_to` to take an `ancestors` set, which is
`::head9999` in this case, and iterate non-master flat segments in it. So it
will skip N0 to N9998 directly by finding the N9999 flat segment and only use
it. The number of heads will have a smaller impact on performance.
Another slowness is `draft::draft_heads`, if there are too many `draft_heads`,
the internal calculation of `::draft_heads` can be slow. Optimize it by
limiting `draft_heads` to `draft:`. Practically this affects `y::` revset as
`y::` is translated to `y::visible_heads` and `visible_heads` can be large.
`cargo bench --bench dag_ops -- '::-master'` shows significant difference:
Before:
range (master::draft) 18.112 s
range (recent_draft::drafts) 2.594 s
After:
range (master::draft) 72.542 ms
range (recent_draft::drafts) 14.932 ms
In my fbsource checkout there were 20k+ heads. The improvement of
`master::recent_draft` (`x::y`) is pretty visible, and `y::` is also improved:
% lhg debugbenchmarkrevsets -m -x 'p1(min(7e8c86ae % master))' -Y 'draft() & 7e8c86ae' -e 'x::y' -e 'y::' --no-default
# x: 168f5228e570fb6b2ff7f851bd82413102748d84 (p1(min(7e8c86ae % master)))
# y: 7e8c86aec68ebc6e0b8254afcb381315991fd21c (draft() & 7e8c86ae)
# before
| revset \ backend | segments | revlog | revlog-cpy |
|------------------|----------|--------|------------|
| x::y | 17ms | 0.1ms | 0.5ms |
| y:: | 3.3ms | 0.7ms | 1.3ms |
# after
| revset \ backend | segments | revlog | revlog-cpy |
|------------------|----------|--------|------------|
| x::y | 0.2ms | 0.1ms | 0.6ms |
| y:: | 1.0ms | 0.7ms | 1.3ms |
Reviewed By: sfilipco
Differential Revision: D23214387
fbshipit-source-id: 4d11db84cd28f4e04e8b991cbc650c9d5781fd27
Summary:
Lots of non-master heads is not an exercised graph in the benchmarks.
Add it as it practically happens. This will be used by the next change.
Reviewed By: sfilipco
Differential Revision: D23259879
fbshipit-source-id: 7fe290d14403e42e6d135bde56e2d5c8519ae530
Summary:
Currently the fuzz test only uses the master group. Let it exercise non-master
group too.
Reviewed By: DurhamG
Differential Revision: D23214388
fbshipit-source-id: 7108a1055fbdda2b012f93c5948fb83ef3b9a96f
Summary:
The calculation can take tens of milliseconds. Cache it.
Invalidate the cache on transaction commit.
This will improve perf on revsets like `descendants` that will use
`head()`.
Reviewed By: DurhamG
Differential Revision: D23196412
fbshipit-source-id: 2913310ebb97e1c0346198c1e2738799799c740a
Summary: Provide a way to see segments.
Reviewed By: sfilipco
Differential Revision: D23196408
fbshipit-source-id: b1418f945a5a3364ac73b0f97466d973dd4b6300
Summary:
Provide a way to print out all segments with resolved names. This will be used
in a debug command.
Reviewed By: sfilipco
Differential Revision: D23196410
fbshipit-source-id: 1712bfda0271aa548699fe4a6b8603c5ec07af7f
Summary:
Use the parent-child index to answer children query quickly.
`cargo bench --bench dag_ops -- children`:
Before:
children (spans) 606.076 ms
children (1 id) 124.105 ms
After:
children (spans) 602.999 ms
children (1 id) 10.777 ms
Reviewed By: sfilipco
Differential Revision: D23196411
fbshipit-source-id: 37195d5ccaa582d35314e0000352ef477287d38c
Summary: This will be used to optimize "children(single vertex)" query.
Reviewed By: sfilipco
Differential Revision: D23196409
fbshipit-source-id: 050c0859faf83b909e3174bb7c7bd6e7725165c0
Summary:
Update the parent index to store non-master group too. To make
"remove_non_master" work, the index contains a "child group" prefix that
allows efficient range invalidation.
This will allow answering "children(single vertex)" query more efficiently.
This diff does not expose an API to query the index yet.
Reviewed By: sfilipco
Differential Revision: D23196406
fbshipit-source-id: 9137da5ffa8306bdafbcabc06b6f0d23f38dcf57
Summary:
Practically, the input of `children` is often one vertex instead of a large set.
Add a benchmark for it.
It looks like:
children (spans) 606.076 ms
children (1 id) 124.105 ms
Reviewed By: sfilipco
Differential Revision: D23196407
fbshipit-source-id: 0645b59ac846836fd061386384f6386a57661741
Summary: They can be figured out at Hints initialization time. So they don't need to be mutable.
Reviewed By: sfilipco
Differential Revision: D23182518
fbshipit-source-id: 133375fdf27a2546a50b63fb130534acdadc5938
Summary:
Both IdSet and IdLazy set require both Dag and IdMap to construct.
This is step 1 torwards making Dag and IdMap immutable in hints.
A misspeall of "lhs" vs "hints" in the union set is discovered by the change
and fixed.
Reviewed By: sfilipco
Differential Revision: D23182520
fbshipit-source-id: 3d052de4b8681d3672ebc45d953d1e784f64b2a4
Summary:
It will be used in places (ex. tests) where a Dag is required but constructing
a real Dag is troublesome.
Reviewed By: sfilipco
Differential Revision: D23182517
fbshipit-source-id: 736911365778e5071c1e0b9615090a4e960392a0
Summary: This is more consistent with `id_map_snapshot`.
Reviewed By: sfilipco
Differential Revision: D23182519
fbshipit-source-id: 62b7fc8bfdc9d6b3a4639a6518ea084c7f3807dd