Summary:
We currently log the server we talk to, which is very nice in order to
benchmark performance of Mononoke vs. hg.
Unfortunately, while this allows for breaking down the samples, it's a little
hard to identify the right samples to break down! Historically, we've
approximated this by looking at the client's hostname, but this isn't always
ideal.
Reviewed By: quark-zju
Differential Revision: D20369686
fbshipit-source-id: 785c67fde09e7b7fc4c30121d14f0bc26e92a355
Summary:
No need for a closure when the function that will be called takes the same
arguments.
Reviewed By: quark-zju
Differential Revision: D20367309
fbshipit-source-id: 986299c4b691330bf17164abd70f3b49ed5263f8
Summary:
In some situations Windows may be unable to remove a file and Mercurial
cannot do anything about it. Instead of raising an error, just ignore it.
In the Python code, errors are ignored in multiple layers, but ultimately
batchremove ignore all OSError errors and just prints a warning about it. A
future diff will build a side channel for errors/warnings reporting, this will
be used to properly report this.
Reviewed By: quark-zju
Differential Revision: D20284803
fbshipit-source-id: 7df80f9e27e720ce819fb4116ea996fca390ad76
Summary:
For now it only checks for symlinks outside of the repo, but can be extended to
check for more.
Caching is used to prevent syscalls to already checked directories, this cache
is intended to be done per-thread, and thus each directory may be checked a
maximum of N, with N being the number of threads. This is deemed reasonable for
now.
Things that the Python pathauditor does that this one doesn't:
- Case checking for Windows,
- Verifies no writes to .hg,
- Windows shortnames aliases,
Reviewed By: quark-zju
Differential Revision: D20265276
fbshipit-source-id: 8e29bedadcaa6df2515e394894ab9e1dd5a9e9a7
Summary:
Symlinks are treated a bit differently from plain files, what is stored in the
ContentStore is the destination of the symlink, not it's content (well, the
content of a symlink really is it's destination).
For now, only unix platforms support symlinks, in reality this should be a
filesystem property as writing to ntfs-3g should have the same behavior as on
Windows.
For executable, we just need to mark the file as executable after writing to
it.
Reviewed By: quark-zju
Differential Revision: D20250943
fbshipit-source-id: 022dabc750125df32953a151df7da60db69b2cec
Summary:
During `hg update`, Mercurial forks multiple processes to write files on disk
concurrently, this is done as fetching blobs from the content store, and
writing them to disk is CPU bound. Usually, threads would be the preferred way
of speeding up such process, but unfortunately, Python has GIL that severely
limit the available concurrency. So, multiple processes were chosen.
Unfortunately, the multi-process solution also brings a lot of other issues,
more recently, we've had cases where the connections to the server and memcache
had to be dropped after the fork. In some other cases, this caused deadlocks.
And the solution is not effective on Windows.
Now that Mercurial is getting more and more Rust, we could instead go back to
the threads solution by using them in Rust, and have Python just push work to
them, this is exactly what this change does.
Things that are left to be done, but I wanted to get a diff out first:
- no file path audit
- no file backup
- no symlink creation
- probably other things I'm missing
Reviewed By: quark-zju
Differential Revision: D20102888
fbshipit-source-id: d47829fd7818b97710586b9851880f178048e27b
Summary:
With this new store, blobs will be transparently written to either an LFS
store, or a non-LFS one, depending on their size.
Initially, and as long as getpackv2 is supported, we also need to support
parsing lfs pointer data that the server is sending and write these to the lfs
pointer store. This code is very adhoc and does manual parsing of the pointer
data, definitively not great, suggestion for a simple and better solution is
welcome :).
From a migration standpoint, the read-only LFS stores are added to the
ContentStore, this allows blobs written in it to be readable at all time even
when `remotefilelog.lfs` isn't set. The code will effecitvely be dormant for a
while until the option is turned on, if we need to disable it, the dormant code
will still be able to read all the blobs written to disk. This forces us to
deploy a release that contains this code to stable first, before setting
`remotefilelog.lfs`.
Reviewed By: quark-zju
Differential Revision: D19986878
fbshipit-source-id: 260f5a542d52e748c0c703bfa7bb8ffac0e7b388
Summary: This makes `RUST_LOG` work for indexedlog tests.
Reviewed By: xavierd
Differential Revision: D20286515
fbshipit-source-id: ff4a1476eb01a9067dabe3622fd598f65fe86a18
Summary:
The tracing / env_logger integration works for hg as a binary. However I'd also
like to use it in library tests. This crate makes it easier to do so.
Reviewed By: xavierd
Differential Revision: D20286507
fbshipit-source-id: f5bf3288ce950591ddfe64b524ad51ce21ee4099
Summary: Those has helped me debugging some issues.
Reviewed By: xavierd
Differential Revision: D20286513
fbshipit-source-id: 012ddb16c2d0efd8f8697a5ecd4564ea31d65630
Summary: Move the scope of spans so the exit code is shown.
Reviewed By: xavierd
Differential Revision: D20286516
fbshipit-source-id: f39cbf60c86ea19a1bb0a09958748f04ff6a42e8
Summary:
Previously env_logger is only initialized if Python is initialized.
This diff makes env_logger initialized for Rust native commands.
Reviewed By: xavierd
Differential Revision: D20286517
fbshipit-source-id: 18fee96c2b41db1da9648d615d1e18809de90a63
Summary:
This means crates like env_logger (which reads $RUST_LOG, and writes to stderr)
can be used for convenient debugging.
Reviewed By: xavierd
Differential Revision: D20286514
fbshipit-source-id: e3b80cc4830ba5cc6dbf7aa1cbb92a4f4f046a54
Summary:
Those metadata include module_path, target, line number, etc, in Rust native
format. They will be used for the upcoming `log` integration.
Reviewed By: xavierd
Differential Revision: D20286510
fbshipit-source-id: 27019b941bef08c0bb3e505bbdae642282dcb141
Summary: Missed cleaning up the other definition in D19956272.
Reviewed By: chadaustin
Differential Revision: D20199689
fbshipit-source-id: 9a00cb7b640f71edc034632e354fb61112a74bc1
Summary: Moving these two files to integrate them better with rest of the Inodes code.
Reviewed By: simpkins
Differential Revision: D19956267
fbshipit-source-id: 5bf8165a6dc70a0c03f71f3f73b383991f21eb56
Summary: Windows sdk doesn't define mode_t. On Windows we use mode to store the file type info which should fix in 16 bits - S_FMT is defined as 0xf000.
Reviewed By: simpkins
Differential Revision: D19956270
fbshipit-source-id: 81acd2cd708f1c82674b8b75cd607d4f5b41a156
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary:
The result of `errnoStr` is often converted to `std::string` so returning `fbstring` adds an extra copy. Make it return `std::string` instead. This will also allow removing dependency between `String.h` and `FBString.h`.
(Note: this ignores all push blocking failures!)
Reviewed By: yfeldblum
Differential Revision: D20195395
fbshipit-source-id: 0dc65f1566911156be3fcb715dd105c58f2a8822
Summary:
Ideally we'd be using mkscratch, but this still isn't shipped
to our Windows systems.
Pick a path that is more friendly to our corp windows environment by default.
Reviewed By: pkaush
Differential Revision: D20342277
fbshipit-source-id: c85bccee6701adc03b26c92ba217b18bd684257a
Summary:
Time filters for file history require fetching changeset or changeset info to decide whether to include the commit into response. To speed up the process instead of sequential mapping, I buffer the map stream in batches of 100.
However it is unfortunate to fetch additional 100 history changesets if there is no time filters and only several commits were requested. Keeping also in mind that the most of the requests don't care about time.
Avoiding that would speed up the history generation,
I changed the changeset_path API so now it just returns the stream of changeset contexts. And commit_path API for history if there is no time filters collects everything to vector, otherwise applies them and also collects to vector. Then this vector is converted into response using FuturesOrdered.
Reviewed By: StanislavGlebik
Differential Revision: D20287497
fbshipit-source-id: 0c6b1315eccddb48f67bf5fa732bdf7c9a54a489
Summary: "tags" are stored as sets, but must be serialized as lists in python. This only affects the `ExternalTelemetryLogger`, which is used on mac builds.
Reviewed By: simpkins
Differential Revision: D20351645
fbshipit-source-id: b097bd6aeb6d109323186f4ae4ef959f831ead2b
Summary:
A lot of callsites want to know repo name. Currently they need to pass it from
the place where repo was initialized, and that's quite awkward, and in some
places even impossible (i.e. in derived data, where I want to log reponame).
This diff adds reponame in BlobRepo
Reviewed By: krallin
Differential Revision: D20363065
fbshipit-source-id: 5e2eb611fb9d58f8f78638574fdcb32234e5ca0d
Summary:
We track this in Scuba right now (and alarm on it), but tracking it in ODS will
make it easier to incorporate in our canary and post-release health check
workflow.
Reviewed By: StanislavGlebik
Differential Revision: D20361803
fbshipit-source-id: 99fb514d41f9cda42c3c9a82f3b8d6681285430a
Summary:
Before we assumed that if the rows_affected length doesn't match the number of
entries we were trying to insert we have a conflict. Let's verify if we really
have conflict or we're trying to insert the same entry twice.
Reviewed By: krallin
Differential Revision: D20343219
fbshipit-source-id: 19e032439fdd65f5fe1afe1a10b401bc2fe33462
Summary: Running blobimport twice on the same commit seems to cause problems.
Reviewed By: krallin
Differential Revision: D20343218
fbshipit-source-id: 4d572630e7c15c219bee8db15cc879b2cb8602fe
Summary:
It might be useful to know how long the end-to-end wireproto command to find
the master bookmark took, so log that.
Reviewed By: quark-zju
Differential Revision: D20344686
fbshipit-source-id: 8459a378a0c828a929eee5f9ffd34875a5b8d3cc
Summary:
For high latency connections we will need to warm up the connection before
starting the download or upload test. Facilitate this by making it possible to
run multiple tests within the same connection.
This changes the protocol for the speed-test command to be line-oriented commands.
Reviewed By: farnz
Differential Revision: D20344687
fbshipit-source-id: 1d2a815736caf974ed1aaaaf365b86343d41fe02
Summary: Add ability to walk all published bookmarks as there may be multiple important bookmarks
Reviewed By: krallin
Differential Revision: D20249806
fbshipit-source-id: aff2ee1ec7d51a9e4fb6e1e803612abd207fd6cb
Summary:
Add a counter to report the number of mounts that we failed to remount during
startup. Mount failures do not prevent EdenFS startup from proceeding. It is
useful to have a metric to report if these errors did occur even though the
start-up as a whole still proceeded otherwise.
Reviewed By: chadaustin
Differential Revision: D20319512
fbshipit-source-id: fd503a1ccc91b476cc9dc2bc6323501bbbeaf2c5
Summary:
On Python 2 the Mercurial JSON can be binary, which can break the telemetry
logger. Ensure the JSON string is valid utf-8 (although it might still be
invalid JSON).
Reviewed By: xavierd
Differential Revision: D20343845
fbshipit-source-id: 61e99e742bddf23c7fd5354a5754d79a0a452c28
