Summary:
Together with `debugmetalogroots`, this allows some kind of "time travel" to
investigate repo states in the past.
Reviewed By: sfilipco
Differential Revision: D20449409
fbshipit-source-id: ed5c134f9e9ee235b24f45c1aa35867a55a71fe5
Summary:
This makes the content of `bookmarks` stable.
It will make the metalog root IDs stable in tests.
Reviewed By: sfilipco
Differential Revision: D20449410
fbshipit-source-id: 969be63ab231f5865ec62e99398b5318d4257093
Summary:
Purge needs to be able to see what directories the walker traversed, so
it can delete them if they are empty. Instead of having the walker call
match.traversedir (which it seems like a bizarre pattern to use the matcher as a
holder for a non-matching related function), let's have the walker return an
enum and have an option to return directories.
At the python layer we then translate this into match.traversedir calls, but we
can clean that up later.
Reviewed By: quark-zju
Differential Revision: D19543795
fbshipit-source-id: cc51c86c91799d3df2c65d25a7b6cfe810206d0a
Summary:
On case insensitive systems we need to normalize file case. I've made a
rust case normalizer, but it requires some more tweaks. In the mean time, let's
handle this at the matching and output stages of the rust walk.
This is probably the pattern we want to follow later anyway, so the walk is
completely decoupled from normalization.
Reviewed By: kulshrax
Differential Revision: D19543797
fbshipit-source-id: 2ef8bdcecb2611a08680441fc030c64c2f4097d1
Summary:
The flat git map file can be huge and writing to it can hang for many
seconds or minutes on loaded systems since it calls fsync. Since we actually
rely on the indexedlog map file instead of the flat map, let's just stop writing
the flat map when indexedlog is enabled. This was previously done to enable
falling back, but we've had no issues and never needed to fallback.
Reviewed By: quark-zju
Differential Revision: D20443392
fbshipit-source-id: a3e7dbdb2b37e2e99df774f540b05d06414fb279
Summary:
Since configparser enforces utf-8 config files (because pest wants Rust strings),
let's migrate from Bytes to Text to remove extra encoding conversions.
Previously this was blocked by the lack of ref-counted text (since the "source"
of each config location is the entire config file). Now minibytes provides Text
so we can use it.
This unfortunately requires dependent code to be updated. The pyconfigparser
interface is in theory wrong - it shouldn't return utf-8 bytes but
local-encoded bytes. I think it's cleaner to make pyconfigparser unaware of
HGENCODING, so I changed pyconfigparser to use unicode, and add compatibility
layer in uiconfig.py.
This also fixes non-ascii encoding issues on user name (especially on Windows).
The hgrc config file should be in utf-8 and the config parser returns explicit
unicode types, and Python code round-trip them with local encodings.
Reviewed By: markbt
Differential Revision: D20432938
fbshipit-source-id: b1359429b8f1c133ab2d6b2deea6048377dfeca1
Summary:
For users on slow connections, hg debugnetwork may not complete within 20s.
Reduce the amount of data we use for the download and upload tests, so that we
have a better chance for the tests completing within the 20s limit.
Reviewed By: farnz
Differential Revision: D20439271
fbshipit-source-id: b3cb0d7e96673c8052022c7606cca80db1788370
Summary: At current, there are some specific AOSP branches that are causing problems for the importer. It would be helpful to be able to run convert jobs that only run on a subset of branches for testing. This commit adds a configuration setting for a whitelist that allows us to specify which repo branches we want to convert, and we only convert those branches.
Reviewed By: tchebb
Differential Revision: D18534728
fbshipit-source-id: bf59605d9c6026e26914c4a7cbe9493bdee77ee5
Summary:
The metalog message is just for display purpose so it does not have to be
byte-to-byte accurate. This solves potential crashes on Python 2 Windows.
File "c:\Tools\hg\python27.zip\edenscm\mercurial\transaction.py", line 587, in _writemetalog
" ".join(map(util.shellquote, pycompat.sysargv[1:])), int(util.timer())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid utf-8
The logic will become more "correct" when we migrate to Python 3.
Reported by: markbt
Reviewed By: markbt
Differential Revision: D20422747
fbshipit-source-id: 41123d132a1e545db77d7321099da611668174f4
Summary:
hg makes sure commit messages are utf-8. So there is no need to check it.
The check logic is also incorrect if the local encoding is not utf-8:
`ctx.description()` returns local-encoding-encoded bytes, which should not
be treated as utf-8 bytes.
Reviewed By: markbt
Differential Revision: D20424241
fbshipit-source-id: 05480e37335c3f2c7971c28c2e23c115dfc6fba7
Summary: This makes it possible to hide draft branches using `hg hide`.
Reviewed By: markbt
Differential Revision: D20403505
fbshipit-source-id: d316e02c24ce636fdc6593f95a5d974b1ba0fc85
Summary:
During cloud sync, the `_applycloudchanges` step will pull all the new commits
from commit cloud, and then record that it has updated to the new workspace
version in the cloud sync state.
If the transaction subsequently fails and gets rolled back, we will roll back
the pull (so the new commits vanish) but we will not roll back the recording of
having updated to the new cloud workspace version.
On the *next* cloud sync, commit cloud will think the user has deliberately
hidden all of those missing commits, and remove them from the cloud workspace.
Write the cloud sync state within the transaction, so if the transaction is
rolled back, we also roll back the recording of the new workspace version.
Reviewed By: quark-zju
Differential Revision: D20420240
fbshipit-source-id: f3d201f299e491cbff47fc6b703a4e1af4015b18
Summary:
Commit cloud sync works at the store granularity, so the status should be
stored in the store vfs. Move it there.
Improve the output of `hg cloud status` by logging more information about the workspace.
Reviewed By: quark-zju
Differential Revision: D20419223
fbshipit-source-id: 4f0d6b9aab55d2bbeaf89b489606f0bc25400de5
Summary:
This diff adds client-side support for the new `designatednodes` mode of the `gettreepack` wireproto command (implemented in D20307442), which allows fetching an exact set of specified tree nodes. This enables the performance benefits of BFS tree fetching over SSH (whereas this sort of fetching was previously only possible over HTTPS via EdenAPI).
To allow us to smoothly control the rollout of this feature, it is gated behind the `treemanifest.ondemandfetch` config option. In theory, this should not be needed since the client will check that the server supports `designatednodes` before attempting to do on-demand fetching, but if users report issues it's nice to have a killswitch.
Reviewed By: quark-zju
Differential Revision: D20382825
fbshipit-source-id: f9aef3fdae08ae84a9720deead3549fe2fba0b9b
Summary:
We cannot differentiate an empty list from a list containing the empty path only. This fixes that.
See D20309700 earlier in this stack for the server-side bits.
Reviewed By: quark-zju
Differential Revision: D20309699
fbshipit-source-id: 02efc747c3c5e2774f949ba0a6370169b55f3cf3
Summary:
In order to support gradual rollout of infinitepush for other backends (e.g. Mononoke), we need the ability to route read vs write requests separately. To achieve this, we need a separate path type: `infinitepush-write` (kind of like `default-push`, the naming inconsistency is a little unfortunate, as I don't want to use `infinitepush-push`).
The desired behavior of the new path type is as follows:
- takes precedence over `infinitepush` path when the user does `hg push -r . --to scratch/bla`
- replaces `infinitepush` path when the user does `hg push infinitepush -r . --to scratch/bla`
- absence of this path means draft pushes will go to `infinitepush` path
- draft pulls always go to `infinitepush` path, and *there's no fallback to `infinitepush-write`*
- commit cloud always talks to `infinitepush-write`, if it is present (meaning that commit cloud pulls do go to `infinitepush-write` path
- this is done, as commitcloud uses infinitepush paths to also check whether something is backed up
- and also, commitcloud may need to pull very soon after something has been pushed
Reviewed By: quark-zju
Differential Revision: D20368158
fbshipit-source-id: 59db174cebbf2b48765dff37bc93aad176c2d7c1
Summary: Those functions are reused in in a future diff.
Reviewed By: sfilipco
Differential Revision: D20367838
fbshipit-source-id: 944babf8c02f0560f8ac8ca5d7c4263432032715
Summary:
The `sparse` extension's dirstate tracking code attempts to read dirstate data
after the command has executed. For commands that didn't require accessing
the dirstate, this can end up reading it for the first time, and can this fail
in some circumstances. Ignore all errors that occur when trying to compute
this telemetry, so they don't break the main command functionality.
In particular, we see reproducible scenarios where `hg` crashes due to this
error when invoked by watchman to compute the list of files changed between
two commits. Watchman currently always invokes Mercurial with `HG_PENDING`
enabled, and this can cause dirstate loading to fail when there is pending
data.
Reviewed By: quark-zju
Differential Revision: D20402221
fbshipit-source-id: 782d9b6eff26d50ef20f080c0cbcbc852e425146
Summary: Remove the old Python, C implementation of getfstype.
Reviewed By: xavierd
Differential Revision: D20313385
fbshipit-source-id: 475c73343aae2fa2f5ad898c7b0879bfa2c80e93
Summary:
A common cause for automigration failure is if the repository has an abandoned
transaction. Just ignore the error and skip migration if this happens.
Otherwise important read-only commands (like `hg debugedenimporthelper`)
refuse to even start in this situation.
Reviewed By: farnz
Differential Revision: D20394546
fbshipit-source-id: abb75fe455e9ee815032431705df7f38ee50283a
Summary:
The code to disable visibility was writing to files without a transaction,
which causes a `ProgrammingError` to be thrown when using metalog.
Reviewed By: farnz
Differential Revision: D20394547
fbshipit-source-id: c529de84ed7b127df18e7f60572cece2169dc520
Summary:
Print output using `repo.ui.write()` rather than directly using `print()` or
`sys.stdout`. This is needed to make the output visible during the
Mercurial `.t` tests.
Reviewed By: farnz
Differential Revision: D20394549
fbshipit-source-id: 7ee50fe26e98c1c34b74ce054d2187050c42b2f3
Summary:
Add type annotations for the `edenscmnative.parsers` methods that handle
serializing and deserializing the dirstate data.
Reviewed By: quark-zju
Differential Revision: D19958218
fbshipit-source-id: 6e20efbc1b0a6ba15b297e47a1e6eec8ed47ee52
Summary:
If `ui.quiet` is set, debugnetwork still ends up outputting the output from the
remote `hostname` command. Prevent this from happening by redirecting the
output to a buffer and logging it separately through `ui.status`.
Reviewed By: quark-zju
Differential Revision: D20384651
fbshipit-source-id: 375687a8e75187a27ef30b441b2ac3a5c0823d00
Summary:
Allow selective running of a subset of the tests by passing options.
* `--connection` runs only the tests for determining connectivity
* `--speed` runs only the speed tests.
Not specifying any option runs all tests.
Reviewed By: quark-zju
Differential Revision: D20384652
fbshipit-source-id: 197e07809bec28ab2ce3fa73cfef6be0fb0fd5b7
Summary:
This command can be used to diff two trees and only print changed paths,
without changed file contents.
With a customized template it can also print changed flags, which can be useful
for watchman use-cases.
Reviewed By: sfilipco
Differential Revision: D20377436
fbshipit-source-id: dad79f1b891182fa612c446114f9daceb4ec5881
Summary: Those experimental revsets are not used in the past month. Remove them.
Reviewed By: sfilipco
Differential Revision: D20355671
fbshipit-source-id: 22d6053af01a56f23b7227b86ebe271aa2b42f41
Summary:
We currently log the server we talk to, which is very nice in order to
benchmark performance of Mononoke vs. hg.
Unfortunately, while this allows for breaking down the samples, it's a little
hard to identify the right samples to break down! Historically, we've
approximated this by looking at the client's hostname, but this isn't always
ideal.
Reviewed By: quark-zju
Differential Revision: D20369686
fbshipit-source-id: 785c67fde09e7b7fc4c30121d14f0bc26e92a355
Summary:
Symlinks are treated a bit differently from plain files, what is stored in the
ContentStore is the destination of the symlink, not it's content (well, the
content of a symlink really is it's destination).
For now, only unix platforms support symlinks, in reality this should be a
filesystem property as writing to ntfs-3g should have the same behavior as on
Windows.
For executable, we just need to mark the file as executable after writing to
it.
Reviewed By: quark-zju
Differential Revision: D20250943
fbshipit-source-id: 022dabc750125df32953a151df7da60db69b2cec
Summary:
During `hg update`, Mercurial forks multiple processes to write files on disk
concurrently, this is done as fetching blobs from the content store, and
writing them to disk is CPU bound. Usually, threads would be the preferred way
of speeding up such process, but unfortunately, Python has GIL that severely
limit the available concurrency. So, multiple processes were chosen.
Unfortunately, the multi-process solution also brings a lot of other issues,
more recently, we've had cases where the connections to the server and memcache
had to be dropped after the fork. In some other cases, this caused deadlocks.
And the solution is not effective on Windows.
Now that Mercurial is getting more and more Rust, we could instead go back to
the threads solution by using them in Rust, and have Python just push work to
them, this is exactly what this change does.
Things that are left to be done, but I wanted to get a diff out first:
- no file path audit
- no file backup
- no symlink creation
- probably other things I'm missing
Reviewed By: quark-zju
Differential Revision: D20102888
fbshipit-source-id: d47829fd7818b97710586b9851880f178048e27b
Summary:
It might be useful to know how long the end-to-end wireproto command to find
the master bookmark took, so log that.
Reviewed By: quark-zju
Differential Revision: D20344686
fbshipit-source-id: 8459a378a0c828a929eee5f9ffd34875a5b8d3cc
Summary:
For high latency connections we will need to warm up the connection before
starting the download or upload test. Facilitate this by making it possible to
run multiple tests within the same connection.
This changes the protocol for the speed-test command to be line-oriented commands.
Reviewed By: farnz
Differential Revision: D20344687
fbshipit-source-id: 1d2a815736caf974ed1aaaaf365b86343d41fe02
Summary:
On Python 2 the Mercurial JSON can be binary, which can break the telemetry
logger. Ensure the JSON string is valid utf-8 (although it might still be
invalid JSON).
Reviewed By: xavierd
Differential Revision: D20343845
fbshipit-source-id: 61e99e742bddf23c7fd5354a5754d79a0a452c28
Summary:
Add `hg debugnetwork` to the things that `hg rage` collects.
Some of the output from `hg debugnetwork` comes from the peer connection, which is not captured when using `ui.pushbuffer`, as it is written using the remote ui object created when the peer connection is set up.
To handle this, add `ui._outputui` as a way to redirect ui output to another ui object, and use this to redirect the output from the remoteui to the local ui object which is buffering output for hg rage.
Reviewed By: quark-zju
Differential Revision: D20307725
fbshipit-source-id: 3b79a77a39c6e2c5f8a7d5cc271ec466653d4db3
Summary:
Add a command that performs various network tests to check the connection to
the Mercurial server for SSH peers.
These tests are:
* Check we can look up the server name in DNS.
* Check we can make a TCP connection to the SSH server.
* Check we can connect and authenticate over SSH and run `hostname`.
* Check we can make a Mercurial wireproto connection and find the `master` bookmark.
* Check the download and upload speed to the server.
Checking the speed requires a companion speed test program on the server.
Reviewed By: mitrandir77, farnz
Differential Revision: D20305227
fbshipit-source-id: adba02a6a1c40ffc20d7bf9d962a5fcf85e06544
Summary: Make a bytestring with 'b""' to fix Python3
Reviewed By: DurhamG
Differential Revision: D20287313
fbshipit-source-id: 7455d1509684bfb0857a3b060bdcca7e658343fd
Summary: We need to encode/decode utf8 when reading/writing to the hgrc file.
Reviewed By: DurhamG
Differential Revision: D20286068
fbshipit-source-id: b1d6828fb62a83ad22414de6883004411f302142
Summary:
We saw >10 timeouts on Mononoke side from people who fetch a lot of files from corp network (https://fburl.com/scuba/mononoke_test_perf/qd15c5f1). They all tried to download a lot of files, and they 90 mins timeout limit.
Let's try to download these files in rather large chunks (e.g. 200000 files). That should help with resumability and prevent timeout errors, and make retries smaller.
It adds an overhead for establishing a new connection after every 200000 files. That shouldn't be a big problem, and even if it's a big problem we can just increase remotefilelog.prefetchchunksize.
Reviewed By: xavierd
Differential Revision: D20286499
fbshipit-source-id: 316c2e07a0856731629447627ae65c094c6eecc5
Summary:
On bad network link (such as on VPN), the reliability of the connection to
Mercurial might be fairly flaky. Instead of failing fetching files, let's retry
a bit first, in the hope that the connection will be back by then.
Reviewed By: quark-zju
Differential Revision: D20295255
fbshipit-source-id: d3038f5e4718b521ae4c3f2844f869a04ebb25e3
Summary:
We've had cases where a git commit goes in that shouldn't be translated
to Mercurial. Let's add an option to skip the commit. Instead of skipping it
entirely (which would require complicated logic to then parent the following
commit on the last converted commit), let's just convert the skipped commit as
an empty commit. This should cover the cases we've encountered so far.
Reviewed By: krallin
Differential Revision: D20261743
fbshipit-source-id: da401863b09c2ac727aae1ceef10a0e8d8f98a7e
