Summary:
This makes the API easier to use.
Practically this makes it easier for configparser to migrate to minibytes.
Reviewed By: markbt
Differential Revision: D20432942
fbshipit-source-id: ad08eb118d2216054dc24c86b0b129ae82b9d17c
Summary:
For users on slow connections, hg debugnetwork may not complete within 20s.
Reduce the amount of data we use for the download and upload tests, so that we
have a better chance for the tests completing within the 20s limit.
Reviewed By: farnz
Differential Revision: D20439271
fbshipit-source-id: b3cb0d7e96673c8052022c7606cca80db1788370
Summary: At current, there are some specific AOSP branches that are causing problems for the importer. It would be helpful to be able to run convert jobs that only run on a subset of branches for testing. This commit adds a configuration setting for a whitelist that allows us to specify which repo branches we want to convert, and we only convert those branches.
Reviewed By: tchebb
Differential Revision: D18534728
fbshipit-source-id: bf59605d9c6026e26914c4a7cbe9493bdee77ee5
Summary:
The metalog message is just for display purpose so it does not have to be
byte-to-byte accurate. This solves potential crashes on Python 2 Windows.
File "c:\Tools\hg\python27.zip\edenscm\mercurial\transaction.py", line 587, in _writemetalog
" ".join(map(util.shellquote, pycompat.sysargv[1:])), int(util.timer())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid utf-8
The logic will become more "correct" when we migrate to Python 3.
Reported by: markbt
Reviewed By: markbt
Differential Revision: D20422747
fbshipit-source-id: 41123d132a1e545db77d7321099da611668174f4
Summary:
hg makes sure commit messages are utf-8. So there is no need to check it.
The check logic is also incorrect if the local encoding is not utf-8:
`ctx.description()` returns local-encoding-encoded bytes, which should not
be treated as utf-8 bytes.
Reviewed By: markbt
Differential Revision: D20424241
fbshipit-source-id: 05480e37335c3f2c7971c28c2e23c115dfc6fba7
Summary: This makes it possible to hide draft branches using `hg hide`.
Reviewed By: markbt
Differential Revision: D20403505
fbshipit-source-id: d316e02c24ce636fdc6593f95a5d974b1ba0fc85
Summary:
During cloud sync, the `_applycloudchanges` step will pull all the new commits
from commit cloud, and then record that it has updated to the new workspace
version in the cloud sync state.
If the transaction subsequently fails and gets rolled back, we will roll back
the pull (so the new commits vanish) but we will not roll back the recording of
having updated to the new cloud workspace version.
On the *next* cloud sync, commit cloud will think the user has deliberately
hidden all of those missing commits, and remove them from the cloud workspace.
Write the cloud sync state within the transaction, so if the transaction is
rolled back, we also roll back the recording of the new workspace version.
Reviewed By: quark-zju
Differential Revision: D20420240
fbshipit-source-id: f3d201f299e491cbff47fc6b703a4e1af4015b18
Summary:
Commit cloud sync works at the store granularity, so the status should be
stored in the store vfs. Move it there.
Improve the output of `hg cloud status` by logging more information about the workspace.
Reviewed By: quark-zju
Differential Revision: D20419223
fbshipit-source-id: 4f0d6b9aab55d2bbeaf89b489606f0bc25400de5
Summary:
Expose the "TreeSpans" type from tracing-collector so filtering can be done on
them.
Reviewed By: DurhamG
Differential Revision: D19797707
fbshipit-source-id: 56b63c8fb79865666ce923612dbd5a9cc512dce6
Summary:
Previously Rust str was serialized into bytes. To be Python 3 friendly, let's
serialize it into `str`.
Reviewed By: markbt
Differential Revision: D19797706
fbshipit-source-id: 388eb044dc7e25cdc438f0c3d6fa5a5740f22e3d
Summary:
This diff adds client-side support for the new `designatednodes` mode of the `gettreepack` wireproto command (implemented in D20307442), which allows fetching an exact set of specified tree nodes. This enables the performance benefits of BFS tree fetching over SSH (whereas this sort of fetching was previously only possible over HTTPS via EdenAPI).
To allow us to smoothly control the rollout of this feature, it is gated behind the `treemanifest.ondemandfetch` config option. In theory, this should not be needed since the client will check that the server supports `designatednodes` before attempting to do on-demand fetching, but if users report issues it's nice to have a killswitch.
Reviewed By: quark-zju
Differential Revision: D20382825
fbshipit-source-id: f9aef3fdae08ae84a9720deead3549fe2fba0b9b
Summary:
We cannot differentiate an empty list from a list containing the empty path only. This fixes that.
See D20309700 earlier in this stack for the server-side bits.
Reviewed By: quark-zju
Differential Revision: D20309699
fbshipit-source-id: 02efc747c3c5e2774f949ba0a6370169b55f3cf3
Summary:
The goal of the stack is to support "rendering" diffs for large files in scs
server. Note that rendering is in quotes - we are fine with just showing a
placeholder like "Binary file ... differs". This is still better than the
current behaviour which just return an error.
In order to do that I suggest to tweak xdiff library to accept FileContentType
which can be either Normal(...) meaning that we have file content available, or
Omitted, which usually means the file is large and we don't even want to fetch it, and we
just want xdiff to generate a placeholder.
Reviewed By: markbt, krallin
Differential Revision: D20389226
fbshipit-source-id: 0b776d4f143e2ac657d664aa9911f6de8ccfea37
Summary:
In order to support gradual rollout of infinitepush for other backends (e.g. Mononoke), we need the ability to route read vs write requests separately. To achieve this, we need a separate path type: `infinitepush-write` (kind of like `default-push`, the naming inconsistency is a little unfortunate, as I don't want to use `infinitepush-push`).
The desired behavior of the new path type is as follows:
- takes precedence over `infinitepush` path when the user does `hg push -r . --to scratch/bla`
- replaces `infinitepush` path when the user does `hg push infinitepush -r . --to scratch/bla`
- absence of this path means draft pushes will go to `infinitepush` path
- draft pulls always go to `infinitepush` path, and *there's no fallback to `infinitepush-write`*
- commit cloud always talks to `infinitepush-write`, if it is present (meaning that commit cloud pulls do go to `infinitepush-write` path
- this is done, as commitcloud uses infinitepush paths to also check whether something is backed up
- and also, commitcloud may need to pull very soon after something has been pushed
Reviewed By: quark-zju
Differential Revision: D20368158
fbshipit-source-id: 59db174cebbf2b48765dff37bc93aad176c2d7c1
Summary: Those functions are reused in in a future diff.
Reviewed By: sfilipco
Differential Revision: D20367838
fbshipit-source-id: 944babf8c02f0560f8ac8ca5d7c4263432032715
Summary:
This will be used in the Python world for legacy reasons. It shouldn't be used
in new Rust node.
To use it, the name `LegacyCodeNeedIdAccess` has to be used so we can do a code
search to find all users of it.
Reviewed By: sfilipco
Differential Revision: D20367834
fbshipit-source-id: 9b93a29f1461ce24bba6f31a2bbb1f327e216c6d
Summary: This will be useful to actually sort commits.
Reviewed By: sfilipco
Differential Revision: D20367835
fbshipit-source-id: 43bc7835277af3a14ef323ce34247e0c03878dc8
Summary:
The old "AllSet" implementation is not very practical - it does not support
iteration. Practically, the "all()" set comes from the DAG. Change the "all"
concept to a hint similar to "is_topo_sorted", and update the fast path
(intersection) accordingly.
Reviewed By: sfilipco
Differential Revision: D20367837
fbshipit-source-id: fdbf370897c93058bfcab0571c1f6fa4b99b0f6b
Summary: The word "snapshot" more accurately describes its purpose.
Reviewed By: sfilipco
Differential Revision: D20367836
fbshipit-source-id: c91a0bd402fa1718b5d805beedc0e062824c53d3
Summary:
The `sparse` extension's dirstate tracking code attempts to read dirstate data
after the command has executed. For commands that didn't require accessing
the dirstate, this can end up reading it for the first time, and can this fail
in some circumstances. Ignore all errors that occur when trying to compute
this telemetry, so they don't break the main command functionality.
In particular, we see reproducible scenarios where `hg` crashes due to this
error when invoked by watchman to compute the list of files changed between
two commits. Watchman currently always invokes Mercurial with `HG_PENDING`
enabled, and this can cause dirstate loading to fail when there is pending
data.
Reviewed By: quark-zju
Differential Revision: D20402221
fbshipit-source-id: 782d9b6eff26d50ef20f080c0cbcbc852e425146
Summary: Remove the old Python, C implementation of getfstype.
Reviewed By: xavierd
Differential Revision: D20313385
fbshipit-source-id: 475c73343aae2fa2f5ad898c7b0879bfa2c80e93
Summary:
Without this:
In [3]: util.getfstype('')
IOError: [Errno 2] No such file or directory (os error 2)
And there is a code path hitting this:
File "edenscm/mercurial/util.py", line 1483, in checknlink
fstype = getfstype(os.path.dirname(testfile))
# testfile = '.'
# os.path.dirname(".") = ""
The old implementation works fine for an empty path:
In [2]: m.util.getfstype('')
Out[2]: 'eden'
So let's make the new Rust implementation consistent.
Reviewed By: xavierd
Differential Revision: D20313387
fbshipit-source-id: 258c424a3e8a796d983e20b0d4656e8e3f413706
Summary: Expose the Rust fstype implementation so we can remove C and Python code.
Reviewed By: xavierd
Differential Revision: D20313390
fbshipit-source-id: 931b4f6bc8c4b81f30bfea06d055fae3c677eedd
Summary: Similar to D13982877. Try to get names like "fuse.ntfs".
Reviewed By: farnz
Differential Revision: D20313392
fbshipit-source-id: 8363d3d92843e6afb53a0003950be083034bd841
Summary:
Only keep type parameters at the top-level function.
This reduces the binary size and speeds up rustc.
Reviewed By: xavierd
Differential Revision: D20313388
fbshipit-source-id: 29d77731ff462fee1f1bb9f234601e3430198ae7
Summary: This makes the code a bit more portable.
Reviewed By: xavierd
Differential Revision: D20313389
fbshipit-source-id: 080538939fa4d2d72e5905f23ad9be987d952748
Summary:
Rename the main method to "fstype". The API has no relation with repo.
So let's rename it.
Reviewed By: xavierd
Differential Revision: D20313386
fbshipit-source-id: 80dd1231ccccfe945150b117b151bce773f0dfeb
Summary:
Thes test proved to be a bit flaky due to some invalid input from quickeck,
let's filter these cases and ask quickcheck for different input.
Reviewed By: quark-zju
Differential Revision: D20397604
fbshipit-source-id: cc97bef0e286009f1394c7aa2c0429d1e1b1c8c2
Summary:
A common cause for automigration failure is if the repository has an abandoned
transaction. Just ignore the error and skip migration if this happens.
Otherwise important read-only commands (like `hg debugedenimporthelper`)
refuse to even start in this situation.
Reviewed By: farnz
Differential Revision: D20394546
fbshipit-source-id: abb75fe455e9ee815032431705df7f38ee50283a
Summary:
The code to disable visibility was writing to files without a transaction,
which causes a `ProgrammingError` to be thrown when using metalog.
Reviewed By: farnz
Differential Revision: D20394547
fbshipit-source-id: c529de84ed7b127df18e7f60572cece2169dc520
Summary:
Print output using `repo.ui.write()` rather than directly using `print()` or
`sys.stdout`. This is needed to make the output visible during the
Mercurial `.t` tests.
Reviewed By: farnz
Differential Revision: D20394549
fbshipit-source-id: 7ee50fe26e98c1c34b74ce054d2187050c42b2f3
Summary:
Add type annotations for the `edenscmnative.parsers` methods that handle
serializing and deserializing the dirstate data.
Reviewed By: quark-zju
Differential Revision: D19958218
fbshipit-source-id: 6e20efbc1b0a6ba15b297e47a1e6eec8ed47ee52
Summary:
Since the mocked memcache is shared between the tests, we need to make sure the
keys used by the tests are different, otherwise they are just caching each
others data.
Reviewed By: ikostia
Differential Revision: D20388783
fbshipit-source-id: 0f2f926e0ffe0e52e55291e46142808ce0921288
Summary:
Some `use`s are not used on Windows. The code was also formatted using the
latest rustfmt.
Reviewed By: xavierd
Differential Revision: D20379704
fbshipit-source-id: ffadcd68e4e0440dcbd2a4e1ad8532b47a9d83e2
Summary: Similarly to the ContentStore, remove the Arc from MetadataStore.
Reviewed By: quark-zju
Differential Revision: D20376838
fbshipit-source-id: 4321600b752c919b6d9fa7bdee6f6cb7ae083b10
Summary:
The clients should use an Rc/Arc if they need the ability to clone it. This
makes it more obvious and reduces the number of pointer indirection.
Reviewed By: quark-zju
Differential Revision: D20376839
fbshipit-source-id: c56e7e8f89ab17727be621894c329e344a7f3adb
Summary:
If `ui.quiet` is set, debugnetwork still ends up outputting the output from the
remote `hostname` command. Prevent this from happening by redirecting the
output to a buffer and logging it separately through `ui.status`.
Reviewed By: quark-zju
Differential Revision: D20384651
fbshipit-source-id: 375687a8e75187a27ef30b441b2ac3a5c0823d00
Summary:
Allow selective running of a subset of the tests by passing options.
* `--connection` runs only the tests for determining connectivity
* `--speed` runs only the speed tests.
Not specifying any option runs all tests.
Reviewed By: quark-zju
Differential Revision: D20384652
fbshipit-source-id: 197e07809bec28ab2ce3fa73cfef6be0fb0fd5b7
Summary:
The dag crate is designed to work with any kind of binary commit hashes (ex. bonsai,
git or hg). The only use of `types` is to convert from binary to hex. Since dag
already has its own `to_hex` logic in `VertexName`. Let's use that instead.
Reviewed By: sfilipco
Differential Revision: D20378447
fbshipit-source-id: 00ecb551ea927fdb60dd91e5e645064f23139bcd
Summary:
This command can be used to diff two trees and only print changed paths,
without changed file contents.
With a customized template it can also print changed flags, which can be useful
for watchman use-cases.
Reviewed By: sfilipco
Differential Revision: D20377436
fbshipit-source-id: dad79f1b891182fa612c446114f9daceb4ec5881
Summary: Those experimental revsets are not used in the past month. Remove them.
Reviewed By: sfilipco
Differential Revision: D20355671
fbshipit-source-id: 22d6053af01a56f23b7227b86ebe271aa2b42f41
Summary:
Recently there are some Windows-related test flakiness in . All of them are
caused by `file.persist(path)` in `atomic_write_plain` failing with
"Access Denied". Since that can be caused by Windows Anti-Virus scans or other
weird stuff, let's workaround around it using automatically retires.
Process Explorer does not provide extra information:
indexedlog-d0c6135fd7ed9ece.exe 5868 SetRenameInformationFile C:\Users\quark\AppData\Local\Temp\.tmpKERc5G\.tmpcfDsQQ ACCESS DENIED ReplaceIfExists: True, FileName: C:\Users\quark\AppData\Local\Temp\.tmpKERc5G\meta
A successful rename looks like:
indexedlog-d0c6135fd7ed9ece.exe 5868 SetRenameInformationFile C:\Users\quark\AppData\Local\Temp\.tmpKERc5G\.tmpbXEVw0 SUCCESS ReplaceIfExists: True, FileName: C:\Users\quark\AppData\Local\Temp\.tmpKERc5G\meta
Reviewed By: ikostia
Differential Revision: D20379618
fbshipit-source-id: db3e6be3d785875486f7a517df11cbf58bf65ddd
Summary:
Now that the ContentStore can automatically strip the metadata header, no need
for duplicated code in the backingstore.
Reviewed By: fanzeyi
Differential Revision: D20376812
fbshipit-source-id: e863e1cc2fcdc8b9e612a464b305fa25ceb66e13
Summary:
We currently log the server we talk to, which is very nice in order to
benchmark performance of Mononoke vs. hg.
Unfortunately, while this allows for breaking down the samples, it's a little
hard to identify the right samples to break down! Historically, we've
approximated this by looking at the client's hostname, but this isn't always
ideal.
Reviewed By: quark-zju
Differential Revision: D20369686
fbshipit-source-id: 785c67fde09e7b7fc4c30121d14f0bc26e92a355
Summary:
No need for a closure when the function that will be called takes the same
arguments.
Reviewed By: quark-zju
Differential Revision: D20367309
fbshipit-source-id: 986299c4b691330bf17164abd70f3b49ed5263f8
Summary:
In some situations Windows may be unable to remove a file and Mercurial
cannot do anything about it. Instead of raising an error, just ignore it.
In the Python code, errors are ignored in multiple layers, but ultimately
batchremove ignore all OSError errors and just prints a warning about it. A
future diff will build a side channel for errors/warnings reporting, this will
be used to properly report this.
Reviewed By: quark-zju
Differential Revision: D20284803
fbshipit-source-id: 7df80f9e27e720ce819fb4116ea996fca390ad76
Summary:
For now it only checks for symlinks outside of the repo, but can be extended to
check for more.
Caching is used to prevent syscalls to already checked directories, this cache
is intended to be done per-thread, and thus each directory may be checked a
maximum of N, with N being the number of threads. This is deemed reasonable for
now.
Things that the Python pathauditor does that this one doesn't:
- Case checking for Windows,
- Verifies no writes to .hg,
- Windows shortnames aliases,
Reviewed By: quark-zju
Differential Revision: D20265276
fbshipit-source-id: 8e29bedadcaa6df2515e394894ab9e1dd5a9e9a7
Summary:
Symlinks are treated a bit differently from plain files, what is stored in the
ContentStore is the destination of the symlink, not it's content (well, the
content of a symlink really is it's destination).
For now, only unix platforms support symlinks, in reality this should be a
filesystem property as writing to ntfs-3g should have the same behavior as on
Windows.
For executable, we just need to mark the file as executable after writing to
it.
Reviewed By: quark-zju
Differential Revision: D20250943
fbshipit-source-id: 022dabc750125df32953a151df7da60db69b2cec
Summary:
During `hg update`, Mercurial forks multiple processes to write files on disk
concurrently, this is done as fetching blobs from the content store, and
writing them to disk is CPU bound. Usually, threads would be the preferred way
of speeding up such process, but unfortunately, Python has GIL that severely
limit the available concurrency. So, multiple processes were chosen.
Unfortunately, the multi-process solution also brings a lot of other issues,
more recently, we've had cases where the connections to the server and memcache
had to be dropped after the fork. In some other cases, this caused deadlocks.
And the solution is not effective on Windows.
Now that Mercurial is getting more and more Rust, we could instead go back to
the threads solution by using them in Rust, and have Python just push work to
them, this is exactly what this change does.
Things that are left to be done, but I wanted to get a diff out first:
- no file path audit
- no file backup
- no symlink creation
- probably other things I'm missing
Reviewed By: quark-zju
Differential Revision: D20102888
fbshipit-source-id: d47829fd7818b97710586b9851880f178048e27b
Summary:
With this new store, blobs will be transparently written to either an LFS
store, or a non-LFS one, depending on their size.
Initially, and as long as getpackv2 is supported, we also need to support
parsing lfs pointer data that the server is sending and write these to the lfs
pointer store. This code is very adhoc and does manual parsing of the pointer
data, definitively not great, suggestion for a simple and better solution is
welcome :).
From a migration standpoint, the read-only LFS stores are added to the
ContentStore, this allows blobs written in it to be readable at all time even
when `remotefilelog.lfs` isn't set. The code will effecitvely be dormant for a
while until the option is turned on, if we need to disable it, the dormant code
will still be able to read all the blobs written to disk. This forces us to
deploy a release that contains this code to stable first, before setting
`remotefilelog.lfs`.
Reviewed By: quark-zju
Differential Revision: D19986878
fbshipit-source-id: 260f5a542d52e748c0c703bfa7bb8ffac0e7b388
Summary: This makes `RUST_LOG` work for indexedlog tests.
Reviewed By: xavierd
Differential Revision: D20286515
fbshipit-source-id: ff4a1476eb01a9067dabe3622fd598f65fe86a18
Summary:
The tracing / env_logger integration works for hg as a binary. However I'd also
like to use it in library tests. This crate makes it easier to do so.
Reviewed By: xavierd
Differential Revision: D20286507
fbshipit-source-id: f5bf3288ce950591ddfe64b524ad51ce21ee4099
Summary: Those has helped me debugging some issues.
Reviewed By: xavierd
Differential Revision: D20286513
fbshipit-source-id: 012ddb16c2d0efd8f8697a5ecd4564ea31d65630
Summary: Move the scope of spans so the exit code is shown.
Reviewed By: xavierd
Differential Revision: D20286516
fbshipit-source-id: f39cbf60c86ea19a1bb0a09958748f04ff6a42e8
Summary:
Previously env_logger is only initialized if Python is initialized.
This diff makes env_logger initialized for Rust native commands.
Reviewed By: xavierd
Differential Revision: D20286517
fbshipit-source-id: 18fee96c2b41db1da9648d615d1e18809de90a63
Summary:
This means crates like env_logger (which reads $RUST_LOG, and writes to stderr)
can be used for convenient debugging.
Reviewed By: xavierd
Differential Revision: D20286514
fbshipit-source-id: e3b80cc4830ba5cc6dbf7aa1cbb92a4f4f046a54
Summary:
Those metadata include module_path, target, line number, etc, in Rust native
format. They will be used for the upcoming `log` integration.
Reviewed By: xavierd
Differential Revision: D20286510
fbshipit-source-id: 27019b941bef08c0bb3e505bbdae642282dcb141
Summary:
It might be useful to know how long the end-to-end wireproto command to find
the master bookmark took, so log that.
Reviewed By: quark-zju
Differential Revision: D20344686
fbshipit-source-id: 8459a378a0c828a929eee5f9ffd34875a5b8d3cc
Summary:
For high latency connections we will need to warm up the connection before
starting the download or upload test. Facilitate this by making it possible to
run multiple tests within the same connection.
This changes the protocol for the speed-test command to be line-oriented commands.
Reviewed By: farnz
Differential Revision: D20344687
fbshipit-source-id: 1d2a815736caf974ed1aaaaf365b86343d41fe02
Summary:
On Python 2 the Mercurial JSON can be binary, which can break the telemetry
logger. Ensure the JSON string is valid utf-8 (although it might still be
invalid JSON).
Reviewed By: xavierd
Differential Revision: D20343845
fbshipit-source-id: 61e99e742bddf23c7fd5354a5754d79a0a452c28
Summary:
Spliting lock file acquisition from `IdDag::prepare_filesystem_sync` to its own
function.
Useful when looking ahead to split IdDag from IndexedLog.
Reviewed By: quark-zju
Differential Revision: D20316443
fbshipit-source-id: a0fd43439730376920706bb4349ce497f6624335
Summary:
This removes an inline use of the indexedlog indexes.
This is going to be useful when we try to separate IndexedLog specifics from
IdDag functionality.
Reviewed By: quark-zju
Differential Revision: D20316058
fbshipit-source-id: 942a0a71660bb327376c81fd3ac435d002ecca6e
Summary:
Add `hg debugnetwork` to the things that `hg rage` collects.
Some of the output from `hg debugnetwork` comes from the peer connection, which is not captured when using `ui.pushbuffer`, as it is written using the remote ui object created when the peer connection is set up.
To handle this, add `ui._outputui` as a way to redirect ui output to another ui object, and use this to redirect the output from the remoteui to the local ui object which is buffering output for hg rage.
Reviewed By: quark-zju
Differential Revision: D20307725
fbshipit-source-id: 3b79a77a39c6e2c5f8a7d5cc271ec466653d4db3
Summary:
Add a command that performs various network tests to check the connection to
the Mercurial server for SSH peers.
These tests are:
* Check we can look up the server name in DNS.
* Check we can make a TCP connection to the SSH server.
* Check we can connect and authenticate over SSH and run `hostname`.
* Check we can make a Mercurial wireproto connection and find the `master` bookmark.
* Check the download and upload speed to the server.
Checking the speed requires a companion speed test program on the server.
Reviewed By: mitrandir77, farnz
Differential Revision: D20305227
fbshipit-source-id: adba02a6a1c40ffc20d7bf9d962a5fcf85e06544
Summary:
Instead of returning `anyhow::Error` wrapping an `ErrorKind` enum
from each Thrift client method, just return an error type specific
to that method. This will make error handling simpler and less
error-prone by removing the need to downcast the returned error.
This diff also removes the `ErrorKind` enums so that we can be sure
that there are no leftover places trying to downcast to them.
(Note: this ignores all push blocking failures!)
Reviewed By: dtolnay
Differential Revision: D20260398
fbshipit-source-id: f0dd96a7b83dd49f6b30948660456539012f82e6
Summary: Make a bytestring with 'b""' to fix Python3
Reviewed By: DurhamG
Differential Revision: D20287313
fbshipit-source-id: 7455d1509684bfb0857a3b060bdcca7e658343fd
Summary: We need to encode/decode utf8 when reading/writing to the hgrc file.
Reviewed By: DurhamG
Differential Revision: D20286068
fbshipit-source-id: b1d6828fb62a83ad22414de6883004411f302142
Summary:
We saw >10 timeouts on Mononoke side from people who fetch a lot of files from corp network (https://fburl.com/scuba/mononoke_test_perf/qd15c5f1). They all tried to download a lot of files, and they 90 mins timeout limit.
Let's try to download these files in rather large chunks (e.g. 200000 files). That should help with resumability and prevent timeout errors, and make retries smaller.
It adds an overhead for establishing a new connection after every 200000 files. That shouldn't be a big problem, and even if it's a big problem we can just increase remotefilelog.prefetchchunksize.
Reviewed By: xavierd
Differential Revision: D20286499
fbshipit-source-id: 316c2e07a0856731629447627ae65c094c6eecc5
Summary:
On bad network link (such as on VPN), the reliability of the connection to
Mercurial might be fairly flaky. Instead of failing fetching files, let's retry
a bit first, in the hope that the connection will be back by then.
Reviewed By: quark-zju
Differential Revision: D20295255
fbshipit-source-id: d3038f5e4718b521ae4c3f2844f869a04ebb25e3
Summary:
The old code does "read, lock, write", which is unsound because after "lock"
the data just read can be outdated and needs a reload.
Reviewed By: xavierd
Differential Revision: D20306137
fbshipit-source-id: a1c29d5078b2d47ee95cf00db8c1fcbe3447cccf
Summary:
We've had cases where a git commit goes in that shouldn't be translated
to Mercurial. Let's add an option to skip the commit. Instead of skipping it
entirely (which would require complicated logic to then parent the following
commit on the last converted commit), let's just convert the skipped commit as
an empty commit. This should cover the cases we've encountered so far.
Reviewed By: krallin
Differential Revision: D20261743
fbshipit-source-id: da401863b09c2ac727aae1ceef10a0e8d8f98a7e
Summary:
I thought the index function could be the bottleneck. However, the Log reading
(xxhash, decoding vlqs) can be much slower for very long entries. Therefore
using bytes as the lag threshold is better. It does leaked the Log
implementation details (how it encodes an entry) to some extend, though.
Reverts D20042045 and D20043116 logically. The lagging calculation is using
the new Index::get_original_meta API, which is easier to verify correctness
(In fact, it seems the old code is wrong - it might skip Index flushes if
sync() is called multiple times without flushing).
This should mitigate an issue where a huge entry (generated by `hg trace`) in
blackbox does not get indexed in time and cause performance regressions.
Reviewed By: DurhamG
Differential Revision: D20286508
fbshipit-source-id: 7cd694b58b95537490047fb1834c16b30d102f18
Summary: This will be used to more reliably detect index lags.
Reviewed By: DurhamG
Differential Revision: D20286518
fbshipit-source-id: c553b6587363a55603b75df12580588e3100e35f
Summary:
This ensures indexes are complete even if index format or definition has been
changed.
Reviewed By: DurhamG
Differential Revision: D20286509
fbshipit-source-id: fcc4ebc616a4501e4b6fd2f1a9826f54f40b99b8
Summary:
This avoids loading all blackbox logs when `init()` gets called multiple times
(for example, once in Rust and once in Python).
Reviewed By: DurhamG
Differential Revision: D20286511
fbshipit-source-id: ef985e454782b787feac90a6249651a882b6552e
Summary: This API has the benefit that it does not trigger loading older logs.
Reviewed By: DurhamG
Differential Revision: D20286512
fbshipit-source-id: 426421691ad1130cdbb2305612d76f18c9f8798c
Summary:
With the new crate-public interfaces and Debug implementations it's possible to
write tests for DagSet. So let's do it.
Reviewed By: sfilipco
Differential Revision: D20242561
fbshipit-source-id: 180e04d9535f79471c79c4307f6ab6e8e8815067
Summary: The compiler was complaining about these on Windows.
Reviewed By: quark-zju
Differential Revision: D20250719
fbshipit-source-id: 89405e155875a4a549b243e93ce63cf3f53b1fab
Summary:
Don't restrict constructing a c_api datapack store to only Unix, we can
construct it on Windows too by assuming that their path will be valid UTF-8.
Reviewed By: quark-zju
Differential Revision: D20250718
fbshipit-source-id: 07234b6a71b50c803cfe3b962fa727f57037c919
Summary: This returns the ancestors in the reverser order as the parents method.
Reviewed By: sfilipco
Differential Revision: D20265277
fbshipit-source-id: 83277cee3d8e9070fc56d20d4c1877e6782c22f7
Summary:
D18538145 introduced a transaction that spans the entire infintepush
pull. This has a couple of unfortunate consequences:
1. hg pull --rebase now aborts the entire pull if the rebase hits a conflict,
since it's unable to commit the transaction.
2. If tree prefetching fails, it aborts the entire pull as well.
Tests seem to work fine if we scope down this lock.
Reviewed By: xavierd
Differential Revision: D20260480
fbshipit-source-id: d84228ababdb5572401645f74e78df035bf1461b
Summary: Those will be reused by nameset::DagSet.
Reviewed By: sfilipco
Differential Revision: D20242563
fbshipit-source-id: 944e9a04aeb15439256ecea64355b67e326e5c89
Summary:
This is useful for `assert_eq!(format!("{:?}", set), "...")` tests.
It will be eventually exposed to Python as `__repr__`, similar to Python's
smartsets.
Reviewed By: sfilipco
Differential Revision: D20242562
fbshipit-source-id: 5373bb180db7cafebf273ace7cf2cb80fbfb8038
Summary:
In the Python world all smartsets have some kind of "debug" information. Let's
do something similar in Rust.
Related code is updated so the test is more readable.
Reviewed By: sfilipco
Differential Revision: D20242564
fbshipit-source-id: 7439c93d82d5d037c7167818f4e1125c5a1e513e
Summary:
Previously, `flush()` will skip writing the file if there are only metadata
changes. Fix it by detecting metadata changes.
This can potentially fix an issue that certain blackbox indexes are empty,
lagging and require scanning the whole log again and again. In that case,
the index itself is not changed (the root radix entry is not changed), but
only the metadata tracking how many bytes in Log the index covered
changed.
Reviewed By: sfilipco
Differential Revision: D20264627
fbshipit-source-id: 7ee48454a92b5786b847d8b1d738cc38183f7a32
Summary:
On filesystems without symlinks, the test fails because ln prints errors.
Fix the test by using `#if symlink`.
Reviewed By: DurhamG
Differential Revision: D20260904
fbshipit-source-id: 1d0ffcc7e95d2718087fb01297369ca276b59013
Summary:
The `lines` renderer doesn't work if the output encoding doesn't support the
curved line drawing characters. In this case we should fall back to
`lines-square`.
Rename `lines` to `lines-curved`, and change `lines` to pick the best renderer
to use based on what is possible with the current output encoding.
Reviewed By: quark-zju
Differential Revision: D20248022
fbshipit-source-id: dfaf359426528a9cb515fb3e1d366fbfb15162ff
Summary:
The pager may accept a different encoding than either the process encoding or
the output encoding.
For example, on Windows:
* the process encoding may be cp1252 (which is used for all `...A` system calls.
* the output encoding may be cp436 (which is used for writing directly to the console).
* the pager encoding may be utf-8 (which is written to the console using more modern system calls).
To fix this, add a `pager.encoding` config option, which, when set, overrides
the output encoding when writing to the pager.
Reviewed By: quark-zju
Differential Revision: D20247650
fbshipit-source-id: 1e4d1246c95f2102763d879f9783d02acc193a73
Summary:
Using `if cfg!` instead of `#[cfg]` allows for the compiler to understand
that the arguments aren't unused, and silence the warnings.
Reviewed By: quark-zju
Differential Revision: D20242280
fbshipit-source-id: 332dfe17b3a80a1096d15c91c9fb6644bd10e0cd
Summary:
Compiling it on Windows produced a bunch of warning due to
`hgrc_configset_load_path` not being compiled on it. Fixed it so it no longer
depends on Unix specific imports.
Reviewed By: quark-zju
Differential Revision: D20241102
fbshipit-source-id: 3002f961191fbb9bc51aa9ac1154d6d50bd7fe23
Summary:
The `.into_iter()` for this object is being deprecated and won't compile in
the future, fix it now.
Reviewed By: quark-zju
Differential Revision: D20241103
fbshipit-source-id: fdee463ed81cd07a65f3cc4c70a96c88928b3b87
Summary:
While compiling on Windows, this file issues a bunch of warnings, use `if
cfg!` instead of `#[cfg]` to silence them. The behavior is the same, but the
later allows the compiler to recognize that some is not unused.
Reviewed By: quark-zju
Differential Revision: D20241104
fbshipit-source-id: 2cd7f171c7a2f7220cc73bea9be3359260de19b2
Summary:
The change is in theory not necessary. However it improves the reliability on
OS crashes a bit, and can potentially workaround some bugs in filesystems
(as we saw in production where the atomic-written files are empty and the
system didn't crash).
The idea is, the `symlink` syscall does the file creation and "content" writing
together, while there is no way to create a file and write specific content
in one syscall. Note that the C symlink call uses 0-terminated string, and
the Rust stdlib exports it as accepting `Path`. To be safe, we encode binary
or non-utf8 content using `hex`.
For downgrade safety, the write path does not use symlink by default unless
format.use-symlink-atomic-write is set to true. This makes downgrade possible:
the read path is rolled out first, then we can turn on and off the write path.
The indexedlog Rust unit tests and test-doctor.t are migrated to use the new
symlink code paths.
Reviewed By: DurhamG
Differential Revision: D20153864
fbshipit-source-id: c31bd4287a8d29575180fbcf7227d2b04c4c1252
Summary:
This makes it possible to implement atomic_write differently (ex. use a
symlink).
Reviewed By: DurhamG
Differential Revision: D20153865
fbshipit-source-id: 07fa78c2f2dac696668f477c75f65cf70950b73f
Summary:
If hghave fails to check a feature because the feature name is unknown, treat
it as a test failure instead of skipping the entire test. This is especially
useful since `#if feature-name` only affects part of the test and failing to
test the feature should not skip the entire test. It also allows us to capture
issues about mis-spelled feature names or stale feature tests.
This has bitten us twice in the past:
- D18819680 removed `pure` and accidently disabled tests including
`test-install.t`, `test-annotate.t` and `test-issue4074.t`. Those tests got
re-enabled as part of D20155399, while they pass Python 2 tests,
the Python 3 tests were failing.
- D18088850 removed svn related feature checks, which has caused some issues
that got fixed by D18713921 and D18713922.<Paste>
Reviewed By: xavierd
Differential Revision: D20231782
fbshipit-source-id: 6adf99bd79b2a295d4e84ce4da5f9425a100936a
Summary: The test should not assert Python version is "2.*".
Reviewed By: kulshrax
Differential Revision: D20231781
fbshipit-source-id: 2e10c37bb4b665bc4d5d4b27329c4c2cb23d54e3
Summary:
This makes it clear that `log` is a math concept, not an append-only file like
`Log`.
Reviewed By: DurhamG
Differential Revision: D20149376
fbshipit-source-id: 67d2e9584b15f48759ca9b6dfce4279a5b1365a0
Summary: This makes it friendly to Python 2.
Reviewed By: sfilipco
Differential Revision: D20162233
fbshipit-source-id: 5beb7a0f52159afc454332ff6e37e13087177cc0
Summary:
When I run `hg doctor` in my www checkout it fails the assertion check of the
first line of visibleheads is "v1". Make it graceful so doctor can check and
fix other components.
Reviewed By: DurhamG
Differential Revision: D20147969
fbshipit-source-id: 6aee2cab962fcd0ef06a0611d288021e86621249
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
Also patch aho-corasick to fix the issue.
The issue was introduced by [an optimization path](063ca0d253) added in aho-corasick 0.7 series (used by globset 0.4.3).
aho-corasick 0.6.x (globset 0.4.2) are not affected.
The next aho-corasick release (0.7.9) contains the fix.
See https://github.com/BurntSushi/aho-corasick/issues/53 for more context.
Reported by: yns88
Reviewed By: DurhamG
Differential Revision: D20125697
fbshipit-source-id: 592375b43d7ee494bb3e916a1cb11c18f9ebe425
Summary:
`parentfunc` is only needed when adding new nodes to the DAG.
Move it to `addheads` methods instead.
Reviewed By: sfilipco
Differential Revision: D20155398
fbshipit-source-id: 0bddd5f46e84c44891928b9f598a38206917aecb
Summary:
Migrate away from some uses of revision numbers.
Some dead code in discovery.py is removed.
I also fixed some test issues when I run tests locally.
Reviewed By: sfilipco
Differential Revision: D20155399
fbshipit-source-id: bfdcb57f06374f9f27be51b0980652ef50a2c8e0
Summary:
`hiddenoverride` is a hacky implementation that preserves part of another hacky
`inhibit` extension. With our modern setup (inhibit or narrow-heads),
`hiddenoverride` is less useful. Therefore just remove it.
Reviewed By: sfilipco
Differential Revision: D20148011
fbshipit-source-id: f4a5f05b67ae6f315e9b07d50ef03018d6d05df5
Summary:
This makes it so that DAG calculations in NameDag are all using commit hashes.
The `id2node`, `node2id` APIs are still using integer ids, and hopefully their
usage can eventually be removed.
Reviewed By: sfilipco
Differential Revision: D20020527
fbshipit-source-id: ee32b1ccacabd5174ff1556e426b5ed32d2b8507
Summary:
This exposes the NameSet type to the Python world.
The code is similar to the SpanSet wrapper that exists in pydag.
Reviewed By: sfilipco
Differential Revision: D20020521
fbshipit-source-id: 840e009eadca7154f11ca61561da4c48022088f6
Summary: This makes it possible to use NameIter in py_class.
Reviewed By: sfilipco
Differential Revision: D20020529
fbshipit-source-id: b9147b7dccb38d18d8361b420507fcbe97e01351
Summary:
Mercurial has a special case that b'\0' * 20 maps to rev -1 and means
"an empty commit". This cannot be cleanly supported by the zstore commit data,
since sha1("") is not '\0' * 20 and zstore does not allow faked SHA1 keys.
Therefore let's add the special case in the bindings layer. It's possible to
do this check in Python, but that'll be slower.
Reviewed By: sfilipco
Differential Revision: D20020520
fbshipit-source-id: 0686832666646f2e201035992e3951b47c32eb5a
Summary: Use the new NameDag as the backing structure and expose its APIs.
Reviewed By: sfilipco
Differential Revision: D20020528
fbshipit-source-id: ccb49e1a5e757bd35a3f71cfb54ceccfb544664e
Summary: This will be used by commit hash prefix lookup.
Reviewed By: sfilipco
Differential Revision: D20020523
fbshipit-source-id: f2905ddf63098704b08dad8eb48272c3ffba7e25
Summary: Export common types at the top-level of the crate so it's easier to use.
Reviewed By: sfilipco
Differential Revision: D20020526
fbshipit-source-id: e9a0a8bc3cc91f81d0bc74e7530cd4613fc1dd61
Summary: Those just delegate to IdDag for the actual calculation.
Reviewed By: sfilipco
Differential Revision: D20020522
fbshipit-source-id: 272828c520097c993ab50dac6ecc94dc370c8e8b
Summary: This will be used to produce NameSet.
Reviewed By: sfilipco
Differential Revision: D20020519
fbshipit-source-id: abf6d73f2b985b74560d6b5db2800ff25450f02e
Summary: DagSet's SpanSet has fast paths for set operations. Use them.
Reviewed By: sfilipco
Differential Revision: D19912104
fbshipit-source-id: 24b55aa14d03be2f1be59c923e0b8e79d6bcbe6d
Summary: This is similar to hg's fullreposet. It'll be useful as a dummy "subset".
Reviewed By: sfilipco
Differential Revision: D19912108
fbshipit-source-id: 33a95bcb3cf5931a431a1201d1a1f3c627cec7a1
Summary: SortedSet is a wrapper to other sets that marks it as topologically sorted.
Reviewed By: sfilipco
Differential Revision: D19912111
fbshipit-source-id: 2637e8fd29b97f6db0c5bae3f0decd7ac382eeb1
Summary:
Wraps SpanSet + IdMap so it only exposes commit names without ids.
There is no equivalent smartset in Mercurial.
Reviewed By: sfilipco
Differential Revision: D19912112
fbshipit-source-id: 0d257de11527dfa8836065ac94f652730a97a468
Summary: Similar to Mercurial's smartset.baseset. All names are statically known.
Reviewed By: sfilipco
Differential Revision: D19912105
fbshipit-source-id: e4fcf2d59291adb3ca01b3b90f1ac32c65ad7eaa
Summary: The python files were missing in the package, let's add them.
Reviewed By: quark-zju
Differential Revision: D20163637
fbshipit-source-id: 0a7870a21c42d9b92a8b78b51e4954db0d96c593
Summary:
Blame can use a templater which doesn't support bytes. Let's just force
all blame output to be unicode, since it doesn't make a ton of sense to blame
binary files anyway.
Also fix test-annotate.py
Reviewed By: quark-zju
Differential Revision: D19907530
fbshipit-source-id: a7a47246368ed50f65486e824f93552872adc09a
Summary:
Notably, we drop all the encoding business when dealing with json
objects, and instead use mercurial.json.
Reviewed By: sfilipco
Differential Revision: D19888130
fbshipit-source-id: 2101c32833484c37ce4376a61220b1b0afeb175a
Summary:
This is an example about how to use the new Bytes type. The performance change
is not obviously visible in benchmarks since the bottleneck is not at the bytes
copying.
Reviewed By: DurhamG
Differential Revision: D19818720
fbshipit-source-id: a431ae206cfa4fa08b2e162a48b3d7cbcd900f7f
Summary: The APIs are compatible so the switch is straightforward.
Reviewed By: DurhamG
Differential Revision: D19818713
fbshipit-source-id: 504e9149567c90eb661804e0dad20580a401aa76
Summary:
D20042045 changes the meaning of "lag_threshold". Update the value in mutation
store accordingly.
Reviewed By: DurhamG
Differential Revision: D20043116
fbshipit-source-id: 154e6dc2aa88ab0a9a9b21929ae5fa6163dcd403
Summary:
Previously indexes are only updated at `sync()` time. This diff makes it so
`open()` can also update lagging indexes. This should make index migration
(ex. D19851355) smoother - indexes are built in time and users suffer less from
the absent of indexes.
Reviewed By: DurhamG
Differential Revision: D20042046
fbshipit-source-id: 20412661a0ca4f5f67b671137c47b6373a42981d
Summary: The logic is currently only used by `sync()`. I'd like to reuse it at `open()`.
Reviewed By: DurhamG
Differential Revision: D20042044
fbshipit-source-id: 5c9734ff68bdcf8f8c8710c6a821b18d3afeaca0
Summary:
This is more friendly for indexedlog users - deciding lag_threshold by number
of entries is easier than by bytes.
Initially, I thought checking `bytes` is cheaper and checking `entries` is more
expensive. However, practically we will have to build indexes for `entires`
anyway. So we do know the number of entries lagging behind.
Reviewed By: DurhamG
Differential Revision: D20042045
fbshipit-source-id: 73042e406bd8b262d5ef9875e45a3fd5f29f78cf
Summary:
This can be useful for users of indexedlog when they want `Bytes` (to get rid
of the lifetime parameter).
This might be useful for storage layer that wants to take the ownership of the
returned bytes.
Reviewed By: xavierd
Differential Revision: D19818714
fbshipit-source-id: cb2d4e7deff921915e07454fee15cb94a3d5c00d
Summary: Those utilities are no longer necessary since the new code uses Bytes.
Reviewed By: xavierd
Differential Revision: D19818717
fbshipit-source-id: 0b43af0f1eae1a4288e84d4170db058b27f80334
Summary: This simplifies the code a bit and makes it cheaper to clone the Log.
Reviewed By: xavierd
Differential Revision: D19818716
fbshipit-source-id: bbf07b8b36009d53b63d8066ec422fc3c3796840
Summary: It's no longer used since Index now has inlined its checksum logic.
Reviewed By: ikostia
Differential Revision: D19850744
fbshipit-source-id: eb134e4c1613573a2d238710b44ad8119c80a5ee
Summary:
Change index filename and metadata name. This makes sure the new format and old
format are separate so upgrading or downgrading won't have issues.
Reviewed By: DurhamG
Differential Revision: D19851355
fbshipit-source-id: 25dee018073a90040f5818b32b753a3f589c10e0
Summary:
Enhance the index format: The Root entry can be followed by an optional
Checksum entry which replaces the need of ChecksumTable.
The format is backwards compatible since the old format will be just
treated as "there is no ChecksumTable", and the ChecksumTable will be built on
the next "flush".
This change is non-trivial. But the tests are pretty strong - the bitflip test
alone covered a lot of issues, and the dump of Index content helps a lot too.
For the index itself without ".sum", checksum, this change is bi-directional
compatible:
1. New code reading old file will just think the old file does not have the
checksum entry, similar to new code having checksum disabled.
2. Old code will think the root+checksum slice is the "root" entry. Parsing
the root entry is fine since it does not complain about unknown data at the
end.
However, this change dropped the logic updating ".sum" files. That part is an
issue blocking old clients from reading new data.
Reviewed By: DurhamG
Differential Revision: D19850741
fbshipit-source-id: 551a45cd5422f1fb4c5b08e3b207a2ffe3d93dea
Summary:
To solve the soundness issue of ChecksumTable raised by the last diff.
I plan to move Checksum logic to Index. This has multiple benefits:
- Solve the soundness issue of ChecksumTable.
- Indexedlog no longer writes the ".sum" files. `atomic_write` can be quite
slow (tens of milliseconds) on Windows. So this should help perf - with
many indexes, it can save hundreds of milliseconds on Windows per
indexedlog sync.
This diff adds the definition and serialization of the new Checksum entry.
The index format is not updated yet.
Reviewed By: markbt
Differential Revision: D19850742
fbshipit-source-id: df6e6ed12a12ef0d2a782dc9d6b4dc5dec3f4b46
Summary:
With the last change, mmap cost is reduced, but ChecksumTable is unsound in a
corner case: the buffer to check is shorter than what ChecksumTable covers:
checksum: |----chunk----|----chunk----|----chunk--|
buf: |-------------------------------| |
^ ^
logic len physical len
The checksum table will be unable to verify the last chunk, since it does not
have enough data in buf.
The issues is exposed by stress testing the multithread sync tests. It's not
always easy to reproduce, though.
Reviewed By: markbt
Differential Revision: D19850745
fbshipit-source-id: a1a96080163b7b9b56dcd6c1673d5d8d10e18a2b
Summary: This avoids some extra mmap syscalls by ChecksumTable.
Reviewed By: xavierd
Differential Revision: D19818721
fbshipit-source-id: dace55193f2b4b0f35e3868781faa2d2998d3b58
Summary:
This simplifies the code a bit (no special cases about 0-sized mmap buffers)
and makes it cheaper to clone the index buffer (just an Arc::clone, without
another mmap syscall).
Reviewed By: xavierd
Differential Revision: D19818718
fbshipit-source-id: e96d42af74c7f0bb11703c5da31cdfbd5d76c372
Summary:
TreeSpans used to use `&str`, which adds a lifetime to the struct, making it
harder to be used in the Python land. Use a type parameter so TreeSpans<String>
can be used.
Reviewed By: DurhamG
Differential Revision: D19797708
fbshipit-source-id: c66429abfaf16d876151ca6f29da976bed91485d
Summary:
The filtering interface allows callsite to select what they want. It's similar
to manifest walk with files or directory matchers in source control.
Reviewed By: DurhamG
Differential Revision: D19784467
fbshipit-source-id: 5cf6e4016d6fa1c90f8aeccc50809baccd4af5ab
Summary: The idea is that instants (events) can be a drop-in replacement for `ui.log`.
Reviewed By: DurhamG
Differential Revision: D19782897
fbshipit-source-id: 795bbba23d921e460f723f19ef529b203aea366a
Summary: This function will be reused by the next diff.
Reviewed By: DurhamG
Differential Revision: D19782895
fbshipit-source-id: 1e636eabee9b0dffd287a1e6784a24ab2259f51f
Summary: This allows us to define methods on the treespans, such as filtering APIs.
Reviewed By: DurhamG
Differential Revision: D19782896
fbshipit-source-id: 2e7bd8344c0196e382728c26a8233abf944bbf29
Summary: The Thrift generated code depends only on futures 0.3, not 0.1. Thus it isn't necessary to depend on renamed:futures-preview and we can depend on futures-preview directly, which is exposed to Rust code as `futures::`.
Reviewed By: jsgf
Differential Revision: D20145921
fbshipit-source-id: 5cae94ec6747a374c2bf05f124ab237c798de005
Summary: It was a list. Make it possible to use it as a string.
Reviewed By: xavierd
Differential Revision: D20144811
fbshipit-source-id: b280c0344215a4c23ab9c63d89f47adf34fb06f3
Summary: This should help reduce test flakiness.
Reviewed By: xavierd
Differential Revision: D19872952
fbshipit-source-id: d66f6c404534b3f47903b478e3cdfdda5ed46284
Summary:
The state entry of a dirstate tuple is a single character. In python 3
it's a unicode string. To parse it, previously we used 'C' which takes a single
character unicode string and (little did I know) returns an int. We were storing
this in a char, which causes corruption.
Let's switch to reading the string, and just grabbing the first byte.
Reviewed By: xavierd
Differential Revision: D20143094
fbshipit-source-id: d9946c0cefdafe0941f4bdac070659fac27f30e3
Summary:
This new method returns the content of a blob without the copy-from metadata
header.
Reviewed By: DurhamG
Differential Revision: D20102889
fbshipit-source-id: e96f636b7d30460b59707a2cb700d667e616116a
Summary:
Python json produces unicode strings in the parsed results. This breaks
when passed to parts of the code that now assert that byte strings are required
(like the wire protocol). Let's switch phabricator stuff to use Mercurial json,
which produces bytes in Python 2 and unicode in Python 3.
Reviewed By: ikostia
Differential Revision: D20123140
fbshipit-source-id: d1b11426736a0f43ff7e74acf709ab1fd70d5bfe
Summary:
The NameSet is something similar to SpanSet and Mercurial's smartset but speaks
VertexNames instead of Ids. The idea is, NameSet will be part of NameDag APIs,
and potentially replace Mercurial's smartset layer (just smartset the container
types, not the revset language), in a way that revision numbers are completely
hidden behind the scenes.
This diff adds some basic abstraction around iteration-related operations.
Other operations will be added later.
Reviewed By: sfilipco
Differential Revision: D19912109
fbshipit-source-id: 504a26c074282ec51f260535ca63e943124f688e
Summary: EdenFS is planning on throwing an error if a user requests a checkout while a checkout is already in progress. Often, this is already disallowed by a mercurial repository lock, but there are instances where these calls can still get through. We would like to disallow these calls to queue, so we will throw an `EdenError` instead. Without this handling, a full stack trace prints, so this just makes it a bit prettier for the user.
Reviewed By: simpkins
Differential Revision: D20106480
fbshipit-source-id: e33df3d0b7aa42867ee752e4c1f3a47b31ade76b
Summary:
The ssh output order issue is a large contributor to test flakiness.
Example test failures are:
```
--- test-unbundlereplay.t
+++ test-unbundlereplay.t.respondfully.err
@@ -154,9 +154,9 @@
remote: [ReplayVerification] Expected: (master_bookmark, c2e526aacb5100b7c1ddb9b711d2e012e6c
69cda). Actual: (master_bookmark, 893d83f11bf81ce2b895a93d51638d4049d56ce2)
remote: pushkey-abort: prepushkey hook exited with status 1
remote: transaction abort!
+ replay failed: error:pushkey
+ unbundle replay batch item #0 failed
remote: rollback completed
- replay failed: error:pushkey
- unbundle replay batch item #0 failed
[1]
$ cat $TESTTMP/reports.txt
unbundle replay batch item #0 failed
--- test-commitcloud-backup-all.t
+++ test-commitcloud-backup-all.t.err
@@ -59,9 +59,9 @@
remote: pushing 1 commit:
remote: eccc11f58a56 D3
backing up stack rooted at 42952ab62cec
+ backing up stack rooted at 4903fdffd9c6
remote: pushing 1 commit:
remote: 42952ab62cec E1
- backing up stack rooted at 4903fdffd9c6
remote: pushing 1 commit:
remote: 4903fdffd9c6 E2
commitcloud: backed up 8 commits
test-fb-hgext-lfspushrebase-verify-blobs.t
--- test-fb-hgext-treemanifest-pushrebase.t
+++ test-fb-hgext-treemanifest-pushrebase.t.err
@@ -127,9 +127,9 @@
$ hg push --to master -B master --config treemanifest.sendtrees=True
pushing to ssh://user@dummy/master
searching for changes
- remote: baz
remote: prepushrebase.cat hook exited with status 1
abort: push failed on remote
+ remote: baz
[255]
- Disable the hook
```
The order is nondeterministic because the stderr reading thread can read the
content before or after ui.write or ui.write_err in the main thread.
This diff introduces an optional feature in dummyssh that buffers all stderr
output and only write them after the wrapped hg serve process has exited, at
which time the hg client should also have completed its operations and has no
reason to ui.write or ui.write_err anything nondeterministically. Then the
dummyssh wrapper writes out the buffered stderr so the output order becomes
well defined.
Reviewed By: xavierd
Differential Revision: D19872612
fbshipit-source-id: 84710f98a8e6b4a1c283ffecf008585cca12be0a
Summary: This makes the next change easier to see.
Reviewed By: xavierd
Differential Revision: D19872609
fbshipit-source-id: 9263a246258ffd18d8d883da7ced435a91fb5ced
Summary:
Right now, all of our manifest parsing and evaluation is in the repo() class, but this is a design mistake. Over a repo's convert lifetime, a single repo will have many different manifests, based on branch, and location in the commit history. What's worse is that the current design makes it hard to build unit tests and new features like include evaluation.
This commit creates a whole new class called repomanifest, that represents a specific manifest (and its included files). It also has unit tests to test the various operations that the manifest performs, such as path and revision mapping. This commit does not modify the existing converter code outside of the class to use this new implementation.
Reviewed By: tchebb
Differential Revision: D19402995
fbshipit-source-id: b97dadcc595c6332f4495460618317194873a780
Summary:
In the past I saw test breakages where the stderr from the remote ssh process
becomes incomplete. It's hard to reproduce by running the tests directly.
But inserting a sleep in the background stderr thread exposes it trivially:
```
# sshpeer.py:class threadedstderr
def run(self):
# type: () -> None
while not self._stop:
buf = self._stderr.readline()
+ import time
+ time.sleep(5)
if len(buf) == 0:
break
```
Example test breakage:
```
--- a/test-commitcloud-sync.t
+++ b/test-commitcloud-sync.t.err
@@ -167,8 +167,7 @@ Make a commit in the first client, and sync it
$ hg cloud sync
commitcloud: synchronizing 'server' with 'user/test/default'
backing up stack rooted at fa5d62c46fd7
remote: pushing 1 commit:
- remote: fa5d62c46fd7 commit1
commitcloud: commits synchronized
finished in * (glob)
....
```
Upon investigation it's caused by 2 factors:
- The connection pool calls pipee.close() before pipeo.close(), to workaround
an issue that I suspect solved by D19794281.
- The new threaded stderr (pipee)'s close() method does not actually closes the
pipe immediately. Instead, it limits the text to read to one more line at
most, which causes those incomplete messages.
This diff made the following changes:
- Remove the `pipee.close` workaround in connectionpool.
- Remove `pipee.close`. Embed it in `pipee.join` to prevent misuses.
- Add detailed comments in sshpeer.py for the subtle behaviors.
Reviewed By: xavierd
Differential Revision: D19872610
fbshipit-source-id: 4b61ef8f9db81c6c347ac4a634e41dec544c05d0
Summary:
This makes `peer.close()` actually close the ssh connection if it's an
sshpeer. This affects the `clone` path to actually clean up the ssh connection
so we don't depend on (fragile) `__del__`.
I traced the code back to peerrepository.close in 2011 [1]. At that time it
seems the codebase depends on `__del__`. Nowadays the codebase calls `close()`
properly so I think it's reasonable to make the change.
[1]: https://www.mercurial-scm.org/repo/hg/rev/d747774ca9da.
Reviewed By: ikostia
Differential Revision: D19911393
fbshipit-source-id: ea640d1cd82ffcb786e22f47da8116c7f50a4690
Summary:
The added function can be used by extensions to run extra logic before the
"clone" function closes the repos or peers.
This is needed to make the next diff work. Otherwise extensions like remotenames will try to write to a closed sshpeer and cause errors.
Reviewed By: DurhamG
Differential Revision: D19911390
fbshipit-source-id: ca1364e808cebb632e051fbbdcfe4bf0dca721bc
Summary:
Update the `print_status()` function to take a `clidispatch::io::IO` object as
a parameter, instead of a simple output object. This will allow us to also
print error messages from this function in a future diff.
Reviewed By: quark-zju
Differential Revision: D19958504
fbshipit-source-id: bf482fdc4420e1350363a730c6a539cd760aef25
Summary: Updates the C code to support unicode filenames and states.
Reviewed By: simpkins
Differential Revision: D19786275
fbshipit-source-id: e7aeb029b792818b1b1a9c5d3028640b56522235
Summary: There is no need to open a transaction otherwise.
Reviewed By: DurhamG
Differential Revision: D20109840
fbshipit-source-id: e47adaaeea2d7565f3629701d8de4a67d4b55182
Summary:
Verifying the changelog is quite slow and we've had more users needing
to run hg recover these days. Let's finally get rid of the verify step.
Reviewed By: simpkins
Differential Revision: D20109706
fbshipit-source-id: a512d9e11716514bce986b0e3a26347fe6afd955
Summary: Most of the fixes related to encoding in `patch.py`
Reviewed By: DurhamG
Differential Revision: D19713378
fbshipit-source-id: 66ccbd0fc7826ab2d4c05173c7e9edb96700d106
Summary: As I work, it's getting harder and harder to keep my multiple changes from introducing merge conflicts between different branches. We need to break out the repo_source's implementation in to a bunch of different files to make it easier to keep things separate.
Reviewed By: zhonglowu, tchebb
Differential Revision: D20015946
fbshipit-source-id: bf954ac581e5ca9e43c091b6b1b4c539c14471f2
Summary:
Fix the PathRelativizer APIs to accept `Path` and even `str` arguments instead
of just `PathBuf`. The old code required a `PathBuf`, which often forced
callers to make a copy of the path data.
Reviewed By: quark-zju
Differential Revision: D19958505
fbshipit-source-id: 6fa40dd4b75df4e3faf9ad2ae4f0e4e6595669f6
Summary:
The bytes 0.5 is a depencency of newer tokio, it's also newer, and thus better.
Staying on 0.4 means that copies between Bytes 0.4 and 0.5 need to be done,
this will be especially bad in the LFS code since 10+MB buffer will have to be
copied...
One main API change is for the configparser. The code used to take Into<Bytes>
for the keys, I switched it to AsRef<[u8]>.
For hg_memcache_client, an extra copy is performed to build a Delta, since this
code uses an old tokio, and is being replaced right now, the effort of
switching to a new tokio and new bytes was not deemed worth it, the copy will
do for now.
Reviewed By: dtolnay
Differential Revision: D20043137
fbshipit-source-id: 395bfc3749a3b1bdfea652262019ac6a086e61e0
Summary:
`treedirstatemap._repacked` is sometimes set in write(), but does not appear
to be used anywhere. Remove it. (I noticed this since Pyre complains about
it if you enable type checking for `write()`)
Reviewed By: xavierd
Differential Revision: D19958219
fbshipit-source-id: a55e237865160191d814ed950f69c3113bec4f64
Summary:
Add type annotations for the propertycache type.
Unfortunately at the moment Pyre still can't properly type check code that
uses this class, as it does not understand the special `__get__()` method.
It looks like support for this is hopefully coming in D19206575.
Reviewed By: xavierd
Differential Revision: D19958223
fbshipit-source-id: 0f8f15fc6935ec3feaef41d3be373a85225276fe
Summary:
Add type annotations for `dirstate.status()` and
`filesystem.pendingchanges()`
Unfortunately Pyre appears to choke when processing the `dirstate.status()`
function, and currently does not actually report type errors inside this
function at the moment. I've let the Pyre team know about this.
(If Pyre did work correctly it would report one issue since it doesn't realy
understand the `rootcache` decorator applied to `dirstate._ignore`)
Reviewed By: xavierd
Differential Revision: D19958226
fbshipit-source-id: a1cd4b9402a0a449481035cee819533c56b9b336
Summary:
This module previously used to handle deciding how a particular module should
be imported if it had multiple versions (e.g., pure Python or native).
However, as of D18819680 it was changed to always import the native C version.
Lets go ahead and remove it entirely now. Using `policy.importmod` simply
makes it harder for type checkers to figure out the actual module that will be
used.
The only functionality that `policy.importmod()` still provided was verifying
that the module contained a "version" field that looked like what was
expected. In practice these version numbers are not bumped often, so this
doesn't really seem to provide much value in checking that we imported the
correct version that we expected to be shipped with this release.
Reviewed By: xavierd
Differential Revision: D19958227
fbshipit-source-id: 05f1d027d0a41cf99c4aa93cb84a51e830305077
Summary:
Add *.pyi type stub files for most of the native C extensions.
This allows Pyre to type check functions that use these extensions.
These type annotations likely aren't complete, but contain enough information
to allow Pyre to pass cleanly on the existing type-checked locations in the
code using these modules.
Reviewed By: xavierd
Differential Revision: D19958220
fbshipit-source-id: 85dc39a16e595595a174a8e59e419c418d3531be
Summary:
This moves the build rules for the extensions in mercurial/cext into a TARGETS
file in this directory.
This will allow us to start writing `*.pyi` files that contain type
information for these modules, and store them alongside the corresponding `.c`
files. By having the build rules in the top-level `eden/scm` directory we
would have needed to keep the `.pyi` files for these modules directly in the
`eden/scm` directory instead, as the namespace for the `pyi` files is assumed
to be the basemodule plus their path relative to the TARGETS file.
Reviewed By: xavierd
Differential Revision: D19958222
fbshipit-source-id: fdc26ead16663036ffa2562a96eb1649f91cba81
Summary:
The last diff fixed this for fsmonitor. Let's skip these same paths for
non-fsmonitor.
Reviewed By: quark-zju
Differential Revision: D20014808
fbshipit-source-id: 02e3cd9aa29d9c024ba3e8e42a46e21a7c8dfc30
Summary:
Watchman may report invalid utf-8 filenames, even after they've been
deleted. Let's skip them, and print a warning.
Reviewed By: sfilipco
Differential Revision: D20012187
fbshipit-source-id: b13550918a8330ef3eb5c546105d1e054dcb7724
Summary:
Error strings were being converted to unicode if they contained certain
characters. This caused python 2 Mercurial to throw various errors when it tried
to turn them into strings to report errors.
Let's return cpython_ext::Str instead of String.
Reviewed By: sfilipco
Differential Revision: D20012188
fbshipit-source-id: af6fa7d98d68e3c188292e4972cfc1bdb758dbdf
Summary:
Whenever remotefilelog.cacheprocess2 is set, remotefilelog.cachekey is also
set, but the later is not be present when remotefilelog.cacheprocess is. Since
remotefilelog.cacheprocess already includes the cachekey, let's not add it
twice.
This also fixes the issue where hg_memcache_client would die early due to being
passed too many arguments.
Reviewed By: DurhamG
Differential Revision: D20014792
fbshipit-source-id: 8ed6775f70cf967d1c069f8acdb5a782ee819090
Summary:
This error handling can be extremely slow: calling `self.node()` can end up
triggering a linkrev scan of the changelog, which can take over 5 minutes.
If we did want to add this back in the future we would need some sort of API
on `filectx` to try and get the node ID only if it was cheap, and that would
fail fast if this is using remotefilelog and trying to get the node ID will
require scanning the changelog.
Note that KeyError can occur fairly regularly when invoked in long-lived
commands like `hg debugedenimporthelper`. If we are asked about data in a new
commit that was added since this repository was originally opened a KeyError
will be thrown here (in which case `debugedenimporthelper` will call
`repo.invalidate()` and then retry).
Reviewed By: quark-zju
Differential Revision: D20010279
fbshipit-source-id: 0e9b4c163cb9256de57daa91eed70a3736cb1075
Summary: There are two copies of pywatchman in fbcode (!) and some changes didn't make it into the edenscm copy.
Reviewed By: quark-zju
Differential Revision: D19794480
fbshipit-source-id: bcc85e0d3efc225d94b8bfa1e433f6e9cc024643
Summary:
Mercurial filenode hash is computed by including the copy information in the
blob header. Before computing the blob content hash, or returning it to the
upper layers, we need to either strip or reconstruct this header appropriately.
Reviewed By: DurhamG
Differential Revision: D19975887
fbshipit-source-id: 7555e7219e50f4d18ec677fdecc216ee705d7af4
Summary: This will make it easier to support more hash schemes in the future.
Reviewed By: DurhamG
Differential Revision: D19975888
fbshipit-source-id: 8b8ce3b20d72199bac3cd20a48475b5ab56bfc52
Summary:
With the Arc embedded into the store themselves, this forces a second
allocation in order to use them as trait objects. Since in most cases, we do
not want the stores themselves to be cloneable, we can move the Arc outside and
thus reduce the number of pointer indirection.
Reviewed By: DurhamG
Differential Revision: D19867568
fbshipit-source-id: 9cd126831fe2b9ee715472ac3299b7a09df95fce
Summary:
The ContentStore now can read LFS blobs from both the shared cache, and the
local store.
Reviewed By: DurhamG
Differential Revision: D19866249
fbshipit-source-id: a6fb3523495e9d3832613b56438f631cfa552b91
Summary:
With the LFS store being added, and the indexedlog being soon used for trees,
this simplification should help in formalizing the hierarchy of files/folders.
It will look like the following:
<root dir>/lfs: for the lfs store
<root dir>/indexedlog*: for the indexedlog
<root dir>/foobar: for a hypothetical foobar store
For manifests, <root dir> will therefore be: <store dir>/manifests. The
unfortunate part is that the current tree data lives under
<store dir>/packs/manifests. As packfiles will be replaced, this small
discrepency is acceptable.
Reviewed By: DurhamG
Differential Revision: D19866248
fbshipit-source-id: 7ef59ef7df19149b19a529b4f4a45a479cc9d23b
Summary:
This is the first step in having a stronger integration between LFS blobs and
the ContentStore abstraction. The 2 main difference between the Python based
LFS implementation and this one are:
- pointers are not stored alongside plain data,
- blobs are split between local and shared blobs
As of now, no reclamation is being performed for shared blobs, blobs aren't
fetched or uploaded. This will come in future diffs.
Reviewed By: DurhamG
Differential Revision: D19859291
fbshipit-source-id: 45000fc574e6fbd6d3487f4966cad4f49dab731c
Summary:
Some of our upcoming repo merges will make it infeasible for someone to
use a full checkout. Let's add a config that will warn users of this. It has a
few levels, starting with a suppressable hint, then a non-suppressable warning,
then a suppressable exception, then a non-suppressable exception.
Reviewed By: ikostia
Differential Revision: D19974408
fbshipit-source-id: bad35a477ad8626dbc0977465368f5d71007e2d5
Summary:
On Windows, there are *two* 8-bit encodings for each process.
* The ANSI code page is used for all `...A` system calls, and this is what
Mercurial uses internally. It can be overridden using the `--encoding`
command line option.
* The OEM code page is used when outputing to the console. Mercurial has no
concept of this, and instead renders to the console using the ANSI code page,
which results in mojibake like "Θ" instead of "é".
Add the concept of an `outputencoding`. If this differs from `encoding`, we
convert from the local encoding to the output encoding before writing to the
console.
On non-Windows platforms, this defaults to the same encoding as the local encoding,
so this is a no-op unless `--outputencoding` is manually specified.
On Windows, this defaults to the codepage given by `GetOEMCP`, causing output
to be converted to the OEM codepage before being printed.
For ordinary strings, the local encoded version is wrapped by `localstr` if the
encoding does not round-trip cleanly. This means the output encoding works
even if the character is not represented in the local encoding.
Unfortunately, the templater is not localstr-clean, which means strings can get
flattened down to the local encoding and the original code points are lost. In
this case we can only output characters which are in the intersection of the
encoding and the output encoding.
Most US English Windows systems use cp1252 for the ANSI code page and cp437 for
the OEM code page. These both contain many accented characters, so users with
accented characters in their names will now see them correctly rendered.
All of this only applies to Python 2.7. In Python 3, everything is Unicode,
the `--encoding` and `--outputencoding` options do nothing, and it just works.
Reviewed By: quark-zju, ikostia
Differential Revision: D19951381
fbshipit-source-id: d5cb8b5bfe2bc131b2e6c3b892137a48b2139ca9
Summary:
`hg rage` generates the rage in the user's encoding. Since pastes are expected
to be in UTF-8, non-UTF-8 encodings result in garbled pastes.
Similarly, the lines-dec graph renderer uses escape sequences that won't work
on web pages, and the lines graph renderer uses curved lines which don't
render very well either. Force the use of the lines-square graph renderer,
which renders well.
Reviewed By: quark-zju
Differential Revision: D19951382
fbshipit-source-id: d1a5fd2ef195658f9bf10210088031474355f168
Summary:
The Rust graph renderer expects the message to be a unicode string, so ensure
we convert it from the local encoding before passing it to Rust.
Reviewed By: quark-zju
Differential Revision: D19951383
fbshipit-source-id: 644862c63873079364cb9902bd1bb49de8aa1ab9
Summary:
See later in this stack for motivation. This seems to work fine, and it allows
characters that don't fit latin1 when rendering diffs.
Reviewed By: markbt
Differential Revision: D19969743
fbshipit-source-id: 79c4afce5a19822d9b075d23ff4c88aa76ce2f42
Summary:
As of 63c471ad8a4ba0bebd1acf70569bcdcefc3fffbf in upstream Dulwich, it
now turns commands into unicode. Unfortunately, _ssh.py in hggit sees that the
type is no longer str or bytes and thinks it's an array and puts spaces between
every letter, causing it to break.
Let's allow unicode. This broke because dulwich was recently upgraded.
Reviewed By: sfilipco
Differential Revision: D19983215
fbshipit-source-id: 059756905bf4b2c73009001b078c8723ae378246
Summary: This should get rid of the extraneous uninitialized attribute errors related to `setUp` and abstract classes.
Reviewed By: simpkins
Differential Revision: D19964487
fbshipit-source-id: 52d5a6496e372d99d4398473f9ed7672228a76f5
Summary:
This is a revised version of D19887220.
D19887220 has 2 problems:
- It can silently ignore the mt.exe error after failures of all retries.
- There is another place that `mt.exe` runs that is not covered by retry.
This diff fixes them by wrapping the `set_long_paths_manifest` function
directly so it covers two `mt.exe` places, and makes sure all retry failure
is still a failure.
Reviewed By: sfilipco
Differential Revision: D19977802
fbshipit-source-id: 774d0c42b247a7e111841cd69f71760a5544d685
Summary:
Update includes to the third-party xdiff.h file to use absolute includes
from the repository root. This allows many parts of our internal build
tooling to work better, including automatic dependency processing.
Reviewed By: xavierd
Differential Revision: D19958228
fbshipit-source-id: 341dd8c94f1138cf4a387b92e1817b2a286d6aa1
Summary:
Update the C files under edenscm/mercurial/cext to use absolute includes from
the repository root. Also update a few of the libraries in edenscm/mercurial
that the cext code depends on.
This makes these files easier to build with Buck in fbsource, and reduces the
number of places where we have to use deprecated Buck functionality to help
find these headers. This also allows autodeps to work with the build targets
for these rules.
Reviewed By: xavierd
Differential Revision: D19958221
fbshipit-source-id: e6e471583a795ba5773bae5f16ed582c9c5fd57e
Summary:
Remove `thirdparty/pyre2/__init__.py` from the `libhg` sources list.
We don't compile the `thirdparty/pyre2/_re2.cc` file in the fbcode build, so
importing the `__init__.py` module from this package just triggers an
ImportError when the code tries to use it. The code then always falls back to
using the version of pyre2 included from the `fb-re2` wheel.
Dropping the `__init__.py` module from our library should simply trigger an
ImportError earlier when we can't even find this file, and the code will still
fall back to using `fb-re2`.
Including this `__init__.py` file just causes issues for type checking, since
it causes us to try and type check this file even though its dependencies are
not present.
Reviewed By: xavierd
Differential Revision: D19958224
fbshipit-source-id: 34ea8806b6ee9377f17a9318c64c91ec242225df
Summary:
Some of the methods in eden_dirstate_map.py had comments that were close to
type annotations that were added a couple years ago. Update them to proper
type comments that can be recognized by Pyre and mypy.
Also remove the unused create_clone_of_internal_map() method.
Reviewed By: chadaustin, xavierd
Differential Revision: D19958225
fbshipit-source-id: b753c030acb15cf4f8d8c536614e657ee1bcba52
Summary:
Update the `eden_dirstate_map` class to store `dirstatetuple` objects instead
of plain tuples in its `_map` member variable. Without this the `filefoldmap`
code that is used on Windows fails, as it directly accesses `self._map` and
expects it to contain `dirstatetuple` objects.
Reviewed By: DurhamG, pkaush
Differential Revision: D19841881
fbshipit-source-id: ddb7523b598cfd8ec8719a8a74446cefcb411358
Summary: Generate the Cargo.toml files inside xdiff with autocargo. This will enable Mononoke to depend on this code easily without sacrificing anything on eden/scm side.
Reviewed By: aslpavel
Differential Revision: D19948741
fbshipit-source-id: 905ff3d64b90830e5f075e4c6ed2b3de959e3f00
Summary:
Not being able to prefetch draft parent trees should not be considered as a
fatal error.
This code path is causing trouble with narrow-heads clone:
1. Streaming clone. The client gets a changelog.
2. The client runs "pull" to get new commits. The prefetchdraftparents code path runs.
3. The client has stale remote names, and public() is lagging. `prefetchdraftparents`
will try to fetch trees at the old master, but the repo is not configured properly.
That causes a stacktrace like:
$ /usr/bin/hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/www.rc ssh://hg.fb.com/repo repo
connected to hg.fb.com
streaming all changes
searching for changes
adding commits
adding manifests
adding file changes
added 1 commits with 0 changes to 0 files # <<<< No traceback if this says "0 commit".
Traceback (most recent call last):
File "edenscm/hgext/remotenames.py", line 1464, in exclonecmd
orig(ui, *args, **opts)
File "edenscm/hgext/remotefilelog/__init__.py", line 433, in cloneshallow
orig(ui, repo, *args, **opts)
File "edenscm/mercurial/commands/__init__.py", line 1615, in clone
shareopts=shareopts,
# shareopts = {'mode': 'identity'}
File "edenscm/mercurial/hg.py", line 741, in clone
exchange.pull(local, srcpeer, revs, streamclonerequested=stream)
File "edenscm/mercurial/util.py", line 621, in __exit__
self.close()
File "edenscm/mercurial/transaction.py", line 46, in _active
return func(self, *args, **kwds)
File "edenscm/mercurial/transaction.py", line 543, in close
self._postclosecallback[cat](self)
# cat = bin('6472616674706172656e74747265656665746368')
File "edenscm/hgext/treemanifest/__init__.py", line 490, in _parenttreefetch
self.prefetchtrees([c.manifestnode() for c in draftparents])
# c = <changectx b5ad643b3009>
# draftparents = [<changectx b5ad643b3009>]
File "edenscm/hgext/treemanifest/__init__.py", line 522, in prefetchtrees
self._prefetchtrees("", mfnodes, basemfnodes, [], depth)
# basemfnodes = [bin('a25f17018d7cd07f1f6bc3076f95c5980ba087a9')]
# mfnodes = [bin('ad717aac7700e783a1d84f3330d13a7731a4726a')]
File "edenscm/hgext/treemanifest/__init__.py", line 529, in _prefetchtrees
fallbackpath = getfallbackpath(self)
File "edenscm/hgext/treemanifest/__init__.py", line 2173, in getfallbackpath
if util.safehasattr(repo, "fallbackpath"):
File "edenscm/mercurial/util.py", line 190, in safehasattr
return getattr(thing, attr, _notset) is not _notset
# attr = 'fallbackpath'
File "edenscm/mercurial/util.py", line 904, in __get__
result = self.func(obj)
File "edenscm/hgext/remotefilelog/shallowrepo.py", line 42, in fallbackpath
"no remotefilelog server " "configured - is your .hg/hgrc trusted?"
Abort: no remotefilelog server configured - is your .hg/hgrc trusted?
abort: no remotefilelog server configured - is your .hg/hgrc trusted?
Fix it by making prefetchdraftparents non-fatal. This would hopefully unblock
narrow-heads rollout.
Reviewed By: xavierd
Differential Revision: D19957251
fbshipit-source-id: e65bbe6bf422776effe49055f7332ec538177a41
Summary:
This will allow us to improve our dashboards filtering out errors we are
responsible for, like missing certs on the machines.
Reviewed By: mitrandir77
Differential Revision: D19950614
fbshipit-source-id: 73503e984dfe8513a700fdcb2fc36b1618c20a4f
Summary:
Commit messages and extras can be unbounded in size. This can cause problems if users create commits with exceptionally large messages or extras. Mercurial will commit these to the changelog, increasing its size. On Mononoke, large commit messages may go over the cacheing threshold, resulting in poor performance for requests involving these commits as Mononoke will need to reload on every access.
Commit messages should not usually be that large. Mostly likely it will happen by accident, e.g. through use of `hg commit -l some-large-file`. Prevent this from happening by accident by adding configuration for soft limits when creating commits.
If a user really does need to create a commit with a very large message or extras, they can override using the config option.
Reviewed By: xavierd
Differential Revision: D19942522
fbshipit-source-id: 09b9fe1f470467237acc1b20286d2b1d2ab25613
Summary:
This parameter was originally removed in D12811551, but re-added in D12855935
due to the fact that at the time the `eden_dirstate.py` and `dirstate.py`
files were deployed in separate RPMs and could not be updated together
atomically. We now deploy these files together, so we can drop this extra
unnecessary argument.
Reviewed By: chadaustin
Differential Revision: D19913057
fbshipit-source-id: 0f0b4fde4b3124a8fc5bb568551b4e67de14d410
Summary:
Sometimes the treestate points to an unknown commit (ex. aborted transaction
might strip commits). While `debugrebuilddirstate -r HASH --hidden` is able to
fix it, it is too slow.
This diff adds treestate repair logic to the `doctor` command. It scans through
the treestate files, find a most recent `Root` entry with `p1` pointing to a
known commit.
This can be much faster than `debugrebuilddirstate` in some cases, because the
watchman clock might still be valid, and the NEED_CHECK file list might still
be small. In that case, `status` can still be fast.
Since treestate atomically updates all information needed for `status`
calculation (parents, need-check-files (or, "non-normal files"), watchman-clock
(only with fsmonitor), and stat for clean files). Reverting to a previous state
is still atomic. Correctness-wise, this is equivalent to aborting a "large"
transaction, and restoring treestate data to the state before the transaction.
It should be consistent, and the next `status` call won't mis-report files like
the dangerous `debugsetparents` command.
Reviewed By: DurhamG
Differential Revision: D19864422
fbshipit-source-id: d5d2f8b43a0c15ea2ac0e3c164edec7deeb8451f
Summary:
See the test change. Without this change repairing the changelog won't give the
user back a working repo.
Reviewed By: markbt
Differential Revision: D19864421
fbshipit-source-id: b84582c5302469828c8cfcb3db362ea82f2eea63
Summary:
Reuse utilities in the fixcorrupt extension to repair changelog.
This is better than fixcorrupt because `hg doctor` does not require a repo
object. Some messages are updated so they become more consistent with the
rest of `hg doctor`.
The main motivation is to get changelog fixed early, so other repair logic can
check if a commit hash is known by changelog or not.
Reviewed By: markbt
Differential Revision: D19864418
fbshipit-source-id: 6f95c6c6191d7db2a474a07a5278a857cf41d8e2
Summary:
Run 'edenfsctl doctor' on an edenfs repo. If there is no current repo, it might
be caused by edenfs daemon stopped running. So let's also run edenfsctl doctor
in that case.
Reviewed By: markbt
Differential Revision: D19864419
fbshipit-source-id: d2a49a126a040845b88b4883d214162326d08d8d
Summary:
We're seeing a user have issues because their username contains unicode
characters and sampling's use of json doesn't handle it well. I've not been able
to repro it unfortunately, but let's go ahead and switch sampling to use
mercurial.json.
Differential Revision: D19895419
fbshipit-source-id: a1f087d1e2c7568488c2b8d54f267bd5c8266202
Summary: This will be used in the LFS store.
Reviewed By: DurhamG
Differential Revision: D19895803
fbshipit-source-id: 4cf447987c10fed0b5c98904f20c841428965d89
Summary:
In some cases, higher level stores may want to store data in either a plain
IndexedLog, or in a RotateLog, for local and shared data. Due to slight
difference between the 2, they can't easily be adapted into a common trait.
Instead let's just wrap both into an enum and implement the main functions that
the higher level stores need.
The first use of this will be the LfsStore, future use will include the
IndexedLogDataStore and the IndexedLogHistoryStores.
Reviewed By: DurhamG
Differential Revision: D19859292
fbshipit-source-id: 920572e0cf5f69bda4901a727a6b0dc0f08fc8d0
Summary: There's still some issues, but it's a lot closer.
Reviewed By: quark-zju
Differential Revision: D19802023
fbshipit-source-id: da539094cbc0ba3542e4b5fd3d49f5f80455ec23
Summary:
There was a spot where we returned bytes for a filepath. Fix this to
make dirstate tests pass more.
Reviewed By: quark-zju
Differential Revision: D19786274
fbshipit-source-id: 7465cae8bb2e3be7758abc6279ed3f5f59581732
