Summary:
Previously, archival.py ran a matchfn on all files reported by the manifest. This is slow for treemanifest repos.
Update this to allow the manifest to run the matchfn itself, which is considerably quicker.
Reviewed By: DurhamG, markbt
Differential Revision: D13624052
fbshipit-source-id: eca022170e6bb0c0cf3c845bcafcb4e35207e825
Summary:
Several people observed this connection closed early message last week
and were unable to go further, so perhaps it would be helpful to
include a bit of additional data in the error message.
Reviewed By: quark-zju
Differential Revision: D13165697
fbshipit-source-id: 8f7d9d29d52697393a8474d6b8697099b0e33442
Summary:
upstream hg started to use separate bundle2 part to push phases but we are stuck with the old
way.
The old way was enabled in facebook.rc but nowhere in the tests except of the one
(where I disabled it) proving that the new way is working.
The original commit introduced the config option as temporary is talking about
pushrebase
changeset: 877f6928075428a4470bd399c82c9b1e9eaba9ad D6156039
user: Durham Goode <durham@fb.com>
date: Wed, 25 Oct 2017 16:39:02 -0800
configs: set devel.legacy.exchange=phases
Summary:
Upstream has enable sending phase bundle parts by default now, but
our server doesn't have the pushrebase fix to make this work. Let's
temporarily disable this until we've updated the servers.
I see in pushrebase support was implemented but the temporary disabling ended
up being permanent.
changeset: f6cf77230612bdf8be0c7fdd6ab21737fdaf35bf D1204
user: Stanislau Hlebik <stash@fb.com>
date: Mon, 23 Oct 2017 09:36:16 -0800
pushrebase: handle pushing phases through separate bundle2 part
Differential Revision: https://phab.mercurial-scm.org/D1204
I would like to switch to this transport because this is the one we are
supporting on Mononoke side and I don't like to have different configs on client side depends on
whether Mononoke is used or not.
Reviewed By: DurhamG
Differential Revision: D13622994
fbshipit-source-id: 63c72654a34584ad31d17b174660166b46f087fb
Summary:
If a user manually disconnects from Commit Cloud Sync, their next background
backup will automatically reconnect them if `commitcloud.autocloudjoin` is set.
Make the `autocloudjoin` setting only work if the user has never connected
to a workspace before. Detect the difference between the two by leaving a
`commitcloudrc` file in place after disconnecting.
Reviewed By: liubov-dmitrieva
Differential Revision: D13621476
fbshipit-source-id: ffccd473cb3da592e5b991dd863b8afed45dc83a
Summary:
If `commitcloud.updateonmove` is set, but backups are disabled, we shouldn't
still follow any new obsmarkers that have appeared (e.g. because of
pullcreatemarkers) when cloud sync runs.
Reviewed By: liubov-dmitrieva
Differential Revision: D13599478
fbshipit-source-id: 10c5c190a08fe5cf72cdfd0165ca61928c0d1800
Summary:
The pullcreatemarkers extension interacts badly with commit cloud sync's
updateonmove. If the current commit has been landed, the next cloud sync
will follow the marker to the landed commit. This is not usually what the user
wants, and slows down usage as they have to wait for the update to finish
before they can continue working.
Reviewed By: liubov-dmitrieva
Differential Revision: D13599477
fbshipit-source-id: f25d50d9dcb023894f2459e632fbd5ff4d172dd0
Summary:
Use of `write!` requires checking for errors, however in this case, there is no
need to use `write!`, as we just want the error as a string.
Reviewed By: ikostia
Differential Revision: D13596497
fbshipit-source-id: 5892025344936936188cf3a8ca227e71eff57d55
Summary:
Teach the fsmonitor extension about Eden, and have it explicitly avoid
wrapping repository objects for Eden-backed repositories.
Reviewed By: quark-zju
Differential Revision: D13523302
fbshipit-source-id: d1114b24311a933fe46baef74d3e514778bd400b
Summary:
When building changegroups, remotefilelog omits filenodes for which the
linknode is known to be available at the server. Since linknodes may
now be null, we need to include these filenodes, as we don't know
whether the linknode is available or not.
Reviewed By: quark-zju
Differential Revision: D13504563
fbshipit-source-id: 8d7106d32f4ec3f2e006b253d68b32c031638b4d
Summary:
When sending a bundle to the server, ensure that the treemanifest linknodes
correctly refer to a commit that is in the bundle.
Treemanifest ignores the linknodes of the subtrees - instead they inherit the
linknode of the root manifest in which they were introduced, so we only need to
fix up the linknode for the root manifest and propagate that to the subtrees.
Reviewed By: quark-zju
Differential Revision: D13504564
fbshipit-source-id: 2f481a4939239784d84d5db12c70d473b3045610
Summary:
When we send a bundle to the server, the treemanifest nodes are sent with
whatever linknode they have. This linknode might not refer to a commit that
the server knows about, which makes the bundle invalid. Add a test that
demonstrates that the server aborts in this case.
Pushrebase works fine as it rewrites the commits, generating new linknodes for
all of the trees.
Reviewed By: quark-zju
Differential Revision: D13504565
fbshipit-source-id: 39894d367c111aea5cef7de2d7da122e39d9debe
Summary:
If we receive a remotefilelog blob from the server where the history
information contains a null linknode, don't store that blob in memcache.
Reviewed By: quark-zju
Differential Revision: D13517715
fbshipit-source-id: 6ea68391ab2488db223ca261e8303ea24e091915
Summary:
Use the cpython-failure crate for generating `PyResult` from `Result` by
mapping to a `PyErr`.
Reviewed By: DurhamG
Differential Revision: D13464988
fbshipit-source-id: d927f89c111dce737b59905ceeab1d30381a8510
Summary:
When I was debugging an eden importer issue with Puneet, we saw errors caused
by important extensions (ex. remotefilelog, lz4revlog) not being loaded. It
turned out that configpaser was checking the "exe dir" to decide where to
load "system configs". For example, If we run:
C:\open\fbsource\fbcode\scm\hg\build\pythonMSVC2015\python.exe eden_import_helper.py
The "exe dir" is "C:\open\fbsource\fbcode\scm\hg\build", and system config is
not there.
Instead of copying "mercurial.ini" to every possible "exe dir", this diff just
switches to a hard-coded system config path. It's now consistent with what we
do on POSIX systems.
The logic to copy "mercurial.ini" to "C:\open\fbsource\fbcode\scm\hg" or
"C:\tools\hg" become unnecessary and are removed.
Reviewed By: singhsrb
Differential Revision: D13542939
fbshipit-source-id: 5fb50d8e42d36ec6da28af29de89966628fe5549
Summary:
`test-check-fix-code.t` was failing due to copyright header missing
from certain files. This commit fixes the files by running
```
contrib/fix-code.py FILE
```
as suggested in the failure message.
Reviewed By: DurhamG
Differential Revision: D13538506
fbshipit-source-id: d8063c9a0e665377a9976abeccb68fbef6781950
Summary:
Even if no new commits are created, absorb might still have "applied" some
changes by deleting commits. Let's fix the end-user message.
Reviewed By: DurhamG
Differential Revision: D13531959
fbshipit-source-id: 4d942f3ccd8201e8b62c8bc1c86227d41021b5f9
Summary:
`scmutil.cleanupnodes` was initially ported from absorb to simplify other
commands. Now use it to simplify absorb itself.
This solves a crash when `self.finalnode` is empty (ex. no new commits are
created, only with commits deletion).
Reviewed By: DurhamG
Differential Revision: D13531961
fbshipit-source-id: 7006b5ac5dfc4db897413d18ccd26eedde3c98d9
Summary: It's used in the upcoming patches.
Reviewed By: DurhamG
Differential Revision: D13531962
fbshipit-source-id: 9caf6c3d5ef079082c9fc677ff2c2ef0e492a1db
Summary:
Previously when a commit does not have a replacment, it will be moved fair back
to a commit that is not being replaced. For example, when A::C is being rebased
to A'::C' and B disappers due to being empty, the old code would move BOOK-B to
Z, while the new code would move BOOK-B to A':
C C'
| |
B BOOK-B -> |
| |
A ------ A' <- new BOOK-B
| /
Z ---- <- old BOOK-B
Note, the current `rebase` implementation overrides the "moves" calcuation used
in cleanupnodes. It already has the new behavior. So there are no changes in
rebase tests.
Right now, the real intended user is absorb, so I'm not adding new tests here.
Without this change, when absorb migrates to cleanupnodes, its test will break.
So I'm not adding new tests here.
Reviewed By: DurhamG
Differential Revision: D13531964
fbshipit-source-id: 03b6afa116e1a7b08b33a2c8856f2e52d6f8043a
Summary:
It was to workaround the upstream obsmarker design which cannot support cycles.
Now that our internal obsmarkers can have cycles just fine, and the upcoming
mutation metadata makes it impossible to have cycles. Drop the workaround.
Reviewed By: DurhamG
Differential Revision: D13531960
fbshipit-source-id: d569172f0d2d5a3b4e1f6589be44ac21a09604f3
Summary: Otherwise absorb never works with HGPLAIN=1.
Reviewed By: DurhamG
Differential Revision: D13531963
fbshipit-source-id: af598b985501db425405f0c851e196e9eddc2350
Summary:
I have been wondering how hg behaves with filenode collision. So I added some
tricky-looking tests about it. It actually shows the existing logic is
problematic :(
Reviewed By: DurhamG
Differential Revision: D13011554
fbshipit-source-id: fffb026e05adc8d8de4a1e5692bbee57293cce4e
Summary:
Use an `asset` to download Cython on demand. So we don't need to install Cython
as build dependency on all supported platforms, and maintain the "Cython"
package for those platforms.
Upgrade to the latest Cython by the way.
Reviewed By: singhsrb
Differential Revision: D13513514
fbshipit-source-id: 5ebe9a3e5b785a8f85cd51624663f9cc1e5c66fd
Summary:
Drop dependency of `python-lz4`.
Add some convertions from bytearray to bytes to make code compatible.
Reviewed By: DurhamG
Differential Revision: D13516212
fbshipit-source-id: 89beb0aa92be4c5442a8e837f509e1eb17bb1512
Summary:
When I replace lz4 to rust.lz4, the test failed. The change fixes it. That
also means datapack corruption detection is not that reliable. However, usually
those files are not changed when they are loaded into an in-memory store, so
it's probaby fine.
Also note the python-lz4 used in production has strange behavior when
compressing an empty string:
In [6]: lz4.compressHC('')
Out[6]: '\x00\x00\x00\x00\xa0#\xd9\x040\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00'
In [7]: lz4.compress('')
Out[7]: '\x00\x00\x00\x00\xa0#\xd9\x040\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00'
In [9]: rustlz4.compress('')
Out[9]: bytearray(b'\x00\x00\x00\x00')
In [10]: rustlz4.compresshc('')
Out[10]: bytearray(b'\x00\x00\x00\x00')
In [13]: lz4.compress('1')
Out[13]: '\x01\x00\x00\x00\x101'
In [14]: rustlz4.compress('1')
Out[14]: bytearray(b'\x01\x00\x00\x00\x101')
Reviewed By: DurhamG
Differential Revision: D13528199
fbshipit-source-id: 9b3e8674f989062928900766156a97d28262c8cb
Summary:
Unfortunately required symbols are not exposed by lz4-sys. So we just declare
them ourselves.
Make sure it compresses better:
In [1]: c=open('/bin/bash').read();
In [2]: from mercurial.rust import lz4
In [3]: len(lz4.compress(c))
Out[3]: 762906
In [4]: len(lz4.compresshc(c))
Out[4]: 626970
While it's much slower for larger data (and compresshc is slower than pylz4):
Benchmarking (easy to compress data, 20MB)...
pylz4.compress: 10328.03 MB/s
rustlz4.compress_py: 9373.84 MB/s
pylz4.compressHC: 1666.80 MB/s
rustlz4.compresshc_py: 8298.57 MB/s
pylz4.decompress: 3953.03 MB/s
rustlz4.decompress_py: 3935.57 MB/s
Benchmarking (hard to compress data, 0.2MB)...
pylz4.compress: 4357.88 MB/s
rustlz4.compress_py: 4193.34 MB/s
pylz4.compressHC: 3740.40 MB/s
rustlz4.compresshc_py: 2730.71 MB/s
pylz4.decompress: 5600.94 MB/s
rustlz4.decompress_py: 5362.96 MB/s
Benchmarking (hard to compress data, 20MB)...
pylz4.compress: 5156.72 MB/s
rustlz4.compress_py: 5447.00 MB/s
pylz4.compressHC: 33.70 MB/s
rustlz4.compresshc_py: 22.25 MB/s
pylz4.decompress: 2375.42 MB/s
rustlz4.decompress_py: 5755.46 MB/s
Note python-lz4 was using an ancient version of lz4. So there could be differences.
Reviewed By: DurhamG
Differential Revision: D13528200
fbshipit-source-id: 6be1c1dd71f57d40dcffcc8d212d40a853583254
Summary:
The `pybuf` provides a way to read `bytes`, `bytearray`, some `buffer` types in
a zero-copy way. The main benefit is to use same code to support different
input types. It's copied to a couple of places. Let's move it to `cpython-ext`.
Reviewed By: DurhamG
Differential Revision: D13516206
fbshipit-source-id: f58881c4bfe651a6fdb84cf317a74c3c8d7a4961
Summary:
Use the newly added zero-copy method to improve Rust lz4 performance. It's now
roughly as fast as python-lz4 when tested by stresstest-compress.py:
Benchmarking (easy to compress data)...
pylz4.compress: 10461.62 MB/s
rustlz4.compress_py: 9379.41 MB/s
pylz4.decompress: 3802.85 MB/s
rustlz4.decompress_py: 3975.61 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5341.69 MB/s
rustlz4.compress_py: 5012.30 MB/s
pylz4.decompress: 6768.17 MB/s
rustlz4.decompress_py: 6651.08 MB/s
(Note: decompress can be visibly faster if we return `bytearray` instead of
`bytes`. However a lot of places expect `bytes`)
Previously, the result looks like:
Benchmarking (easy to compress data)...
pylz4.compress: 10810.05 MB/s
rustlz4.compress_py: 11175.36 MB/s
pylz4.decompress: 3868.92 MB/s
rustlz4.decompress_py: 634.56 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 4565.91 MB/s
rustlz4.compress_py: 622.94 MB/s
pylz4.decompress: 6887.76 MB/s
rustlz4.decompress_py: 2854.79 MB/s
Note this changes the return type from `bytes` to `bytearray` for the
`compress` function. `decompress` still returns `bytes`, which is important for
compatibility. Note that zero-copy `bytes` can not be implemented `compress` -
the size of `PyBytes` is unknown and cannot be pre-allocated.
Reviewed By: DurhamG
Differential Revision: D13516211
fbshipit-source-id: b21f852c390722c086aa2f37a758bf3f58af31b4
Summary: Make it possible to write content directly into a PyBytes buffer.
Reviewed By: DurhamG
Differential Revision: D13528202
fbshipit-source-id: 8c0a4ed030439a8dc40cdfbd72b1f6734a8b2036
Summary:
This allows decompressing into a pre-allocated buffer. After some experiments,
it seems `bytearray` will just break too many things, ex:
- bytearray is not hashable
- bytearray[index] returns an int
- a = bytearray('x'); b = a; b += '3' # will mutate 'a'
- ''.join([bytearray('')]) will raise TypeError
Therefore we have to use zero-copy `bytes` instead, which is less elegent. But
this API change is a step forward.
Reviewed By: DurhamG
Differential Revision: D13528201
fbshipit-source-id: 1cfaf5d55efdc0d6c0df85df9960fe9682028b08
Summary:
I need to convert `Vec<u8>` to a Python object in a zero-copy way for rustlz4
performacne.
Assuming Python and Rust use the same memory allocator, it's possible to transfer
the control of a malloc-ed pointer from Rust to Python. Use this to implement
zero-copy. PyByteArrayObject is chosen because its struct contains such a pointer.
PyBytes cannot be used as it embeds the bytes, without using a pointer.
Sadly there are no CPython APIs to do this job. So we have to write to the raw
structures. That means the code will crash if python is replaced by
python-debug (due to Python object header change). However, that seems less an
issue given the performance wins. If python-debug does become a problem, we can
try vendoring libpython directly.
I didn't implement a feature-rich `PyByteArray` Rust object. It's not easy to
do so outside the cpython crate. Most helper macros to declare types cannot be
reused, because they refer to `::python`, which is not available in the current
crate.
Reviewed By: DurhamG
Differential Revision: D13516209
fbshipit-source-id: 9aa089b309beb71d4d21f6c63fcb97dbc798b5f8
Summary:
This is intended to replace the python-lz4 library so we have a unified code
path.
However, added benchmark indicates the Rust version is significantly slower
than python-lz4:
Benchmarking (easy to compress data)...
pylz4.compress: 10964.14 MB/s
rustlz4.compress_py: 12126.00 MB/s
pylz4.decompress: 3908.29 MB/s
rustlz4.decompress_py: 798.68 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5615.86 MB/s
rustlz4.compress_py: 740.32 MB/s
pylz4.decompress: 6145.68 MB/s
rustlz4.decompress_py: 2423.99 MB/s
The only case where the Rust version is fine is when the returned data is
small. That suggests rust-cpython was likely doing some memcpy unnecessarily.
Reviewed By: DurhamG
Differential Revision: D13516207
fbshipit-source-id: 72150b15c38bc8d8c7e7717a56a41f48d114db19
Summary:
This gives some sense about how fast it is.
Background: I was trying to get rid of python-lz4, by exposing this to Python.
However, I noticed it's 10x slower than python-lz4. Therefore I added some
benchmark here to test if it's the wrapper or the Rust lz4 code.
It does not seem to be this crate:
```
# Pure Rust
compress (100M) 77.170 ms
decompress (~100M) 67.043 ms
# python-lz4
In [1]: import lz4, os
In [2]: b=os.urandom(100000000);
In [3]: %timeit lz4.compress(b)
10 loops, best of 3: 87.4 ms per loop
```
Reviewed By: DurhamG
Differential Revision: D13516205
fbshipit-source-id: f55f94bbecc3b49667ed12174f7000b1aa29e7c4
Summary:
This exposes the underlying lookup functions from `Index`.
Alternatively we can allow access to `Index` and provide an `iter_started_from`
method on `Log` which takes a raw offset. I have been trying to avoid exposing
raw offsets in public interfaces, as they would change after `flush()` and cause
problems.
Reviewed By: markbt
Differential Revision: D13498303
fbshipit-source-id: 8b00a2a36a9383e3edb6fd7495a005bc985fd461
Summary:
This is the missing API before `indexedlog::Index` can fit in the
`changelog.partialmatch` case. It's actually more flexible as it can provide
some example commit hashes while the existing revlog.c or radixbuf
implementation just error out saying "ambiguous prefix".
It can be also "abused" for the semantics of sorted "sub-keys". By replace
"key" with "key + subkey" when inserting to the index. Looking up using "key"
would return a lazy result list (`PrefixIter`) sorted by "subkey". Note:
the radix tree is NOT efficient (both in time and space) when there are common
prefixes. So this use-case needs to be careful.
Reviewed By: markbt
Differential Revision: D13498301
fbshipit-source-id: 637856ebd761734d68b20c15866424b1d4518ad6
Summary: This will be used in prefix lookups.
Reviewed By: markbt
Differential Revision: D13498300
fbshipit-source-id: 3db7a21d6f35a18699d9dc3a0eca71a5410e0e61
Summary:
As we tell users that hg cloud join -w <name> is lgtm, this should be fixed.
hg cloud leave
hg cloud join -w <name>
worked correctly but users could skip cloud leave
currently, it leads to excessive unused connection to IceBreaker.
Differential Revision: D13528559
fbshipit-source-id: 91f36089ccac6718f7a29413c8a3dae80b6b25c6
Summary:
When pulling an infinitepush bundle from the server to a client, it
uses remotefilelog's excludepattern functionality to force the server to serve
filelogs to the client. Unfortunately, it didn't specify what type of pattern,
and therefore it was treated as a regular expression. This meant that any path
that happened to be a regular expression would generally not match itself,
resulting in data not being sent to the client.
The fix is to just prefix these with 'path:' since we know they are all exact
paths.
Reviewed By: phillco, quark-zju
Differential Revision: D13521814
fbshipit-source-id: 8afc35b37d5858913b80ed0babfa0b5e401f0ab4
Summary:
If a filename is a valid regex and it doesn't match itself, that file
data will not be delivered to the client when doing a infinitepush pull that
requires a rebundle. This adds a test. The next diff fixes it.
Reviewed By: phillco
Differential Revision: D13522579
fbshipit-source-id: 6f8c5df20c31b834a33dbb9b73dc9031e26e969b
Summary: Adds the current flavor of the eden rust+python deps to LFS and have setup.py bake them into the IPython.zip.
Reviewed By: quark-zju
Differential Revision: D13516179
fbshipit-source-id: 64a0a86bc97b187f35475a3df4b580e0f0bc5deb
Summary:
Rather than only making it available to `hg dbsh`,
add it for all commands/extensions to use.
Reviewed By: quark-zju
Differential Revision: D13505762
fbshipit-source-id: c79fee888d9394c5ad70d3d8b7f59addef1381a1
Summary:
This is the library shared between the eden cli and the
eden hg extension.
Reviewed By: quark-zju
Differential Revision: D13503730
fbshipit-source-id: 45ab550da3126042cb3baacaf8469b8acd6b1c4a
Summary:
makes it possible to `import thrift` from python2.
This is implemented as a subclass of `asset`, but rather than downloading
files we copy them from their location in fbsource. Extraction
copies them from their fbsource path, optionally excluding specific files
(eg: python3 only files) to the build path.
Reviewed By: quark-zju
Differential Revision: D13503711
fbshipit-source-id: fbe69400de31376ff6135c2c5173a984ff97f282
Summary:
In addition to `six` (already listed), these packages are
required to use the eden extension thrift deps.
I downloaded these using `pip2 download URL` and then uploaded using:
```
$ ../../tools/lfs/lfs.py upload /tmp/futures-3.2.0-py2-none-any.whl -l fb/tools/.lfs-pointers
```
Reviewed By: quark-zju
Differential Revision: D13503336
fbshipit-source-id: 87a21984dbe544882bfc4a818f7b5bff46907693
Summary:
dsp implemented the remotefilelog prefetch logic in https://phab.mercurial-scm.org/D732. With that change, LFS download should only be triggered once per update for a remotefilelog repo since remotefilelog/__init__.py wrapped around checkunknownfiles() (called by calculateupdates() in hg update) to first run prefetch(). However, lfs prefetch is only called when there's missing data in either datastore or historystore.
This diff reuse the logic for lfs prefetch in remotefilelog/fileserverclient.py, but enable lfs prefetch regardless of whether there's missing data in hg stores. The logic to prefetch lfs files is very similar to that of lfs update, except that files to download are batch-processed.
Reviewed By: quark-zju, ikostia
Differential Revision: D13515102
fbshipit-source-id: 5e14c615b192d775db238bacd5e16ceec8141efa
Summary:
Sandcastle jobs are using mercurial in conjunction with Phabricator CATs as a part of Phabricato security effort.
Here is an example of a mercurial command that failed because of this:
https://our.intern.facebook.com/intern/sandcastle/job/1498468091/
This adds logic to support parsing CATs from .arcrc for mercurial commands
Reviewed By: quark-zju
Differential Revision: D13468442
fbshipit-source-id: 033806d0e0779f9e7ade054d21e4cdbbdef08ed0
Summary:
Tests don't close the child process stdout. On newer versions of Python, this
can lead to ResourceWarnings when the test runner thread terminates.
Reviewed By: HarveyHunt
Differential Revision: D13517425
fbshipit-source-id: 6cedf4f39efe1299c41dbde784daf8c159309640
Summary:
Some of the escape sequences in run-tests.py are invalid. These cause
DeprecationWarnings on newer versions of Python.
In both cases, there are `\` characters that need to be escaped as `\\`.
Reviewed By: HarveyHunt
Differential Revision: D13517137
fbshipit-source-id: a899c3c28d55210f5972a515474a2fa69d051671
Summary:
The duringundologlock config and hook were used to test the undolog using
timing. We've replaced that, so remove the hook.
Reviewed By: quark-zju
Differential Revision: D13504644
fbshipit-source-id: a6b5fb308bc8938eec72788d93c9be6c237b72d7
