Summary:
Move all Rust bindings to a single python extension, `bindings`. This should
improve compilation time and make things simpler.
Reviewed By: quark-zju
Differential Revision: D13923866
fbshipit-source-id: 560592b5a6c0c4f1b836c755ef123666a1059164
Summary:
We will probably not be supporting secret phase in the future anyway
so lets get rid of it.
Reviewed By: quark-zju
Differential Revision: D13928907
fbshipit-source-id: 7edcd84d2ee3c37124dcf9d3ba92c0610d074157
Summary:
Add an option that let's us specify the date of each commit in a pushrebase.
It should be specified via a config option, which accepts a name of a file in
which json dictionary should be stored. Key of this dictionary is a commit hash, value
is a date.
Reviewed By: ikostia
Differential Revision: D13803174
fbshipit-source-id: 6271c18c61399b89c92dce7a4fe63c8fae8dae7c
Summary:
The rename to edenscm broke the test.
(Note: this ignores all push blocking failures!)
Reviewed By: phillco, ikostia
Differential Revision: D13900989
fbshipit-source-id: ad04fcb6889a273bb39901b2288ae1aef74531a6
Summary:
Original commit changeset: 1cdd43658889
Trying to enable this again.
The details are in the original diff: D13622994
Reviewed By: quark-zju
Differential Revision: D13801916
fbshipit-source-id: a7a1aee6faa0d1db7535dfd3cfab523829c62900
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
The tempfile rust crates opens the file with RW permissions for the user only,
but once written out to disk, the permissions needs to be readable by everyone.
Unfortunately, rust doesn't have a portable way of doing this, so we have to
resort to using `if cfg!(unix)` conditions for doing this.
Reviewed By: DurhamG
Differential Revision: D13703406
fbshipit-source-id: 688bc679b5c1a7943ceab723c1f649d555b61a7a
Summary:
Loosefiles are still being fetched from memcache, and thus until we switch
memcache to provide pack files, repack needs to be able to handle loosefiles.
We could either implement the loosefile repack logic in rust, or we could just
let the python code do it. Since we plan on having memcache serve pack files
soon, the second approach is prefered as it result in the least amount of work.
Reviewed By: DurhamG
Differential Revision: D13707886
fbshipit-source-id: 5373b61f9749c158e6dfb026f6b12a7dd2a67802
Summary:
Replace the `-w` option of the commit cloud commands that process workspace
names with general options of the form:
* `-u` or `--user` - to select the workspaces of a particular user.
* `-w` or `--workspace` - to select a particular workspace for that user.
A (hidden) `--raw-workspace` option is provided to allow use a workspace
by its exact name, should that be needed.
The `-w` option still accepts full user workspaces, but shows a warning.
These options apply uniformly to any command that takes a workspace.
Reviewed By: liubov-dmitrieva
Differential Revision: D13730839
fbshipit-source-id: d1b884b100caf909619511db4861eb1b880a0d3e
Summary:
Commit cloud maintains the infinitepush backup state to match what it has
synced to commit cloud. It should only include locally backed-up heads.
When syncing with the server and omitting some heads, don't include the omitted
ones.
Reviewed By: liubov-dmitrieva
Differential Revision: D13719477
fbshipit-source-id: 817c1a73a34af6b5550f6254e2b08d01283760a5
Summary:
We should not be attaching copy headers to lfs metadata blobs since
they store the data separately. All this logic was a little overly complicated.
The file text we receive should be passed through unmodified in all cases except
when storing in the loose file format, which needs to strip the copy metadata
from the blob.
Reviewed By: ikostia
Differential Revision: D13703708
fbshipit-source-id: d7e4eb703599d2355009efe070833013d1fcd4fc
Summary:
Enabling packlocaldata is causing local move data contents to be
corrupted by the copy header being tacked on the front. Let's add a test
demonstrating the failure.
Reviewed By: ikostia
Differential Revision: D13703709
fbshipit-source-id: d891042a5cab639a4533a89a508865115c482ba1
Summary:
In our linux deployments it was relatively straightforward
to import the mercurial runtime from a python process running the
system python executable. Our macOS deployments are a lot more
complex because they do not use the system python and do not install
the mercurial python packages in the python path of the target
python executable.
It is simpler to move the import helper functional into a mercurial
command that we can invoke instead of our own helper program.
This diff moves the script to be a debug command and adjusts its
argument parsing to match the mercurial dispatcher requirements.
There are some stylistic mismatches between this code and the
rest of mercurial; I'm suggesting that we ignore those as the
medium term solution is that this command is replaced by eden
directly consuming the rust config parsing code and by native
rust code to perform the data fetching that we need.
Reviewed By: pkaush
Differential Revision: D13522225
fbshipit-source-id: 28d751c5de4228491924df4df88ab382cfbf146a
Summary:
Directory listing is different in every OS, and due to the current repack
implementation, this directly affect the order in which the packfiles are added
to the new one. Since the resulting packfile name depends on the hash of its
content, the name was influenced by the directory order.
By sorting the files in list_packs, the packfile name will be independent of
the directory listing and thus be the same for all the OSes.
Reviewed By: singhsrb
Differential Revision: D13700935
fbshipit-source-id: 01e055a0c1bcf7fb2dc4faf614dfb20cd4499017
Summary:
This ports the logic from `eden/hg/eden/` to `scm/hg/mercurial/`.
Note this does not delete the logic from `eden/hg/eden` as part of this
change because we may continue to do Eden releases before we roll out a
version of Hg with this code. Only once Hg has been rolled out everywhere
[that is using Eden] can we consider removing
`/usr/local/fb-mercurial/eden/hgext3rd/eden`.
Reviewed By: quark-zju
Differential Revision: D10316761
fbshipit-source-id: cae1dfad831ad6505590628cf969897167e84b30
Summary:
The user, description, and bookmarks must be converted from the local encoding
before they can be included in the JSON object, as JSON can only contain UTF-8
strings.
Note that `crdump` will not work with filenames that are not UTF-8 encoded.
This doesn't fix that.
Reviewed By: HarveyHunt
Differential Revision: D13635999
fbshipit-source-id: 5379656e39b73d7d7f905d06de50851f17c54e8b
Summary: For now, combine all files smaller than 100MB that accumulate to less than 4GB.
Reviewed By: DurhamG
Differential Revision: D13603760
fbshipit-source-id: 3fa74f1ced3d3ccd463af8f187ef5e0254e1820b
Summary:
Update pest to 2.1.0.
This version has a new behaviour for parser error messages: the line feed at
the end of the line is shown in the error output.
Reviewed By: wez
Differential Revision: D13671099
fbshipit-source-id: b8d1142a44a56a0b21b3b72cf027f3f8a30f421e
Summary: In D13363853, a basic rust based repack was implemented, but couldn't be called from Python. For now, this version will be called when the repack.userust config is set, and when requesting a pack-only repack, as the rust version doesn't support loose objects (?).
Reviewed By: DurhamG
Differential Revision: D13555855
fbshipit-source-id: 83d53b33f2d3b6ae3167801d0594cd1490a0a09d
Summary:
D13646642 used the `--date` flag for the `touch` command which
unfortunately does not work on our prod OSX hosts. Therefore, lets switch to
the more compatible `-t` flag.
Reviewed By: DurhamG
Differential Revision: D13673021
fbshipit-source-id: 6f9e0234e7fd3d4c345bedecdd2197949d71c420
Summary:
Generalise the `migrateonpull` mechanism of `treestate` into a generic
`automigrate` step that is invoked at the start of pulling. This will be used
for other migrations in the future.
Reviewed By: liubov-dmitrieva
Differential Revision: D13608718
fbshipit-source-id: d558dc21176a6b8d786836d06414e3fc88a20d47
Summary:
We've observed some users with very large hgcache directories that were filled
with temporary pack files that for some reasons were not removed/renamed on a
previous repack.
These files can appear due to a variety of reasons, such as forcibly killing
hg, or a host power-off, or simply due to a bug in mercurial. It is likely that
the later case is what causes some of the hgcache directories to grow and this
patch doesn't attempt on finding the underlying mercurial issue. Rather, let's
alleviate the issue by simply removing the temporary files older than 24h.
Reviewed By: ikostia
Differential Revision: D13646642
fbshipit-source-id: faa0605e322d440a75187e2517cbbcb13031dae0
Summary: If the there is no changeset left after sparse skip in `--command` mode, show the result and return, as it's done in manual testing mode.
Reviewed By: markbt
Differential Revision: D13650568
fbshipit-source-id: 8e867a38858d84d9a10078b74e2087318c81b01e
Summary: Adding new test to show that sparse skip with `--command` doesn't show the correct answer
Reviewed By: markbt
Differential Revision: D13650567
fbshipit-source-id: f4b6670fe67d6ef2543efedd91d9760e1e6bc74c
Summary:
Original commit changeset: 63c72654a345
Backout as there are issue in implementation of this transport on Mononoke side.
It requires better testing.
Reviewed By: StanislavGlebik
Differential Revision: D13635800
fbshipit-source-id: 1cdd43658889f68cd13df757ca6e21de01140dc9
Summary:
Several people observed this connection closed early message last week
and were unable to go further, so perhaps it would be helpful to
include a bit of additional data in the error message.
Reviewed By: quark-zju
Differential Revision: D13165697
fbshipit-source-id: 8f7d9d29d52697393a8474d6b8697099b0e33442
Summary:
upstream hg started to use separate bundle2 part to push phases but we are stuck with the old
way.
The old way was enabled in facebook.rc but nowhere in the tests except of the one
(where I disabled it) proving that the new way is working.
The original commit introduced the config option as temporary is talking about
pushrebase
changeset: 877f6928075428a4470bd399c82c9b1e9eaba9ad D6156039
user: Durham Goode <durham@fb.com>
date: Wed, 25 Oct 2017 16:39:02 -0800
configs: set devel.legacy.exchange=phases
Summary:
Upstream has enable sending phase bundle parts by default now, but
our server doesn't have the pushrebase fix to make this work. Let's
temporarily disable this until we've updated the servers.
I see in pushrebase support was implemented but the temporary disabling ended
up being permanent.
changeset: f6cf77230612bdf8be0c7fdd6ab21737fdaf35bf D1204
user: Stanislau Hlebik <stash@fb.com>
date: Mon, 23 Oct 2017 09:36:16 -0800
pushrebase: handle pushing phases through separate bundle2 part
Differential Revision: https://phab.mercurial-scm.org/D1204
I would like to switch to this transport because this is the one we are
supporting on Mononoke side and I don't like to have different configs on client side depends on
whether Mononoke is used or not.
Reviewed By: DurhamG
Differential Revision: D13622994
fbshipit-source-id: 63c72654a34584ad31d17b174660166b46f087fb
Summary:
If a user manually disconnects from Commit Cloud Sync, their next background
backup will automatically reconnect them if `commitcloud.autocloudjoin` is set.
Make the `autocloudjoin` setting only work if the user has never connected
to a workspace before. Detect the difference between the two by leaving a
`commitcloudrc` file in place after disconnecting.
Reviewed By: liubov-dmitrieva
Differential Revision: D13621476
fbshipit-source-id: ffccd473cb3da592e5b991dd863b8afed45dc83a
Summary:
When sending a bundle to the server, ensure that the treemanifest linknodes
correctly refer to a commit that is in the bundle.
Treemanifest ignores the linknodes of the subtrees - instead they inherit the
linknode of the root manifest in which they were introduced, so we only need to
fix up the linknode for the root manifest and propagate that to the subtrees.
Reviewed By: quark-zju
Differential Revision: D13504564
fbshipit-source-id: 2f481a4939239784d84d5db12c70d473b3045610
Summary:
When we send a bundle to the server, the treemanifest nodes are sent with
whatever linknode they have. This linknode might not refer to a commit that
the server knows about, which makes the bundle invalid. Add a test that
demonstrates that the server aborts in this case.
Pushrebase works fine as it rewrites the commits, generating new linknodes for
all of the trees.
Reviewed By: quark-zju
Differential Revision: D13504565
fbshipit-source-id: 39894d367c111aea5cef7de2d7da122e39d9debe
Summary:
Even if no new commits are created, absorb might still have "applied" some
changes by deleting commits. Let's fix the end-user message.
Reviewed By: DurhamG
Differential Revision: D13531959
fbshipit-source-id: 4d942f3ccd8201e8b62c8bc1c86227d41021b5f9
Summary:
`scmutil.cleanupnodes` was initially ported from absorb to simplify other
commands. Now use it to simplify absorb itself.
This solves a crash when `self.finalnode` is empty (ex. no new commits are
created, only with commits deletion).
Reviewed By: DurhamG
Differential Revision: D13531961
fbshipit-source-id: 7006b5ac5dfc4db897413d18ccd26eedde3c98d9
Summary:
It was to workaround the upstream obsmarker design which cannot support cycles.
Now that our internal obsmarkers can have cycles just fine, and the upcoming
mutation metadata makes it impossible to have cycles. Drop the workaround.
Reviewed By: DurhamG
Differential Revision: D13531960
fbshipit-source-id: d569172f0d2d5a3b4e1f6589be44ac21a09604f3
Summary:
I have been wondering how hg behaves with filenode collision. So I added some
tricky-looking tests about it. It actually shows the existing logic is
problematic :(
Reviewed By: DurhamG
Differential Revision: D13011554
fbshipit-source-id: fffb026e05adc8d8de4a1e5692bbee57293cce4e
Summary:
Drop dependency of `python-lz4`.
Add some convertions from bytearray to bytes to make code compatible.
Reviewed By: DurhamG
Differential Revision: D13516212
fbshipit-source-id: 89beb0aa92be4c5442a8e837f509e1eb17bb1512
Summary:
When I replace lz4 to rust.lz4, the test failed. The change fixes it. That
also means datapack corruption detection is not that reliable. However, usually
those files are not changed when they are loaded into an in-memory store, so
it's probaby fine.
Also note the python-lz4 used in production has strange behavior when
compressing an empty string:
In [6]: lz4.compressHC('')
Out[6]: '\x00\x00\x00\x00\xa0#\xd9\x040\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00'
In [7]: lz4.compress('')
Out[7]: '\x00\x00\x00\x00\xa0#\xd9\x040\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00'
In [9]: rustlz4.compress('')
Out[9]: bytearray(b'\x00\x00\x00\x00')
In [10]: rustlz4.compresshc('')
Out[10]: bytearray(b'\x00\x00\x00\x00')
In [13]: lz4.compress('1')
Out[13]: '\x01\x00\x00\x00\x101'
In [14]: rustlz4.compress('1')
Out[14]: bytearray(b'\x01\x00\x00\x00\x101')
Reviewed By: DurhamG
Differential Revision: D13528199
fbshipit-source-id: 9b3e8674f989062928900766156a97d28262c8cb
Summary:
Unfortunately required symbols are not exposed by lz4-sys. So we just declare
them ourselves.
Make sure it compresses better:
In [1]: c=open('/bin/bash').read();
In [2]: from mercurial.rust import lz4
In [3]: len(lz4.compress(c))
Out[3]: 762906
In [4]: len(lz4.compresshc(c))
Out[4]: 626970
While it's much slower for larger data (and compresshc is slower than pylz4):
Benchmarking (easy to compress data, 20MB)...
pylz4.compress: 10328.03 MB/s
rustlz4.compress_py: 9373.84 MB/s
pylz4.compressHC: 1666.80 MB/s
rustlz4.compresshc_py: 8298.57 MB/s
pylz4.decompress: 3953.03 MB/s
rustlz4.decompress_py: 3935.57 MB/s
Benchmarking (hard to compress data, 0.2MB)...
pylz4.compress: 4357.88 MB/s
rustlz4.compress_py: 4193.34 MB/s
pylz4.compressHC: 3740.40 MB/s
rustlz4.compresshc_py: 2730.71 MB/s
pylz4.decompress: 5600.94 MB/s
rustlz4.decompress_py: 5362.96 MB/s
Benchmarking (hard to compress data, 20MB)...
pylz4.compress: 5156.72 MB/s
rustlz4.compress_py: 5447.00 MB/s
pylz4.compressHC: 33.70 MB/s
rustlz4.compresshc_py: 22.25 MB/s
pylz4.decompress: 2375.42 MB/s
rustlz4.decompress_py: 5755.46 MB/s
Note python-lz4 was using an ancient version of lz4. So there could be differences.
Reviewed By: DurhamG
Differential Revision: D13528200
fbshipit-source-id: 6be1c1dd71f57d40dcffcc8d212d40a853583254
Summary:
Use the newly added zero-copy method to improve Rust lz4 performance. It's now
roughly as fast as python-lz4 when tested by stresstest-compress.py:
Benchmarking (easy to compress data)...
pylz4.compress: 10461.62 MB/s
rustlz4.compress_py: 9379.41 MB/s
pylz4.decompress: 3802.85 MB/s
rustlz4.decompress_py: 3975.61 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5341.69 MB/s
rustlz4.compress_py: 5012.30 MB/s
pylz4.decompress: 6768.17 MB/s
rustlz4.decompress_py: 6651.08 MB/s
(Note: decompress can be visibly faster if we return `bytearray` instead of
`bytes`. However a lot of places expect `bytes`)
Previously, the result looks like:
Benchmarking (easy to compress data)...
pylz4.compress: 10810.05 MB/s
rustlz4.compress_py: 11175.36 MB/s
pylz4.decompress: 3868.92 MB/s
rustlz4.decompress_py: 634.56 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 4565.91 MB/s
rustlz4.compress_py: 622.94 MB/s
pylz4.decompress: 6887.76 MB/s
rustlz4.decompress_py: 2854.79 MB/s
Note this changes the return type from `bytes` to `bytearray` for the
`compress` function. `decompress` still returns `bytes`, which is important for
compatibility. Note that zero-copy `bytes` can not be implemented `compress` -
the size of `PyBytes` is unknown and cannot be pre-allocated.
Reviewed By: DurhamG
Differential Revision: D13516211
fbshipit-source-id: b21f852c390722c086aa2f37a758bf3f58af31b4
Summary:
This is intended to replace the python-lz4 library so we have a unified code
path.
However, added benchmark indicates the Rust version is significantly slower
than python-lz4:
Benchmarking (easy to compress data)...
pylz4.compress: 10964.14 MB/s
rustlz4.compress_py: 12126.00 MB/s
pylz4.decompress: 3908.29 MB/s
rustlz4.decompress_py: 798.68 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5615.86 MB/s
rustlz4.compress_py: 740.32 MB/s
pylz4.decompress: 6145.68 MB/s
rustlz4.decompress_py: 2423.99 MB/s
The only case where the Rust version is fine is when the returned data is
small. That suggests rust-cpython was likely doing some memcpy unnecessarily.
Reviewed By: DurhamG
Differential Revision: D13516207
fbshipit-source-id: 72150b15c38bc8d8c7e7717a56a41f48d114db19
Summary:
When pulling an infinitepush bundle from the server to a client, it
uses remotefilelog's excludepattern functionality to force the server to serve
filelogs to the client. Unfortunately, it didn't specify what type of pattern,
and therefore it was treated as a regular expression. This meant that any path
that happened to be a regular expression would generally not match itself,
resulting in data not being sent to the client.
The fix is to just prefix these with 'path:' since we know they are all exact
paths.
Reviewed By: phillco, quark-zju
Differential Revision: D13521814
fbshipit-source-id: 8afc35b37d5858913b80ed0babfa0b5e401f0ab4
Summary:
If a filename is a valid regex and it doesn't match itself, that file
data will not be delivered to the client when doing a infinitepush pull that
requires a rebundle. This adds a test. The next diff fixes it.
Reviewed By: phillco
Differential Revision: D13522579
fbshipit-source-id: 6f8c5df20c31b834a33dbb9b73dc9031e26e969b
Summary:
Sandcastle jobs are using mercurial in conjunction with Phabricator CATs as a part of Phabricato security effort.
Here is an example of a mercurial command that failed because of this:
https://our.intern.facebook.com/intern/sandcastle/job/1498468091/
This adds logic to support parsing CATs from .arcrc for mercurial commands
Reviewed By: quark-zju
Differential Revision: D13468442
fbshipit-source-id: 033806d0e0779f9e7ade054d21e4cdbbdef08ed0
Summary:
Tests don't close the child process stdout. On newer versions of Python, this
can lead to ResourceWarnings when the test runner thread terminates.
Reviewed By: HarveyHunt
Differential Revision: D13517425
fbshipit-source-id: 6cedf4f39efe1299c41dbde784daf8c159309640
Summary:
Some of the escape sequences in run-tests.py are invalid. These cause
DeprecationWarnings on newer versions of Python.
In both cases, there are `\` characters that need to be escaped as `\\`.
Reviewed By: HarveyHunt
Differential Revision: D13517137
fbshipit-source-id: a899c3c28d55210f5972a515474a2fa69d051671
Summary:
The duringundologlock config and hook were used to test the undolog using
timing. We've replaced that, so remove the hook.
Reviewed By: quark-zju
Differential Revision: D13504644
fbshipit-source-id: a6b5fb308bc8938eec72788d93c9be6c237b72d7
Summary:
The undo tests use timing to detect when the lock is being taken. This is
flaky. Instead add extra logging to detect when the lock is taken.
Reviewed By: quark-zju
Differential Revision: D13504643
fbshipit-source-id: 07b80e416047d11b4ba3e1631c2385e5f12fa36f
