Summary:
Drop dependency of `python-lz4`.
Add some convertions from bytearray to bytes to make code compatible.
Reviewed By: DurhamG
Differential Revision: D13516212
fbshipit-source-id: 89beb0aa92be4c5442a8e837f509e1eb17bb1512
Summary:
Unfortunately required symbols are not exposed by lz4-sys. So we just declare
them ourselves.
Make sure it compresses better:
In [1]: c=open('/bin/bash').read();
In [2]: from mercurial.rust import lz4
In [3]: len(lz4.compress(c))
Out[3]: 762906
In [4]: len(lz4.compresshc(c))
Out[4]: 626970
While it's much slower for larger data (and compresshc is slower than pylz4):
Benchmarking (easy to compress data, 20MB)...
pylz4.compress: 10328.03 MB/s
rustlz4.compress_py: 9373.84 MB/s
pylz4.compressHC: 1666.80 MB/s
rustlz4.compresshc_py: 8298.57 MB/s
pylz4.decompress: 3953.03 MB/s
rustlz4.decompress_py: 3935.57 MB/s
Benchmarking (hard to compress data, 0.2MB)...
pylz4.compress: 4357.88 MB/s
rustlz4.compress_py: 4193.34 MB/s
pylz4.compressHC: 3740.40 MB/s
rustlz4.compresshc_py: 2730.71 MB/s
pylz4.decompress: 5600.94 MB/s
rustlz4.decompress_py: 5362.96 MB/s
Benchmarking (hard to compress data, 20MB)...
pylz4.compress: 5156.72 MB/s
rustlz4.compress_py: 5447.00 MB/s
pylz4.compressHC: 33.70 MB/s
rustlz4.compresshc_py: 22.25 MB/s
pylz4.decompress: 2375.42 MB/s
rustlz4.decompress_py: 5755.46 MB/s
Note python-lz4 was using an ancient version of lz4. So there could be differences.
Reviewed By: DurhamG
Differential Revision: D13528200
fbshipit-source-id: 6be1c1dd71f57d40dcffcc8d212d40a853583254
Summary:
The `pybuf` provides a way to read `bytes`, `bytearray`, some `buffer` types in
a zero-copy way. The main benefit is to use same code to support different
input types. It's copied to a couple of places. Let's move it to `cpython-ext`.
Reviewed By: DurhamG
Differential Revision: D13516206
fbshipit-source-id: f58881c4bfe651a6fdb84cf317a74c3c8d7a4961
Summary:
Use the newly added zero-copy method to improve Rust lz4 performance. It's now
roughly as fast as python-lz4 when tested by stresstest-compress.py:
Benchmarking (easy to compress data)...
pylz4.compress: 10461.62 MB/s
rustlz4.compress_py: 9379.41 MB/s
pylz4.decompress: 3802.85 MB/s
rustlz4.decompress_py: 3975.61 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5341.69 MB/s
rustlz4.compress_py: 5012.30 MB/s
pylz4.decompress: 6768.17 MB/s
rustlz4.decompress_py: 6651.08 MB/s
(Note: decompress can be visibly faster if we return `bytearray` instead of
`bytes`. However a lot of places expect `bytes`)
Previously, the result looks like:
Benchmarking (easy to compress data)...
pylz4.compress: 10810.05 MB/s
rustlz4.compress_py: 11175.36 MB/s
pylz4.decompress: 3868.92 MB/s
rustlz4.decompress_py: 634.56 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 4565.91 MB/s
rustlz4.compress_py: 622.94 MB/s
pylz4.decompress: 6887.76 MB/s
rustlz4.decompress_py: 2854.79 MB/s
Note this changes the return type from `bytes` to `bytearray` for the
`compress` function. `decompress` still returns `bytes`, which is important for
compatibility. Note that zero-copy `bytes` can not be implemented `compress` -
the size of `PyBytes` is unknown and cannot be pre-allocated.
Reviewed By: DurhamG
Differential Revision: D13516211
fbshipit-source-id: b21f852c390722c086aa2f37a758bf3f58af31b4
Summary:
This is intended to replace the python-lz4 library so we have a unified code
path.
However, added benchmark indicates the Rust version is significantly slower
than python-lz4:
Benchmarking (easy to compress data)...
pylz4.compress: 10964.14 MB/s
rustlz4.compress_py: 12126.00 MB/s
pylz4.decompress: 3908.29 MB/s
rustlz4.decompress_py: 798.68 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5615.86 MB/s
rustlz4.compress_py: 740.32 MB/s
pylz4.decompress: 6145.68 MB/s
rustlz4.decompress_py: 2423.99 MB/s
The only case where the Rust version is fine is when the returned data is
small. That suggests rust-cpython was likely doing some memcpy unnecessarily.
Reviewed By: DurhamG
Differential Revision: D13516207
fbshipit-source-id: 72150b15c38bc8d8c7e7717a56a41f48d114db19
Summary:
Rather than only making it available to `hg dbsh`,
add it for all commands/extensions to use.
Reviewed By: quark-zju
Differential Revision: D13505762
fbshipit-source-id: c79fee888d9394c5ad70d3d8b7f59addef1381a1
Summary:
Be compatible with `hgext.` or `hgext/` prefix. But print a warning saying it's
deprecated.
Reviewed By: DurhamG
Differential Revision: D13490362
fbshipit-source-id: ef13bd57a74be810df409af18a6259bc7b2b6dad
Summary:
Add a third argument to the `get(dict, key)` template function that serves as a
default.
Reviewed By: DurhamG, quark-zju
Differential Revision: D12980774
fbshipit-source-id: d619275dd3ae880f6aba4c2b3d91aea4b45ea6d6
Summary:
The test failed because "hgext" is no longer a namespace package. The check
becomes unnecessary since we almost always use the in-repo modules.
Reviewed By: DurhamG
Differential Revision: D13453160
fbshipit-source-id: 0c5a72b3abeb0a12426d8ed15c045a4ce478a4c6
Summary:
Add a new `--print` option to drawdag which prints the commit hashes and
descriptions of all the nodes that were generated by drawdag.
Reviewed By: quark-zju
Differential Revision: D10149266
fbshipit-source-id: 55b4c133b3c98c0258419811d7c00a3ec73a02cc
Summary:
Update drawdag to generate mutation metadata in commits when mutation recording
is enabled.
In order for this to work, we need to add the mutation relationships as edges
between the commits, so that when walking the DAG the predecessors are already
committed and we can find their hashes.
Reviewed By: ikostia
Differential Revision: D9989024
fbshipit-source-id: 671c1eb6c4ae6e87760efb4d3aa47e5e0585c94d
Summary:
Add a convenience function that generates the nodes that match a revset, rather
than the contexts or rev numbers.
Reviewed By: DurhamG, quark-zju
Differential Revision: D10149262
fbshipit-source-id: 5c889b1c7f03e3fcda7dccd297674c40729ccd90
Summary: Add support for mutation information being added when a commit is pushrebased.
Reviewed By: DurhamG, quark-zju, ikostia
Differential Revision: D9975558
fbshipit-source-id: 55444c05df32e5be159fbc19610f7a13ab103109
Summary:
Add support for mutation information being added when a commit is split.
The top of the final stack is the commit that stores the mutation metadata.
The other commits in the stack are just normal new commits, although the final
commit does record their hashes in its metadata.
Determining when the last commit in the stack happens before it is committed is
a little hard, as the code previously relied on the dirstate ending up clean
to detect a finished split. Instead we look at the patches that come out of
the filter function and see if they match the original patches that were sent
in.
Reviewed By: DurhamG, ikostia
Differential Revision: D9975469
fbshipit-source-id: acec485f9b561952f4ccdbaaf9491c9d48a70f58
Summary:
Add support for mutation information being added when a commit is metaedited or
folded.
The folded commit uses the top of the original stack as its predecessor. The
other commits that are folded are also stored in the metadata.
Reviewed By: DurhamG, quark-zju, ikostia
Differential Revision: D9975471
fbshipit-source-id: 92089d14cd5d65df283bbdc80a8b352b5f3982e9
Summary: Add support for mutation information being added when a commit is rebased.
Reviewed By: DurhamG, quark-zju, ikostia
Differential Revision: D9975481
fbshipit-source-id: 02a1ace0d6c80d73b1ab76e4d1b23a8d46ab049b
Summary: Add support for mutation information being added when a commit is amended.
Reviewed By: DurhamG, ikostia
Differential Revision: D9975472
fbshipit-source-id: 57d51823e709f71a88ad0fb86992a0a38541527c
Summary: Add a new command to debug-dump mutation information in commits
Reviewed By: DurhamG, quark-zju
Differential Revision: D9975474
fbshipit-source-id: a9ed1578dffc80da3e375837cede8d8685ee0427
Summary:
This is the first commit in a series that will replace obsolescence markers
with separate tracking of commit mutation and visibility.
Mutation information is metadata about which commit was mutated and in which
way that led to the creation of this commit. It is embedded in the extras of
the commit object.
This commit adds the fields to the commit context. Future commits will create
the mutation information when commits are mutated.
Reviewed By: DurhamG, quark-zju, ikostia
Differential Revision: D9975470
fbshipit-source-id: 0107308731dc021cb72734db8b5dc0a02e6e86e1
Summary:
In order for the help text for aliases to be valid RST, it must be possible to
indent some of the lines. This isn't possible with normal config, as leading
whitespace is stripped. Make it valid for lines in alias documentation to be
quoted, preserving leading whitespace.
Reviewed By: singhsrb
Differential Revision: D13431946
fbshipit-source-id: eefada39bcd92ce5162a0f34027b9380b5a420c9
Summary:
This is subtle.
`.hgignore`'s current behavior: if `^foo$` rule exists, then directory `foo/`
is ignored (ex. 'foo/bar' should be ignored).
However, that imposes problems for the sparse ignore matcher, which is the
"negate" of the "include" matcher. A user can write `[include]` glob patterns
like:
a*{b*,c*/d*}/e*
The ignore matcher will be the negate of the above patterns. Then because
`a1b2` does not match `a*{b*,c*/d*}/e`, the negate matcher returns "True",
and the ignore matcher will ignore the directory. So even if file `a1b2/e3`
should be selected, the parent directory being ignored cause the file to
be ignored.
That is clearly incorrect for sparse's usecase.
I think the issue is fundementally a layer violation - it's the *matcher*'s
responsibility to check whether one of the parent directory is matched (or
ignored), not the directory walker's responsibility.
This diff fixes the walker so it uses the visitdir interface, and moves back
the directory check to hgignore matcher to maintain compatibility. For three
matchers involved in ignore handling:
- hgignore matcher: updated to do the recursive directory handling on its own
- gitignore matcher: work out of box. already consider parent directories!
- sparse matcher: want the new behavior
`test-sparse-issues.t` is now green.
With this change, the `forceincludematcher` subdir hack used in sparse is no
longer necessary. Therefore removed.
Besides, all ignore matchers can handle "visitdir" correctly. That is, if
`visitdir('x')` returns `'all'`, then `visitdir('x/y')` will also return `'all'`.
Therefore the parent directory logic in `dirstate.dirignore` becomes
unnecessary and dropped.
Reviewed By: DurhamG
Differential Revision: D10861612
fbshipit-source-id: aa0c181ae64b361b85f08b8fecfdfe6331e9a4c2
Summary:
The extension was mainly for users `pip install <3rd-party-hg-ext>`, which
usually installs them into `hgext3rd`. Since we no longer search into
`hgext3rd`, this extension can be removed.
Reviewed By: DurhamG
Differential Revision: D13440401
fbshipit-source-id: d67a655ece9a1f6d5feb18d8aeac84852edfdcb9
Summary:
Our codebase is already unfriendly to 3rd-party extensions and we will likely
break the compatibility as part of revisising / replacing components in Rust
anyway. So just stop supporting `--config extensions.3rdpartyfoo=`.
Note `--config extensions.3rdpartyfoo=foo.py` still works.
This also makes `ImportError` more meaningful. Previously, when `hgext.foo`
imports `bar` which does not exist, the error is "cannot import foo", while
it should really be "cannot import bar". That is because extensions.py falls
back to import `hgext3rd.foo`, and `foo`. Now extensions.py only tries
`hgext.foo` and will suface the right error.
Reviewed By: DurhamG
Differential Revision: D13440400
fbshipit-source-id: c659f6d05abbc0d09f2811f36c0dc31e49491f19
Summary:
This diff adds the `callsite` parameter to worker.worker and checks whether the passed-in callsite is enabled in the worker.enablecallsites config flag. This change essentially enabled parallization for lfs prefetch, which is called during initial repo clone(when mercurial cache is clean). Since blobstore.py seems to be thread-safe, we could also anticipate similar improvements for the more generalized `hg udpate` case by batching files to download.
Details see T37718264
Reviewed By: ikostia
Differential Revision: D13284680
fbshipit-source-id: c3b825033a28344e19ba5ca1621b59fe7b46b322
Summary:
See user report https://fburl.com/latdfq5n. `fcd` in the code path is not
always a `workingfilectx` and calling `p1` might raise if it does not have
a parent.
Reviewed By: phillco
Differential Revision: D13144605
fbshipit-source-id: 8961943e65504b43b4a9d1a4828755fdf8ca0915
Summary:
Now that `parselist` is available in Rust. Let's switch to that.
It's 50x faster.
```
In [1]: s='command, commandfinish, commandexception, exthook, pythonhook, fsmonitor, fsmonitor_details,
...: fsmonitor_status, treedirstate, watchman-command, merge-resolve'
In [3]: p1=m.config.parselist
In [4]: %timeit p1(s)
10000 loops, best of 3: 97.2 µs per loop
In [5]: from mercurial.rust import config
In [6]: p2=config.parselist
In [9]: %timeit p2(s)
100000 loops, best of 3: 2.03 µs per loop
```
Reviewed By: DurhamG
Differential Revision: D9332596
fbshipit-source-id: e7b7d5d41f537d663d36169e14d27ed3d395a59c
Summary:
The last use of it was the bugzilla extension, which was removed.
Therefore remove it.
Reviewed By: DurhamG
Differential Revision: D13285203
fbshipit-source-id: 47d1fc9df68369660a63313fe4f39aaaffd7f239
Summary:
Backports my upstream patch (https://phab.mercurial-scm.org/D2289)
Discovered by nemo on #mercurial freenode IRC, some months did not parse.
Reviewed By: DurhamG
Differential Revision: D12983571
fbshipit-source-id: b33227b0c986180966579afbb2b7c11fd48d275b
Summary:
An upcoming diff will need the ability to iterate over all the keys in
the store. So let's expose that functionality.
Reviewed By: quark-zju
Differential Revision: D13062575
fbshipit-source-id: a173fcdbbf44e2d3f09f7229266cca6f3e67944b
Summary:
When the curses interface cannot be used, prompt the user about the text
interface to unblock.
Reviewed By: akushner
Differential Revision: D12987094
fbshipit-source-id: 3eff84d9daaaf19aaa08ebf28cbd7c7bf98f9e9a
Summary:
Oculus wants to preserve the "do not show binary diff" config while running
`hg export` in automation, since printing the binary diff might just OOM the
container. Let's add a plain exception for it.
Reviewed By: zhh95
Differential Revision: D13111212
fbshipit-source-id: 34af58ac0917de3b3231e637774896d882585e26
Summary: Fix test failures (`test-check-code.t` and `test-check-config.t`) introduced by my Mononoke API diff stack. I thought Sandcastle would have surfaced these, but I guess not.
Reviewed By: quark-zju
Differential Revision: D13325133
fbshipit-source-id: fd14f4c7e7280155d6f677f18b20691ef54ceca3
Summary:
Make repacks flush the newly created pack files to disk before proceeding to
delete the old datapacks. This ensures the data is committed to disk before we
delete the old pack files. There had been user complaints of data corruption.
The changes affects repack only to avoid slowing down performance for other
operations with manipulates datapack and historypack.
Reviewed By: DurhamG
Differential Revision: D12920147
fbshipit-source-id: 907b64d7763a6212fb49487cfc3bc403f8e3dce2
Summary:
Turn BookmarkStore rust implementation into indexed-log backed.
Note that this no longer matches existing mercurial bookmark store
disk representation.
Reviewed By: DurhamG
Differential Revision: D13133605
fbshipit-source-id: 2e0a27738bcec607892b0edab6f759116929c8e1
Summary:
debugcommand for receiving several lfs objects from server
Analog to the funtion `debuglfsreceive`, but for several objects
Will be used in replay traffic for Mononoke LFS.
Requires URL and an iterable with oids and sizes.
Format required:
url, (oid1, size1, oid2, size2, ... )
Write all files to the console.
Reviewed By: quark-zju
Differential Revision: D12956274
fbshipit-source-id: f83dd0636b2ad197cace9633222d0f1ed8191dab
Summary:
We had remotefilelog specific logic in core mercurial code. Now that
we have the fileslog abstraction we can hide remotefilelog specific invalidation
in a remotefilelog specific fileslog implementation.
Reviewed By: quark-zju
Differential Revision: D12959495
fbshipit-source-id: ccf224bf9799eb1af74f0dff6021fcbc2eb20d68
Summary:
In upcoming diffs I want to introduce lifetime management for
structures receiving writes for file content. We need to be able commit these at
the end of a transaction, and roll them back in the event of an abort.
We already have this, but it's adhoc and inconsistent. Let's introduce the
concept of a fileslog on the repository that is responsible for all filelog read
writes. At the moment it doesn't manage reads, and only governs remotefilelog
writes, but we can extend this later.
This pattern of a top level repo.*log property that manages the reading/writing
of a type of object is already used in the manifestlog and has proven to be
relatively clean.
Reviewed By: quark-zju
Differential Revision: D12959494
fbshipit-source-id: 676aa86c313cb7e48512091a9c19b9452e8f114a
Summary:
Previously, remotefilelog tried to hide which store should receive
writes behind the union store abstraction. This is starting to make things a
little complicated though. For instance, it means that right now both local data
and local history are written via one api on the remotefilelogcontentstore, even
though the remotefilelogcontentstore should only be about data. It also means
that remotefilelogcontentstore had to become aware of pack files, which it can't
read, so it no longer upholds the guarantee that anything written to it can be
immediately read. Overall it's just confusing.
This diff rips out the writestore concept, and instead has the store setup logic
store the writable stores directly on the repo, and changes the write path to
write to those stores directly. Thus removing the notion of pack files from
remotefilelogcontentstore.
A future diff will clean this up even further, and fix the bug where you can't
read data that was just written to a local pack.
Reviewed By: quark-zju
Differential Revision: D12959502
fbshipit-source-id: 85c39c0696febd0972a21f22f3640fd6954901c1
Summary:
The Rust condition implemenation seems working okay. Therefore drop the support
to switch to buggy `threading.Condition`.
Reviewed By: phillco
Differential Revision: D12875128
fbshipit-source-id: 15f41bdc3fceeaa7de7d8cdbd76ee6d70951c107
Summary:
`@@ -1,1 +-1,0 @@` is not a valid patch hunk header.
Change it to `@@ -1,1 +0,0 @@`.
This is a backport of my upstream patch https://phab.mercurial-scm.org/D3737
Reviewed By: singhsrb
Differential Revision: D12970975
fbshipit-source-id: eb769e93076efc932a8c4be7387ba21933aae331
Summary: This has been broken after D12832280.
Reviewed By: quark-zju
Differential Revision: D12981581
fbshipit-source-id: 3f411260e24e623abb6f18814fa5b0bf87b416cd
Summary:
The way I wrote this, `if e.errno != errno.EEXIST or e.errno != errno.EACCES`
this condition never succeeds. So we're re-raising always, rather than falling
back to a `delete/rename` strategy.
Differential Revision: D12974035
fbshipit-source-id: 0b500d009315ba0094019a82576d5a3698b624b2
