Summary:
Creates used by hg and tp2 should be consistent since there are both buck and
setup.py builds. Switch to use vendored crates generated from tp2 crates-io
Cargo.toml to get the consistency.
Reviewed By: DurhamG
Differential Revision: D8956612
fbshipit-source-id: 5223eafa75fb1da8bcbc9ddb6d81c18adc2d45f7
Summary:
The file is already quite big, and I'm going to add request recording
functionality soon. Let's move it in a separate package
Reviewed By: quark-zju
Differential Revision: D8787566
fbshipit-source-id: c0202d2ebf1e1662d4b02b966035f6cce9d2344e
Summary:
I cloned the GitHub export and found it does not build. It's easy to fix so
let's do so.
Reviewed By: phillco
Differential Revision: D8772772
fbshipit-source-id: 8e86576175270b541c03803e7357a3f76d29e25a
Summary: This was breaking perfsuite runs on the megarepo because it was not getting included in the package.
Reviewed By: singhsrb
Differential Revision: D8402910
fbshipit-source-id: 84115e1d81e09b35729f23e656655dc9e5c480ca
Summary:
Re-apply D8302882. This was causing problems because I didn't realize
commitcloud was using zstd bundles. Now we do have Rust-backed zstd
compression support so we can remove the python bindings.
Reviewed By: DurhamG
Differential Revision: D8361250
fbshipit-source-id: 981289734793a4c3401577426180649fdc7eb1b9
Summary:
This allows us to remove the Python binding without breaking existing zstd
users (commitcloud bundles). It might also make the future Rust migration
easier.
Using `zstd` create for its streaming APIs. `zstdelta`'s APIs cannot be used
since it requires decompressed length to be known, which wouldn't work for
streaming compressed data.
Note: For easier implementation, the Python land no longer processes data
in a streaming way. This is probably fine for the current bundle use-case.
In the long term, we might want to revisit the bundle format entirely.
As we're here, also expose zstdelta's APIs and add a test for it.
Reviewed By: DurhamG
Differential Revision: D8342421
fbshipit-source-id: 89902d551f4616469d6e1bc9b334a1c37c884775
Summary:
Several of our c extensions and libraries are missing their header
dependencies. This means `make local` doesn't correctly rebuild when you
change only header files.
It turns out that distutils' implementation of `build_libraries` doesn't
support the `depends` argument, but we're patching that anyway, so this commit
also adds support for that argument.
Reviewed By: quark-zju
Differential Revision: D8332423
fbshipit-source-id: 9b1a49a52b865143c4e28ded8e303b3e5035076d
Summary: This causing problems with the new alpha build w/ pulling std bundles and we need to back kit out until quark-zju can ship a Rust version.
Reviewed By: singhsrb
Differential Revision: D8339855
fbshipit-source-id: 2ff8b4d023e3d248ecc7e70924302bbf394ca268
Summary:
We use lz4 compression in production. Regarding on zstd, our approach would
be using the Rust `lib/zstdelta` library. So there is no need to keep the
Python binding.
This makes `make local` faster and also makes internal code search about
zstd cleaner.
Reviewed By: DurhamG, phillco
Differential Revision: D8302882
fbshipit-source-id: a6c34d6fea59140caeac158274388ba75a28fb29
Summary:
build rust binaries as part of mercurial build
by default they appear together with hg and chg in the folder called scripts but it can be configured.
Reviewed By: quark-zju
Differential Revision: D8270436
fbshipit-source-id: df928b89e6530c285c3eab438769c53f545d4098
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
Going to make changes to `mercurial/` for cleaner fsmonitor support
directly. So let's move the Rust python bridge there first.
Reviewed By: markbt
Differential Revision: D7909174
fbshipit-source-id: 454d784b5dca18a3af9328fc7b2f342cd4188cf6
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
Summary:
Generate a `u64` integer about the "version" at build time, and make chg
client check the version before connecting to the server.
This would ensure a chg client would only connect to a matched version of
the server.
- In setup.py, compute the "versionhash", write it as
`mercurial.__version__.versionhash`.
- In dispatch.py, `mercurial.__version__` needs to be explicitly loaded
before forking.
- In commandserver.py, send the versionhash to the client with the "hello"
message.
- In chg.c, verify the versionhash. If it does not match, unlink the socket
path and reconnect.
Reviewed By: farnz
Differential Revision: D7978131
fbshipit-source-id: 50acc923e72e40a4f66a96f01a194cf1a57fe832
Summary:
This splits out the logic in the `fsmonitor` extension that is responsible for
publishing `hg.filemerge` and `hg.update` state changes to Watchman into
its own extension, `hgevents`. This is because we want the behavior of
`hgevents` when Hg is running in Eden, but we do not want the remaining
behavior of `fsmonitor` when Hg is running in Eden, so splitting the logic
into separate extensions is the most straightforward way to achieve that.
To achieve the split, we move some more logic that is common to both
`fsmonitor` and `hgevents` out of `hgext/fsmonitor/__init__.py` and into
`hgext/extlib/watchmanclient/__init__.py`. Then we move these lines
out of `extsetup()` in `fsmonitor` to create `extsetup()` in `hgevents`:
```
extensions.wrapfunction(merge, 'update', wrapupdate)
extensions.wrapfunction(filemerge, '_xmerge', _xmerge)
```
We also have to pull all of the transitive dependencies for this logic
into `hgevents`.
Finally, we also have to define a `reposetup()` function in `hgevents`
that does a subset of what `reposetup()` does in `fsmonitor`. Specifically,
it ensures that a Watchman client is created for a `repo`, as appropriate,
so that it can be used to dispatch state changes to Watchman in
`state_filemerge` and `state_update`.
Note that the utility functions `createclientforrepo()` and
`getclientforrepo()` have been added to ensure that only one
Watchman client is created (and shared) when both `fsmonitor`
and `hgevents` are enabled.
Today, when an Hg repo is created in Eden, we set `extensions.fsmonitor=!`
in the `.hg/hgrc`:
diffusion/FBS/browse/master/fbcode/eden/hooks/hg/post-clone.py$69
Therefore, to get existing repos (both Eden and non-Eden) to pick up
the `hgevents` extension automatically, we add it to the list of
`[extensions]` in `common.rc`:
diffusion/FBS/browse/master/fbcode/scm/hg/fb/staticfiles/etc/mercurial/repo-specific/common.rc$53-60
as this is where `fsmonitor` is configured. We do not enable it in
`scm/hg/fb/staticfiles/etc/mercurial/facebook.rc` because
there is no reason to enable `hgevents` on Hg servers. Therefore, we
also decline to add `hgevents` to the set of `DEFAULT_EXTENSIONS` in
`scm/hg/mercurial/extensions.py`.
Reviewed By: quark-zju
Differential Revision: D8003628
fbshipit-source-id: 4f23881f8c25f4638f5475c292537b0352ae8d15
Summary:
There is some logic in `hgext/fsmonitor/watchmanclient.py` that needs
to be shared between fsmonitor and the new extension we are trying to
split out of it as part of T29379329.
Reviewed By: quark-zju
Differential Revision: D8003255
fbshipit-source-id: de01c5ba1460d7dde22969980b296afb3a942dd7
Summary:
This is a precursor to splitting the fsmonitor extension, as both
it and the new extension will use pywatchman.
Reviewed By: quark-zju
Differential Revision: D8002713
fbshipit-source-id: 37983fe2898d23223d1178eb3f15685f17ff8868
Summary:
This will contain all the Python centric hg store code that will let
Python call into the Rust storage layer.
Reviewed By: quark-zju
Differential Revision: D7632406
fbshipit-source-id: 6b7bcc8f47a23e9c0121e1f92de1137369bf584e
Summary:
I need a no-op change to test landing logic.
(Note: this ignores all push blocking failures!)
Reviewed By: phillco
Differential Revision: D7747338
fbshipit-source-id: 49388721418572b97b99971def951704059fc36a
Summary:
Swap the include paths so it looks for re2 source specified by `--re2-path`
first. This should resolve one of the recent OSX build failure issues.
Reviewed By: singhsrb
Differential Revision: D7702478
fbshipit-source-id: 303f9efc6009da3b876b63b0d5162437d06e7a04
Summary:
This allows us to build hg with a bundled re2. It can avoid issues like ABI
incompatibility with a shared re2 library. That might happen, if the RPM
dependencies are not strongly guaranteed (which might happen, although
shouldn't).
Reviewed By: simpkins
Differential Revision: D7622815
fbshipit-source-id: da6199aab7982f1e72466c676ebb0d1ed2100d6f
Summary:
`install_requires` is not well supported. And the subverty detection is not
meaningful - it checks at package build time, not package install time.
```
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)
```
Reviewed By: markbt
Differential Revision: D7626055
fbshipit-source-id: 061147f02698ad46561c95d2039f0a8ad26182d2
Summary:
I did some extra xdiff changes in upstream, namely:
- Remove unused features
- Replace "long" (32-bit in MSVC) with int64_t to support large files
- Add comment on some key variables
This backports them. It also includes Matt's fixes about Windows compatibility.
Reviewed By: ryanmce
Differential Revision: D7223939
fbshipit-source-id: 9287d5be22dae4ab41b05b3a4c160d836b5714a6
Summary:
On Windows, the `mman` and `dirent` functionality can be provided by external
libs, which we would not like to mention in the `setup.py`. This change allows
us to to call `setup.py` with the right env and link with the right libs, same
approach as with headers.
Reviewed By: quark-zju
Differential Revision: D7535358
fbshipit-source-id: d0364cb305a9114260106844f9a3575a731419e9
Summary: We don't really need this separation.
Reviewed By: quark-zju
Differential Revision: D7535364
fbshipit-source-id: a480ddb62883e2e9044e06ff7b27201151632b71
Summary:
`mercurial.rust` needs to be an explicit package for importing from it to
work.
(Note: this ignores all push blocking failures!)
Reviewed By: DurhamG
Differential Revision: D7449336
fbshipit-source-id: 36113a6a2e4fbb12d99b538b1c06cfa1023a035b
Summary: Build the new Rust matcher with both buck and setup.py
Reviewed By: markbt
Differential Revision: D7319607
fbshipit-source-id: c5944a28602495a9127acb20b59eb95632a9a1f5
Summary:
This fixes `hg serve` [1] by adding templates to the distribution and also puts copying of `CONTRIBUTING` and `CONTRIBUTORS` in the right place.
[1] https://fburl.com/w1oerx2f
Reviewed By: ryanmce
Differential Revision: D7341585
fbshipit-source-id: e6b022309f10ef0dc958e06bc7955c626cc6cee7
Summary:
Implement a `mercurial.cext.xdiff` module that exposes the xdiff algorithm.
`xdiff.blocks` should be a drop-in replacement for `bdiff.blocks`.
In theory we can change the pure C version of `bdiff.c` directly. However
that means we lose bdiff entirely. It seems more flexible to have both at
the same time so they can be easily switched via Python code. Hence the
Python module approach.
Reviewed By: ryanmce
Differential Revision: D7135205
fbshipit-source-id: 48cd3b5be7fd5ef41b64eab6c76a5c8a6ce99e05
Summary: `fastverifier` was sometimes being overriden by `shallowverifier` when remotefilelog was enabled. Since the latter is a subset of the former, let's just fold both into the core verifier code backed by a config, `verify.skipmanifests`, that we can default to true.
Reviewed By: DurhamG
Differential Revision: D6882222
fbshipit-source-id: 9f337ca031a070425ccdc9ee02f6765e68436da9
Summary:
Backport the build script so it's now buildable on Linux.
Add a hard-coded fallback version `4.4.2` in `setup.py` when `hg` cannot be
found. This makes the build not depending on hg - to allow bootstrapping on
Arch Linux - you can scp the files needed to do the build, without requiring
a special hg that can read fbsource.
Note: an hg that can work with the LFS extension is still needed. But that
is just `sudo pacman -S mercurial`.
Reviewed By: DurhamG
Differential Revision: D6827349
fbshipit-source-id: 45ea75175d8890a845d987cfe2be33251cd1a806
Summary:
This is a resend of https://phab.mercurial-scm.org/D1430, without breaking
Windows.
I encountered "too many opened files" problem due to treemanifest packs on my
laptop. This patch seems to be the easiest solution without side effects. Other
choices are deleting files (seem like an non-ideal workaround), forcing a
repack (could be slow), and rewriting using Rust (could take too long).
The root cause is Python's `mmap` implementation has to keep a fd internally
to support `mmapobj.resize` API. We only need read-only operation on the
mmap object so the fd is unnecessary. Re-implement a minimal mmap interface
for this purpose.
Reviewed By: DurhamG
Differential Revision: D6835890
fbshipit-source-id: 74c429e957cb8677682604eb02fc38b5b8d13ef7
Summary:
This "fixes" setup.py error:
```
python setup.py \
build_py -c -d . \
build_clib \
build_ext -i \
build_rust_ext -i \
build_hgexe -i \
build_mo
Traceback (most recent call last):
File "setup.py", line 969, in <module>
depends=common_depends),
File "/usr/lib64/python2.7/distutils/extension.py", line 106, in __init__
assert type(name) is StringType, "'name' must be a string"
AssertionError: 'name' must be a string
```
While the most correct fix would be possibly prepending prefixes
to strings in `setup.py`, it feels so much churn for so little
gain. So let's just disable unicode literal for now.
Reviewed By: phillco
Differential Revision: D6836945
fbshipit-source-id: 7d44969661c26b681f9d2e12d2e10026c4673a1c
Summary:
py2exe allows us to reduce startup time by 0.4s to 1s in some cases.
Depends on D6798345
Reviewed By: DurhamG
Differential Revision: D6798433
fbshipit-source-id: 566cd01f33c7bce32f577bd7b662642a4316c92b
Summary:
Adds some basic building blocks to build hg using buck.
Header files are cleaned up, so they are relative to the project root.
Some minor changes to C code are made to remove clang build
warnings.
Rust dependencies, fb-hgext C/Python dependencies (ex. cstore,
mysql-connector), and 3rd-party dependencies like python-lz4
are not built yet. But the built hg binary should be able to run
most tests just fine.
Reviewed By: wez
Differential Revision: D6814686
fbshipit-source-id: 59eefd5a3ad86db2ad1c821ed824c9f1878c93e4
Summary:
This test wasn't running because it had a hg10 check. Let's drop that
and fix the one failure.
Differential Revision: D6798754
fbshipit-source-id: 9d1f1f8b58ff14cba8228f9dc4c83ce21cab6c48
Summary:
The last diff lost most of its content after v2 was submitted. Resend.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D6795055
fbshipit-source-id: ed5f6d5eba9f8cfd505ed4ba1724110414b685f8
Summary:
./setup.py's logic to pick a version no longer works in fbsource because it only works if `.hg` is in the current
directory. (It set it to 'unknown'.)
We already moved the build logic to a streamlined system of (YYMMDD_HHmmSS_hash), so let's do the same with the actual version that
gets shipped with the release.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D6794773
fbshipit-source-id: 5de44b9a3541babddeff06e096b5b1faa8733e47
Summary: We probably want this script to work going forward. A lot of cleanup is needed, but at least now it produces a working binary.
Differential Revision: D6770776
fbshipit-source-id: dc2d44bd66e16c5bee53aadf262d6d3dee2fea96
Summary:
`lfs` is the better large file solution. `largefiles` is rarely used, and
its implementation is less clean. So let's remove it.
Test Plan:
Ran all tests. A subrepo test was removed instead of cleaned up since the
longer term plan is to also drop subrepo support.
Reviewers: phillco, #mercurial
Reviewed By: phillco
Differential Revision: https://phabricator.intern.facebook.com/D6740361
Signature: 6740361:1516225594:555e3803571ad05e0434021897a2823ac99347ae
