Summary: Move the edenapi Python bindings into the common `bindings` crate.
Reviewed By: quark-zju
Differential Revision: D13963179
fbshipit-source-id: 76dead82af992615a9e452ee6fbb9f66639c822c
Summary: This wasn't being included before.
Reviewed By: vipannalla, quark-zju
Differential Revision: D14071697
fbshipit-source-id: 15f78c74a9629a8000ec1125c3e9e30e124c3d51
Summary:
On windows we use zip files to load the mercurial packages, but this
was failing because the edenscm module did not have an __init__.py file. Let's
include that in the build.
Reviewed By: singhsrb
Differential Revision: D13974190
fbshipit-source-id: 90e2cc5256fa08281202e11292e3ddf4cc000c01
Summary: Rename Mononoke API to Eden API, per war room discussion.
Reviewed By: quark-zju
Differential Revision: D13908195
fbshipit-source-id: 94a2fe93f8a89d0c5e9b6a24939cc4760cfaade0
Summary:
As requested by DurhamG, bundle DotSlash in the same way that scmtelemetry
is bundled rather than as directly part of the Mercurial package.
This approach makes it possible for DotSlash to leverage Hg's contbuild
while still being in its own package.
To implement this, I copypasta'd scmtelemetry's setup and modified it
for DotSlash, as appropriate.
Although I tried to port the Windows configuration over in addition to
Mac and Linux, I'm going to disable it for now because I want to limit
how many things I'm going to try to get working on the initial rollout.
Reviewed By: singhsrb
Differential Revision: D13950939
fbshipit-source-id: 5d8393d756e23f1ac228c7315a4b95a36288ac3a
Summary:
Move all Rust bindings to a single python extension, `bindings`. This should
improve compilation time and make things simpler.
Reviewed By: quark-zju
Differential Revision: D13923866
fbshipit-source-id: 560592b5a6c0c4f1b836c755ef123666a1059164
Summary:
This change is to get DotSlash (https://fb.quip.com/7BbiAmxAK09T)
in the Mercurial release. It renames the build rule from `dotslash` to
`dsl` under the assumption that that determines the name of the binary
to be deployed.
Reviewed By: quark-zju
Differential Revision: D13868779
fbshipit-source-id: dd44d874a60ab85b3dcca79ba5076a6f2fa64135
Summary:
`.hg-vendored-crates` is no longer a thing.
Now, the hg vendored crates always match tp2 crates, managed by the
`fbcode/tools/lfs/crates-io.py` script.
Reviewed By: ikostia
Differential Revision: D13873209
fbshipit-source-id: 6b5782b4834c49fa7d88aa1870703138eb9feb00
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
When I was debugging an eden importer issue with Puneet, we saw errors caused
by important extensions (ex. remotefilelog, lz4revlog) not being loaded. It
turned out that configpaser was checking the "exe dir" to decide where to
load "system configs". For example, If we run:
C:\open\fbsource\fbcode\scm\hg\build\pythonMSVC2015\python.exe eden_import_helper.py
The "exe dir" is "C:\open\fbsource\fbcode\scm\hg\build", and system config is
not there.
Instead of copying "mercurial.ini" to every possible "exe dir", this diff just
switches to a hard-coded system config path. It's now consistent with what we
do on POSIX systems.
The logic to copy "mercurial.ini" to "C:\open\fbsource\fbcode\scm\hg" or
"C:\tools\hg" become unnecessary and are removed.
Reviewed By: singhsrb
Differential Revision: D13542939
fbshipit-source-id: 5fb50d8e42d36ec6da28af29de89966628fe5549
Summary:
Use an `asset` to download Cython on demand. So we don't need to install Cython
as build dependency on all supported platforms, and maintain the "Cython"
package for those platforms.
Upgrade to the latest Cython by the way.
Reviewed By: singhsrb
Differential Revision: D13513514
fbshipit-source-id: 5ebe9a3e5b785a8f85cd51624663f9cc1e5c66fd
Summary:
This is intended to replace the python-lz4 library so we have a unified code
path.
However, added benchmark indicates the Rust version is significantly slower
than python-lz4:
Benchmarking (easy to compress data)...
pylz4.compress: 10964.14 MB/s
rustlz4.compress_py: 12126.00 MB/s
pylz4.decompress: 3908.29 MB/s
rustlz4.decompress_py: 798.68 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5615.86 MB/s
rustlz4.compress_py: 740.32 MB/s
pylz4.decompress: 6145.68 MB/s
rustlz4.decompress_py: 2423.99 MB/s
The only case where the Rust version is fine is when the returned data is
small. That suggests rust-cpython was likely doing some memcpy unnecessarily.
Reviewed By: DurhamG
Differential Revision: D13516207
fbshipit-source-id: 72150b15c38bc8d8c7e7717a56a41f48d114db19
Summary: Adds the current flavor of the eden rust+python deps to LFS and have setup.py bake them into the IPython.zip.
Reviewed By: quark-zju
Differential Revision: D13516179
fbshipit-source-id: 64a0a86bc97b187f35475a3df4b580e0f0bc5deb
Summary:
This is the library shared between the eden cli and the
eden hg extension.
Reviewed By: quark-zju
Differential Revision: D13503730
fbshipit-source-id: 45ab550da3126042cb3baacaf8469b8acd6b1c4a
Summary:
makes it possible to `import thrift` from python2.
This is implemented as a subclass of `asset`, but rather than downloading
files we copy them from their location in fbsource. Extraction
copies them from their fbsource path, optionally excluding specific files
(eg: python3 only files) to the build path.
Reviewed By: quark-zju
Differential Revision: D13503711
fbshipit-source-id: fbe69400de31376ff6135c2c5173a984ff97f282
Summary:
In addition to `six` (already listed), these packages are
required to use the eden extension thrift deps.
I downloaded these using `pip2 download URL` and then uploaded using:
```
$ ../../tools/lfs/lfs.py upload /tmp/futures-3.2.0-py2-none-any.whl -l fb/tools/.lfs-pointers
```
Reviewed By: quark-zju
Differential Revision: D13503336
fbshipit-source-id: 87a21984dbe544882bfc4a818f7b5bff46907693
Summary:
It makes testing duplicated - now `cargo test` would try running tests on 2 entry points:
lib.rs and indexedlog_dump.rs. Move it to a separate crate to solve the issue.
Reviewed By: markbt
Differential Revision: D13498266
fbshipit-source-id: 8abf07c1272dfa825ec7701fd8ea9e0d1310ec5f
Summary:
Many code paths assume 'build' exists. So let's create it on demand.
Use the "scratch" tool to make it more friendly on an Eden checkout.
Reviewed By: markbt
Differential Revision: D13471293
fbshipit-source-id: cce461ab67b984c53a00a98d481a821ad1f11c35
Summary:
Our codebase is already unfriendly to 3rd-party extensions and we will likely
break the compatibility as part of revisising / replacing components in Rust
anyway. So just stop supporting `--config extensions.3rdpartyfoo=`.
Note `--config extensions.3rdpartyfoo=foo.py` still works.
This also makes `ImportError` more meaningful. Previously, when `hgext.foo`
imports `bar` which does not exist, the error is "cannot import foo", while
it should really be "cannot import bar". That is because extensions.py falls
back to import `hgext3rd.foo`, and `foo`. Now extensions.py only tries
`hgext.foo` and will suface the right error.
Reviewed By: DurhamG
Differential Revision: D13440400
fbshipit-source-id: c659f6d05abbc0d09f2811f36c0dc31e49491f19
Summary:
Fix final pieces to make IPython work on Windows:
- Install `win_unicode_console`. `pip2 download ipython` running on Linux won't
include it. But running on Windows will.
- Make `build_pyzip` command support `-i`.
- Make `Makefile` run `build_pyzip -i` so IPython.zip gets generated under
mercurial/thirdparty properly.
As we're here, also clean up IPython.zip so it no longer contains `setup.pyc`.
Reviewed By: kulshrax
Differential Revision: D13426947
fbshipit-source-id: 91db6cb85de4c689a4e5c10043debbc26bb94c18
Summary:
IPython is pretty handy (ex. `%timeit`, `?` etc). It should probably be the
recommended way to explore the code base for new people. This is a step towards
making it available for all platforms.
It's also smaller when compiled directly. The zip file containing all `.pyc`
files is just 8MB. When installed using CentOS's system package, a lot of GUI
dependencies will be installed, which need hundreds of MBs.
Note the zip only contains pure Python modules. The only native dependency
seems to be `scandir`. But `scandir` has a pure Python fallback. So it still
runs, just slower.
Reviewed By: DurhamG
Differential Revision: D13352617
fbshipit-source-id: 2ecbef69438ffc478389c5bec67bec5f83f7edeb
Summary:
To build IPython locally, its dependencies need to be fetched.
The direct motivation behind IPython is, recently when new people tried to hack
on Mercurial, they don't have a great way to understand the dynamic codebase.
Some of them use static code search, which does not work well. IPython's `?`
feature is pretty handy to get the location of related definitions. Together
with `hg dbsh`, it's much easier to explore the code base easily.
So let's just ship hg with IPython bundled.
Note installing IPython is also not very smooth:
- IPython on CentOS 7 is too old (3.x) and installing it brings in too many
GUI dependencies unnecessarily.
- IPython on Windows is not that easy to install, as we shipped our own
python.exe without corresponding pip.
Reviewed By: DurhamG
Differential Revision: D13352616
fbshipit-source-id: 91750664170c99f556ca406e718b030bf509f275
Summary:
Previously we had 2 different places to fetch re2 source code: Makefile and
Windows-only build_nupkg.py. We now have a command in setup.py that fetches them.
Let's just call the "fetch_build_deps" command as part of "build_ext".
This also makes re2 a required component. It's no longer optional.
Reviewed By: DurhamG
Differential Revision: D13352619
fbshipit-source-id: 0bd93560acfbc2e900005a20e4b33a236aad5f98
Summary:
The way to download build dependencies are not consistent and reinvented in a
couple of places. For example:
- Cython: error out if it's not avaialable
- Rust: (the fb part of) distutils_rust auto-installs it
- Re2: Makefile and build_nupkg.py are doing the same thing
- Various Windows dependencies: downloaded by the fb part of build_nupkg.py
This is an attempt to unify part of them to be a command of setup.py:
"fetch_build_deps".
The motivation was to get IPython bundled. For now I just add re2 to the asset
list. Upcoming diffs will add IPython. Cython might be a good fit, too.
Reviewed By: DurhamG
Differential Revision: D13352622
fbshipit-source-id: 151d299663eba9bb49c9577be0e224f9de8a9912
Summary:
While running `make local` on Windows, I got:
running build_rust_ext
downloading vendored crates 'tp2-crates-io'
[crates-io.py] downloading vendored crates archive from LFS
[crates-io.py] removing outdated vendor directory
Traceback (most recent call last):
File "fbcode\tools\lfs\crates-io.py", line 101, in <module>
download()
File "fbcode\tools\lfs\crates-io.py", line 69, in download
shutil.rmtree(VENDOR_DIRNAME)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 250, in rmtree
os.remove(fullname)
WindowsError: [Error 5] Access is denied: 'vendor\\lalrpop\\src\\parser\\lrgrammar.rs'
error: download of Rust vendored crates 'tp2-crates-io' failed
make.exe: *** [local] Error 1
This is caused by the file being deleted has "read-only" attribute set on Windows.
Fix it by removing the read-only attribute automatically.
As we're here, also change `crates-io.py` to do the same thing as it's used by `setup.py`.
Reviewed By: ikostia
Differential Revision: D13413081
fbshipit-source-id: b10c44fd152a61c021edf6a8d86cb82a339f366f
Summary: The pymononokeapi Rust extension currently fails to build on Windows due to issues with OpenSSL. For now, don't build it on Windows to unbreak the build. This is acceptable because this crate does not yet have any significant functionality.
Reviewed By: DurhamG
Differential Revision: D13371991
fbshipit-source-id: ab3c3de116fc6a04b4a706919ceb541349eb353e
Summary:
This diff adds Python FFI bindings for the `mononokeapi` crate introduced in the previous diff in this stack. It exposes a `PyMononokeClient` Python class which wraps the Rust `MononokeClient` class internally.
This class currently only has one method, `health_check()`, which hits the API server's health check endpoint, returning `None` on success or raising an exception otherwise.
Reviewed By: DurhamG
Differential Revision: D13055688
fbshipit-source-id: a3703617c919f5317c90029fb40d275f3d63d56f
Summary:
In order to be able to run `chg` from the main `hg.rust` binary, we decided
that we will turn it into a static lib. This diff teaches our current build
scripts to do this.
Reviewed By: quark-zju
Differential Revision: D10853906
fbshipit-source-id: 8e0f37aa7e52d4a0610f36d7903eb0a318c193ad
Summary:
Python 2's `threading.Condition` and `threading.RLock` implementation are in
pure Python. Part of `RLock.acquire` looks like (simplified):
def acquire(self, blocking=1):
me = _get_ident()
if self.__owner == me:
self.__count = self.__count + 1
return 1
rc = self.__block.acquire(blocking)
########## Here #########
if rc:
self.__owner = me
self.__count = 1
return rc
If an interruption (ex. SIGTERM) happens at "HERE". The lock would be in an
inconsistent state. And if some `finally` block, or `__exit__` in a context
manager tries to release the lock, it could deadlock.
Similar problems also apply to `release`, `_acquire_restore`, and
`_release_save`. Basically, `self.__owner`, `self.__count` and `self.__block`
(i.e. the real lock) cannot be guaranteed in a consistent state in pure Python
world, because interruption can happen before any Python bytecode instruction
(but not inside a single Python bytecode instruction).
Therefore the interruption-safe implementation cannot be done in pure Python.
Use Rust to rescue.
The added test `streetest-condint.py` has a high chance to reproduce the
deadlock issue with Python 2.
Python 3 has a native RLock implementation, which makes things better. The
"Condition" implementation is not native and I haven't checked whether it
is sound or not.
Unfortunately, as part of testing, I hit https://bugs.python.org/issue29988 and
confirmed from the Rust world. That is, `__exit__` is not guarnateed called (!!)
That means native implementations still have a chance to be wrong, and there is
no easy way to fix it. `streetest-condint.py` was then updated to expose the
issue more easily.
The implementation is better than Python 2 stdlib, though.
Reviewed By: markbt
Differential Revision: D10517920
fbshipit-source-id: 394c9050c512ce2a0f9743c28ccfafe0f560141a
Summary:
The script uses fanotify to answer "who changes this file" questions on Linux.
Sometimes people complain about files showing up unexpectedly on "hg status".
This could be helpful to debug those kind of issues.
This is to answer "what process" questions for a few selected long-living files
without guarnateed correctness. It does not handle cases like replacing files
(changing inode), renaming, etc.
It is Linux only right now. It seems an OSX version could be built using
fsevents.
Reviewed By: DurhamG
Differential Revision: D10418991
fbshipit-source-id: 55b75d0a324a1e561d85431c92b59a13da87fba0
Summary:
Significant chunks of `build_nupkg.py` and `setup.py` are no longer needed.
This diff does a superficial work of cleanign it up.
Reviewed By: phillco
Differential Revision: D10390559
fbshipit-source-id: fc0f03d2e400fe9f400df9b4c1aaf6ba5d498619
Summary:
This relies on adding chef's unix toolchain to the PATH rather than copying stuff from LFS.
Plus, this sets the `HGDEV` env var, which causes `exec\hgmain\src\hgenv.rs` to load
environment from `build\env`. Thus, we don't need to copy `build\hg\hg-python` to the repo dir.
Reviewed By: quark-zju
Differential Revision: D9559617
fbshipit-source-id: de70b262d81befffc651ee82dd5f5e7e0a9c6ed3
Summary:
This is mainly to be used on Windows (although I tried to be xplat-friendly).
The idea is to make sure setup.py alone can produce the built version of Mercurial,
while `build_nupkg.py` should only concern itself with the packaging logic.
Cleanup of `build_nupkg.py` to follow later.
"Embedded" version of Mercurial is teh `py2exe`-like behavior:
- the entire pure Python stdlib is compiled into `.pyc` files
- all of the Mercurial `.py` files are compiled into `.pyc` files
- all of the `.pyc` files are zipped together
- all of the Python `.pyd` files are copied into an embedded distribution dir
- all of the Mercurial `.pyd` files are renamed to include the full path within their respective packages (e.g. `mercurial/cext/osutil.pyd` => `mercurial.cext.osutil.pyd`)
- `hgdemandimport` (should be renamed) now knows to check if it's loaded from a `.zip` file and if it is, add import hooks to respect the renaming from the point above
Reviewed By: quark-zju
Differential Revision: D9919805
fbshipit-source-id: efa96f452f9eaec1fc825be76a314c6ad693cc72
Summary:
The functionality we care about is provided by the `fbamend`
extension. Therefore, lets replace the `amend` extension with the `fbamend`
extension.
Reviewed By: farnz
Differential Revision: D10320739
fbshipit-source-id: 5700d39f488777fcc4033f60ce0a51cda15ef2ad
Summary: Reformat all opted-in python code with version `18.9b0` of Black.
Reviewed By: ambv
Differential Revision: D10126605
fbshipit-source-id: 82af0d645dd411ce8ae6b8d239e151b3730cd789
Summary:
We have seen build errors on platforms like Ubuntu due to an old version of
Rust was installed. Upgrading Rust on all platforms supported by Mercurial
could be fairly painful. Therefore just install Rust directly as part of the
build step.
This patch changes `distutils_rust` to install Rust to `build/` and use `cargo`
from there.
Reviewed By: singhsrb
Differential Revision: D9985861
fbshipit-source-id: 07ddd3880b11ac67822333352290ad30b3b14b79
Summary:
Create Python bindings so that the BookmarkStore can be interacted with
from Python.
Reviewed By: quark-zju
Differential Revision: D9768565
fbshipit-source-id: 00d75b5250d8bc7dbeddd90d80ff4a23c60d00f9
Summary:
The prepare step is used to setup environments.
Changed functions to be remembered and do not repeat work.
Reviewed By: phillco
Differential Revision: D9504560
fbshipit-source-id: 6d5833ee1652d8470b9b681c68b41f832e8dfa05
Summary:
This causes `make local` to build the new binary, copy the result into the
`.../hg` dir and rename it into `hg.rust` (while `hgmain` seems like a good
crate name, it seems like the binary should be called `hg.rust`, at least for
now)
Reviewed By: quark-zju
Differential Revision: D9218057
fbshipit-source-id: 49a0e09ae78b8cdb64c7158da3bb4179a47d4af9
Summary: This allows the Python world to access its features.
Reviewed By: DurhamG
Differential Revision: D8790922
fbshipit-source-id: e7e561c86596159c3766d6da8e0834c6fe441cbd
