Summary: We want to have just one entry point to Mercurial, namely the Rust binary. Getting rid of the `hg` Python script means that we finally can do things that only exist in Rust and not in Python.
Reviewed By: simpkins
Differential Revision: D13186374
fbshipit-source-id: f3c8cfe4beb7bf764172a8af04fd25202eca9af2
Summary: This will hopefully make the `eden` module available on Windows.
Reviewed By: pkaush
Differential Revision: D15526731
fbshipit-source-id: 6c4fe5d6d298ae271fbc86b68ffb9905199cbf7b
Summary:
devservers have both `scratch` and `mkscratch`. `mkscratch` is
the binary that gets distributed with the hgbuild and is the one that is
available on mac (and maybe also windows?).
I plan to kill the old `scratch` package at some point on linux.
Reviewed By: simpkins
Differential Revision: D15479682
fbshipit-source-id: db78cc640ccaeb174b23d808fecbe7875fb656dd
Summary:
After D15266191, the bindings crate directly depends on pyrevisionstore, and
since then we have seen really weird compilation errors that magically go away
when some files are touched.
From my observations, here is what I came up with. The pyrevisionstore crate is
both compiled as a dependency of the bindings crate, and as a library to be
used in python. Therefore, its Cargo.toml contains a '[lib]' section, the
presence of this section forces the crate to be compiled into a
"libpyrevisionstore.{rlib,so}", while all the regular crates have a hashed
suffix like "libedenapi-2b9975ec9b865e87.rlib".
None of this would usually matter, but our build system re-uses the build
directory to then compile the pyrevisionstore library. While doing so, it will
re-create the "libpyrevisionstore.{rlib,so}", but not in an identical fashion.
After this, the rest of the build succeeds.
Once a file in the bindings crate is touched, recompiling will only recompile
its file, and not the pyrevisionstore crate, but since
"libpyrevisionstore.{rlib,so}" is different, it now fails to compile...
A previous effort tried to compile each top level target into a separate
directory, but that directly affected some of our tests. Since the underlying
issue is that pyrevisionstore is compiled twice, let's simply not compile it as
a top-level target and simply fold it into the bindings lib. Ideally, we would
want to do the opposite, but let's do that at a later time.
Reviewed By: DurhamG
Differential Revision: D15381038
fbshipit-source-id: 047cfab794371464942f19084ffb9240e836cb40
Summary: Similar to D7562864, but work with cargo workspaces.
Reviewed By: singhsrb
Differential Revision: D15084323
fbshipit-source-id: 3c15f2ceabb73dd54028523b6da5eb7857e7c842
Summary:
Using cargo for development using mercurial will result in use of random target
directories depending on where cargo is invoked from.
This diff sets the target directory for cargo in the on disk config file so
that all invocations of cargo from our project reuse the previously built
artifacts.
This may make it more problematic to build for multiple targets because cargo
does not recompile when flags change.
Reviewed By: quark-zju
Differential Revision: D14606468
fbshipit-source-id: 56c0b3b192ee94e1ddafe0949cc2267bb995a696
Summary:
Otherwise `hg.rust` might use a stale libchg.a and cannot pass the
client/server version check when installed.
Reviewed By: sfilipco
Differential Revision: D14531275
fbshipit-source-id: 6d8f04a75d91ad2c25ef2002ee3c3f9848094578
Summary:
BuildInfo symbols are generated for fbcode binaries, and scanned by the
internal procprint tool. The procprint tool also scans hg binaries, but cannot
extract any build information.
The procprint data is used by Push4Push to find out job owners running ancient
software.
This diff inserts BuildInfo symbols so Push4Push (and we) can use them to
figure out people running old versions of hg and send tasks to them.
Note rpmbuild strips symbols by default and is hard to customize. For now, I
just disable the debuginfo packages to preserve the symbols. If space usage or
runtime performance becomes an issue, we can try other workarounds (ex. provide
a shim of `%_find_debuginfo_dwz_opts` that skips `hg.rust`, or provide a shim
of `%__strip`, etc).
By default, the symbols only get added when `setup.py` is run by `rpmbuild`.
So `make local` won't rebuild `hg.rust` every time.
Reviewed By: singhsrb
Differential Revision: D14464056
fbshipit-source-id: ac6f9dedf28cee5fc8d65e144e3d7d1c544d1eda
Summary: We don't run this binary anymore, no reason to build and ship it.
Reviewed By: quark-zju
Differential Revision: D14437317
fbshipit-source-id: dd6da521783f18a2a518a7aa042be98950894e89
Summary:
There are not a lot of users of fbcode lfs pointers. Let's just merge the hg
one into it. This makes it easier to upload things without worrying about which
lfs-pointer file to use.
Reviewed By: singhsrb
Differential Revision: D14381672
fbshipit-source-id: f3ecb996e094d41605e3a883a83bd3e96b5bcd55
Summary:
This commit is to start including the memcommit extension in the
Mercurial package.
Reviewed By: quark-zju
Differential Revision: D14293241
fbshipit-source-id: c6e840d66e7e31b35d6c6aea176043cf56a16ad2
Summary: Build openssl using vcpkg. Then zip its build results.
Reviewed By: kulshrax
Differential Revision: D14197306
fbshipit-source-id: 7f019f52fb93118cb7c20c95cf41aedf3a1fbefc
Summary:
My goal is to have `mkscratch` get deployed with the mercurial
release on all platforms. This is currently deployed via packman
as `scratch` on devservers, and that packman build should continue to
succeed because the TARGETS file in here is left unchanged.
That will allow both `scratch` and `mkscratch` to coexist for a while
until we've made it through a few mercurial releases, and then we can
either fixup existing users of `scratch` to use `mkscratch`, or re-do
the `fb-scratch` rpm so that it symlinks to `mkscratch`.
I don't really know if this diff is sufficient to include the binary
in the mercurial package; I'm cargo culting the `scm_daemon` stuff
and that seems to be included in the omnibus, so maybe this is it?
Reviewed By: quark-zju
Differential Revision: D14060864
fbshipit-source-id: 5fd74a877437d84fe7676819c2cefb902fb7002d
Summary: Move the edenapi Python bindings into the common `bindings` crate.
Reviewed By: quark-zju
Differential Revision: D13963179
fbshipit-source-id: 76dead82af992615a9e452ee6fbb9f66639c822c
Summary: This wasn't being included before.
Reviewed By: vipannalla, quark-zju
Differential Revision: D14071697
fbshipit-source-id: 15f78c74a9629a8000ec1125c3e9e30e124c3d51
Summary:
On windows we use zip files to load the mercurial packages, but this
was failing because the edenscm module did not have an __init__.py file. Let's
include that in the build.
Reviewed By: singhsrb
Differential Revision: D13974190
fbshipit-source-id: 90e2cc5256fa08281202e11292e3ddf4cc000c01
Summary: Rename Mononoke API to Eden API, per war room discussion.
Reviewed By: quark-zju
Differential Revision: D13908195
fbshipit-source-id: 94a2fe93f8a89d0c5e9b6a24939cc4760cfaade0
Summary:
As requested by DurhamG, bundle DotSlash in the same way that scmtelemetry
is bundled rather than as directly part of the Mercurial package.
This approach makes it possible for DotSlash to leverage Hg's contbuild
while still being in its own package.
To implement this, I copypasta'd scmtelemetry's setup and modified it
for DotSlash, as appropriate.
Although I tried to port the Windows configuration over in addition to
Mac and Linux, I'm going to disable it for now because I want to limit
how many things I'm going to try to get working on the initial rollout.
Reviewed By: singhsrb
Differential Revision: D13950939
fbshipit-source-id: 5d8393d756e23f1ac228c7315a4b95a36288ac3a
Summary:
Move all Rust bindings to a single python extension, `bindings`. This should
improve compilation time and make things simpler.
Reviewed By: quark-zju
Differential Revision: D13923866
fbshipit-source-id: 560592b5a6c0c4f1b836c755ef123666a1059164
Summary:
This change is to get DotSlash (https://fb.quip.com/7BbiAmxAK09T)
in the Mercurial release. It renames the build rule from `dotslash` to
`dsl` under the assumption that that determines the name of the binary
to be deployed.
Reviewed By: quark-zju
Differential Revision: D13868779
fbshipit-source-id: dd44d874a60ab85b3dcca79ba5076a6f2fa64135
Summary:
`.hg-vendored-crates` is no longer a thing.
Now, the hg vendored crates always match tp2 crates, managed by the
`fbcode/tools/lfs/crates-io.py` script.
Reviewed By: ikostia
Differential Revision: D13873209
fbshipit-source-id: 6b5782b4834c49fa7d88aa1870703138eb9feb00
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
When I was debugging an eden importer issue with Puneet, we saw errors caused
by important extensions (ex. remotefilelog, lz4revlog) not being loaded. It
turned out that configpaser was checking the "exe dir" to decide where to
load "system configs". For example, If we run:
C:\open\fbsource\fbcode\scm\hg\build\pythonMSVC2015\python.exe eden_import_helper.py
The "exe dir" is "C:\open\fbsource\fbcode\scm\hg\build", and system config is
not there.
Instead of copying "mercurial.ini" to every possible "exe dir", this diff just
switches to a hard-coded system config path. It's now consistent with what we
do on POSIX systems.
The logic to copy "mercurial.ini" to "C:\open\fbsource\fbcode\scm\hg" or
"C:\tools\hg" become unnecessary and are removed.
Reviewed By: singhsrb
Differential Revision: D13542939
fbshipit-source-id: 5fb50d8e42d36ec6da28af29de89966628fe5549
Summary:
Use an `asset` to download Cython on demand. So we don't need to install Cython
as build dependency on all supported platforms, and maintain the "Cython"
package for those platforms.
Upgrade to the latest Cython by the way.
Reviewed By: singhsrb
Differential Revision: D13513514
fbshipit-source-id: 5ebe9a3e5b785a8f85cd51624663f9cc1e5c66fd
Summary:
This is intended to replace the python-lz4 library so we have a unified code
path.
However, added benchmark indicates the Rust version is significantly slower
than python-lz4:
Benchmarking (easy to compress data)...
pylz4.compress: 10964.14 MB/s
rustlz4.compress_py: 12126.00 MB/s
pylz4.decompress: 3908.29 MB/s
rustlz4.decompress_py: 798.68 MB/s
Benchmarking (hard to compress data)...
pylz4.compress: 5615.86 MB/s
rustlz4.compress_py: 740.32 MB/s
pylz4.decompress: 6145.68 MB/s
rustlz4.decompress_py: 2423.99 MB/s
The only case where the Rust version is fine is when the returned data is
small. That suggests rust-cpython was likely doing some memcpy unnecessarily.
Reviewed By: DurhamG
Differential Revision: D13516207
fbshipit-source-id: 72150b15c38bc8d8c7e7717a56a41f48d114db19
Summary: Adds the current flavor of the eden rust+python deps to LFS and have setup.py bake them into the IPython.zip.
Reviewed By: quark-zju
Differential Revision: D13516179
fbshipit-source-id: 64a0a86bc97b187f35475a3df4b580e0f0bc5deb
Summary:
This is the library shared between the eden cli and the
eden hg extension.
Reviewed By: quark-zju
Differential Revision: D13503730
fbshipit-source-id: 45ab550da3126042cb3baacaf8469b8acd6b1c4a
Summary:
makes it possible to `import thrift` from python2.
This is implemented as a subclass of `asset`, but rather than downloading
files we copy them from their location in fbsource. Extraction
copies them from their fbsource path, optionally excluding specific files
(eg: python3 only files) to the build path.
Reviewed By: quark-zju
Differential Revision: D13503711
fbshipit-source-id: fbe69400de31376ff6135c2c5173a984ff97f282
Summary:
In addition to `six` (already listed), these packages are
required to use the eden extension thrift deps.
I downloaded these using `pip2 download URL` and then uploaded using:
```
$ ../../tools/lfs/lfs.py upload /tmp/futures-3.2.0-py2-none-any.whl -l fb/tools/.lfs-pointers
```
Reviewed By: quark-zju
Differential Revision: D13503336
fbshipit-source-id: 87a21984dbe544882bfc4a818f7b5bff46907693
Summary:
It makes testing duplicated - now `cargo test` would try running tests on 2 entry points:
lib.rs and indexedlog_dump.rs. Move it to a separate crate to solve the issue.
Reviewed By: markbt
Differential Revision: D13498266
fbshipit-source-id: 8abf07c1272dfa825ec7701fd8ea9e0d1310ec5f
Summary:
Many code paths assume 'build' exists. So let's create it on demand.
Use the "scratch" tool to make it more friendly on an Eden checkout.
Reviewed By: markbt
Differential Revision: D13471293
fbshipit-source-id: cce461ab67b984c53a00a98d481a821ad1f11c35
Summary:
Our codebase is already unfriendly to 3rd-party extensions and we will likely
break the compatibility as part of revisising / replacing components in Rust
anyway. So just stop supporting `--config extensions.3rdpartyfoo=`.
Note `--config extensions.3rdpartyfoo=foo.py` still works.
This also makes `ImportError` more meaningful. Previously, when `hgext.foo`
imports `bar` which does not exist, the error is "cannot import foo", while
it should really be "cannot import bar". That is because extensions.py falls
back to import `hgext3rd.foo`, and `foo`. Now extensions.py only tries
`hgext.foo` and will suface the right error.
Reviewed By: DurhamG
Differential Revision: D13440400
fbshipit-source-id: c659f6d05abbc0d09f2811f36c0dc31e49491f19
Summary:
Fix final pieces to make IPython work on Windows:
- Install `win_unicode_console`. `pip2 download ipython` running on Linux won't
include it. But running on Windows will.
- Make `build_pyzip` command support `-i`.
- Make `Makefile` run `build_pyzip -i` so IPython.zip gets generated under
mercurial/thirdparty properly.
As we're here, also clean up IPython.zip so it no longer contains `setup.pyc`.
Reviewed By: kulshrax
Differential Revision: D13426947
fbshipit-source-id: 91db6cb85de4c689a4e5c10043debbc26bb94c18
Summary:
IPython is pretty handy (ex. `%timeit`, `?` etc). It should probably be the
recommended way to explore the code base for new people. This is a step towards
making it available for all platforms.
It's also smaller when compiled directly. The zip file containing all `.pyc`
files is just 8MB. When installed using CentOS's system package, a lot of GUI
dependencies will be installed, which need hundreds of MBs.
Note the zip only contains pure Python modules. The only native dependency
seems to be `scandir`. But `scandir` has a pure Python fallback. So it still
runs, just slower.
Reviewed By: DurhamG
Differential Revision: D13352617
fbshipit-source-id: 2ecbef69438ffc478389c5bec67bec5f83f7edeb
Summary:
To build IPython locally, its dependencies need to be fetched.
The direct motivation behind IPython is, recently when new people tried to hack
on Mercurial, they don't have a great way to understand the dynamic codebase.
Some of them use static code search, which does not work well. IPython's `?`
feature is pretty handy to get the location of related definitions. Together
with `hg dbsh`, it's much easier to explore the code base easily.
So let's just ship hg with IPython bundled.
Note installing IPython is also not very smooth:
- IPython on CentOS 7 is too old (3.x) and installing it brings in too many
GUI dependencies unnecessarily.
- IPython on Windows is not that easy to install, as we shipped our own
python.exe without corresponding pip.
Reviewed By: DurhamG
Differential Revision: D13352616
fbshipit-source-id: 91750664170c99f556ca406e718b030bf509f275
Summary:
Previously we had 2 different places to fetch re2 source code: Makefile and
Windows-only build_nupkg.py. We now have a command in setup.py that fetches them.
Let's just call the "fetch_build_deps" command as part of "build_ext".
This also makes re2 a required component. It's no longer optional.
Reviewed By: DurhamG
Differential Revision: D13352619
fbshipit-source-id: 0bd93560acfbc2e900005a20e4b33a236aad5f98
Summary:
The way to download build dependencies are not consistent and reinvented in a
couple of places. For example:
- Cython: error out if it's not avaialable
- Rust: (the fb part of) distutils_rust auto-installs it
- Re2: Makefile and build_nupkg.py are doing the same thing
- Various Windows dependencies: downloaded by the fb part of build_nupkg.py
This is an attempt to unify part of them to be a command of setup.py:
"fetch_build_deps".
The motivation was to get IPython bundled. For now I just add re2 to the asset
list. Upcoming diffs will add IPython. Cython might be a good fit, too.
Reviewed By: DurhamG
Differential Revision: D13352622
fbshipit-source-id: 151d299663eba9bb49c9577be0e224f9de8a9912
Summary:
While running `make local` on Windows, I got:
running build_rust_ext
downloading vendored crates 'tp2-crates-io'
[crates-io.py] downloading vendored crates archive from LFS
[crates-io.py] removing outdated vendor directory
Traceback (most recent call last):
File "fbcode\tools\lfs\crates-io.py", line 101, in <module>
download()
File "fbcode\tools\lfs\crates-io.py", line 69, in download
shutil.rmtree(VENDOR_DIRNAME)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "fbcode\scm\hg\build\hg-python\lib\shutil.py", line 250, in rmtree
os.remove(fullname)
WindowsError: [Error 5] Access is denied: 'vendor\\lalrpop\\src\\parser\\lrgrammar.rs'
error: download of Rust vendored crates 'tp2-crates-io' failed
make.exe: *** [local] Error 1
This is caused by the file being deleted has "read-only" attribute set on Windows.
Fix it by removing the read-only attribute automatically.
As we're here, also change `crates-io.py` to do the same thing as it's used by `setup.py`.
Reviewed By: ikostia
Differential Revision: D13413081
fbshipit-source-id: b10c44fd152a61c021edf6a8d86cb82a339f366f
Summary: The pymononokeapi Rust extension currently fails to build on Windows due to issues with OpenSSL. For now, don't build it on Windows to unbreak the build. This is acceptable because this crate does not yet have any significant functionality.
Reviewed By: DurhamG
Differential Revision: D13371991
fbshipit-source-id: ab3c3de116fc6a04b4a706919ceb541349eb353e
Summary:
This diff adds Python FFI bindings for the `mononokeapi` crate introduced in the previous diff in this stack. It exposes a `PyMononokeClient` Python class which wraps the Rust `MononokeClient` class internally.
This class currently only has one method, `health_check()`, which hits the API server's health check endpoint, returning `None` on success or raising an exception otherwise.
Reviewed By: DurhamG
Differential Revision: D13055688
fbshipit-source-id: a3703617c919f5317c90029fb40d275f3d63d56f
