Summary:
Made changes to the commit cloud sync to support remote bookmarks. Here I only declared steps of future remote bookmarks sync and will add the implementation further in the stack.
Processing a merge state between cloud remote bookmarks state, last sync and local state is done earlier in the workflow, than, for example, for local bookmarks, because to update remote bookmarks we may need to fetch new public heads from the server and so need to get them first.
Reviewed By: mitrandir77
Differential Revision: D15898740
fbshipit-source-id: ca77f66b0f1244a811b4b549e5b189c5f1772708
Summary: Adding support of remote bookmarks to the get/set references of Commit Cloud client service.
Reviewed By: mitrandir77
Differential Revision: D15852515
fbshipit-source-id: 9d5331955f5fc95ecb4bfe8e060d3b1d67952a41
Summary:
PYTHONPATH is no longer a thing on Windows. Let's not rely on it
for our `heredoctest.py`-based tests
Reviewed By: mitrandir77
Differential Revision: D16028830
fbshipit-source-id: ae66d9cf194c7e3c15e44a81a18dfeeb4b68e281
Summary: When Eden API falls back to SSH, log the exception type and message (in addition to the traceback) to the `hg_errors` table. Note that previously, we were attempting to log the error message but failing because we were using the wrong column name. (`msg` instead of `exception_msg`).
Reviewed By: xavierd
Differential Revision: D16014833
fbshipit-source-id: 06460574f66999b1293ea31b43b5c7ec89737144
Summary: `hg debugedenimporthelper --get-file-size PATH:REV` will print the size of the given file
Reviewed By: chadaustin, xavierd
Differential Revision: D16009279
fbshipit-source-id: 6edb01dd154a467cdd26c0ced1d4ae82411088f1
Summary: Now that Eden API returns specific exception types rather than always raising a `RuntimeError` when things go wrong, the fallback path needs to be updated to catch all exception types, not just `RuntimeError`.
Reviewed By: xavierd
Differential Revision: D16013115
fbshipit-source-id: d5c2d88acede7e70519eda8915401bb8ee394038
Summary: This was causing `Popen` to crash, since `env` had non-strings (`NoneType`)
Reviewed By: mitrandir77
Differential Revision: D16009578
fbshipit-source-id: cbed384b7d773e3de23f9a8920d85296c5deec85
Summary: If `subprocess.Popen` crashes, the lock is never released.
Reviewed By: mitrandir77
Differential Revision: D16009579
fbshipit-source-id: e27137826ffb29aa1f497a3c4c85160b183f2a51
Summary:
`--fixup` is not really a great flag name and it came from a legacy
implementation. Suggest `restack` instead.
Reviewed By: kulshrax
Differential Revision: D15978952
fbshipit-source-id: 4e8c2e79be01f6d8f13628409a446d42ae22c0af
Summary:
See the test change for what this change is about.
The new code has side effect on linearizing things for `<rebase_src>::`. This
is reflected on the second last test change. The new behavior is arguably more
desiable - The new code can fail if rebasing D onto C2 causes conflicts. If
that happens, and rebasing C+C2+D onto the new B does not cause conflicts, then
the old behavior is better. We can add fallback code to the old behavior if
there are conflicts later, by calling a plain `rebase` instead of `restack`.
Reviewed By: kulshrax
Differential Revision: D15978953
fbshipit-source-id: 20b9d62c6125ce2faf8e21bd86f6aad31ac38a0c
Summary: This allows the callsite to control what revs to rebase.
Reviewed By: kulshrax
Differential Revision: D15978954
fbshipit-source-id: 73174475411c3986f3a8e93a1a61ce5b857f454b
Summary:
I run into this frequently and it's pretty annoying. Basically, amending X
tries to rebase things that are outside `X::`, which is unexpected.
Also, rebasing commits outside `X::` can easily fail due to conflicts. That
rollbacks the *entire* transaction. It'd be better if it only rolls back
the transaction rebasing a single branch.
I have a stack like:
o 86685181 79 minutes ago quark D15755281
| [hg] logging: log ui.write blocked time
|
o 218029b5 79 minutes ago quark D15755284
| [hg] logging: log ui.prompt blocked time
|
o da4d16a3 79 minutes ago quark D15755283
| [hg] logging: migrate crecord "blocked" logging to new API
|
o 6d0574e1 Today at 15:17 quark D15710672
| [hg] logging: migrate fsmonitor to new blackbox API
|
o
.
.
o 388aabb8 Today at 11:41 quark D15685475
| [hg] blackbox: add pattern matching for filtering events
|
| o e7d08fbd Today at 11:41 quark D15710677
| | wip
| |
| x 55895cb8 [Rewritten into 6d0574e15334] Today at 11:41 quark D15710672
| | [hg] logging: migrate fsmonitor to new blackbox API
| |
| | o de3b561f Today at 11:41 quark
| | | wip
| | |
| | o 5898b7a0 Today at 11:41 quark
| | | wip
| | |
| | x 478da1be [Rewritten into 55895cb88484] Today at 11:41 quark D15710672
| | | [hg] logging: migrate fsmonitor to new blackbox API
| | |
| | x e7c5c7d3 [Rewritten into e7d08fbdb3e0] Today at 11:41 quark D15710677
| | | [hg] blackbox: add more event types
| | |
| | x 7d715385 [Rewritten into 55895cb88484] Today at 11:41 quark
| |/ [hg] blackbox: add fsmonitor event
| |
| x c9a01966 [Rewritten into 00c67719c09d] Today at 11:41 quark
| . [hg] blackbox: change session id to u64
| .
| x b4a0a8ab [Rewritten into 388aabb8f9f6] Today at 11:41 quark D15685475
|/ [hg] blackbox: add pattern matching for filtering events
|
o 708540a9 Today at 11:41 quark D15685471
[hg] blackbox: add Event::to_value
The current behavior rebases the wip commits when amending commits on the top
like 6d0574e1. It always causes conflicts and rolls back the entire rebase.
Reviewed By: kulshrax
Differential Revision: D15978955
fbshipit-source-id: 1573407958fec647fca4ca261f6807cd8eae9803
Summary: When the Eden API Rust client raises a CredsError exception, print out a configurable error message to the user (defined by the `edenapi.badcertmessage` config option). This allows us to provide specific instructions on how the user should renew their certificate.
Reviewed By: xavierd
Differential Revision: D15992903
fbshipit-source-id: 8316e33c3a5d86da272deea6402271d4a65548f4
Summary: The future is the Rust stores, not C ones.
Reviewed By: kulshrax
Differential Revision: D15993202
fbshipit-source-id: fbdb796eafcfdaccd1f98177e83a5c0a851bee5a
Summary:
We've had these 2 options turned on for a while now. Let's stop pretending
we'll switch them off, and just hardcode them over the codebase.
Some places are still using either the C code, or the Python one explicitely,
future changes will switch them to the Rust code.
Reviewed By: kulshrax
Differential Revision: D15981631
fbshipit-source-id: c23e474a84e887a2b92c9a304d63ef8d680cdf2d
Summary:
The requirements field is only interesting for non-remotefilelog or
generaldelta repos. Nowadays it's probably safe to assume clients are
remotefilelog and "requirements" is less useful.
Reviewed By: xavierd
Differential Revision: D15710676
fbshipit-source-id: 65f04b64760c652432471c4a8dda7acc4cf45466
Summary:
The in-core blackbox command displays blackbox entries in the given time range.
The `blackbox` command provided by the blackbox extension was removed. They
can still be accessed via `.hg/blackbox.log`, though.
Tests are updated. Most `| grep` patterns were changed to use structured pattern
matching `--pattern` instead. Tests that are not interesting (ex. bundlebackup,
since we are moving away from bundle files slowly) are just removed.
Reviewed By: markbt
Differential Revision: D15640718
fbshipit-source-id: 7e5da60ca2b15ae9495d0242b340a066979d5a4f
Summary:
Add a `mercurial.blackbox` module which just delegates to the Rust
binding. This means blackbox is no longer optional.
Change `ui.log` to log to the native blackbox as `LegacyLog` event.
The plan is to slowly migrate users from `ui.log` to `blackbox.log`,
which supports well-defined event types (instead of `LegacyLog`).
This does not change the `blackbox` command, which still uses the
legacy blackbox implementation. That will be changed in a later diff.
Reviewed By: markbt
Differential Revision: D15640720
fbshipit-source-id: de171f46e1430060083c9b7aee0a96dde315d021
Summary:
The downside of regex test output is that it is not `run-tests.py -i` friendly.
So stop suggesting that.
Reviewed By: markbt
Differential Revision: D15755287
fbshipit-source-id: fa9ededb358955bc2ac78a5d6d5f71265385b40d
Summary:
This is a simple but effective way to speed up frequent writes.
Before:
blackbox insertion (400 entries) 99.856 ms 12.896 KB/ 54.109 KB 16.012 KB
After:
blackbox insertion (400 entries) 0.940 ms 39 B / 20.207 KB 16.012 KB
Since writing is fast, the benchmark was changed to insert 4000 entries instead:
blackbox insertion (4000 entries) 3.918 ms 39 B /209.836 KB 160.012 KB
Reviewed By: markbt
Differential Revision: D15685472
fbshipit-source-id: 62a53512a3e0ad83120e625998e7a853b43dbe41
Summary:
This gives us some data about blackbox performance, and interesting areas to improve.
blackbox insertion (400 entries) 99.856 ms 12.896 KB/ 54.109 KB 16.012 KB
blackbox filter by index (4000 entries) 2.259 ms 0 B / 0 B
blackbox filter by pattern (4000 entries) 9.592 ms 0 B / 0 B
(wall time) (IO read / write ) (log size)
(not counting mmap IO)
Takeaways:
- Insertion performance is bad.
- Blackbox size looks okay.
- Pattern matching alone can be slow for large data (ex. 100MB log requires ~6s
matching). Therefore using indexes to narrow down entries seems worthwhile.
Reviewed By: markbt
Differential Revision: D15685473
fbshipit-source-id: 44ffffefd77f9e22699755b2ec520b1fc57ae244
Summary:
In case an IO measurement is run multiple times, it should not sum the bytes,
but just pick a max, or average.
This affects a future use-case.
Reviewed By: kulshrax
Differential Revision: D15685476
fbshipit-source-id: 8b98088c2596b8a39f09a9a1358e364c21ba50ae
Summary:
This just renames it to make it clear that the filter is backed by index, while
the pattern matching isn't.
Reviewed By: kulshrax
Differential Revision: D15685477
fbshipit-source-id: 64145caea00baa336c8c98a633129bc0f063ce8c
Summary:
This allows filtering events by flexible patterns. Conceptually it's reduentant
with the existing "filter". The difference is performance - one backed by
indexes, the other does linear scans.
The next diff will rename `Filter` to `IndexFilter` to make the difference more
obvious.
Reviewed By: markbt
Differential Revision: D15685474
fbshipit-source-id: ac533c25676256e6463d68bb72ba9a0a7e3cc867
Summary:
Treat Event as a JSON value, then use flexible pattern matching to match it.
The matching logic is somewhat flexible (and powerful), but not designed to be
efficient, though. Practically, it's probably always a good idea to filter
events by a time range first, then filter by the (expensive) pattern matching.
Reviewed By: markbt
Differential Revision: D15685475
fbshipit-source-id: aae033c39dd6e1845818a6b738de2ed5b0139721
Summary:
The `argv[0]` is used by libpython to decide the "executable path". However,
libpython cannot handle `argv[0]` being a relative path in the buck test
environment. Looking at the logic, it seems rather fragile. Therefore, use
the Rust stdlib to resolve the full path of the current executable. Pass
it as `argv[0]` to libpython, to make it able to insert the executable path
correctly.
This does affect `sys.argv[0]` seen from Python code. It is changed from a
relative path (ex. `hg`) to a full path (ex. `/bin/hg`). It seems to be
more executable and therefore might avoid some surprises.
Reviewed By: markbt, xavierd
Differential Revision: D15989622
fbshipit-source-id: 5a0d5b2f3e0b9ca1d97c84cb72d5dc079e9d3c1b
Summary: Export Rust-defined Python exceptions from the Eden API bindings module to Mercurial's Python code.
Reviewed By: quark-zju
Differential Revision: D15975434
fbshipit-source-id: eb681cfbd0c81d51180f04eb2c13394cc0e57f6f
Summary: This diff adds wrapper class around the Rust `edenapi.client` class defined in the bindings crate whose purpose is to catch any exceptions raised by the Rust bindings. For now, the wrapper class simply re-raises all exceptions; later diffs will use this to add special-case handling for certain exception types. (For example, for showing helpful messages when an error is due to a client certificate issue.)
Reviewed By: quark-zju
Differential Revision: D15975435
fbshipit-source-id: 21cef44dac84d5f468cb0e1d9eaddc2a1e93bfda
Summary:
Our test framework as it stands right now is a light passthrough to the hg `run-tests.py` test framework, which attempts to place all the files it needs to run (including tests) into a `python_binary`, then runs the hg test runner from that directory.
It heavily relies on how Buck works to offer functionality:
- It expects that all the sources it registers for its master binary will all be in the same directory when it builds
- It expects that the sources will be symlinks to the real files so that `--interactive` can work.
This has a few problems:
- It doesn't work in `mode/opt`. The archive that gets built in `mode/opt` doesn't actually have all the sources we registered, so it's impossible to run tests.
- To add a new test, you must rebuild everything. We don't do that very often, but it'd be nice if we didn't have to.
- Iterating on the runner itself is painful, because as far as Buck is concerned, it depends on the entire world. This means that every change to the runner has to scan a lot more stuff than necessary. There's some functionality I'd like to get into the runner (like reporting test timings) that hasn't been easy to add as a result.
This diff attempts to solve these problems by separating concerns a little more:
- The runner is now just a simple `python_binary`, so it's easier to make changes to it.
- The runner now provides the logic of working from local files when needed (this means you can add a new test and it'll work immediately),
- All the binaries we need are dependencies of the integration test target, not the runner's. However, to make it possible to run the runner incrementally while iterating on something, there's a manifest target that points at all the various paths the runner needs to work. This will also help integrate the test runner with other build frameworks if necessary (e.g. for open-sourcing).
- We have separate targets for various assets we need to run the tests (e.g. the hg test framework).
- The runner now controls whether to use the network blackhole. This was necessary because the network blackhole breaks PAR archives (because tmp is no longer owned by the right owner, because we use a user namespace). We should be able to bring this back at some point if we want to by using a proper chroot for opt tests.
I included a README to explain this new design as well.
There are some things that could yet stand to be improved here (notably, I think we should put assets and tests in different directories for the sake of clarity), but so far I've been aiming at providing a 1-1 translation of the old system into the new one. I am planning to make further improvements in followup diffs.
Reviewed By: farnz
Differential Revision: D15921732
fbshipit-source-id: 09052591c419acf97f7e360b1e88ef1f412da6e5
Summary: The output of `find` is wrong on OSX if the path is given with a trailing slash.
Reviewed By: ikostia, krallin
Differential Revision: D15984473
fbshipit-source-id: 57977b01310d95dd2b4828d62d53baf54a2f00bb
Summary: Now that we have an error type that we can pattern match on, use this to raise an appropriate Python exception in the bindings. Note that due to Rust's coherence rules, we need to implement the conversion to Python errors in the `edenapi` crate. Since not all users will need this FFI functionality, it is gated behind a feature flag (which is enabled by default for Buck builds due to limitations in how Rust targets are specified).
Reviewed By: xavierd
Differential Revision: D15914529
fbshipit-source-id: 3f6a5a651fc62d2506231dd2a33fbc06528d2bce
Summary:
The getpackv1 protocol doesn't unfortunately support LFS blobs, which is
therefore blocking deploying remotefilelog.fetchpacks on ovrsource on the
clients.
The easiest way to get there was to simply add a getpackv2 API that is similar
in every way to getpackv1, but with the addition of a metadata field. While
full support for this was added to Mercurial, the Mononoke support is the
absolute minimum required as Mononoke doesn't support LFS.
I'm expecting that EdenAPI will completely remove the need for getpackv2 and
therefore for this code should be fairly short-lived.
Reviewed By: farnz
Differential Revision: D15954031
fbshipit-source-id: 465ac13ed8987191ccf9a7cec198d913143aaf13
Summary:
The getpackv1 API doesn't support LFS blobs as no metadata flags are sent over
the wire. This means that repositories like ovrsource can't use it, which
prevents these repos from moving away from loosefiles...
The getpackv2 is similar to getpackv1, except that it also sends the metadata
alongside the data. This means that the LFS flag is properly sent over the
wire.
Reviewed By: DurhamG
Differential Revision: D15905425
fbshipit-source-id: dab8dcb75c8d9db8c661f11f87318feca7d0f6a9
Summary:
The getpackv1 is never sending delta chains, as the full content is always
requested. We don't have plans to change this, so let's simplify the code a
bit.
Reviewed By: DurhamG
Differential Revision: D15905431
fbshipit-source-id: f853da08391185b525d8b3da2382519b5f8d76b2
Summary:
This just rewrites the code to be shorter by avoiding testing `is_system_exit`
multiple times and avoiding some temporary variables.
Reviewed By: ikostia
Differential Revision: D15924835
fbshipit-source-id: 2bd17538ebbfde2e2865991dd9bfd778774d33b4
Summary:
The most important template files are now shipped in the code. Assuming hgweb
is not used, the rest of template files are not necessary. Therefore do not
distribute them. The contributing guide is also heavily outdated and was
removed.
Reviewed By: ikostia
Differential Revision: D15915003
fbshipit-source-id: b080337bbd07775f9ec0643c34d7634ade1afdc7
Summary:
Switch to the new Python runtime. Remove parts that are incompatible and no
longer necessary, including:
- Building curses, python-lz4, and urllib3 (in build_nupkg.py).
- Mangling sys.path (in hgpython and entrypoint.py).
- Zip-related logic (in hgpython).
- Cargo features controlling build environment (in hgpython and hgmain).
- Zipping Python stdlib (in setup.py).
- Shipping files in `templates`, `help`, and most `contrib` files (in
setup.py).
For the hgpython part, the new expectation is: in all cases (windows, linux,
make local, installed, buck), `edenscm.mercurial.entrypoint` and
`edenscmnative` modules are always directly importable and are always the
expected modules if imported. So the `hg` logic just imports and runs it
without having any `sys.path` related logic.
To explain it further:
- When installed on a POSIX system, the default `sys.path` contains
site-packages. Both edenscm and edenscmnative are in site-packages.
- When installed on Windows, the executable (hg.real.exe), python27.dll,
python27.zip, and edenscmnative/ are all in a same directory. Therefore the
default sys.path (exe dir + python27.zip) would be sufficient.
- When using `make local`, and run via `scm/hg/hg`, the `PySys_SetArgv` API
(called by `hgpython`) inserts the `scm/hg` directory as `sys.path[0]`,
therefore edenscm and edenscmnative in `scm/hg` will be imported as expected.
Since we no longer hard code paths to search for modules, this should fix
issues on systems with a different sys.path, ex. debian and ubuntu uses
"dist-packages" instead of "site-packages".
Note: IPython is broken. It seems to be a combination of newer Python version,
newer compiler and 64 bit (see [1]). It looks like prompt_toolkit incorrectly
uses untyped ctypes APIs where it passes "int/long"s to places expecting a
HANDLE. The ctypes library uses 4-byte integers for plain "int/long"s where a
HANDLE is 8 byte on 64 bit Windows. The new interpreter is stricter somehow
and will error out in this case (also explains why D15912516 is needed).
The fix to prompt_toolkit was sent as
https://github.com/prompt-toolkit/python-prompt-toolkit/pull/930.
[1]: https://github.com/prompt-toolkit/python-prompt-toolkit/issues/406
Reviewed By: ikostia
Differential Revision: D15894694
fbshipit-source-id: 560d11ae28c1e65d58b760eac93701e753bd397e
Summary:
Use _n for plurals of strings, rather than hard-coding the suffix, and don't
retranslate the string after is has already been translated.
Reviewed By: kulshrax
Differential Revision: D15940244
fbshipit-source-id: 26ef175fe201bcc5211ad05c8b3894fbc22d61b6
Summary:
In S179901, we've seen a couple of instances of `hg status` consuming as much
memory as they could until getting OOM killed. Looking at backtraces for these
processes (P67246356) the code spins trying to read data from memcache. The
connection is however closed and therefore `recv` returns an empty string,
which is then added to a list and the `recv` is retried again.
Reviewed By: kulshrax
Differential Revision: D15947154
fbshipit-source-id: f1216e26b0ce159377ff8e82bedc13a7cc3d0444
Summary:
This makes it possible for `python setup.py` to import `distutils_rust` without
changing sys.path. The `argv[0]` path being part of `sys.path` is the expected
Python interpreter behavior.
Note: the `argv[0]` path is used in hgmain for POSIX, but not Windows because
of `0005-PC-aggressively-simplify-sys.path.patch`. That is desirable since on
Windows, the "main exe path" is the `argv[0]` path, and it already exists in
the default `sys.path`.
Reviewed By: ikostia
Differential Revision: D15935893
fbshipit-source-id: a759440ef3103dfdd49d97727ea74d09afe1d881
Summary:
Some features of the PDCurses binding requires some marcos to compile. Define them.
Also removed duplicated source code that got accidentally copied.
The macros are copied from setup.py of the Python binding [1],
with `NCURSES_MOUSE_VERSION` removed, which breaks complication.
[1]: curses‑2.2‑source.zip at https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses
Reviewed By: ikostia
Differential Revision: D15935894
fbshipit-source-id: 944466f083666b68c55435881d627d8a51acffa2
Summary: This updates `run-tests.py` to report times in milliseconds if we're running under TestPilot. This is useful because TestPilot diverges a bit from the JUnit spec in the sense that it expects time to be in milliseconds, not seconds.
Reviewed By: StanislavGlebik
Differential Revision: D15938672
fbshipit-source-id: a421b523931abfdf5da10e6702739cecf00a4cdd
Summary: Instead of just throwing `RuntimeError` for every failure, we should return more appropriate built in exceptions on a case-by-case basis.
Reviewed By: quark-zju
Differential Revision: D15913954
fbshipit-source-id: 753aa95111bca5ce18c3b0a210e6439f60f4894a
Summary: This diff adds a new `ApiError` type that will serve as the general error type for Eden API operations. This error type is a based on the ["Error and ErrorKind pair" pattern](https://rust-lang-nursery.github.io/failure/error-errorkind.html) described in the `failure` crate's documentation. It uses a `failure::Context` to wrap an underlying error, and provides both the ability to provide users with high-quality custom error messages as well as the ability to programmatically consume such errors via pattern matching. See the doc comment in the code for usage details.
Reviewed By: xavierd
Differential Revision: D15913955
fbshipit-source-id: cb61f8c40dc04c78a5c281535163788e95c4f4e8
Summary: Add a new `errors` module with a new `ApiResult` type. For now, this type is just an alias for `failure::Fallible`. The goal of this diff is to simply plumb the new result type throughout the crate. In a later diff, we will then leverage this to change the error type for the whole crate without needing to update all the function signatures.
Reviewed By: xavierd
Differential Revision: D15913956
fbshipit-source-id: be3c64051c67722b517add3c006006bcd9116962
