Summary:
The in-core blackbox command displays blackbox entries in the given time range.
The `blackbox` command provided by the blackbox extension was removed. They
can still be accessed via `.hg/blackbox.log`, though.
Tests are updated. Most `| grep` patterns were changed to use structured pattern
matching `--pattern` instead. Tests that are not interesting (ex. bundlebackup,
since we are moving away from bundle files slowly) are just removed.
Reviewed By: markbt
Differential Revision: D15640718
fbshipit-source-id: 7e5da60ca2b15ae9495d0242b340a066979d5a4f
Summary:
Add a `mercurial.blackbox` module which just delegates to the Rust
binding. This means blackbox is no longer optional.
Change `ui.log` to log to the native blackbox as `LegacyLog` event.
The plan is to slowly migrate users from `ui.log` to `blackbox.log`,
which supports well-defined event types (instead of `LegacyLog`).
This does not change the `blackbox` command, which still uses the
legacy blackbox implementation. That will be changed in a later diff.
Reviewed By: markbt
Differential Revision: D15640720
fbshipit-source-id: de171f46e1430060083c9b7aee0a96dde315d021
Summary:
Our test framework as it stands right now is a light passthrough to the hg `run-tests.py` test framework, which attempts to place all the files it needs to run (including tests) into a `python_binary`, then runs the hg test runner from that directory.
It heavily relies on how Buck works to offer functionality:
- It expects that all the sources it registers for its master binary will all be in the same directory when it builds
- It expects that the sources will be symlinks to the real files so that `--interactive` can work.
This has a few problems:
- It doesn't work in `mode/opt`. The archive that gets built in `mode/opt` doesn't actually have all the sources we registered, so it's impossible to run tests.
- To add a new test, you must rebuild everything. We don't do that very often, but it'd be nice if we didn't have to.
- Iterating on the runner itself is painful, because as far as Buck is concerned, it depends on the entire world. This means that every change to the runner has to scan a lot more stuff than necessary. There's some functionality I'd like to get into the runner (like reporting test timings) that hasn't been easy to add as a result.
This diff attempts to solve these problems by separating concerns a little more:
- The runner is now just a simple `python_binary`, so it's easier to make changes to it.
- The runner now provides the logic of working from local files when needed (this means you can add a new test and it'll work immediately),
- All the binaries we need are dependencies of the integration test target, not the runner's. However, to make it possible to run the runner incrementally while iterating on something, there's a manifest target that points at all the various paths the runner needs to work. This will also help integrate the test runner with other build frameworks if necessary (e.g. for open-sourcing).
- We have separate targets for various assets we need to run the tests (e.g. the hg test framework).
- The runner now controls whether to use the network blackhole. This was necessary because the network blackhole breaks PAR archives (because tmp is no longer owned by the right owner, because we use a user namespace). We should be able to bring this back at some point if we want to by using a proper chroot for opt tests.
I included a README to explain this new design as well.
There are some things that could yet stand to be improved here (notably, I think we should put assets and tests in different directories for the sake of clarity), but so far I've been aiming at providing a 1-1 translation of the old system into the new one. I am planning to make further improvements in followup diffs.
Reviewed By: farnz
Differential Revision: D15921732
fbshipit-source-id: 09052591c419acf97f7e360b1e88ef1f412da6e5
Summary: The output of `find` is wrong on OSX if the path is given with a trailing slash.
Reviewed By: ikostia, krallin
Differential Revision: D15984473
fbshipit-source-id: 57977b01310d95dd2b4828d62d53baf54a2f00bb
Summary:
The getpackv1 protocol doesn't unfortunately support LFS blobs, which is
therefore blocking deploying remotefilelog.fetchpacks on ovrsource on the
clients.
The easiest way to get there was to simply add a getpackv2 API that is similar
in every way to getpackv1, but with the addition of a metadata field. While
full support for this was added to Mercurial, the Mononoke support is the
absolute minimum required as Mononoke doesn't support LFS.
I'm expecting that EdenAPI will completely remove the need for getpackv2 and
therefore for this code should be fairly short-lived.
Reviewed By: farnz
Differential Revision: D15954031
fbshipit-source-id: 465ac13ed8987191ccf9a7cec198d913143aaf13
Summary:
The getpackv1 API doesn't support LFS blobs as no metadata flags are sent over
the wire. This means that repositories like ovrsource can't use it, which
prevents these repos from moving away from loosefiles...
The getpackv2 is similar to getpackv1, except that it also sends the metadata
alongside the data. This means that the LFS flag is properly sent over the
wire.
Reviewed By: DurhamG
Differential Revision: D15905425
fbshipit-source-id: dab8dcb75c8d9db8c661f11f87318feca7d0f6a9
Summary:
Switch to the new Python runtime. Remove parts that are incompatible and no
longer necessary, including:
- Building curses, python-lz4, and urllib3 (in build_nupkg.py).
- Mangling sys.path (in hgpython and entrypoint.py).
- Zip-related logic (in hgpython).
- Cargo features controlling build environment (in hgpython and hgmain).
- Zipping Python stdlib (in setup.py).
- Shipping files in `templates`, `help`, and most `contrib` files (in
setup.py).
For the hgpython part, the new expectation is: in all cases (windows, linux,
make local, installed, buck), `edenscm.mercurial.entrypoint` and
`edenscmnative` modules are always directly importable and are always the
expected modules if imported. So the `hg` logic just imports and runs it
without having any `sys.path` related logic.
To explain it further:
- When installed on a POSIX system, the default `sys.path` contains
site-packages. Both edenscm and edenscmnative are in site-packages.
- When installed on Windows, the executable (hg.real.exe), python27.dll,
python27.zip, and edenscmnative/ are all in a same directory. Therefore the
default sys.path (exe dir + python27.zip) would be sufficient.
- When using `make local`, and run via `scm/hg/hg`, the `PySys_SetArgv` API
(called by `hgpython`) inserts the `scm/hg` directory as `sys.path[0]`,
therefore edenscm and edenscmnative in `scm/hg` will be imported as expected.
Since we no longer hard code paths to search for modules, this should fix
issues on systems with a different sys.path, ex. debian and ubuntu uses
"dist-packages" instead of "site-packages".
Note: IPython is broken. It seems to be a combination of newer Python version,
newer compiler and 64 bit (see [1]). It looks like prompt_toolkit incorrectly
uses untyped ctypes APIs where it passes "int/long"s to places expecting a
HANDLE. The ctypes library uses 4-byte integers for plain "int/long"s where a
HANDLE is 8 byte on 64 bit Windows. The new interpreter is stricter somehow
and will error out in this case (also explains why D15912516 is needed).
The fix to prompt_toolkit was sent as
https://github.com/prompt-toolkit/python-prompt-toolkit/pull/930.
[1]: https://github.com/prompt-toolkit/python-prompt-toolkit/issues/406
Reviewed By: ikostia
Differential Revision: D15894694
fbshipit-source-id: 560d11ae28c1e65d58b760eac93701e753bd397e
Summary: This updates `run-tests.py` to report times in milliseconds if we're running under TestPilot. This is useful because TestPilot diverges a bit from the JUnit spec in the sense that it expects time to be in milliseconds, not seconds.
Reviewed By: StanislavGlebik
Differential Revision: D15938672
fbshipit-source-id: a421b523931abfdf5da10e6702739cecf00a4cdd
Summary: Similar to the previous diff, but for another function.
Reviewed By: ikostia
Differential Revision: D15912517
fbshipit-source-id: 292875e13377e2d6d2a205e02815324238d169e1
Summary:
Instead of relying on ctypes' default type guess being correct, let's provide
the function signature of GetProcessTimes explicitly.
This resolves an issue where the new interpreter raises
`WindowsError: [Error 6] The handle is invalid.`.
Reviewed By: ikostia
Differential Revision: D15912516
fbshipit-source-id: 4fbb0de918d8a95ff665ae1b89494ed91e3fa8d3
Summary:
This just moves things around. So native and pure Python modules are split to
different Python packages. This makes it possible to use the standard zip
importer without hacks (ex. `hgdemandimport/embeddedimport`).
This diff is mostly about moving things. While `make local` still works,
it does break nupkg build, which will be fixed in a later diff.
Reviewed By: kulshrax
Differential Revision: D15798642
fbshipit-source-id: 5d83f17099aa198df0acd5b7a99667e2f35fe7b4
Summary:
Verify that the behavior of the Rust history store is the same as the Python
one. A handful of changes needed to be made to both to account for small
difference in behavior regarding getancestors, and empty copyfrom paths.
Reviewed By: kulshrax
Differential Revision: D15721524
fbshipit-source-id: e03b7e8dcbc6f9f1941c64536b8de39b77f637c9
Summary:
Now that linkrev are no longer added to mutablehistorypack, let's remove the
code that deals with it. This will enable using a Rust based historystore in
the future.
Reviewed By: kulshrax
Differential Revision: D15684623
fbshipit-source-id: d305e01cdfc0366c9432e0418854fd3b43a35539
Summary:
We had a user that had a folder created in .hg/edit-tmp, this prevented them
from commiting their work as `hg commit` would crash due to the folder. Let's
handle this case.
Reviewed By: quark-zju
Differential Revision: D15857754
fbshipit-source-id: 087096f3c14ca2ac490ae1af20651abc813df8f3
Summary:
This adds a new queue to replay scratch bookmark changes into Mononoke, which will allow us to replay them there.
There's not a lot going on in Mercurial (i.e. in this diff) to achieve this: we simply record bookmark changes as they happen in infinitepush, and allow for excluding some bookmarks. Specifically, we'll want to exclude backup branches, which we don't want to copy, since a) there's way too many of them and b) they're deprecated in favor of Commit Cloud.
Currently, this does not allow for replaying deletions. That would require further rework of how we delete things, since right now we do it my matching on bookmark names in the DB, which means the Python side of things is not aware of which bookmarks exactly were deleted. I'm not aware of how much use this is currently getting, but I'll research that and add it in if necessary.
Finally, one thing that's worth calling out here is the `bookmark_hash` column in this table. This is here in case we need to scale out the replication of bookmarks across multiple workers.
Indeed, we'll always want the replication of any given bookmark to happen sequentially, so we should perform it in a single worker. However, if we have too many bookmarks to replicate, then that could become a bottleneck. If that happens, we'll want to scale out workers, which we can do by having each worker operate on separate bookmarks.
The `bookmark_hash` column allows us to evenly divide up the space of bookmarks across workers if that becomes necessary (e.g. we could have 16 workers: one for each first hex digit of the hash). We won't use `bookmark_hash` immediately, but since it's very cheap to add (just compute one hash in Mercurial and put it in the table), I'm adding it in this diff now in case we need it later to avoid the friction of having to re-redeploy hg servers for that.
Reviewed By: StanislavGlebik
Differential Revision: D15778665
fbshipit-source-id: c34898c1a66e5bec08663a0887adca263222300d
Summary:
The `b2x:rebase` part is just like a changegroup, but it has a different name, and that results in `debugbundle` not telling us what the contents are.
This updates `debugbundle` to the contents of `b2x:rebase` parts.
Reviewed By: farnz
Differential Revision: D15822108
fbshipit-source-id: 60c28e4d965dabb7a889a14c179c1b69d0c5196e
Summary: The hg pasterage is the command that people run to share debug info when something is wrong with their repository. Having the information about CommitCloud related stuff will help with all the future debugging of that.
Reviewed By: mitrandir77
Differential Revision: D15744065
fbshipit-source-id: 094ccdf79c38fed78f5106a1617a5af09e1870e8
Summary:
When a revlog cannot be opened, include the broken revlog filename in
the error message.
Reviewed By: quark-zju
Differential Revision: D15503605
fbshipit-source-id: 46aff78a65eaa35c921af5034d175d95a4380c08
Summary: This adds (behing a feature flag) the option for Mercurial servers to record bundles they ingest into a `forwardfillerqueue`. This queue is consumed by the commit cloud forwardfiller (see stack rooted at D15712911 for that) to replay those commits into Mononoke.
Reviewed By: farnz
Differential Revision: D15759638
fbshipit-source-id: 70660343408bbaa865fc8f51f49f590eaf525cce
Summary:
Selective pull, when just enabled, will filter only remote bookmark for the active remote repo at that time. When user will try to pull next time from the different remote, bookmarks for that remote won't be filtered, because the selective pull will think that it's already enabled and the remotenames file already has only interesting bookmarks.
I changed `selectivepullenabled` file to keep track of the remotes which selectivepull has been enabled for.
Reviewed By: markbt
Differential Revision: D15583506
fbshipit-source-id: dec09baf1e1a0c80d1d5987ac6f85fe1202b2dab
Summary: Now slective pull, when enabled, will keep not only configured remote bookmarks, but accessed, that were recoreded earlier, as well.
Reviewed By: markbt
Differential Revision: D15583505
fbshipit-source-id: 8c8d300afc333a94427513d9844c1c1af3cf7f32
Summary:
If selective pull is enabled, the old implementation of `--remote` doesn't show all available remote bookmarks, but only locally available ones.
New implementation uses `listkeys` API to fetch remote bookmarks from the server.
However because `remote-path` flag belongs to the infinitepush extension, if the extension is not enabled, than `--remote` will show only bookmarks from the default remote.
Reviewed By: markbt
Differential Revision: D15547513
fbshipit-source-id: b7c79a0cc7237bbf086c64a71d6ed313aa520582
Summary:
With selective pull feature we need to have ability to list locally available remote bookmarks - the bookmarks the user is subscribed to.
As current implementation of `--remote` does exactly that, pointing subscription to the same implementation.
Reviewed By: markbt
Differential Revision: D15547515
fbshipit-source-id: d3ac7295fee3391076855fbe1bed00124dab973f
Summary: When selective pull will be enabled, `hg log` won't be able to show any information about the remote bookmarks that are not in subscriptions for particular user. So we need to hint the user that they may want to explicitly pull the remote bookmark first, if hg log fails to find it.
Reviewed By: quark-zju
Differential Revision: D15516462
fbshipit-source-id: 5be77b0048d8e175a737f76a8e89768f4c837f60
Summary:
In D15640721 the LFS components of the p4fastimport tests were removed. This
breaks the build. Restore them.
Original commit changeset: 3e3bd248526c
Reviewed By: ikostia
Differential Revision: D15737561
fbshipit-source-id: 41da16304732728de8ba5f8463f38901370db70a
Summary:
As we're working towards moving exclusively to Rust based mutable stores, let's
modify the existing datapack unit tests to test them.
Reviewed By: kulshrax
Differential Revision: D15673688
fbshipit-source-id: 847ec0f5f5316e6f8ba72f27a7665dd3bf46c5fc
Summary:
We don't care about tag caches. Remove them. This makes it easier for the new
blackbox migration in a later diff, since I don't have to deal with those
tag cache tests using blackbox.
Reviewed By: xavierd
Differential Revision: D15685470
fbshipit-source-id: 4e56daaddc4e0a9f9d590ddd51ed164c8f5228ab
Summary: Those tests are failing on my devserver (with p4 installed).
Reviewed By: xavierd
Differential Revision: D15640721
fbshipit-source-id: 3e3bd248526cab47a26bd1ae2631b8cbf65271df
Summary: We want to have just one entry point to Mercurial, namely the Rust binary. Getting rid of the `hg` Python script means that we finally can do things that only exist in Rust and not in Python.
Reviewed By: simpkins
Differential Revision: D13186374
fbshipit-source-id: f3c8cfe4beb7bf764172a8af04fd25202eca9af2
Summary:
Use just the predecessor and successor information to detect duplicate obsmarkers.
If multiple obsmarkers relate the same sets of commits, just ignore the later ones.
Reviewed By: mitrandir77
Differential Revision: D15679886
fbshipit-source-id: 92d468b41af039804c6047a0ec58580514aceee7
Summary:
If duplicated obsmarker information got propagated into the store, then commits
may have multiple copies of the same predecessor or successor. This causes the
predecessorsset and successorsset algorithms to become exponential.
Make sure that whenever we look-up predecessors or successors sets, or when we
expand them to the next or previous step in the chain, we remove any duplicates
that have resulted.
Reviewed By: mitrandir77
Differential Revision: D15679887
fbshipit-source-id: 9e396c1422674cf2406d44bf55cf961a8f118117
Summary:
Duplicate predecessors shouldn't exist in mutation data, but if it does,
highlight in the debug output and don't recurse multiple times.
Reviewed By: mitrandir77
Differential Revision: D15679889
fbshipit-source-id: 438dbd95152b33ccff7494afd684088a86a118ca
Summary:
Obsmarkers can get duplicated with the timestamp field slightly different due
to floating point precision differences. The obsmarker code compares
obsmarkers exactly, and so these duplicates are not detected.
When backfilling these obsmarkers into mutation information, we end up turning
the duplicate obsmarkers into folding and divergence.
This test demonstrates the problem.
Reviewed By: mitrandir77
Differential Revision: D15679888
fbshipit-source-id: 0d7e7b4f4365e53988e1679ccd8510334b9035b8
Summary: D13838772 introduce validation of hashes when fetching data from mercurial/memcache. We'd like to get insights in to how often this would happen, and potentially attach alarms to notify.
Differential Revision: D15472456
fbshipit-source-id: ad2f28e91824e4bbfc02284f230111e2d4cb3f00
Summary:
The clients don't have any mapping from `globalrev->hash` locally but
they can do this lookup quickly using ScmQuery. We do not want to invest too
much effort on supporting globalrev based lookups on the clients as this is a
workflow we want to discourage anyway. But, for now, existing workflows may
break if this lookup is slow on the clients. Therefore, for the interim, lets
just have this lookup backed by ScmQuery. We can later improve or completely
discard this based on the future direction for globalrevs.
Reviewed By: quark-zju
Differential Revision: D15588420
fbshipit-source-id: 61f91414248ca1defe6eac4311243ee8029a92cf
Summary:
Support bundlerepos with visibility.
The heads of the commits in the bundle are added to the set of visible heads.
Reviewed By: mitrandir77
Differential Revision: D15604477
fbshipit-source-id: 1d287075fa69eeee21bf963ddf28b731eee441f2
Summary:
Support bundlerepos with mutation data.
Infinitepush is currently the only place where bundles may include mutation
data. This is used when rebundling infinitepush bundles to send back to the
client.
Reviewed By: mitrandir77
Differential Revision: D15604475
fbshipit-source-id: 111a3211044c742d3d1c75d06c6e6114f754e396
Summary:
Update `hg githelp -- git branch -d` suggestions to suggest `hg hide -B`.
Also fix up the code so that trailing `-B` options are not produced.
Reviewed By: farnz
Differential Revision: D15575864
fbshipit-source-id: bd0d0b23f2a509e02554bcbb453718bebe8ea11a
Summary:
Add support for `hg hide -B bookmarkname`. This hides all commits that are
uniquely reachable by the provided bookmarks. This means if a bookmark is at
the head of a stack, `hg hide -B bookmark` will hide the stack.
Reviewed By: quark-zju
Differential Revision: D15562715
fbshipit-source-id: 9fdb3383b534faea982396a4f4782c03e4910dc3
Summary:
Make the code for calculating which revisions the `-B` options to `hg prune`
and `hg debugstrip` affect common to both commands.
The remotenames modification of `repair.stripbmrevset` is wrong. It should
exclude all ancestors of commits that have remotenames, not just those that
don't share a bookmark with the remotename. Fix it.
Reviewed By: quark-zju
Differential Revision: D15562716
fbshipit-source-id: 507002505100e7d60c395f242cc8e1062b91cc20
Summary:
Through Scuba, we can identify quite a few users that are doing `hg push default` for an infinitepush: https://fburl.com/scuba/6vz3ien7.
This is typically happening in repositories where the `default-push` destination is not the hg repository.
In fact, we actually prompt users to use hg push default if they try to perform a scratch push to a svn repository a few lines below!
markbt suggested we should just replicate the push in this case as well, which I think is a good idea.
Simply replicating the push in this case is not quite enough, however!
Indeed, for our infinitepushes to go to both Mercurial and Mononoke, we would have to maintain an number of consistency rules between `default`, `infinitepush`, and `infinitepush-other`:
- `default` and `infinitepush-other` must point to opposite repositories (i.e. one Mercurial, one Mononoke), to serve users using `hg push default`.
- `infinitepush` and `infinitepush-other` must point to opposite repositories, to serve users using `hg push` without a path.
This effectively means `default` and `infinitepush` must be the same, and `infinitepush-other` must be the other. We could do this with path markers, but this is getting so complicated that I don't foresee us getting it right.
At the end of the day, the root cause of this complexity stems from the fact that we're using a destination that is now effectively a read path (`default`) for infinitepush writes. I think this was perfectly valid when default meant "Mercurial, not subversion", but not so much now that we have Mononoke in the mix.
Of course, we could just update the message and ask our users to use `hg push infinitepush` instead, but this would mess with their muscle memory as well as their shell history (not to mention that `hg push default` would silently do the wrong thing).
So, this patch updates the code to use the infinitepush "write" destination for writes when that is what the user intended.
---
As a side note, note that the current behavior is actually a little broken. Indeed, if one were to do `hg push default --to scratch/$FOO` while Mononoke is serving reads, the scratch would go to Mononoke. In this scenario, the scratch push would in theory have been accepted but would have had its bookmarks discarded (at least until we support htose in Mononoke :) ).
This probably never happened in practice, however, for two reasons:
- Most users and systems that actively push scratch bookmarks are actually pushing to a specific `hg.vip` path instead of `default` (one notable exception is On Demand WWW).
- `hg push` to Mononoke for a scratch bookmark doesn't actually work if you have the pushrebase extension code loaded in some way (either by enabling the extension, or enabling an extension that uses it). See D15576199 for the details.
Reviewed By: farnz
Differential Revision: D15576545
fbshipit-source-id: c28b808632505bb8e8f4d114029f7d8c17c9749e
Summary:
The hgsubversion tests are flaky because they select a random port between
`socket.IPPORT_USERRESERVED` (5000) and 65535 for the subversion server to
listen on. The port that is selected may not be free, as it might be in use by
another service running on the same host.
Ports above 5000 are a wild west, so there is no guarantee of any particular
port being available for svnserve to use. Check that the randomly selected
port is actually available by trying to bind to it. If the bind fails, assume
the port is not available and pick a different one.
Reviewed By: quark-zju
Differential Revision: D15554733
fbshipit-source-id: c37d61488449bb0b93907eeee91f6fa404c959cf
Summary:
We will eventually get rid of the `hgsubversion` extension and
therefore, we want to remove any dependency on it in the long term. This commit
removes any dependency on the `hgsubversion` extension for retrieving the
`globalrev` corresponding to a commit. Note that the `globalrev` for the older
commits is just the `svnrev`.
Reviewed By: quark-zju
Differential Revision: D15579401
fbshipit-source-id: 021e6d93d008b07591e1b7801062d9c5fbfc7651
Summary:
Aaron recently noticed that `hg log -p -l 1 fbcode/assistant` ran for a very
long time while it only produce one diff. The profile shows that most of the
time is spent collecting ancestors of files that involves lots of downloads.
Even when everything is cached locally, its run time is terrible at over 5min.
Looking at the profile, the code spent most of its time in the `populate`
function that tries to build the set of files present in a specific revision.
This operation has a O(N*M) complexity, with N the number of files, and M the
number of revisions per files.
To speed this up, we split the problem: 1) for individual files and 2) for
directories.
For 1), the user intention is to get correct history, so let's keep the code as
it is. For 2) let's simply match files in the given directory, with no
consideration for the revision or the ancestors.
Reviewed By: quark-zju
Differential Revision: D15542664
fbshipit-source-id: 63028894babbb3efd41d09454d60d5241d7c3787
Summary:
Overriding `util.getpid` breaks tests that test locks as these require
different processes to have different PIDs.
Remove the general override and only override in tests that test blackbox
output and don't care about the PIDs.
Reviewed By: DurhamG
Differential Revision: D15558451
fbshipit-source-id: fb640cedca4ad53e29ee57d81fc4cc610ab27260
