Summary:
Adjust link node needs to read the node for a given file in a manifest,
and it can inspect manifests far back in time. The tree fetching code attempts
to download the entire tree, which makes each lookup take 40+ seconds in some
situations. Let's tweak adjust linknodes so downloads during it will download
trees one-by-one as needed.
Let's also record how long adjust link node takes in the scuba table.
Reviewed By: quark-zju
Differential Revision: D8629725
fbshipit-source-id: 4f1f9a3694c45a10a53f23ee081ba5f3c9d296b6
Summary:
We're experiencing cases where users are pulling down far more trees
than they need. Now that we can download just portions of a tree, let's add a
depth limit to the fetch code path. This will let us avoid downloading trees
that are never used.
The default is still to fetch all trees, but future diffs will make certain code
paths download only portions of trees.
Reviewed By: quark-zju
Differential Revision: D8523217
fbshipit-source-id: 066c5aef14b00ea0a366cf43e8a89ee053f82d22
Summary:
We're experiencing cases where users are pulling down far more trees
than they need. Let's enable the ability to download a certain depth of trees.
In this diff we just implement the actual iteration limiting in native code. In
a future diff we'll make the appropriate python changes and add a test.
Reviewed By: quark-zju
Differential Revision: D8523215
fbshipit-source-id: ccfeadee1525b86b1a5578631dbafe666de41f4c
Summary:
Previously, we could only ask `logginghelper` to log some `section.name`
config option via `ui.log` under the `name` name. This might be not enough
when section keys are generic and only make sense when context is available.
The entire list of config options to log was also supposed to be provided
under a single config option, `logging.configoptions`, therefore it was
impossible to extend it in the included config files, just completely
overwrite.
Let's solve this by dedicating the entire `[logging]` config section to
the purpose of defining other config options to log, where each key would
be a name to use and each value would be a `section.key` of the desired option
to log.
Reviewed By: DurhamG
Differential Revision: D8639436
fbshipit-source-id: 442792c2ea11cb4c9e7191b92cd4a8c0c67e9f27
Summary:
Update the buck build to use the "hg" script as its main module, rather than
using its own custom mercurial/main.py file.
This also changes how the extensions code decides whether to perform strict
checks or not. Previously this was disabled for buck builds by checking the
entrypoint argument. Now that buck builds use the same main module the
entrypoint is the same. Therefore we now distinguish buck-based builds based
on the presence of the `__manifest__` module.
Reviewed By: ryanmce
Differential Revision: D8601651
fbshipit-source-id: abfea4c6d91596149c240baa1bead740dffe8750
Summary:
Remove the code from the highlights extension that tries to eagerly load the
pygments lexers. This prevents commands that load the highlight extension
from failing in some circumstances.
This code was added in https://phab.mercurial-scm.org/D1619 since it avoided
"really weird traceback problems caused by demandimport messing with some of
the lexer plugins." It also claimed that eagerly loading these lexers
shouldn't affect most users since the highlight extension is only relevant for
servers. However the highlight module is loaded by other client-side commands
in several cases (for instance, "hg help"). When combined with the strict
foreign module import checking this causes this extension to get loaded
triggering this code.
This `find_plugin_lexers()` is using pkg_resources internally to find its
dependencies. This fails if some of the dependencies were installed as plain
python packages rather than as setuptools packages or eggs.
Reviewed By: quark-zju
Differential Revision: D8656400
fbshipit-source-id: 0b16852208e56823c5f8ab055b69020ac338447b
Summary:
When we apply draft commits to the repository, through
pull/unbundler/etc, we want to prefetch their public parent trees. If we don't,
when the draft commit is accessed it has the root node, but not some of the
children, and therefore attempts to download all the necessary trees ondemand
without any base trees. This is very expensive.
Let's prefetch the public parents, which should be relatively cheap since it can
delta against a nearby tree, and which is probably required anyway since
applying draft commits usually indicates they will be accessed soon.
Reviewed By: quark-zju
Differential Revision: D8654617
fbshipit-source-id: 63ee1e958fc0a0abdb8fa7bdb8c5d9e01181b6a7
Summary:
A future diff will want to use the base wirepack.receivepack with some
existing data and history packs, so let's refactor it so it can be used this
way.
Reviewed By: phillco
Differential Revision: D8404524
fbshipit-source-id: b1ab053870042933a2e2125f9bd58175479da255
Summary:
This commit just moves out the updates to the revision_references
table to a separate method so that it can be easily wrapped around.
Reviewed By: DurhamG, phillco
Differential Revision: D8406981
fbshipit-source-id: 8046a3e5b41dcf6c2f569699ac03de78611ba0bd
Summary:
p4fastimport has been replaced by p4seqimport months ago, on top of that previous assumptions have changed that might make p4fastimport not work (e.g. allowing import to connect to LFS).
This change cleans up a lot of unused code and a bunch of tests that no longer make sense.
Reviewed By: zhh95
Differential Revision: D8525286
fbshipit-source-id: 91d33e7530bf6df1e1ec92fae3acde0345230247
Summary: Otherwise, tests that use seq.py to create files, like test-rebase-inmemory-abort-tool.t, will end up with different content, and thus commit hashes, on Windows.
Reviewed By: quark-zju
Differential Revision: D8643553
fbshipit-source-id: 8bb533765f3f6bd794aa1f7ccaf754cfbe5c3bfb
Summary:
this command will be use in fbclone
it does sync and join if a person has been used already the commit cloud workspace in that repo
Reviewed By: markbt
Differential Revision: D8640667
fbshipit-source-id: 9c8d96497cf7930b995a49c8bdaebe7048e0c1e8
Summary: show commit cloud workspace state and infinitepush state as pretty jsons
Reviewed By: farnz
Differential Revision: D8611360
fbshipit-source-id: 97c0f38cd25bc0cc73f9a41d474b94234e463d3c
Summary: This commit just fixes the messaging for the errors.
Reviewed By: DurhamG
Differential Revision: D8553820
fbshipit-source-id: 73f2cd13e7538b6870b16a0e47e657a6d08af9e3
Summary:
The scenario we are handling is as follows:
1. the user runs a long-running Mercurial command with a lot of output, like
`hg log`
2. Mercurial runs pager, which is a subprocess, like `less` and changes its
stdout/stderr to point to pager's stdin.
3. The user exits from the pager (by pressing `q`, for example)
4. The pipe connecting Mercurial stdout/stderr with pager's stdin is now dead.
5. Mercurial tries to write something into its stdout, a failure occurs
6. The finally section restores stdout/stderr to be a tty
7. A failure gets caught up in the stack and printed to stderr, now a tty
A failure is vague here. On Linux, it means a `SIGPIPE`, on Windows, just and
exception that is manifested in Python as `IOError` with `errno=EINVAL`.
This is how Linux deals with it:
1. registed a signal handler for `SIGPIPE`, make this handler raise a
Mercurial-defined `error.SingalInterrupt`, which is a child exception class to
`KerboardInterrupt`
2. catch `KeyboardInterrupt` in `dispatch.dispatch`, handle the broken pipe
case there
For Windows this does not work, since there's no `SIGPIPE`, so
`SignalInterrupt` never gets raised, so we never handle the `IOError`.
This diff just explicitly handles the `IOError` in `dispatch.dispatch` and
silently consumes the `errno.EINVAL` case.
Reviewed By: quark-zju
Differential Revision: D8610555
fbshipit-source-id: 6cadba6a9ec296f03b1b616b763e56ec55da3a70
Summary:
Use the registrar to register template keywords and revset predicates, and
remove code for running the extension in old versions of Mercurial.
Reviewed By: farnz
Differential Revision: D8633454
fbshipit-source-id: aa0ebf86ff644072a3e78df8142dd8da7a188309
Summary:
Assume the stack:
```
o - C
|
o - B
|
o - A
|
o - Root
```
then if we need to metaedit commits A and B the stack becomes:
```
o - C' (metaedited)
| o - B'
| |
| o - A' (metaedited)
| /
o - Root
```
This bug affects also Jellyfish while submitting the stack of commits where some commit in the middle is already assotiated with Phab diff.
The bug is caused by the logic where we firstly metaedit and rebase the commits to metaedit in their topological order and only after all their descendants. So for now the given commits are processed with their descendants in topological order to track new parents for each commit.
Reviewed By: mitrandir77
Differential Revision: D8591897
fbshipit-source-id: 330a598e084af60cff42255510aca420bfe389de
Summary:
Add new revsets for finding remote bookmarks and remote branches by their exact
name, or by regular expression.
Reviewed By: quark-zju
Differential Revision: D8588923
fbshipit-source-id: 11e6d3636a0cccc5145c7dbd8824135b51dcbab3
Summary:
Add a config option `fsmonitor.detectrace` and environment variable
`HGDETECTRACE`. Once set, perform an additional query to check if there are
writes to the working copy while `wctx._buildstatus()` is being calculated.
Also enable the race detector for all fsmonitor tests.
Reviewed By: wez
Differential Revision: D8597957
fbshipit-source-id: 5d83d529641325dc8d5e72ad059eed8db27d1f2a
Summary:
2f205fdc948fbea made remote stderr get written to local stderr, but
didn't catch this spot where when a connection closes we read the remainder of
stderr and write it out.
Reviewed By: quark-zju
Differential Revision: D8627272
fbshipit-source-id: c846ac6bb7425114fb29c6374c74b35057c14e63
Summary: D8601652 fixed a bug which broke this test. So let's update the test.
Reviewed By: quark-zju
Differential Revision: D8629814
fbshipit-source-id: 661a406f6b978836644f79ae80d007be1ff98ffa
Summary:
For some reason, some users still have `perftweaks.cachenoderevs` enabled.
That actually slows down things because `clindex` is expected to make node
lookup faster without special hacks. So let's disable
`perftweaks.cachenoderevs` if clindex is enabled in config. This would also
allow us to clean up related configs in Chef (blocking D7826336).
Reviewed By: DurhamG, singhsrb
Differential Revision: D8626199
fbshipit-source-id: 69a6f22ee25ff0a00e870f7d07c73ca0431720c9
Summary:
use -w option instead of -p for keychain command security add-generic-password
-p Specify password to be added (legacy option, equivalent to -w)
-w Specify password to be added
options are the same, but -p seems to be legacy one
Reviewed By: DurhamG
Differential Revision: D8481013
fbshipit-source-id: 7c2c47f0b4e4737612416449c2a7c000ca09ab29
Summary:
Fix the `_collectimport()` code to more accurately figure out module paths.
When used from a statement like `import foo.bar` the `__import__()` function
returns the `foo` module. However, if used like `from foo.bar import baz` it
returns the `foo.bar` module. The `_collectimport()` code did not expect this
behavior and assumed that it always returned `foo`. This resulted in it doing
the wrong thing when trying to resolve nested imports.
In most cases it would just fail with an attribute error and it would then
swallow this exceptions. However in some cases it was able to find a
non-module object. For instance, when processing the `mercurial.extensions`
module it would find the `extensions()` function inside this module. Calling
`inspect.getabsfile()` on this function doesn't always produce the correct
result. If loaded from a pre-compiled .pyc file the function may have a
relative path baked in as its function name. The `inspect.getabsfile()` code
then simply prepends this with the current working directory, despite the fact
that this wasn't actually imported from the current directory.
I also changed the code to use inspect.getfile() rather than
inspect.getabsfile(). The getabsfile() function attempts to do other
normalization and string manipulation that we don't need. (It tries to find
`*.py` names rather than `*.pyc` or `*.pyo` names, and it normalizes case.)
Reviewed By: ryanmce, quark-zju
Differential Revision: D8601652
fbshipit-source-id: db9be3c5cbbea83a880851036093aaa28b5b9b19
Summary:
The chg code now always attempts to import __version__.py
Previously this was not generated during buck-based builds. This updates the
buck build to generate a __version__.py file now.
Reviewed By: ryanmce
Differential Revision: D8601650
fbshipit-source-id: 59bd0605394feeabcc41766a7e55cbdd253f07ad
Summary:
`size = -1` does not have special meaning in dirstate. Only `size = -2` is
special.
Reviewed By: DurhamG
Differential Revision: D8623318
fbshipit-source-id: 7293a7b492ba44cb8ffc49ce54a908e9612148e2
Summary:
The previous version of this tool raised on all merges, even non-conflicting ones.
We need to attempt a three-way merge first, _then_ raise an exception if it produces conflicts.
Reviewed By: DurhamG
Differential Revision: D8607184
fbshipit-source-id: 3f208caf54fa1ace28e1ee9011b34f5ec7bacec4
Summary:
For treestate, `copies` is more expensive as it's not a cache but calculates
from the source of truth. So avoid using `copies` but read the copy
information directly from the tree.
Reviewed By: DurhamG
Differential Revision: D8595357
fbshipit-source-id: bcea254358d66c9fa9f933d73221bbeb0bd8d5b2
Summary:
Let's advertise sparse to all the people who still need to hear about this.
This is behind config know. I think it's reasonable to enable it for ovrsource
and fbsource (possibly with different tresholds).
Reviewed By: quark-zju
Differential Revision: D8591973
fbshipit-source-id: 99d22705eb81e0b646b0c1537ddec395322069e6
Summary:
Some scripts are parsing all hg outputs. The remote stderr output could
contain random noise that break the scripts. Therefore let's forward
remote stderr to local stderr, instead of local stdout.
Reviewed By: DurhamG
Differential Revision: D8514952
fbshipit-source-id: 2f205fdc948fbeacd20b5af9d6d52eaa8212e90e
Summary: This will allow us to trigger hints from functions that wrap runcommand itself.
Reviewed By: quark-zju
Differential Revision: D8591974
fbshipit-source-id: e8e4742d6728ca347b85d10adc2e2ab7ab06ef16
Summary:
Using `wdir()` in dagrange expressions (e.g. `master::wdir()`) causes Mercurial
to crash with `IndexError: head out of range`. This is because the dagrange
operations can't use the wdir rev number as it is out of range for the
changelog.
Instead, modify the revisions that we are passing in to calculate the matching
revs, and add the wdirrev at the last moment if appropriate.
Reviewed By: ryanmce
Differential Revision: D8592129
fbshipit-source-id: 8742cb72c22d1ba26f5d03f9fcf56bfdb785c25a
Summary:
provide sockpath in the config, rather than calling `watchman get-sockname` command from python.
the config variable was already deployed on all mac
this improves hg st by 250-300 ms
(the logic gives more priority when env var if specified (the env var override is usually used for testing))
the logic has a retry mechanism if we can't connect to the socket path specified in the config
next step: deploy config on dev servers (but we will have only few ms speed up there)
we use %i (i for identity) notation for user
[298] → hg conf | grep sockpath
fsmonitor.sockpath=/opt/facebook/watchman/var/run/watchman/%i-state/sock
Reviewed By: farnz
Differential Revision: D8549130
fbshipit-source-id: 31df9ebf3eb5133d3e6f64794022cda1b0fadec4
Summary:
This config has been True everwhere for a while, and seems uncontroversial. Let's fold it in everywhere as True.
This simplifies a test case.
Reviewed By: DurhamG
Differential Revision: D8578289
fbshipit-source-id: f66a6427e435d80750b4da14358c5e4f0528b6f1
Summary:
It's exceeding unix domain socket length limit:
AssertionError: 2018-06-22T13:48:50,820: [] /tmp/hgtests.8M8JR4/child3/test-fb-hgext-treemanifest-prefetch.t-remotefilelog.true.shallowrepo.true.watchman/sock: path is too long
Talked with wez. Watchman does not want to workaround the issue by using
chdir like chg[1]. The main reason is changing cwd could break
thread-safety, especially when used as a library. chg is single-threaded and
does not have the concern.
[1]: c80c16a8a0
Reviewed By: DurhamG, singhsrb
Differential Revision: D8596333
fbshipit-source-id: 347a42fcd40d4ab075ca9c24f7c9ce4c8fa519b0
Summary:
D8560049 made `{reviewers}` work with commit template, but it breaks empty
`committemplate.reviewers` with `abort: "" not in template map`. This diff
fixes it.
Reviewed By: DurhamG
Differential Revision: D8596389
fbshipit-source-id: e1c046d59097181ef5c8d56a1cffc7c2407f0ba5
Summary:
Treestate tracks untracked files while treedirstate doesn't. The
treedirstate importmap logic changed by D7912557 will remove untracked files
from the tree. But it missed other places like nonnormalset, otherparentset
and copymap. This patch changes it to make sure paths in those sets are
tracked.
Reviewed By: DurhamG
Differential Revision: D8560022
fbshipit-source-id: 8ac39bfe634804ea94125fe42e11382acae35d94
Summary:
Previously `state._usetreestate` is a value set in `state.__init__`. In the
treestate migration case, `state._usetreestate` needs to change accordingly.
Therefore let's use a property to always calculate that value from the
source of truth.
Reviewed By: DurhamG
Differential Revision: D8560019
fbshipit-source-id: a313f603275321b959502e7f2c528c4b54a11ce9
Summary:
D8576826 broke the tests by allowing the importer to use lfs objects,
which therefore causes the lfs objects directory to exist, which breaks the
tests. Let's update the tests.
Reviewed By: quark-zju
Differential Revision: D8593012
fbshipit-source-id: 2897f55a8d75c3784d67f0d0e11870092f7dad71
Summary:
Revset predicates that work substantially through `subset.filter` should have a
higher weight than the default value of 1, to prevent the optimizer incorrectly
optimizing operations like:
```
reverse(ancestors(node, limit)) & filterpredicate()
```
in a way that filters all the revisions in the repo according the predicate
before performing the dag search.
Reviewed By: ikostia
Differential Revision: D8588027
fbshipit-source-id: 57be642761af013be8c8aa27ec126b3919c34256
Summary:
The expression `::revs & age("<x")` can be slow, as all ancestors must be
analyzed in case there are commits with ancestors that have later dates. Most
of this time this is unnecessary, as commits are usually monotonic in their
dates.
The `ancestorsaged` revsetpredicate evaluates to all ancestors within an age
range, optimized as if commit dates are monotonic.
Reviewed By: ikostia
Differential Revision: D8489319
fbshipit-source-id: 9b9055019c2afa70497a9df02c6fe3504e2c381a
Summary:
The `public()` revsetpredicate has the default weight of 1, which means
expressions like `::x & public()` are optimized to `('andsmally', ::x, public())`,
which will actually result in iterating over all commits in the repo.
Increase the weight to 3 to ensure it's optimized in a way that favours
following ancestry links and then checking the phase.
Reviewed By: ikostia
Differential Revision: D8549645
fbshipit-source-id: f58750ecb1918079b7e350ce27b7a61771489233
Summary:
Upstream Mercurial renamed peer.cleanup to peer._cleanup last August,
but our connectionpool logic wasn't updated. This meant that expired connections
were left alive and just hanging there until the process died. This defeated the
entire purpose of expiring connections (to free memory on the server).
Reviewed By: singhsrb
Differential Revision: D8583327
fbshipit-source-id: 561b6ed8b76247d996051b0536c5f58050fb83a2
Summary:
In the recent refactor we changed the definition of "is this tree
local" on the server to also include if the tree didn't exist in the store,
since then it might be an ondemand conversion from a flat manifest in a bundle.
This caused issues on peer-to-peer pulls, since most trees don't exist locally.
Let's fix it by only considering a missing tree to be a local tree if we're on
the server.
Reviewed By: quark-zju
Differential Revision: D8586779
fbshipit-source-id: ea8c8ee3e2a4dc054fae9caa122cfc733df1ab81
Summary:
The treemanifest manifest revlog store was keeping all the revlogs it
had ever opened. This was causing unnnecessary memory usage. Let's use an lru
dict instead.
Reviewed By: singhsrb
Differential Revision: D8579992
fbshipit-source-id: a0df30401714c93eebe0448b0ce92571f704bd96
Summary:
It seems the datapackstore c++ implementation is very leaky, which is
causing memory issues on the server. We're in the process of rewriting the
datapack logic to be pure rust, so let's just disable this feature for now.
Disabling this slows down tree downloads by 25%.
Reviewed By: quark-zju
Differential Revision: D8580519
fbshipit-source-id: 7e42e7ff211008d5461f95386da019ba5696e07b
Summary: It turned out that a waiting loop is necessary, at least on my laptop.
Reviewed By: DurhamG
Differential Revision: D8560023
fbshipit-source-id: 8efd8dca4c630a30e5ccb140eb4f2da8a0cc5717
