Summary:
Analogous to `localrepo._wlockfreeprefix`, this is the prefix of files in the
store that can be safely accessed without the store lock.
Reviewed By: mitrandir77
Differential Revision: D8480907
fbshipit-source-id: 6d6c625c7f2230384ee361ff6b585f42feea6d10
Summary: Added to commonancestorshead function explaining how the algorithm works.
Reviewed By: DurhamG
Differential Revision: D8415752
fbshipit-source-id: 68e77b0b1f7afb54b88b77084eb371abeeae8ab0
Summary: This adds a merge tool to only rebase if there aren't conflicts, which I think will be widely useful. It's for IMM only at the moment.
Reviewed By: DurhamG, quark-zju
Differential Revision: D8493065
fbshipit-source-id: 32ccc0e53af27d275a56bcac80861a421d7abfc6
Summary:
hg filename encoding will double the length of upper-case paths. That could
be problematic if there is a long file name with many upper-case characters.
Workaround it by detecting the long name case, and keep upper-case
unchanged.
This unblocks a production issue where a long upper-case file name was
committed to the source of truth and the hg mirror cannot import it. But it
makes Windows hosting less "correct".
Reviewed By: DurhamG
Differential Revision: D8527475
fbshipit-source-id: da25fcce7e3173769788bd164629c28466158cd2
Summary:
When building a bundle that will be used for a pull that can't use the linkrev
fastpath, we need to process the manifests of the commits for which we will
send files so that we can correctly set up their linkrevs. Currently this is
sometimes happening for all commits, which can be very slow when pulling a
large number of commits.
First, make it clearer what `LocalFiles` and `LocalTrees` means, and make it
easier to test for trees being local.
Second, when processing manifests looking for files that will be included in
the bundle, don't process non-local trees if we are only interested in local
trees.
Third, correctly determine whether to send all, local, or no trees or files
depending on what combinations of treemanifest and remotefilelog the client and
server support, and what operation is taking place.
Finally, we aren't passing `b2caps` through to `changegroup.makestream` in the
changegroup part generator for bundle2 parts. This means the above test for
whether the client supports treemanifest always failed.
With these combined changes, a pull of a specific public revision (so not
following the fastpath) where the client supports both remotefilelog and
treemanifest, will result in the server using `LocalFiles` and `LocalTrees`,
and therefore not processing the manifests of public commits looking for which
files to send.
Reviewed By: DurhamG, ikostia
Differential Revision: D8446137
fbshipit-source-id: 9a4247af75fe79c1962a29fcb8bf0181868699f4
Summary:
With treestate, `dirstate.copied` is always calculated from the source of
truth and there is no caching. So let's get the `copies` information outside
the loop.
Reviewed By: phillco
Differential Revision: D8498997
fbshipit-source-id: 4e79f601defa63066624f7498270cdd0b754d2ae
Summary:
With treestate, nonnormalset and otherparentset no longer need separate
maintenance, since they are always calculated from the source of truth.
Reviewed By: phillco
Differential Revision: D8498996
fbshipit-source-id: a9d52f0cbbf2049dc873ca5ab83420a436408f36
Summary:
Implement gc for treestate. Dirstate writes are protected by wlock. So it's
fine to do a gc inside the write method.
Reviewed By: markbt
Differential Revision: D8394736
fbshipit-source-id: 672087a2c14b855d1ca0449eb5b7d92d65bba418
Summary: Similar to treedirsate, if the file size exceeds configured limit, do a repack.
Reviewed By: markbt
Differential Revision: D8394735
fbshipit-source-id: 21a73cd32b33ed462bab8050c3d42a5e7545d3a4
Summary:
Avoiding an additional rename + delete cycle for every file on Windows
can provide us with significant performance gains. While this change
introduces a race condition (and there's no way to hide it behind
a config flag today), I think it is still worth it.
Reviewed By: markbt
Differential Revision: D8455681
fbshipit-source-id: aa09083afc074e5198894ca602628efb6d16a2ca
Summary:
Now, it is impossible to turn `wvfs` auditor caching from repo configs, as
they are read after `wvfs` is instanciated and thus don't influence this
decision.
A proper fix would be to refactor the `localrepository.__init__` method to do
these things in the right order, but for now I would like to just fix what is
currently broken.
Reviewed By: markbt
Differential Revision: D8447069
fbshipit-source-id: d45c1424a5b680c234cd804664599c2074371668
Summary: Avoid code duplication. It is used by an upcoming patch.
Reviewed By: markbt, phillco
Differential Revision: D8394733
fbshipit-source-id: ccaa27fa223503db2e3a475b31d19e89a813da8f
Summary:
The new treestate was designed to store fsmonitor state. Use it to make
fsmonitor and dirstate state consistent, and avoid fsmonitor state
invalidation.
The "fsmonitor identity" check was removed as we now rely on the dirstate
identity check - dirstate and fsmonitor state must be updated consistently -
both updated or neither updated.
Since this is the first dirstate that tracks "untracked" files, several places
are adjusted (ex. dmap.dropfile, dmap.keys) to take the new untracked files
into consideration.
Reviewed By: wez
Differential Revision: D7909172
fbshipit-source-id: 05fd64b25c67ae4b07bc8cfee2731c748205975e
Summary:
The new entry point was added by D7840237.
With it, it is now pointless to pass `--config` arguments to the chg server.
So let's just remove the related logic, to avoid accidental profiling
related configs to the server (although the code path should ignore the
config flags).
Since we no longer use the old `hg serve` command code path, raise a
ProgrammingError explicitly to avoid surprises.
Reviewed By: singhsrb
Differential Revision: D8370965
fbshipit-source-id: 6a54cd54b41dc66c10f87e821ceb8e79adef09c7
Summary:
The previous logic restarted the *entire* rebase if one commit had merge conflicts or needed a mergedriver run. This was unnecessarily slow, plus the abort triggered a full repo scan.
Also, this logic is actually simpler and easier to reason about when done here, vs. `rebase()`. We can delete all the wrapping that function does after this rolls out.
This is config-gated, so we can roll it out gradually. Old config = old behavior.
Reviewed By: quark-zju
Differential Revision: D8272566
fbshipit-source-id: 9e598eea10c8a634497cd211a468dd2f58c5ff47
Summary:
The fancyrenderer is a little loud for progress bars with no totals and
spinners. Tone it down a little.
Reviewed By: phillco
Differential Revision: D8403456
fbshipit-source-id: ff8316f4098a02fbafb083e5ae7baa4a243c93bd
Summary:
After `fork()`, the chg worker no longer needs to detect incoming
connections using the selector, or `accept()` a connection. So let's just
close those fds explicitly.
Reviewed By: phillco
Differential Revision: D8384549
fbshipit-source-id: 9bd43692a575d425b61da8ad9c6b069113b2ae4b
Summary:
Avoid writing `clock` and other debug values if clock is empty.
This helps generate stable output in tests.
Reviewed By: markbt
Differential Revision: D8394734
fbshipit-source-id: 95c6fdc1448eb382c51303ac6ccf400c2c6e578a
Summary: This makes it easier to test whether a "dirstate" file is treestate or not.
Reviewed By: markbt
Differential Revision: D8394737
fbshipit-source-id: 07066da1fc0478cbf460e5b0d46800c7ba20b2ed
Summary:
This helps expose internal flags like NEED_CHECK etc. They would be useful
for precisely testing treestate.
Reviewed By: markbt
Differential Revision: D7935821
fbshipit-source-id: 21117cec148c113ec24f94d9fae849cb7a924284
Summary: This allows us to dogfood the new treestate.
Reviewed By: markbt
Differential Revision: D7912559
fbshipit-source-id: bfb30cd9989460b0eae859aeed402bd92776ab0a
Summary:
Previously, there are 2 configs: `treedirstate.useinnewrepos` and
`format.usetreestate`. They are both related to dirstate format and conflict
with each other. This patch unifies them into a single config
`format.dirstate`.
As we're here, merge `test-fb-hgext-treedirstate-x.t` to `test-dirstate-x.t`
if they were previously copied from `test-dirstate-x.t`
Reviewed By: markbt
Differential Revision: D8393878
fbshipit-source-id: 57abeea22ce732d93205e4d4308923afa90693f4
Summary:
Importing via Python is too slow. Therefore let's implement a native code
path for importing from other dirstate maps.
As we're here, modify treedirstate's import implementation so it skips
untracked files.
Reviewed By: markbt
Differential Revision: D7912557
fbshipit-source-id: ff1673ece2396a89fcc01e8b04041a6754cad885
Summary:
Previously, hg returns -1 (255) on "unknown errors". That could conflict with
other things. For example, http://tldp.org/LDP/abs/html/exitcodes.html suggests
1, 2, 126 ... 255 have special meanings defined by a common shell. Namely,
`ssh` also returns 255.
Another long complaints about hg exit code is the use of 1 to indicate
"no changes". Although that's explained at [1], a lot of scripts still find it
surprising.
This diff adds a `ui.exitcodemask` config option, which makes it to alter the
exit code. Set it to 254 will change 1 to 0 for some script use-cases. Set it
to 63 or so will avoid conflicts with software like `ssh`.
Many existing scripts still expect 255 to be the return code. So we cannot
change the default value that easily. To avoid surprises (like, put the config
option in `/etc/mercurial/hgrc.d/exitcode.rc`), the config set by config
files is ignored if `HGPLAIN` is set and `HGPLAINEXCEPT` does not contain
`exitcode`.
[1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2012-January/037711.html
Reviewed By: DurhamG
Differential Revision: D7921817
fbshipit-source-id: 764b0de030fc727aa5df7305c2b8bc92f576cd33
Summary:
When requesting trees from the server we try to provide a base tree
that we already have. In the hybrid manifest days, when we were fetching
manifest X, we would look up the linknode for X in the flat manifest revlog,
then scan the changelog up and down from that linknode. In the treeonly world we
don't have that revlog anymore though, so the current algorithm just searches from
the tip. This ended up being a really bad algorithm, since if you have tip and you
request tip~1000, then tip~2000, then tip~3000, you end up basing them all
against tip and redownloading a lot of data. This makes hg blame and hg log -p
super slow.
Let's change the algorithm to base off of a hint linknode, and let's change
changectx to set that hint every time it attempts to read a manifest. In manual
testing this significantly sped up hg log -p
Reviewed By: phillco
Differential Revision: D8399613
fbshipit-source-id: 771a94ee8b82be682ea0091b8d6c0fcd5f4e6646
Summary:
This allows us to remove the Python binding without breaking existing zstd
users (commitcloud bundles). It might also make the future Rust migration
easier.
Using `zstd` create for its streaming APIs. `zstdelta`'s APIs cannot be used
since it requires decompressed length to be known, which wouldn't work for
streaming compressed data.
Note: For easier implementation, the Python land no longer processes data
in a streaming way. This is probably fine for the current bundle use-case.
In the long term, we might want to revisit the bundle format entirely.
As we're here, also expose zstdelta's APIs and add a test for it.
Reviewed By: DurhamG
Differential Revision: D8342421
fbshipit-source-id: 89902d551f4616469d6e1bc9b334a1c37c884775
Summary: This should be self._repo.ui.
Reviewed By: quark-zju
Differential Revision: D8382375
fbshipit-source-id: b3269d8fdac3d2159f547b6220280109d8400dcb
Summary:
If the repo has treemanifests, we can use this to do the case check faster.
Note that with treemanifests we only find the first path component that
conflicts (subsequent conflicts won't be found), as these conflicts occur in
different trees.
Reviewed By: farnz
Differential Revision: D8332426
fbshipit-source-id: 243dab9bf711c6fdb62a0e0ba4346faea3b6b0ad
Summary:
This is clearly the right thing to do, given what the other `lookup` function in this file does.
Without `None` as a default `get` argument, OSX tests fail with `missing "default" argument1 error.
Reviewed By: ryanmce
Differential Revision: D8379304
fbshipit-source-id: ff66551221eb95cd5e59dfb5a5ab66b15211ad7f
Summary:
Remove the special-case code path where m.always() is true. The non-always
code path also behaves correctly with always matchers and should perform just
as efficiently. The only two additional things it does is to customize the
matcher's `bad()` callback (which should never be invoked for matchers that
match everything) and to loop through `m.files()`, which is guaranteed to be
empty for matchers where `m.always()` is true.
Reviewed By: quark-zju
Differential Revision: D8229267
fbshipit-source-id: adb9f7cfe269febd53ff417ec0d3817833ac5338
Summary:
The new treestate could track untracked files, and casecollisionauditor
should not iterate all files in the treestate. So let's implement a lazy
version of casecollisionauditor for treestate similar to what was done in
treedirstate.
Reviewed By: markbt
Differential Revision: D7912558
fbshipit-source-id: 7a95275a430aed109a49c4cdcd3a474c5961470b
Summary:
Add a config option `format.usetreestate`, which would turn on the new
treestate for new repos.
Treestate does not need manually maintaining "copymap", "otherparentset",
"nonnormalset". The one single tree is the source of truth of everything.
So some code like `copymap.pop` etc. are disabled on treestate path.
Reviewed By: markbt
Differential Revision: D7909175
fbshipit-source-id: 96a405fccd4099cefc152ec17204cb5407efbe7e
Summary:
This is subtle. If visitor changes file state, `Node.id` should be set to
`None` to mark it as "changed".
In practise, treedirstate uses visitor to rewrite mtime to -1 if mtime is
"fsnow". Those rewritten mtime all belong to "changed" nodes (because "fsnow"
can only increase, and on-disk entries cannot have "mtime == fsnow" because
they would be written to -1 during the previous write), so it's not a problem
yet.
It is safer to not depend on the fact that "visitor" can only change "changed"
nodes. On the other hand, detecting changes for all filestate fields could be
undesirably expensive. So let's make the visitor provide the "changed or not"
information. Surely the visitor knows what it does.
Reviewed By: markbt
Differential Revision: D7909167
fbshipit-source-id: 21e71302cf1db86c1330b294baddd51cc8a96026
Summary:
The reason caching is disabled seems to be that symlinks can change where
paths points to, while some properties of that path are cached.
Sometimes we know that symlinks are not used in the repo, so we'd like hg
to let us shoot ourselves in the foot.
The motivation here is that I have a specially crafted rebase operation,
which makes on the order of 400000 stat-like syscalls without caching and
on the order of 6000 with caching. This can save ~20s on Windows.
Note that this setting is disabled by default so we can't break anything.
FWIW, we already have an unsafe config option, which allows us to opt into the "we don't do crazy stuff with symlinks" behavior, it's called `unsafe.filtersuspectsymlink`, setting it to false speeds up Windows `status` and we do so for `ovrsource` on Windows.
Reviewed By: quark-zju
Differential Revision: D8297129
fbshipit-source-id: ec751d18b50b042f28f68c9e1a3634ab94a4e7c3
Summary:
An updated version of D8174246 with a test. Max memory usage is very useful to us to understand which commands are using too much memory, and the existing wrapper metrics don't work when chg is used (they measure chg's usage).
Windows support will come later as it is a bit more involved.
Reviewed By: DurhamG
Differential Revision: D8318584
fbshipit-source-id: 323450bc7ab376014d70106beb5d4fdcc7fba0c8
Summary:
Needed for windows to allow us write to a tempfile created in Rust. Should do it in a different way?
Context: we create a NameTempFile from Rust that we pass as argument to hg as profiling.output=named_temp_file. When open with 'w', the mercurial command fails with "abort: Permission denied". As far as I read, it might be because of how the Win32 API works. Under the hood, Python's open function is calling the CreateFile function, and if that fails, it translates the Windows error code into a Python IOError. From CreateFile documentation:
If CREATE_ALWAYS and FILE_ATTRIBUTE_NORMAL are specified, CreateFile fails and sets the last error to ERROR_ACCESS_DENIED if the file exists and has the FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM attribute. To avoid the error, specify the same attributes as the existing file.
I might be wrong though... any other suggestions?
Reviewed By: ikostia
Differential Revision: D8099420
fbshipit-source-id: c3077cc5d7bc03c8f2eeafdac3467db62c20a669
Summary:
Template function requires input arguments. They do not always exist.
If any argument is missing, just return an empty value, instead of crash.
Reviewed By: mitrandir77
Differential Revision: D8221246
fbshipit-source-id: ff3839398bdc620bff61d1b19b1512010c06388e
Summary: Will be used by the next change.
Reviewed By: mitrandir77
Differential Revision: D8221077
fbshipit-source-id: c9713175dfaab0b090f81d2c9d4f4d9d63a0771c
Summary:
There mercurial template language is a powerful language. But it has some
limitations right now:
- It's focused on "commit". If there is no commit, things become harder.
- The related formatter concept is coupled with "a list of things". If there
is no list, things become impossible.
- Some commands use formatter/template partially, and that does not play
very well with `ui.write`, and is difficult to document.
This patch starts to solve all the issues, by introducing command-level
template state, and with the vision:
- Anything could be used as an "environment" passed to the template
language, a commit, a file, a line, a "command" with customized
properties, etc.
- If a template keyword function is incompatible with the environment, it
should fail gracefully.
- The template language is a super set of other things. `-Tjson` should not
be magical, but an alias of `{some complex expression | json}`.
- Discourage the use of formatter.
- Template keywords can be registered and documented.
This would remove the limitations of the template language and unleash its
full potential.
Note there are still some cases where the formatter is desirable. For example,
`hg log` with a large revset should be able to stream its output. The command
template feature added right now only renders things at the end of a command,
and is not (and requires significant effort to be) streaming-friendly.
Reviewed By: mitrandir77
Differential Revision: D8221078
fbshipit-source-id: 26fed50025c4676af277686382ce2d36f1faed97
Summary:
The added method takes an environment (`props`), and a program written in
the template language, execute the program, and pass output to `ui.write`.
The API is similar to the upstream [1]. But it avoids `ctx` intentionally.
Since the "environment" should be more flexible than having to have a `ctx`.
[1]: aa32940279
Reviewed By: mitrandir77
Differential Revision: D8221079
fbshipit-source-id: 94ee5c563f943330ef91966a4883c5733cc8dac9
Summary: Implement most methods, similar to treedirstatemap.
Reviewed By: markbt
Differential Revision: D7909176
fbshipit-source-id: d20e4642ee381cc1cfe7c774071c8c3c1e493f45
Summary:
Add a class with methods defined in `dirstate.dirstatemap` so it could be
used as a drop-in replacement. The actual logic is not implemented yet.
Reviewed By: markbt
Differential Revision: D7909170
fbshipit-source-id: 5c1ae1bf4341d6368f5c9824487e7702f93043a7
Summary:
They're actively fighting against the clang-format config
and don't have an auto-fix.
Reviewed By: quark-zju
Differential Revision: D8283622
fbshipit-source-id: 2de45f50e6370a5ed14915c6ff23dc843ff14e8a
Summary:
Re-using long-running connections can cause problems, as the `hg serve` process
on the server may be using large amounts of resources. Add a new configuration
option `connectionpool.lifetime`. Connections that are older than that time
will be closed and not re-used.
Reviewed By: DurhamG
Differential Revision: D8204415
fbshipit-source-id: ac937778e3dc79e77e367017db2141066bd17263
Summary:
A new renderer that doesn't print anything. This ensures the progress engine
continues to run (other ways of disabling progress completely disable progress
calculations), and can be used in conjunction with the progressfile extension
to give progress output there without anything on stdout.
Reviewed By: phillco, farnz
Differential Revision: D8258548
fbshipit-source-id: 78cb78d84c6d1dd22744d84d34bf0f43f7962ace
Summary:
The default value of `rev` for `debugcheckcasecollisions` should be `""`, not
`[]` (the latter causes an exception).
Differential Revision: D8236465
fbshipit-source-id: 841e0218664c46e70abd5b774463d4b4e40fdfb8
Summary:
When running `hg hint --ack ...`, pick the file that exists to write,
instead of using the first default one.
Reviewed By: singhsrb
Differential Revision: D8230173
fbshipit-source-id: 90375653ffa16cfbf77eed4f967b5250b649a532
Summary:
When called with any matcher parameters, `hg revert` would first walk both the
working directory and the commit in question to find all files that match the
pattern. Only after it found all matching files did it check to see which
ones actually need updating.
This made revert performance `O(matched files)` rather than
`O(files needing revert)`. This makes operations like
`hg revert large_directory` quite expensive, especially when only a handful of
files need to be updated.
This should help address the performance of `hg revert .`, which previously
was much slower than `hg revert --all`. Previously it enumerated every file
in the repository twice (once in the working directory and once in the current
commit) before checking which ones needed reverting. This should also help
fix the performance of `hg revert <directory>` in Eden as well.
Reviewed By: quark-zju
Differential Revision: D8217972
fbshipit-source-id: c18e72c8a5ed0b4c3a5ed69e79f6a88d3ea04a9f
