Summary:
Avoiding an additional rename + delete cycle for every file on Windows
can provide us with significant performance gains. While this change
introduces a race condition (and there's no way to hide it behind
a config flag today), I think it is still worth it.
Reviewed By: markbt
Differential Revision: D8455681
fbshipit-source-id: aa09083afc074e5198894ca602628efb6d16a2ca
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
Summary:
The root issue is described in the new ``workingflags()`` function -- ``workingfilectx.flags()`` ought to read from the working copy, but instead reads from the working copy's p1.
Rather than force a big rewrite to adapt all the old code (which might be a waste if we end up rewriting the context classes), let's add a new function with the behavior we want, and callers can detect for it and call it.
simplemerge is the only callsite that needs this behavior, AFAICT.
Previous attempt: D6694292
Reviewed By: singhsrb
Differential Revision: D7083239
fbshipit-source-id: 6f30b877958d40106203e088fe724ab491b541b9
This disables background file closing when in not in main thread
Test Plan:
Ran pull, update, sparse commands and watched the closer threads created and destroyed in procexp.exe
ran test on CentOS. No tests broken compared to the base
Differential Revision: https://phab.mercurial-scm.org/D1457
The initial attempt was to discard cache when appropriate, but it appears
to be error prone. We had to carefully inspect all places where audit() is
called e.g. without actually updating filesystem, before removing files and
directories, etc.
So, this patch disables the cache of audited paths by default, and enables
it only for the following cases:
- short-lived auditor objects
- repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories
and considered sort of append-only (a file/directory would never be
replaced with a symlink)
There would be more cacheable vfs objects (e.g. mq.queue.opener), but I
decided not to inspect all of them in this patch. We can make them cached
later.
Benchmark result:
- using old clone of http://selenic.com/repo/linux-2.6/ (38319 files)
- on tmpfs
- run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null
- try 4 times and take the last three results
original:
real 7.480 secs (user 1.140+22.760 sys 0.150+1.690)
real 8.010 secs (user 1.070+22.280 sys 0.170+2.120)
real 7.470 secs (user 1.120+22.390 sys 0.120+1.910)
clearcache (the other series):
real 7.680 secs (user 1.120+23.420 sys 0.140+1.970)
real 7.670 secs (user 1.110+23.620 sys 0.130+1.810)
real 7.740 secs (user 1.090+23.510 sys 0.160+1.940)
enable cache only for vfs and svfs (this series):
real 8.730 secs (user 1.500+25.190 sys 0.260+2.260)
real 8.750 secs (user 1.490+25.170 sys 0.250+2.340)
real 9.010 secs (user 1.680+25.340 sys 0.280+2.540)
remove cache function at all (for reference):
real 9.620 secs (user 1.440+27.120 sys 0.250+2.980)
real 9.420 secs (user 1.400+26.940 sys 0.320+3.130)
real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)
We want to be able to do more precise check when auditing a path depending of
the intend of the file access (eg read versus write). So we now pass the 'mode'
value to 'audit' and update the audit function to accept them.
This will be put to use in the next changeset.
Now, files (to be truncated) are opened with checkambig=True, only if
localrepository caches it.
Therefore, straightforward "copy if EPERM" is always reasonable to
avoid file stat ambiguity at closing.
This patch makes checkambigatclosing close wrapper copy the target
file, and advance mtime on it after renaming, if EPERM. This can avoid
file stat ambiguity, even if the target file is owned by another (see
issue5418 and issue5584 for detail).
This patch factors main logic out instead of changing
checkambigatclosing._checkambig() directly, in order to reuse it.
There are no remaining users of 'mustaudit' so we can safely drop the API.
External user are unlikely from a quick research so no deprecation is added.
The whole 'mustaudit' API is quite complex compared to its actual usage by its
unique user in stream clone.
Instead we add a "auditpath" parameter to 'vfs.__call_'. The stream clone code
then explicitly open files with path auditing disabled.
The 'mustaudit' API will be cleaned up in the next changeset.
In order to fix issue5418, 0d920bcb0fd1 made vfs.rename(checkambig=True)
omit advancing mtime of renamed file, if renamed file is owned by
another (EPERM is raised in this case).
But this omission causes rewinding mtime at restoration in such
situation, and makes avoiding file stat ambiguity difficult, because
ExactCacheValidationPlan assumes that mtime should be advanced, if a
file is changed in same ctime.
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
Ambiguity of file stat also requires issue5584 to be fixed with other
than file stat, but "hash of file", "generation ID" and so on were
already rejected ideas (please see original RFC linked from "Outline
of issue" in ExactCacheValidationPlan page).
This omission occurs:
- only for non append-only files (dirstate, bookmarks, and phaseroots), and
- only if previous transaction is rollbacked by another user
The latter means "sharing a repository clone via group permission".
This is reasonable usecase, but not ordinary for many users, IMHO.
"hg rollback" itself has been deprecated since Mercurial 2.7, too.
Therefore, increasing the cost at rollbacking previous transaction
executed by another a little seems reasonable, for avoidance of file
stat ambiguity.
This patch does:
- create copy of (already renamed) source file, if advancing mtime
fails for EPERM
- rename from copied file to destination file, and
- advance mtime of renamed file, which is now owned by current user
This patch also factors "self.join(src)" out to reduce redundancy.
See the previous commit for why. Marked as API change since osutil.listdir()
seems widely used in third-party extensions.
The win32mbcs extension is updated to wrap both util. and windows. aliases.
Thoughout hg code, we see a pattern of attempting to remove a file and then
catching and ignoring any errors due to a missing file in the calling code.
Let's unify this pattern in a single implementation in the vfs layer.
The 'scmutil' is growing large (1500+ lines) and 2/5 of it is related to vfs.
We extract the 'vfs' related code in its own module get both module back to a
better scale and clearer contents.
We keep all the references available in 'scmutil' for now as many reference
needs to be updated.