Summary:
Manifestlog needs to use `00manifesttree.i` for filecache if treeonly is turned
on.
Teach `filecache` to also cache non-existed files and add `00manifesttree.i`
to `repo.manifestlog` check list.
Assuming inode cannot be 0, `cacheable` is a constant `True`. Drop code
dealing with `cacheable = False` accordingly.
Reviewed By: DurhamG
Differential Revision: D10417795
fbshipit-source-id: c8aedd36dc39592c86847bf4327ed9c46736bab0
Summary:
Improve the lock handling when fork / exec happens:
- Unlock explicitly so forked process won't cause surprises.
- Set O_CLOEXEC to also reduce surprises.
Reviewed By: wez
Differential Revision: D9646559
fbshipit-source-id: 7b192635594761eee6ddc8c86ab2107c0d95d221
Summary:
D9328199 made them skip eden mounts. This diff expands the whitelist to other
common filesystems.
Reviewed By: wez
Differential Revision: D9416599
fbshipit-source-id: 5f1dddf5a181833a2fa6e0954c2579bb5adcd170
Summary:
This is done by Matt Harbison at Yahoo. See
2062f7c2ac
for the original commit.
As we're here, also make the POSIX version report "eden" directly.
Reviewed By: wez
Differential Revision: D9416596
fbshipit-source-id: 812d02b35f149f9019c2307e6246a29c27c48cfe
Summary:
To correctly create the hg command to shell out to, we need to know whether
we run from a binary or from a shebang script. If we run from a script,
the hg command should be `argv[0]` (as seen by Python), whereas when this is
done from a binary, we need to report the `sys.executable` (which Python pops
from `argv`).
Once I learn how to pass data down from Rust to Python, we can replace this
with something like a `rustdata` module and thus
```
try:
import rustdata
# we are called from Rust
except ImportError:
# we are called from Python
```
will let us distinguish the two cases (given appropriate `hgdemandimport`
fixups).
Reviewed By: StanislavGlebik
Differential Revision: D9338851
fbshipit-source-id: d135bfd9a5d695e5d15fd79c5438758951656f0d
Summary:
We recently ran into issues with locks in pid namespaces [1]. Let's fix that
by using flock.
flock is more reliable in Linux's pid namespace use-case than file-existence
test, because it works without a /proc filesystem and does not have deadlock
issue if an hg process is killed unexpectedly (ex. OOM or SIGKILL).
The transition should be transparent:
- If the new code saw a symlink lock file generated by the old code.
`open(..., O_NOFOLLOW)` will fail and it's considered lock taken by the old
process correctly.
- If the old code saw a new lock file. It will treat it as system without
symlink support and it's considered lock taken by the new process correctly.
A non-symlink stale lock (regardless of whether it contains pid information or not)
will be confidently removed automatically by the new code.
The change is complicated because it works when both new and old hg
run at the same time. If we have migrated most users to the new code path,
the code can be cleaned up significantly.
[1]: https://fburl.com/85fxjisi
Reviewed By: DurhamG
Differential Revision: D9004614
fbshipit-source-id: d501c4f3a7bc8ad73c9556be1c6a265ffd0d0686
Summary:
An updated version of D8174246 with a test. Max memory usage is very useful to us to understand which commands are using too much memory, and the existing wrapper metrics don't work when chg is used (they measure chg's usage).
Windows support will come later as it is a bit more involved.
Reviewed By: DurhamG
Differential Revision: D8318584
fbshipit-source-id: 323450bc7ab376014d70106beb5d4fdcc7fba0c8
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
Summary:
This helps to avoid the following problem:
1. hg creates a temporary lock file, writes some stuff there
2. os writes this stuff into its buffer
3. hg closes the file, the metadata is written out (or journaled)
4. hg renames the file, which is again a metadata-only operation
5. the buffer is still not flushed
6. the OS crashes
7. upon reload, the os has a file with a correct name and a correct length,
but unexpected contents
Reviewed By: quark-zju
Differential Revision: D7889111
fbshipit-source-id: a0a152c9e7efef34847fa2d2ab9b94191bde43f4
Summary:
`GetFileInformationByHandle` returns a `BY_HANDLE_FILE_INFORMATION` structure,
which is similar to what a `stat` call returns. In particular, this structure
contains:
- the `VolumeSerialNumber` field
- the `CreationTime` fields
- the `LastWriteTime` fields
- the `FileSize` field
- the `FileIndex` fields
All of these are self-explanatory, except for the `FileIndex`. Here's what MSDN says:
```
The identifier that is stored in the nFileIndexHigh and nFileIndexLow members is called the file ID.
...
In the NTFS file system, a file keeps the same file ID until it is deleted.
You can replace one file with another file without changing the file ID
by using the ReplaceFile function. However, the file ID of the replacement file,
not the replaced file, is retained as the file ID of the resulting file.
```
Basically, every change to a file, except replacing it with some other file,
results in a changed file Id. Calling `ReplaceFile` however results in
`CreationTime` preserved from the replaced file and `LastWriteTime` preserved
from the replacement file:
```
C:\Code\tries\windowstries
λ python fileinfo.py
1.txt: Attr;32;Create;4064609256;30663014;Write;3046340864;30663166;Volume;1792064959;Size;0;5;Idx;655360;547898
2.txt: Attr;32;Create;3030045984;30663166;Write;3030172944;30663166;Volume;1792064959;Size;0;5;Idx;786432;565725
Replacing 1.txt with 2.txt; result is: 1
1.txt: Attr;32;Create;4064609256;30663014;Write;3030172944;30663166;Volume;1792064959;Size;0;5;Idx;786432;565725
```
Thus comparing all of these fields seems to be enough to replicate the `cachestat` beharior from `posix.py` (We
cache the `stat` of a file, which we almost always expect to change by renaming into it. We only use this `cachestat`
while our process is alive. One notable exception is the `.hgignore` file, which the user can change as they please,
but which we still `cachestat`.)
This change has performance implications for `status` if we use `.hgignore`: it's nearly 0.1s faster.
If we use `.gitignore`, there are no performance implications (at least I did not find any), but I'd still like
to land it for the sake of feature parity between Posix and Windows.
Reviewed By: quark-zju
Differential Revision: D7843746
fbshipit-source-id: f6f69ee12bdce054d7ea77917e83a95bcec17f83
Summary:
Quick experimentation shows that existing lock file logic is not enoug for
frequently run and killed Mercurial processes (Mercurial run by tools, such as
Nuclide is an example of such scenario)
I wrote the following two files:
```
c:\Code\tries\pythontries λ cat lockcreator.py
import os, random
def makelock(info, pathname):
ld = os.open(pathname, os.O_CREAT | os.O_WRONLY | os.O_EXCL)
os.write(ld, info)
# os.fsync(ld)
os.close(ld)
name = os.path.join('locks', 'lock.pid' + str(os.getpid()) + ".rand" + str(random.randint(0, 10000)))
makelock('contents', name)
```
and
```
c:\Code\tries\pythontries λ cat lockracer.py
import os, subprocess, time, random
for i in xrange(10000):
proc = subprocess.Popen('python lockcreator.py')
time.sleep(0.001*random.randint(0, 500))
proc.terminate()
```
After runnning `python lockracer.py`, I did `ls -l locks | grep "0 Apr"`, this way it showed all the 0-byte files created in April. This shows a non-empty output. Uncommenting the `os.fsync` line does not help much.
Rewriting `lockcreator.py` to use temp lock file approach helps greatly.
Reviewed By: quark-zju
Differential Revision: D7653186
fbshipit-source-id: 48e9eeeca34075ea2ec78f3319491bcebc0e88c7
It's been there since 84af5a079c7d (2007-02-19), but seems wrong since any
I/O operations to a closed file would raise ValueError, not IOError. We should
keep the file object open even if the underlying file descriptor is half dead.
See the previous commit for why. Marked as API change since osutil.listdir()
seems widely used in third-party extensions.
The win32mbcs extension is updated to wrap both util. and windows. aliases.
I stumbled into this in the next patch. The difference between getting a
context manager capable object or not from vfs classes was as subtle as adding a
'+' to the file mode.
Previously, there were two slightly different versions of unlinkpath between
windows and posix, but these differences were eliminated in previous patches.
Now we can unify these two code paths inside of the util module.
os.pathsep returns unicode on Python 3. We already have pycompat.ospathsep
which return bytes on Python 3. This patch replaces all the occurrences of
os.pathsep in the codebase (excluding tests) to pycompat.ospathsep.
Now we don't use sys.__stdout__ except for getting its fileno(), so we no
longer have to wrap it by winstdout.
This helps adding pycompat.stdin/out/err.
I'm going to get rid of sys.stderr|out|in references from posix.termwidth().
In order to do that, termwidth() needs to take a ui, but functions in util.py
shouldn't depend on a ui object. So moves termwidth() to scmutil.py.
_winreg module is renamed to winreg in python 3. Added the conditionalize
statements in the respective file because adding this in pycompat will result
in pycompat throwing error as this is a windows registry module and we have
buildbots and most of the contributors on linux.
fopen() and fdopen() have a unique-to-Windows requirement that
transitions between read and write operations in files opened
in modes r+, w+, and a+ perform a file positioning call
(fsetpos, fseek, or rewind) in between. While the MSDN docs don't
say what will happen if this is not done, observations reveal
that Python raises an IOError with errno 0. Furthermore, I
/think/ this behavior isn't deterministic. But I can reproduce
it reliably with subsequent patches applied that open revlogs
in a+ mode and perform both reads and writes.
This patch introduces a proxy class for file handles opened
in r+, w+, and a+ mode on Windows. The class intercepts calls
and audits whether a file positioning function has been called
between read and write operations. If not, a dummy, no-op seek
to the current file position is performed. This appears to be
sufficient to "trick" Windows into allowing transitions between
read and writes without raising errors.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Python 2.6 introduced a new octal syntax: "0oXXX", replacing "0XXX". The
old syntax is not recognized in Python 3 and will result in a parse
error.
Mass rewrite all instances of the old octal syntax to the new syntax.
This patch was generated by `2to3 -f numliterals -w -n .` and the diff
was selectively recorded to exclude changes to "<N>l" syntax conversion,
which will be handled separately.
We'll use it to detect when a sshpeer have server output to be displayed.
The implementation is super basic because all case support is not the focus of
this series.
This is actual test coverage for issue4629. The test changes in 6723e40c7c37
were simply the addition of quotes to the output, not ensuring that strings with
backslashes are quoted.
The '~' in the bug report is being expanded to a path with Windows style slashes
before being passed to shellquote() via util.shellquote(). But shlex.split()
strips '\' out of the string, leaving an invalid path in dispatch.aliasargs().
This regressed in 72640182118e.
For now, the tests need to be conditionalized for Windows (because those paths
are quoted). In the future, a more complex regex could probably skip the quotes
if all component separators are double '\'. I opted to glob away the quotes in
test-rename-merge2.t and test-up-local-change.t (which only exist on Windows),
because they are in very large blocks of output and there are way too many diffs
to conditionalize with #if directives. Maybe the entire path should be globbed
away like the following paths in each changed line. Or, letting #if directives
sit in the middle of the output as was mentioned a few months back would work
too.
Unfortunately, I couldn't figure out how to test the specific bug. All of the
'hg serve' tests have a #require serve declaration, causing them to be skipped
on Windows. Adding an alias for 'expandtest = outgoing ~/bogusrepo' prints the
repo as '$TESTTMP/bogusrepo', so the test runner must be changing the
environment somehow.
According to 6b1369445b7b introducing "windows._removedirs()":
If a hg repository including working directory is a reparse point
(directory symlinked or a junction point), then using
os.removedirs will remove the reparse point erroneously.
"windows._removedirs()" should be used instead of "os.removedirs()" on
Windows.
This patch adds "removedirs" as platform depending function to replace
"os.removedirs()" invocations for portability and safety
It appears that the read() in readpipe() never actually ran before (in
test-ssh.t anyway). A print of the size returned from os.fstat() is 0 for every
single print output in test-ssh.t, so the data in the pipe ends up being read
later instead of when it is available. This is the same problem as Linux, as
mentioned in e20a5309b88d.
There are several places in the Windows SSH tests where the order of local
output vs remote output differ from the other platforms. This only fixes one of
those cases (and interstingly, not the one added in order to test e20a5309b88d),
so there is more investigation needed. However, without this patch, test-ssh.t
also has this diff:
--- c:/Users/Matt/Projects/hg/tests/test-ssh.t
+++ c:/Users/Matt/Projects/hg/tests/test-ssh.t.err
@@ -397,11 +397,11 @@
$ hg push --ssh "sh ../ssh.sh"
pushing to ssh://user@dummy/*/remote (glob)
searching for changes
- remote: Permission denied
- remote: abort: prechangegroup.hg-ssh hook failed
- remote: Permission denied
- remote: pushkey-abort: prepushkey.hg-ssh hook failed
updating 6c0482d977a3 to public failed!
+ remote: Permission denied
+ remote: abort: prechangegroup.hg-ssh hook failed
+ remote: Permission denied
+ remote: pushkey-abort: prepushkey.hg-ssh hook failed
[1]
$ cd ..
Output with this change was stable over 600+ runs of test-ssh.t. I initially
tried a background thread to read the pipe[1], but this was simpler and the test
results were exactly the same. I also tried SetNamedPipeHandleState(), but the
PIPE_NOWAIT is for compatibility with LANMAN 2.0, not for async I/O (the results
were identical though).
[1] http://eyalarubas.com/python-subproc-nonblock.html
The doc string of osutil.posixfile includes (line 611):
"On error, this function may raise either a WindowsError or an IOError."
which is most likely correct, but does not fit for this function here anymore,
as we do fold WindowsError to IOError here specifically.
And this function is now a bit more than just an exception-wrapper, as it has
been expanded to additionally sanitize the unloved seek/tell behavior
of Windows.
(Self-disclosure: This patch is entirely untested at the time of its
publication, as I'm currently not using this version myself. I send it
in hopes that it will reduce potential future confusion. CC-ing Matt Harbison)
