# skip-blame because this was mechanically rewritten the following script. I
ran it on both *.t and *.py, but none of the *.py changes were proper. All *.t
ones appear to be, and they run without addition failures on both Windows and
Linux.
import argparse
import os
import re
ap = argparse.ArgumentParser()
ap.add_argument('path', nargs='+')
opts = ap.parse_args()
globre = re.compile(r'^(.*) \(glob\)(.*)$')
for p in opts.path:
tmp = p + '.tmp'
with open(p, 'rb') as src, open(tmp, 'wb') as dst:
for line in src:
m = globre.match(line)
if not m or '$LOCALIP' in line or '*' in line:
dst.write(line)
continue
if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
dst.write(line)
continue
dst.write(m.group(1) + m.group(2) + '\n')
os.unlink(p)
os.rename(tmp, p)
fsmonitor can significantly speed up operations on large working
directories. But fsmonitor isn't enabled by default, so naive users
may not realize there is a potential to make Mercurial faster.
This commit introduces a warning to working directory updates when
fsmonitor could be used.
The following conditions must be met:
* Working directory is previously empty
* New working directory adds >= N files (currently 50,000)
* Running on Linux or MacOS
* fsmonitor not enabled
* Warning not disabled via config override
Because of the empty working directory restriction, most users will
only see this warning during `hg clone` (assuming very few users
actually do an `hg up null`).
The addition of a warning may be considered a BC change. However, clone
has printed warnings before. Until recently, Mercurial printed a warning
with the server's certificate fingerprint when it wasn't explicitly
trusted for example. The warning goes to stderr. So it shouldn't
interfere with scripts parsing meaningful output.
The OS restriction was on the advice of Facebook engineers, who only
feel confident with watchman's stability on the supported platforms.
.. feature::
Print warning when fsmonitor isn't being used on a large repository
Differential Revision: https://phab.mercurial-scm.org/D894
Extract 'experimental.evolution' = createmarkers as
'experimental.evolution.createmarkers'.
We keep the new option in the 'experimental.evolution' namespace in order to
stay coherent with other options ('experimental.evolution.bundle-obsmarker'
and 'experimental.evolution.track-operation') ease the renaming as possibly
'evolution.createmarkers'.
Differential Revision: https://phab.mercurial-scm.org/D1149
Upon pull or unbundle, we display a message with the range of new revisions
fetched. This revision range could readily be used after a pull to look out
what's new with 'hg log'. The algorithm takes care of filtering "obsolete"
revisions that might be present in transaction's "changes" but should not be
displayed to the end user.
evolution* config has been rewritten in stabilization* in the previous patch,
update tests file to use the new names.
Differential Revision: https://phab.mercurial-scm.org/D249
This vulnerability was fixed by the previous patch and there were more ways
to exploit than using '|shellcmd'. So it doesn't make sense to reject only
pipe character.
Test cases are updated to actually try to exploit the bug. As the SSH bridge
of git/svn subrepos are not managed by our code, the tests for non-hg subrepos
are just removed.
This may be folded into the original patches.
When unquoted, MSYS sees the colon between the drive letter and path as a Unix
path separator and unhelpfully splits on it, feeding only the drive letter as
the command. Much chaos ensues.
I vaguely remember trying to get the test runner to use /letter/path/to/exe
syntax the last time this happened, without success. I doubt a check-code rule
would work, since sometimes it is quoted, and sometimes the quotes are escaped.
This patch replaces hardcoded 127.0.0.1 with $LOCALIP in all tests.
Till now, the IPv6 series should make tests pass on common IPv6 systems
where the local device has the address "::1" and the hostname "localhost"
resolves to "::1".
This allows us to write doctests depending on a ui object, but not on global
configs.
ui.load() is a class method so we can do wsgiui.load(). All ui() calls but
for doctests are replaced with ui.load(). Some of them could be changed to
not load configs later.
The NamedTemporaryFile file is cleared up so checklink ends up as a dangling
symlink, causing cp -r in tests to complain on both Solaris and OS X. Use
a permanent file instead when there is a .hg/cache directory.
util.checklink would create a symlink and remove it again. That would sometimes
happen multiple times. Write operations are relatively expensive and give disk
tear and noise for applications monitoring file system activity.
Instead of creating a symlink and deleting it again, just create it once and
leave it in .hg/cache/check-link . If the file exists, just verify that
os.islink reports true. We will assume that this check is as good as symlink
creation not failing.
Note: The symlink left in .hg/cache has to resolve to a file - otherwise 'make
dist' will fail ...
test-symlink-os-yes-fs-no.py does some monkey patching to simulate a platform
without symlink support. The slightly different testing method requires
additional monkeying.
Before, Mercurial would create a new temporary file every time, stat it, change
its exec mode, stat it again, and delete it. Most of this dance was done to
handle the rare and not-so-essential case of VFAT mounts on unix. The cost of
that was paid by the much more common and important case of using normal file
systems.
Instead, try to create and preserve .hg/cache/checkisexec and
.hg/cache/checknoexec with and without exec flag set. If the files exist and
have correct exec flags set, we can conclude that that file system supports the
exec flag. Best case, the whole exec check can thus be done with two stat
calls. Worst case, we delete the wrong files and check as usual. That will be
because temporary loss of exec bit or on file systems without support for the
exec bit. In that case we check as we did before, with the additional overhead
of one extra stat call.
It is possible that this different test algorithm in some cases on odd file
systems will give different behaviour. Again, I think it will be rare and
special cases and I think it is worth the risk.
test-clone.t happens to show the situation where checkisexec is left behind
from the old style check, while checknoexec only will be created next time a
exec check will be performed.
We can't predict where those will show up and they're not
super-important for the contents of this particular test, so just drop
them. Further reduces the flakiness of the test to zero.
Healthy output (one log file mentioning "existing pooled" and one
mentioning "new pooled") will now print in a stable order, but
unhealthy output will print some sort of error.
This reduces the flakiness of the test from 55% to 38%. My next patch
makes it completely stable.
Before, if performing a clone+share from a repo that was itself
using shared storage, the share code would copy paths.default from
the underlying repo being shared, not from the source given by
the user.
This patch teaches hg.clonewithshare to resolve paths.default
and pass it to share so it can be written to the hgrc accordingly.
The changed test lines were added in 6a934a7b8d54, which
landed on stable. When this changeset merged to the default
branch, the test failed because 31ed0254094d (on default
but not stable) changed the order of working directory update
when performing a share-based clone from pooled storage.
The changes in this patch are similar to test changes in
31ed0254094d.
There are race conditions between clients performing a shared clone
to pooled storage:
1) Clients race to create the new shared repo in the pool directory
2) 1 client is seeding the repo in the pool directory and another goes
to share it before it is fully cloned
We prevent these race conditions by obtaining a lock in the pool
directory that is derived from the name of the repo we will be
accessing.
To test this, a simple generic "lockdelay" extension has been added.
The extension inserts an optional, configurable delay before or after
lock acquisition. In the test, we delay 2 seconds after lock acquisition
in the first process and 1 second before lock acquisition in the 2nd
process. This means the first process has 1s to obtain the lock. There
is a race condition here. If we encounter it in the wild, we could
change the dummy extension to wait on the lock file to appear instead
of relying on timing. But that's more complicated. Let's see what
happens first.
When pooled storage is enabled, `hg clone` will initialize a repo
from a local repo using the store sharing mechanism then pull from
the originally requested repo.
Before this patch, the working directory update occurred between
these steps. This meant that we would only update to revisions that
were already present in the local pooled storage.
This patch moves the update to after we pull from the originally
requested repository so we may check out a revision that didn't yet
exist locally. In other words, it makes the behavior like normal
`hg clone`.
Before this patch, repo could be set to None for wrong -R. It's okay for
commands that can reject repo=None, but the command server have a problem
because:
- it accepts repo=None for "unbound" mode
- and it reenters dispatch() where repo object is created for cwd by default
Test outputs are changed because the error is detected earlier. I think new
message is better than ".hg not found".
Before this patch, when auto sharing is enabled, 'hg.clone()' tries to
create local clone regardless of locality of the clone destination on
the host, and causes failure.
To avoid auto sharing when the clone destination is remote, this patch
adds examination of 'islocal(dest)' before auto sharing in
'hg.clone()'.
'islocal(dest)' is examined after 'sharepool', because:
- the former is more expensive than the latter
- without enabling share extension, the later is always negative
Many 3rd party consumers of Mercurial have created wrappers to
essentially perform clone+share as a single operation. This is
especially popular in automated processes like continuous integration
systems. The Jenkins CI software and Mozilla's Firefox release
automation infrastructure have both implemented custom code that
effectively perform clone+share. The common use case here is that
clients want to obtain N>1 checkouts while minimizing disk space and
network requirements. Furthermore, they often don't care that a clone
is an exact mirror of a remote: they are simply looking to obtain
checkouts of specific revisions.
When multiple third parties implement a similar feature, it's a good
sign that the feature is worth adding to the core product. This patch
adds support for an easy-to-use clone+share feature.
The internal "clone" function now accepts options to control auto
sharing during clone. When the auto share mode is active, a store will
be created/updated under the base directory specified and a new
repository pointing to the shared store will be created at the path
specified by the user.
The share extension has grown the ability to pass these options into
the clone command/function.
No command line options for this feature are added because we don't
feel the feature will be popular enough to warrant their existence.
There are two modes for auto share mode. In the default mode, the shared
repo is derived from the first changeset (rev 0) in the remote
repository. This enables related repositories existing at different URLs
to automatically use the same storage. In environments that operate
several repositories (separate repo for branch/head/bookmark or separate
repo per user), this has the potential to drastically reduce storage
and network requirements. In the other mode, the name is derived from the
remote's path/URL.
This warning exists to prevent git users from prematurely polluting
their namespace when trying out Mercurial. But for repos that already
have multiple branches, understanding what branches are is not
optional so we should just shut up.
When the progress extension is not enabled, each call to 'ui.progress' used to
issue a debug message. This results is a very verbose output and often redundant
in tests. Dropping it makes tests less volatile to factor they do not meant to
test.
We had to alter the sed trick in 'test-rename-merge2.t'. Sed is used to drop all
output from a certain point and hidding the progress output remove its anchor.
So we anchor on something else.
The revbranchcache is read on demand before it will be used for updating the
branch map. It is written back when the branchmap is written and it will thus
use the same locking as branchmap. The revbranchcache instance is short-lived;
it is only stored in the branchmap from .update() is invoked and until .write()
is invoked. Branchmap already assume that the repo is locked in that case.
The use of revbranchcache for branch map updates will make sure that the
revbranchcache "always" is kept up-to-date.
The perfbranchmap benchmark is somewhat bogus, especially when we can see that
the caching makes a significant difference between the realistic case of a
first run and the rare case of rerunning it with a full cache. Here are some
'base' numbers on mozilla-central:
Before:
! wall 6.912745 comb 6.910000 user 6.840000 sys 0.070000 (best of 3)
After - initial, cache is empty:
! wall 7.792569 comb 7.790000 user 7.720000 sys 0.070000 (best of 3)
After - cache is full:
! wall 0.879688 comb 0.880000 user 0.870000 sys 0.010000 (best of 4)
The overhead when running with empty cache comes from checking, missing and
updating it every time.
Most of the performance improvement comes from not having to extract the branch
info from the changelog. The last doubling of performance comes from no longer
having to convert all branch names to local encoding but reuse the few already
converted branch names.
On the hg repo:
Before:
! wall 0.715703 comb 0.710000 user 0.710000 sys 0.000000 (best of 14)
After:
! wall 0.105489 comb 0.110000 user 0.110000 sys 0.000000 (best of 87)
Local clones copy/hardlink over files directly, so the branchcaches should all
be valid. There's a slight chance that a read operation would update one of the
branchcaches, but we hold a lock over the source repo so they shouldn't cause
an invalid branchcache to be copied over, just a different, valid one.
Changset 39a4d61c40d6 rewriting "hg.copystore()" with vfs uses
'dstbase + "/lock"' instead of "os.path.join()", because target files
given from "store.copyfiles()" already uses "/" as path separator
But in the repository using revlog format 0, "dstbase" becomes empty
("data" directory is located under ".hg" directly), and 'dstbase +
"/lock"' is treated as "/lock": in almost all cases, write access to
"/lock" causes "permission denied".
This patch uses "os.path.join()" to join path components which may be
empty in "hg.copystore()".
This adds a new root hghave to test against. Almost all of these are a
subset of unix-permissions, but that is also used for checking exec
bit handling.
Mercurial would sometimes exit with:
abort: No such file or directory
where str of the actual OSError exception was the more helpful:
[Errno 2] No such file or directory: ''
The exception will now always show the filename and quote it:
abort: No such file or directory: ''
