Sometimes when running tests you introduce a ton of exceptions.
The most extreme example of this is running Mercurial with Python 3,
which currently spews thousands of exceptions when running the test
harness.
This commit adds an opt-in feature to run-tests.py to aggregate
exceptions encountered by `hg` when running tests.
When --exceptions is used, the test harness enables the
"logexceptions" extension in the test environment. This extension
wraps the Mercurial function to handle exceptions and writes
information about the exception to a random filename in a directory
defined by the test harness via an environment variable. At the
end of the test harness, these files are parsed, aggregated, and
a list of all unique Mercurial frames triggering exceptions is
printed in order of frequency.
This feature is intended to aid Python 3 development. I've only
really tested it on Python 3. There is no shortage of improvements
that could be made. e.g. we could write a separate file containing
the exception report - maybe even an HTML report. We also don't
capture which tests demonstrate the exceptions, so there's no turnkey
way to test whether a code change made an exception disappear.
Perfect is the enemy of good. I think the current patch is useful
enough to land. Whoever uses it can send patches to imprve its
usefulness.
Differential Revision: https://phab.mercurial-scm.org/D1477
And sort arguments so help output is more legible.
There are probably a ton of ways to group things. I tried to
picture the test harness as a pipeline and attempted to draw boundaries
around stages in that pipeline to create the groupings.
Differential Revision: https://phab.mercurial-scm.org/D1475
optparse has been deprecated since Python 3.2. Best to get on the new
boat before the old one sinks.
It looks like argparse formats its usage string differently than
optparse. Meh.
Differential Revision: https://phab.mercurial-scm.org/D1474
In large repositories, updates involving the creation of many files check the
same directories repeatedly in the wctx manifest. Move these checks out to a
separate loop to avoid repeated checks hitting the manifest.
Differential Revision: https://phab.mercurial-scm.org/D1226
As mentioned in D1222, the recent pathconflicts change regresses update
performance in large repositories when many files are being updated.
To mitigate this, we introduce two caches of directories that have
already found to be either:
- unknown directories, but which are not aliased by files and
so don't need to be checked if they are files again; and
- missing directores, which cannot cause path conflicts, and
cannot contain a file that causes a path conflict.
When checking the paths of a file, testing against this caches means we can
skip tests that involve touching the filesystem.
Differential Revision: https://phab.mercurial-scm.org/D1224
If this feature is enabled, early options are parsed using the global options
table. As the parser stops processing options when non/unknown option is
encountered, it won't mistakenly take an option value as a new early option.
Still "--" can be injected to terminate the parsing (e.g. "hg -R -- log"), I
think it's unlikely to lead to an RCE.
To minimize a risk of this change, new fancyopts.earlygetopt() path is enabled
only when +strictflags is set. Also the strict parser doesn't support '--repo',
a short for '--repository' yet. This limitation will be removed later.
As this feature is backward incompatible, I decided to add a new opt-in
mechanism to HGPLAIN. I'm not pretty sure if this is the right choice, but
I'm thinking of adding +feature/-feature syntax to HGPLAIN. Alternatively,
we could add a new environment variable. Any bikeshedding is welcome.
Note that HGPLAIN=+strictflags doesn't work correctly in chg session since
command arguments are pre-processed in C. This wouldn't be easily fixed.
The next patch will add a flag for strict parsing of early options, where
we'll have to parse all early options at once instead of processing them
one-by-one by dispatch._earlygetopt(). That's why I decided to hook
fancyopts().
All dispatch._early*opt() functions is planned to be replaced with this
function. But in this stable series, only the strict mode will be handled
by fancyopts.earlygetopt().
When the option is set, the repository will reject any transaction adding
multiple heads to the same named branch.
For now we reject all scenario with multiple heads. One could imagine handling
closed branches differently. We prefer to keep things simple for now. The
feature might get extended later. Branch closing is not the best experience
Mercurial has to offer anyway.
Previously, the largefile for a dropped standin would be deleted here, and then
restored from the cache. This had the effect of clobbering uncommitted changes
if a revert caused the file to be forgotten, which is not what happens with a
normal file. Now the removal and update is skipped for dropped largefiles, and
the corresponding standin is deleted from disk.
This was noticed when working on issue5738 because the forgotten standin files
were left behind, and that changes the behavior of the next rename to that
directory. My first attempt was to cleanup the standins before calling this.
That failed, because this function deletes the largefile if the corresponding
standin is missing.
This function is called by the revert command, merge (and therefore update), and
patch, via the scmutil.marktouched() override. So it should be pretty narrow in
scope.
I didn't mark issue5738 as fixed because the move related issues can still
happen if the main tree and the .hglf subtree get out of sync somehow. I don't
see an easy fix for that, but that should be an edge case. If whoever queues
this thinks it is good enough to close out the bug and can cram it into the
summary, go for it.
These things were uncovered looking at issue5738.
First, if the destination directory exists under .hglf, the source is moved
under the destination instead of renaming the last component for `hg mv srcdir
dstdir`. This is extra confusing, because it occurs even if the user visible
destination (i.e. the path _not_ under .hglf) does not exist.
Additionally, when a largefile is forgotten via revert, any modifications end up
getting clobbered. For normal files, the forgotten file is left unchanged, as
shown by test-import.t. The forget command on a largefile will correctly leave
the file unmodified.
The `hg lfconvert --to-normal` command uses the convert extension internally to
work its magic, but that produced devel-warn messages if the convert extension
wasn't loaded by the user. The test in 658e7a6d93e0 (modified here) wasn't
showing the warnings because the convert extension was loaded via $HGRCPATH.
Most of the config options default to None/False, but 'hg.usebranchnames' and
'hg.tagsbranch' are supposed to default to True and 'default' respectively.
The first iteration of this was to ui.setconfig() inside lfconvert, to force the
convert extension to load. But there really is no precedent for doing this, and
check-config complained that 'extensions.convert' isn't documented. Yuya
suggested this alternative.
This partially backs out 448e09d8859d.
Repoview can have a different life cycle, causing issue in some corner
cases. The particular instance that revealed this comes from localpeer. The
localpeer hold a reference to the unfiltered repository, but calling 'local()'
will create an on-demand 'visible' repoview. That repoview can be garbaged
collected any time. Here is a simplified step by step reproduction::
1) tr = peer.local().transaction('foo')
2) tr.close()
After (1), the repoview object is garbage collected, so weakref used in (2)
point to nothing.
Thanks to Sean Farley for helping raising and debugging this issue.
Before, early options were stripped from args, and because of this, some
kind of parsing errors weren't reported. For example,
$ hg ci -m -Ra file
would execute "hg ci -m file" in repository "a".
This patch fixes the issue by parsing early options again by real getopt-based
parser, and verifying the results. If the early parsing appears wrong, hg just
aborts. The current error message seems not nice, and should be improved, maybe
in V2 or follow-up.
Note that this isn't a security feature because we can still do anything by
using shell aliases.
So we can easily compare it with the corresponding getopt() result.
There's a minor behavior change. Before, "hg --cwd ''" failed with ENOENT.
But with this patch, an empty cwd is silently ignored. "hg -R ''" has always
worked as such, so -R has no BC.
This allows us to parse the original args later by full-blown getopt() in
order to verify the result of the faulty early parsing. Still we need the
'strip=True' behavior for shell aliases.
Note that this series is RFC because it seems to change too much to be
included in stable release.
Perhaps we'll need to restrict the parsing rules of --debugger and --profile,
where this patch will help us know why the --debugger option doesn't work.
I have another series to extend this feature to --config/--cwd/-R, but even
with that, shell aliases can be used to get around the restriction.
This is a minimal copy of localrepo.commit(). As the current amend() function
heavily depends on the wctx API, it wasn't easy to port it to use a separate
status tuple. So for now, wctx._status is updated in-place.
This demonstrates that missing/untracked files are handled incorrectly. The
correct outputs are suppressed by (false !), and wrong outputs are added with
(true !) instead.
The setup code is copied from test-status-rev.t.
When origbackuppath is set, when looking to see if a file we are backing up
conflicts with a directory in the origbackuppath, we incorrectly match on
symlinks to directories. This means we try to call vfs.rmtree on the
symlink, which fails.
Differential Revision: https://phab.mercurial-scm.org/D1311
If origbackups are in use, a symlink to a valid directory is backed up, and an
update is made that attempts to backup a file or link over that symlink, we
abort with a bad error message instead of successfully updating.
Differential Revision: https://phab.mercurial-scm.org/D1310
We change subrepos.allowed from a list of allowed subrepo types to
a combination of a master switch and per-type boolean flag.
If the master switch is set, subrepos can be disabled wholesale.
If subrepos are globally enabled, then per-type options are
consulted. Mercurial repos are enabled by default. Everything else
is disabled by default.
These config items control share behavior that is implemented in core.
Since the functionality is implemented in core, extensions may
leverage it.
Mozilla has one such extension. And, it needs to access share.pool.
Before this patch, a devel warning regarding accessing an unregistered
config option would be issued unless the share extension were loaded.
Moving the registration of the config options to core fixes this.
We have a security issue with git subrepos. I'm not sure if svn subrepo is
vulnerable, but it seems not 100% safe to allow writing arbitrary data into
a metadata directory. So for now, only hg subrepo is enabled by default.
Maybe we should improve the help to describe why git/svn subrepos are
disabled.
This allows us to minimize the behavior change introduced by the next patch.
I have no idea which config style is preferred in UX POV, but I decided to
get things done.
a) list: 'allowed = hg, git, svn'
b) sub option: 'allowed.hg = True' or 'allowed:hg = True'
c) per-type action: 'hg = allow', 'git = abort'
This is an alternative workaround for the issue5730.
Perhaps this is the simplest way of disabling subrepo operations. It does
nothing clever, but just aborts if Mercurial starts accessing to a subrepo.
I think Greg's patch is more useful since it allows us to at least check
out the parent repository. However, that would be confusing if the default
is flipped to checkout=False and subrepos are silently ignored.
I don't like the config name 'allowed', but I couldn't get any better name.
Previously, if there were unresolved files and the CWD drive was different from
the repo drive, `hg status -v` would page the normal status, followed by the
exception header. A stacktrace was waiting when the pager exited. The
underlying cause was the same as 12441ef4442f.
Unfortunately, I don't see any reasonable way to write a test this [1].
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-November/107401.html
This is a followup to 12441ef4442f. Since there's no way to ensure that more
drive letters than C: exist, this seems like the only way to test it. This is
enough to catch the 12441ef4442f scenario, as well as CWD outside of the repo
when the path isn't prefixed with path/to/repo.
It wasn't easy to extend the pathauditor to check symlink traversal across
subrepos because pathauditor._checkfs() rejects a directory having ".hg"
directory. That's why I added the explicit islink() check.
No idea if this patch is necessary after we've fixed the issue5730 by
splitting submerge() into planning and execution phases.
os.path.relpath() exploded if the 'root' and 'cwd' directories had different
drive letters. I noticed this in TortoiseHg when typing a fileset into the
filter, and it kept complaining until the closing '()' was typed. This was
reproducible on the command line with:
$ cd /d
$ hg -R /c/Users/Matt/Projects/hg files 'set:e'
Traceback (most recent call last):
...
File "mercurial\pathutil.pyc", line 182, in canonpath
File "ntpath.pyc", line 529, in relpath
ValueError: path is on drive c:, start on drive d:
