I've seen the following error a few times recently when running the tests with
`yes | ./run-tests.py --local -j9 -i`:
Errored test-add.t: Traceback (most recent call last):
File "./run-tests.py", line 821, in run
self.runTest()
File "./run-tests.py", line 910, in runTest
if self._result.addOutputMismatch(self, ret, out, self._refout):
File "./run-tests.py", line 1774, in addOutputMismatch
rename(test.errpath, test.path)
File "./run-tests.py", line 571, in rename
os.remove(src)
WindowsError: [Error 32] The process cannot access the file because it is being
used by another process: 'c:\\Users\\Matt\\projects\\hg\\tests\\test-add.t.err'
This change doesn't fix the problem, but it seems like a simple enough
improvement.
The goal is to reduce the amount of hand tuning of new/changed tests that is
required on Windows. Since the OS prints the proper paths everywhere else, this
is limited to Windows. These are based on the check-code rules that were
dropped in 217bd5cb0914.
There are some minor tweaks, because those were trying to detect '/' paths
without a '(glob)' at the end, whereas these detect '\' paths. Also, it looks
like the 'no changes made to subrepo' one was broke, because the path to the
subrepo has been getting output but was not in the pattern. End anchors are
dropped because '(glob)' is no longer required, but '(feature !)' annotations
are a possibility.
The 'saved backup bundle' pattern dropped from run-tests.py was simply carrying
over the first capture group. The replace() method runs prior to evaluating
'\1', but it wasn't doing anything because of the 'r' prefix on '\\'.
The 'not recording move' entry is new, because I stumbled upon it searching for
some of these patterns. There are probably others.
This is the first step to dropping the existing globs for '\' matches, now that
it is handled automatically. Instead of just dropping it, this pattern is now
used to convert to '/' paths, to reduce the amount of manual cleanup required
when creating tests on Windows.
Having to constantly adjust these is a hassle. It's easy for this to slip by
when not testing on Windows, and then when it happens on stable, the tests fail
for the next 3 months if we follow the rules for stable.
This works the same way the EOL differences are ignored, namely to adjust on the
fly and recheck on Windows. I can't think of any situation where there would be
a '\' on Windows, a '/' elsewhere, and the '/' should be considered a failure on
Windows.
This fixes the obvious output problems where (glob) is missing. Without this,
test-alias.t, test-remotenames.t and test-largefiles-misc.t are failing. The
flip side (not handled by this) is the case where an unnecessary glob is
present. There seems to be two separate behaviors. f3517e22bfa1 is an example
of where the test has been autocorrecting (with output differences), and
ed159a9fcf2a is an example where the test fails and reports 'no result code from
test'. Hopefully those cases will become even more rare if people don't need to
guess at when a glob is needed for a Windows path.
It's probably unreasonable to submit a single patch that wipes out all of the
(glob) instances that were only used to hide path differences, given the churn
from other contributors. Since their presence isn't harming the tests, these
can be removed through attrition.
We already do this for lines ending in '\n', such that the test only needs to be
run with --interactive and the changes accepted at the end. But that wasn't
working with list-tree.py output for example, and required manual fixup.
This is the same mechanism in place for largefiles, and solves several problems
working with multiple local repositories. The existing largefiles method is
reused in place, because I suspect that there are other functions that can be
shared. If we wait a bit to identify more before `hg cp lfutil.py ...`, the
history will be easier to trace.
The push between repo14 and repo15 in test-lfs.t arguably shouldn't be uploading
any files with a local push. Maybe we can revisit that when `hg push` without
'lfs.url' can upload files to the push destination. Then it would be consistent
for blobs in a local push to be linked to the local destination's cache.
The cache property is added to run-tests.py, the same as the largefiles
property, so that test generated files don't pollute the real location. Having
files available locally broke a couple existing lfs-test-server tests, so the
cache is cleared in a few places to force file download.
Sometimes when running tests you introduce a ton of exceptions.
The most extreme example of this is running Mercurial with Python 3,
which currently spews thousands of exceptions when running the test
harness.
This commit adds an opt-in feature to run-tests.py to aggregate
exceptions encountered by `hg` when running tests.
When --exceptions is used, the test harness enables the
"logexceptions" extension in the test environment. This extension
wraps the Mercurial function to handle exceptions and writes
information about the exception to a random filename in a directory
defined by the test harness via an environment variable. At the
end of the test harness, these files are parsed, aggregated, and
a list of all unique Mercurial frames triggering exceptions is
printed in order of frequency.
This feature is intended to aid Python 3 development. I've only
really tested it on Python 3. There is no shortage of improvements
that could be made. e.g. we could write a separate file containing
the exception report - maybe even an HTML report. We also don't
capture which tests demonstrate the exceptions, so there's no turnkey
way to test whether a code change made an exception disappear.
Perfect is the enemy of good. I think the current patch is useful
enough to land. Whoever uses it can send patches to imprve its
usefulness.
Differential Revision: https://phab.mercurial-scm.org/D1477
And sort arguments so help output is more legible.
There are probably a ton of ways to group things. I tried to
picture the test harness as a pipeline and attempted to draw boundaries
around stages in that pipeline to create the groupings.
Differential Revision: https://phab.mercurial-scm.org/D1475
optparse has been deprecated since Python 3.2. Best to get on the new
boat before the old one sinks.
It looks like argparse formats its usage string differently than
optparse. Meh.
Differential Revision: https://phab.mercurial-scm.org/D1474
We make "foo (re)" match the entire line by adding a \Z to the regular
expression before matching. However, that doesn't help when the
regular expression is something like "| foo", because that gets
translated to "| foo\Z", where the "|" has lower precedence and it
thus matches the empty string, which is of course a prefix of every
string. Fix by wrapping expression in a group before adding the \Z to
the end.
Differential Revision: https://phab.mercurial-scm.org/D1546
Following 30a570fd0165, running run-tests.py from outside tests/ would lead to
the creation of .testtimes and test-*.t.err in $PWD instead of $TESTDIR. This
patch fixes that and updates the relevant test.
We add a 'common-pattern.py' file that allow to define extra pattern. This seems
a cleaner approach than editing the 'run-test.py' file over and over. In
addition allowing arbitrary pattern registration will also help extension.
The format used is a python file is picked out of convenience defining a list of
tuple in 'substitutions' variable. This is picked out of convenience since it is
dead simple to implement.
The end goal is to register more pattern for Mercurial test. There are multiple
common patterns that change over time. That impact is annoying. Using pattern
emplacement for them would be handy.
The next patches will define all the needed patterns and the last patch will
mass-update the tests outputs as it was easier to do in a single pass.
Commit 30a570fd0165 made TESTDIR be the location of the arguments that were
passed to run-tests.py instead of just PWD. It assumed that these tests were
specified using relative paths, so if pwd was /tmp/foo, and the first argument
was /tmp/baz, it would set TESTDIR to /tmp/foo//tmp/baz.
Differential Revision: https://phab.mercurial-scm.org/D1433
Currently `run-tests.py` automatically discovers test only in the current
directory if no argument is provided. This patch makes it possible to pass a
number of tests and folders as arguments.
$TESTDIR is expected to be the directory were the test lives, and is often used
to reference helper functions. However, it is now set to $PWD at test invocation
time, so if `run-tests.py` is called from another folder, the test will fail.
The solution is to force $TESTDIR to be the base directory of the test itself,
irrespective of where the runner is called from.
The leading newline before "ERROR:" led to an incorrect lexing of the
message and the newline got lost.
The fixed formatting can be seen in the test case. The reason we
didn't notice before was that the bad formatting just led to the loss
of a blank line in the test. However, there are other cases where it
would be joined with a line saying "Accept this change? [n]" or
"Reference output has changed (run again to prompt changes)".
Differential Revision: https://phab.mercurial-scm.org/D1159
As far as I can tell, default values are evaluated far earlier on
Python 3.6 than 2.7, meaning we've got to be more careful about when
we read the defaults dictionary for the kwarg defaults. Sigh.
With this change, test-run-tests.t is significantly less of a mess
under 3.6.
Add `--bisect-repo` flag which accepts a different repo to bisect.
3rd party extensions may reuse `run-tests.py` from core to run tests. Test
failure could be caused by either a core hg change or the 3rd party
extension code itself. Having a way to specify which repo to bisect is
useful.
Differential Revision: https://phab.mercurial-scm.org/D578
This makes `run-tests.py -l test-run-tests.t` 23 seconds faster on my
laptop. Inside the test, `$ rt --known-good-rev=0 test-bisect.t` took 24.9
seconds before, and 1.2 seconds after.
Differential Revision: https://phab.mercurial-scm.org/D576
7340465bd added multiple test cases support. The latter has a problem -
output lines cannot be made conditional with `#if`:
```
# COUNTEREXAMPLE: DOES NOT WORK
#testcases A B
$ command-foo
common ouput
#if A
A's ouput
#else
B's ouput
#endif
common ouput
```
That's not trivial to fix (even if it works in test, `run-tests.py -i` may
be suboptimal because diff algorithm does not know how to skip the `#if`
lines, even if it does, it may have trouble figuring out whether a changed
line belongs to inside a `#if` block or outside).
Matching output lines conditionally is useful. 4eec2f04a added per-line
condition support for hghave. This patch extends that to also support test
case names.
Differential Revision: https://phab.mercurial-scm.org/D466
Some of the recent work on run-tests.py didn't work on Python 3. This
fixes the regressions.
Differential Revision: https://phab.mercurial-scm.org/D270
Essentially, these were acting as a verbose (?) flag, since they weren't being
dropped when required. Foozy has a nice description [1]. Basically, a couple
more places needed to check the features before treating it as optional.
I don't like how test-run-tests.py had to be hacked, but _hghave() can't be made
a static method. The test change was a change while developing `debugssl`,
prior to tightening up the cases where the message is printed, that this fix
would have caught.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101941.html
