Duplicating entire tests just because the output is different is both error
prone and can make the tests harder to read. This harnesses the existing '(?)'
infrastructure, both to improve readability, and because it seemed like the path
of least resistance.
The form is:
$ test_cmd
output (hghave-feature !) # required if hghave.has_feature(), else optional
out2 (no-hghave-feature2 !) # req if not hghave.has_feature2(), else optional
I originally extended the '(?)' syntax. For example, this:
2 r4/.hg/cache/checkisexec (execbit ?)
pretty naturally reads as "checkisexec, if execbit". In some ways though, this
inverts the meaning of '?'. For '(?)', the line is purely optional. In the
example, it is mandatory iff execbit. Otherwise, it is carried forward as
optional, to preserve the test output. I tried it the other way, (listing
'no-exec' in the example), but that is too confusing to read. Kostia suggested
using '!', and that seems fine.
Previously, if a series of optional output lines marked with '(?)' had a (glob)
in one of the first lines, the output would be reordered such that it came last
if none of the lines were output. The (re) declaration wasn't affected, which
was helpful in figuring this out. There were no tests for '(re) (?)' so add
that to make sure everything plays nice.
This is similar to what 9704c8e70d2d does. Since 1363aaf74791 has changed
"127.0.0.1" to "$LOCALIP". The glob pattern needs update accordingly. It is
expected to fix tests running in some BSD jails.
Some systems don't have a 127/8 address for localhost (I noticed this
on a FreeBSD jail). In order to work around this, use 127.0.0.1 as a
glob pattern. A future commit will update needed output lines and add
a requirement to check-code.py.
In py2, json.dumps includes a trailing space after a comma at the
end of lines. The py3 behavior which omits the trailing space is
preferable, so we're going to strip it.
72b40f92c680 enabled lines that were not matched to be found later in cases of jitter.
Unfortunately, in this model an optional line would always jitter to the end when
it is not present. That is not ideal.
It would be possible to do better, by queuing all writes until the end in case
an optional line jitters, but for now, it is simpler to assume optional lines
have a fixed place in the stream.
72b40f92c680 added the ability to remember lines (including their flags, like glob)
and tolerate them even if the output order varies.
This test ensures that the code will continue to work in the future.
dffce3bb1563 added arguments to run-tests.py calls to propagate --pure when
--with-hg is not used.
6bb9fddace23 unified places to use --with-hg via rt(), which obviated the
need for passing HGTEST_RUN_TESTS_PURE.
Previously, after matching a single line, any contiguous subsequent lines ending
with (?) would be added to the output and removed from the expected output.
This is a problem if the subsequent test output would have matched the consumed
(?) line, because it kept the optional line and then added a duplicate without
the (?) [1]. Instead, wait until there is nothing more to match before handling
the leftovers.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-February/080197.html
When reloading tests, run-tests.py was assuming that it could look
up the test by the basename, which only works if you are running
tests which are in the current directory.
This patch changes that lookup to use the full path. This is all
that was needed, and does not appear to cause any problems for
any of the existing testing work flows based on running the
suggested commands at the top of run-tests.py.
Motivation: In order to test Mercurial with Hypothesis (according
to https://www.mercurial-scm.org/wiki/HypothesisPlan) it is
useful to be able to generate temporary test files and execute
them. Generating temporary files in the tests/ directory leads to
a lot of suboptimal clutter.
This patch fixes a crash when both --json and --blacklist were given as
arguments of run-tests.py. Now, instead of crashing, we add an entry for
blacklisted tests in the json output to show that the tests were skipped.
This patch adds to the json report the "diff" between expected and observed
result. This diff can be useful for automatically filing bug report on failing
tests.
This internal test is piped through 'grep -v pwd' to eliminate the pwd alias set
when running with MSYS. Unfortunately, the '.' from the successful run of the
prior internal test precedes the pwd alias for the next test on the same line,
so grep filters out '.' too, except for the final test.
It also looks like there may be a bug with --debug: the output of the internal
test that had this diff says 2 ran, 0 failed (one test being test-failure.t),
but if --debug is omitted from the internal test, then it says 2 ran, 1 failed.
With this longstanding issue fixed, the test suite finally runs cleanly on
Windows (except subrepo merge documented in issue 4988), with 88 skips. \o/
The variable uniformly uses '\' separators, so the straight equality check with
'/' separating the last component fails. It also doesn't like having the quote
appear in the middle of the string when testing.
There are two other unrelated failures in this test. For some reason child1 has
a space after it, thus 2 spaces before the glob, otherwise the glob is ignored
and removed.
test-run-tests and test-hghave call run-tests;
if you don't have a working build environment, and you are trying
to use --pure, then if they don't use --pure or --with-hg,
they'll break.
I want to add tests for our packaging rules, but those necessarily run
a whole build, or possibly two if both native packaging and docker are
available. This lets us flag such tests with a `#require slow` so that
they don't unnecessarily slow down normal test runs.
Before this patch, `RUNTESTDIR` is added to `PATH`, but `TESTDIR`
isn't.
This doesn't cause any problems, if `run-tests.py` runs in `tests`
directory of Mercurial source tree. In this case, `RUNTESTDIR` should
be equal to `TESTDIR`.
On the other hand, if `run-tests.py` runs in `tests` of third party
tools, commands in that directory should be executed with explicit
`$TESTDIR/` prefix in `*.t` test scripts. This isn't suitable for the
policy "drop explicit $TESTDIR from executables" of Mercurial itself
(see fcb1c7d8c36e).
BTW, fcb1c7d8c36e describes that "$TESTDIR is added to the path" even
though `TESTDIR` isn't added to `PATH` exactly speaking, because
`TESTDIR` and `RUNTESTDIR` weren't yet distinguished from each other
at that time.
This is a one of preparations for issue4677.
Before this patch, there is no way to refer files under `tests` or so
of Mercurial source tree, when `run-tests.py` runs in `tests` of third
party tools. In this case, `TESTDIR` refers the latter `tests`.
This prevents third party tools from using useful tools in Mercurial
source tree (e.g. `contrib/check-code.py`).
This patch adds `RUNTESTDIR` environment variable to refer `tests` of
Mercurial source tree, in which `run-tests.py` now running is
placed. For example, tests of third party tools can refer
`contrib/check-code.py` in Mercurial source tree as
`$RUNTESTDIR/../contrib/check-code.py`.
BTW, for similarity with `TESTDIR` referring `test*s*` directory,
newly added environment variable isn't named as `RUNTEST*S*DIR`. In
addition to it, the corresponded local variable is also named as
`runtestdir`.
This is a one of preparations for issue4677.
Before this patch, `run-tests.py` executes `hghave` by the path
relative to `TESTDIR` (= cwd of `run-tests.py` running).
This prevents third party tools for Mercurial from running
`run-tests.py`, which is placed in `tests` of Mercurial source tree,
in `tests` of own source tree. In such cases, `TESTDIR` refers the
latter `tests`, and `hghave` doesn't exist in it.
This is a one of preparations for issue4677.
$TESTDIR is added to the path, so this is superfluous. Also,
inconsistent use of quotes means we might have broken on tests with
paths containing spaces.
When the test engine fails to match output on a line marked with (?),
it will simply continue to the next expected line and try again. This
allows simplifying tests that have either version-specific or
non-fixed behavior, for instance:
$ coin-flip
heads (?)
tails (?)
(There's no form of back-tracking attempted, so optional matches
should be specific.)
This is useful information to understand what is taking time in tests. Both are
included because I can see myself sorting this output using shell script. Having
both data makes it much easier than extracting 'start'+'real'.
We currently have information about how long each test took, but we have no data
about their actual scheduling. So we now track when a test started and stopped
(in the referential of the whole tests run) to expose this information.
The data is currently in the json only because the json output is meant to be
extensible. Later changeset will includes this data in the text output and we
will be happy to bikeshed its formating there.
Yes, "end" is actually just "start" + "time", but computing it an including it
in the output is simple, cheap and convenient.
As part of writing another test, I triggered an array index error in
glob match processing code by having a (glob) line end in a single
backslash (which is the escape character).
Adding a simple bounds check prevents the error in run-tests.py.
The test runner has the ability to stop on first error.
Tests are executed in new Python threads. The test runner starts new
threads when it has capacity to do so. Before this patch, the "stop on
first error" logic would return immediately from the "run tests"
function, without waiting on test threads to complete. There was thus
a race between the test runner thread doing cleanup work and the test
thread performing activity. For example, the test thread could be in
the middle of executing a test shell script and the test runner
could remove the test's temporary directory. Depending on timing, this
could result in any number of output from the test runner.
This patch eliminates the race condition by having the test runner
explicitly wait for test threads to complete before continuing.
I discovered this issue as I modified the test harness in a subsequent
patch and was reliably able to tickle the race condition.
The xunit writer was passing a str to a minidom API. An implicit
.decode('ascii') was performed somewhere, causing UnicodeDecodeError
if test output contained non-ascii sequences.
This patch converts test output to utf-8 before passing it to minidom.
We use the "replace" strategy to ensure invalid utf-8 sequences get
munged into �.
Before this patch, "test-run-tests.t" doesn't test "run-tests.py" with
"--pure", even if outer "run-tests.py" is executed with it.
This patch uses not "HG_RUN_TESTS_PURE" but "HGTEST_RUN_TESTS_PURE",
because "HG_" prefixed environments are forcibly dropped in "_getenv()".
This is also useful to run "run-tests.py" successfully by
"run-tests.py --pure" on Windows without any compilation tools (like
VisualStudio).
Avoid spending too much time adding (glob) after running run-tests -i. This
doesn't handle all cases but it helps.
The run-tests tests add a bit of escaping of trailing (glob) in the output to
avoid interference from the outer test runner.
The regexp for matching the output lines contains a group for making multiline
substitute in a way that works with Python before 2.7.
