Previously, after matching a single line, any contiguous subsequent lines ending
with (?) would be added to the output and removed from the expected output.
This is a problem if the subsequent test output would have matched the consumed
(?) line, because it kept the optional line and then added a duplicate without
the (?) [1]. Instead, wait until there is nothing more to match before handling
the leftovers.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-February/080197.html
At one point run-tests.py and test-run-tests.t worked and passed
under Python 3.5. Various changes to run-tests.py over the past
several months appear to have broken Python 3.5 compatibility.
This patch implements various fixes (all related to str/bytes type
coercion) to make run-tests.py and test-run-tests.t mostly work
again. There are still a few failures in test-run-tests.t due to
issues importing mercurial.* modules. But at least run-tests.py
seems to work under 3.5 again.
When reloading tests, run-tests.py was assuming that it could look
up the test by the basename, which only works if you are running
tests which are in the current directory.
This patch changes that lookup to use the full path. This is all
that was needed, and does not appear to cause any problems for
any of the existing testing work flows based on running the
suggested commands at the top of run-tests.py.
Motivation: In order to test Mercurial with Hypothesis (according
to https://www.mercurial-scm.org/wiki/HypothesisPlan) it is
useful to be able to generate temporary test files and execute
them. Generating temporary files in the tests/ directory leads to
a lot of suboptimal clutter.
The only consumer was test-treemanifest.t, which has been fixed.
In general, you should be able to use killdaemons.py to recycle
ports instead of going over 3 ports (HGPORT, HGPORT1, HGPORT2).
In the future, if you want to add a port, be sure to change
portneeded in _getport.
Adding a port reservation was too hard and someone did it wrong.
By refactoring, such reservations can be managed more safely.
This also adds documentation so that the next person who tries
is more likely to update all the places correctly.
Note that in this commit the reservation and consumers do not
match, that will be fixed in the next commit.
Because the temporary installation directory is shared between hg and chg,
--chg is not allowed if --with-hg option is specified. Also, --chg option
does not work on FreeBSD because "make" command is hard-coded. These
limitations can be improved later.
Almost all tests will fail with chg right now.
Unlike --with-hg=/path/to/chg, this option allows us to start and clean up
command servers in isolated environment. And we can specify the hg command
as well as the chg command.
If the executable is not named as "hg", TTest runner inserts alias. This
way, we can run tests with chg. But it is still warned because the alias
does not always work. We do "$BINDIR"/hg in a few places.
Similar to the previous patch, the .hg/store/meta/ directory does not
get copied when when using "hg clone --uncompressed". Fix by including
"meta/" in store.datafiles(). This seems safe to do, as there are only
a few users of this method. "hg manifest" already filters the paths by
"data/" prefix. The calls from largefiles also seem safe. The use in
verify needs updating to prevent it from mistaking dirlogs for
orphaned filelogs. That change is included in this patch.
Since the dirlogs will now be in the fncache when using fncachestore,
let's also update debugrebuildfncache(). That will also allow any
existing treemanifest repos to get their dirlogs into the fncache.
Also update test-treemanifest.t to use an a directory name that
requires dot-encoding and uppercase-encoding so we test that the path
encoding works.
Laurent's commit 56cdfddbd2ed still suffers from a race: by the
time the "job" function tries to assign to channels[channel], that
list has been truncated to empty. The result is that every job
thread raises an IndexError.
Earlier, I tried an approach of correctly locking channels, but
that caused run-tests to hang on KeyboardInterrupt sometimes.
This approach is strictly hackier, but seems to actually work
reliably.
This patch fixes a crash when both --json and --blacklist were given as
arguments of run-tests.py. Now, instead of crashing, we add an entry for
blacklisted tests in the json output to show that the tests were skipped.
Before this patch, it was possible for run-tests to crash on a race condition.
The race condition happens in the following case:
- the last test finishes and calls: done.put(None)
- the context switches to the main thread that clears the channels list
- the context switches to the last test mentioned above, it tries to access
channels[channel] and crashes
This happened to me while running run-tests.
This patch fixes the issue by clearing the channel before considering that the
test is done.
Threading is incompatible with most Python debuggers,
which makes debugging run-tests.py a real pain.
If there is only one test to run, skip using a thread for it.
Note that --debug is not compatible with debugging tests,
since it bypasses the output handling, which is where
much of the excitement is.
This patch adds to the json report the "diff" between expected and observed
result. This diff can be useful for automatically filing bug report on failing
tests.
Vaguely empirical observations:
* ".py" tests are about an order of magnitude faster than ".t" tests
* dividing size by 1000 gives an approximation to wall-clock
run time (in seconds) that is not completely ridiculous.
This gives one line of output per second with one column per -j level
that allows analyzing test scheduling problems. First 24 seconds of
output at -j 30 looks like this:
0 .
1 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = s.
2 c c o c r l g r s s = c p = c h c a h c g c h c b c c l l c ss
3 h o b o e a e u u u c o a h o e o c g o l h g h u o = a o = s
4 e n s n b r n n b b m t g n l n l w n o e w e n n e r g i .
5 c t o = a g d - r r = m c w v p v . e v g c e c d v x g . m
6 k r l r s e o t e e b a h e e . e . b e . k b k l e t e . p
7 - i e e e f c e p p u n b b r . r . - r . - - - e r e f . o .
8 p b t v - i . s o o n d o d t . t . c t . c s = 2 t n i . r
9 y - e s c l . t - . d - m i - . - . o - . o y r - - s l . t
10 3 p - e h e . s s . l t b r s . s . m s . d m e f s i e . .
11 - e c t e s . . v . e e . . v . v . m v . e r n o v o s . .
12 c r h . c - . . n . 2 m . . n . n . a n . . e a r n n . . .
13 o f e . k u . . . . - p . . - . - . n - . . v m m - . . . .
14 m . c . - p . . . . e l . . s . m . d s . . . e a e . . . .
15 p . k . r d . . . . x a . . i . o . s o . . . - t n . . . .
16 a . h . e a . . . . c t . . n . v . . u . . . m . c . . . .
17 t . e . s t . . . . h e . . k . e . . r . . . e . o . . . .
18 . . a . t e . . . . a . . . . . . . . c . . . r . d . . . .
19 . . d . o . . . . . n . . . . . . . . e . . . g . i . . . .
20 . . s . r . . . . . g . . . . . . . . . . . . e . n . . . .
21 . . . . e . . . . . e . . . . . . . . . . . . 2 . g . . . .
22 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24 . . . . . . . . . . . . . . . . . . . . . . . . . = . . . . ^C
Test names read off vertically, beginning with '='. Idle time (not
shown) appears as blank space.
The scheduler would like to order test execution by expected run-time,
but doesn't know much about how long a test will run. It thus uses
test size as a proxy for run-time. By tweaking these weights we can
keep CPUs more evenly busy and thus finish sooner.
In particular, this change pushes the three currently longest-running
tests closer to the beginning:
test-largefiles-update.t
test-run-tests.t
test-gendoc.t
As the largefiles test is currently the long pole of the test suite
with higher -j factors, the sooner it's started, the sooner the tests
can end.
We also up the weight on some shorter but long-running tests that
could have previously delayed completion with low -j factors by
running very close to the end.
test-run-tests and test-hghave call run-tests;
if you don't have a working build environment, and you are trying
to use --pure, then if they don't use --pure or --with-hg,
they'll break.
I want to add tests for our packaging rules, but those necessarily run
a whole build, or possibly two if both native packaging and docker are
available. This lets us flag such tests with a `#require slow` so that
they don't unnecessarily slow down normal test runs.
When running tests with -j100 or so on a large machine, I see this
os.remove call failing semi-regularly. Since it's not really a problem
when the file is already gone, just suppress the error in that case.
Before this patch, `RUNTESTDIR` is added to `PATH`, but `TESTDIR`
isn't.
This doesn't cause any problems, if `run-tests.py` runs in `tests`
directory of Mercurial source tree. In this case, `RUNTESTDIR` should
be equal to `TESTDIR`.
On the other hand, if `run-tests.py` runs in `tests` of third party
tools, commands in that directory should be executed with explicit
`$TESTDIR/` prefix in `*.t` test scripts. This isn't suitable for the
policy "drop explicit $TESTDIR from executables" of Mercurial itself
(see fcb1c7d8c36e).
BTW, fcb1c7d8c36e describes that "$TESTDIR is added to the path" even
though `TESTDIR` isn't added to `PATH` exactly speaking, because
`TESTDIR` and `RUNTESTDIR` weren't yet distinguished from each other
at that time.
This is a one of preparations for issue4677.
Before this patch, there is no way to refer files under `tests` or so
of Mercurial source tree, when `run-tests.py` runs in `tests` of third
party tools. In this case, `TESTDIR` refers the latter `tests`.
This prevents third party tools from using useful tools in Mercurial
source tree (e.g. `contrib/check-code.py`).
This patch adds `RUNTESTDIR` environment variable to refer `tests` of
Mercurial source tree, in which `run-tests.py` now running is
placed. For example, tests of third party tools can refer
`contrib/check-code.py` in Mercurial source tree as
`$RUNTESTDIR/../contrib/check-code.py`.
BTW, for similarity with `TESTDIR` referring `test*s*` directory,
newly added environment variable isn't named as `RUNTEST*S*DIR`. In
addition to it, the corresponded local variable is also named as
`runtestdir`.
This is a one of preparations for issue4677.
Before this patch, `run-tests.py` executes `hghave` by the path
relative to `TESTDIR` (= cwd of `run-tests.py` running).
This prevents third party tools for Mercurial from running
`run-tests.py`, which is placed in `tests` of Mercurial source tree,
in `tests` of own source tree. In such cases, `TESTDIR` refers the
latter `tests`, and `hghave` doesn't exist in it.
This is a one of preparations for issue4677.
When the test engine fails to match output on a line marked with (?),
it will simply continue to the next expected line and try again. This
allows simplifying tests that have either version-specific or
non-fixed behavior, for instance:
$ coin-flip
heads (?)
tails (?)
(There's no form of back-tracking attempted, so optional matches
should be specific.)
We have started to isolate extra usecases for developer-only output
that is not a warning. As the section has the fairly generic name
'devel' it makes sense to tuck them there. As a result, 'all' becomes
a bit misleading so we rename it to 'all-warnings'. This will break
some developer setups but the tests are still fine and developers will
likely spot this change.
Now that http://bugs.python.org/issue24230 is fixed (thanks to Gregory
Smith for that quick response!) we can drop one more ugly hack around
path handling. Tests still pass in 3.5 with this cleaner version, as
well as in 2.6.
This fixes the python3.5 build. We'll presumably want to build our own
helper function or use 2to3 for this on the main source tree, but for
run-tests we want a single-source version that works in 2.6 and 3.5.
Python 2.7.3 on Windows doesn't have os.WIFEXITED, and the test output looked
like this before I interrupted it.
$ ./run-tests.py --local -j2 -t700
EEEEEEEEEEEEEEEEEEEEEE
This also cleans up the mkdtemp code mentioned in the previous patch.
At this point, the remaining callsites of .{en,de)code() are in the
following categories:
Handling escaped lines in .t files
-----------------------------------
It seems eminently reasonable to me for us to declare that .t files
are valid utf-8, and that any escape sequences we see in .t files
should be valid unicode_escape sequences.
Making error text safe for cdata blocks for xml error reports
-------------------------------------------------------------
This is a point where we're already basically screwed, and we're
simply trying to do something "good enough" that the xml output will
be vaguely useful to the user. Punting here seems fine, and we should
probably stick to the same encoding here that we used in the previous
section.
