This patch starts the process of removing global variables from
run-tests.py. The goal of this is to make it easier to run tests
differently without having to concern yourself with global state.
As a safety precaution, we kill daemons in Test.cleanup(). This is
necessary for a subsequent patch that will reraise KeyboardInterrupt
before killdaemons() runs as part of run().
Reference output should be constant and doesn't need to be computed at
test execution time. We calculate it earlier.
This patch is the first in a mini series that will change how the
TestResult object works.
Test parsing is somewhat complicated. This patch extracts it into its
own function.
The impetus of this patch is folding tsttest() into the TTest class.
Subsequent patches will continue this work until tsttest() no longer
exists.
threadtmp is an implementation detail. We move the cleanup of this
directory to Test.cleanup() and make the variable internal. The
cleanup function will eventually disappear into unittest machinery.
Test.run() can now be executed multiple times on the same Test instance.
This feature is currently unused and there are no plans to implement it.
The main reason for this work was to refactor testtmp, replacements, and
env to be run-time specific as opposed to Test instance specific.
Some implementation details of test execution still live outside of
Test. These include determining what a result means and cleaning up
after the test.
To move to the world where more of this logic can live inside Test or a
derived object, the logic for test execution needs to be refactored.
Specifically, exception trapping and opportunities for result processing
need to be moved into Test.
This patch starts the process by establishing a TestResult class for
holding the results of a test execution. In order to actually use this
class, exception trapping and execution time recording needed to be
moved into Test.run().
testtmp is an implementation detail. It didn't need to be exposed to the
world.
threadtmp is derived from count. It is now created as part of the
constructor and mostly hidden from the outside world.
Environment variables are an implementation detail of how tests are
executed. This patch moves environment variable logic into Test and
completely hides it from the outside.
With this patch, a Test can be executed with two lines: init + run().
Tests are still single-use and take a more arguments to the constructor
than likely necessary. These will get addressed in subsequent patches.
createhgrc() is an implementation detail of how tests are run. It makes
sense to move it into Test.run().
Note that this will cause the test execution time to include the
creation of hgrc. The author does not believe this is a significant
change worth worrying about.
This patch starts the process of moving test-specific variables into the
Test class. The ultimate goal is to be able to instantiate a Test with
minimal arguments and to call run() on it without too much thinking.
This will make it much easier to run tests from other contexts. It will
also enable things like running a test multiple times.
Currently, the state for an individual test is scattered across a number
of functions and variables. This patch begins a process of isolating a
single test's state into instances of a class. It does this by
establishing a new Test base class and child classes for Python tests
and T tests. The class currently has a run() API that proxies into the
existing "runner" functions. Upcoming patches will move the logic for
each test type into the class.
Failing to start a server happens regularly, at least on windows buildbot.
Such a failure often has nothing to do with the test, but with the environment.
But half the test output can change because some data is missing. Therefore this
is worth an extended error message.
Detect the server failure in the diff output because it is most reliable
there. Checking the output only does not show if the server failure was
expected.
Old failure message when server start failed:
Failed test-serve.t: output changed
New message:
Failed test-serve.t: serve failed and output changed
Previously, test paths were assumed to be in the same directory and
wouldn't have a directory component. If a path with a directory
component was specified, it would be filtered out. This change allow
paths to contain directories. This in turn allows tests from other
directories to be executed.
Executing tests in other directories may break assumptions elsewhere in
the testing code. However, on initial glance, things appear to "just
work." This approach of running tests from other directories is
successfully being used at
https://hg.mozilla.org/hgcustom/version-control-tools/file/7085790ff3af/run-mercurial-tests.py
This patch moves the OptionParser population into its own function so
consumers may modify the OptionParser before arguments are evaluated.
This will allow consumers to add custom options, set different defaults,
etc.
Before, arguments were not passed into the optparse.OptionParser
instance and were coming from sys.argv. This patch enables consumers to
define the list of arguments to parse without having to adjust sys.argv.
Convenient when polishing patches and changing details of how they change test
output.
This will probably break in weird ways for revsets with special quoting ... but
it is good enough for run-tests.
Usage example:
yes | ./run-tests.py -li --changed qparent
This extension has always had correctness issues and has been
unmaintained for years. It is now removed in favor of the third-party
hgwatchman which is maintained and appears to be correct.
Users with inotify enabled in their config files will fall back to
standard status performance.
The state "warned" was reported too often. The problem fixed here is that
warnonly was only reset when a line did not match. When there was a line too
much, warnonly remained set.
Fix this by setting more states to warnonly.
More negative testing (testing on result "Failed") has been done this time.
The state "warned" was reported too often. The main problem was that
"False == 0" is true in python. Therefore use an empty string instead of 0
for reporting warn only for a line.
The other problem is fixed in the next patch.
Previously we'd always assume that --with-hg is a script in a user directory,
and would write out a 'python' symlink to the same location. That didn't work
if --with-hg was set to a system installation of hg, e.g. /usr/bin/hg.
Introduce a TMPBINDIR directory which is used to write out the python symlink.
heredoctest.py directory must be in python path to use heredoctest (>>>) in
out-of-tree extension tests like:
$ cd ext/tests
$ python /some/hg/install/tests/run-tests.py test-ext.t
When a glob is unnecessary, now there's a diff output and 'run-tests.py -i'
works for accepting the output.
On windows, some tests which have "passed" currently (with some info lines
printed) will now be reported as "warned". (I recommend to recognize "warned"
by buildbot before applying this patch.)
A test result is recognized as "warned" when the test runner returns the exit
code False. (False is similar to 0, which is reporting a command has run
sucessfully.)
The only difference in display is that the failure message while running writes
"Warning:" instead of "ERROR:". The diff output is the same as when the test
fails. Runing "run-tests.py -i" asks to accept the changed result also for
tests reported as "warned".
When running tests, a "warned" test would look like this:
..
--- xxxx\tests\test-something.t
+++ xxxx\tests\test-something.t.err
@@ -1279,7 +1279,7 @@
$ echo anything
$ hg commit -S -m whatever
committing subrepository s
- committing subrepository s/sbs
+ committing subrepository s/sbs (glob)
warning: something happened
committing subrepository t
$ echo something
Warning: xxxx\tests\test-sOMETHING.t output changed
~.s...s...s..
Reporting a test result as "warned" will be used in following patches.
While running, a test resulting in 'warned' is shown as '~'.
Test results with state warned are listed between the skipped and the failed
tests. Example:
Skipped test-revert-flags.t: missing feature: executable bit
Skipped test-inotify-lookup.t: missing feature: inotify extension support
Warned test-something.t: output changed
Failed test-largefiles.t: output changed
Failed test-subrepo.t: output changed
# Ran 11 tests, 2 skipped, 1 warned, 2 failed.
The test result "warned" will be used in later patches.
Extend the message with the test name and the approximate line number. (The
line number is the one of the command producing the output.)
Finding the line to fix is easier now.
old message:
......
Info, unnecessary glob: at a/b/c (glob)
..
new message:
......
Info, unnecessary glob in test-example.t (after line 9): at a/b/c (glob)
..
The test result is still pass as before.
When the line does not match because of \ instead of / (on windows), append
(glob) in the expected output.
This allows to rename test-bla.t.err to test-bla.t for getting a correct
output. This worked for other failures like missing (esc), but not here.
Output example (only +- lines of diff):
Before:
- path/with/local/sep
+ path\\with\\local/sep
Now:
- path/with/local/sep
+ path/with/local/sep (glob)
This has several advantages.
* Each match function can return some information to the caller runone (used in
the next patch).
* It is not checked that the line ends in " (glob)" when rematch() returns
false.
* And it looks more readable.
This does not happen when running normal. But when fiddling around with
the test infrastructure, this helps a lot.
Old traceback messge
Exception in thread Thread-7:
Traceback (most recent call last):
...
new traceback message
Exception in thread test-something.t:
Traceback (most recent call last):
...
Before no message was returned to the main thread. No result was registered
and no new thread was started.
This does not happen when running normal. But when fiddling around with
the test infrastructure, this helps a lot.
This extension saves shelved changes using a temporary draft commit,
and bundles the temporary commit and its draft ancestors, then
strips them.
This strategy makes it possible to use Mercurial's bundle and merge
machinery to resolve conflicts if necessary when unshelving, even
when the destination commit or its ancestors have been amended,
squashed, or evolved. (Once a change has been unshelved, its
associated unbundled commits are either rolled back or stripped.)
Storing the shelved change as a bundle also avoids the difficulty
that hidden commits would cause, of making it impossible to amend
the parent if it is a draft commits (a common scenario).
Although this extension shares its name and some functionality with
the third party hgshelve extension, it has little else in common.
Notably, the hgshelve extension shelves changes as unified diffs,
which makes conflict resolution a matter of finding .rej files and
conflict markers, and cleaning up the mess by hand.
We do not yet allow hunk-level choosing of changes to record.
Compared to the hgshelve extension, this is a small regression in
usability, but we hope to integrate that at a later point, once the
record machinery becomes more reusable and robust.
Recently this regexp was only appended when running a python test. When running
a tsttest there was a separate handling for each line type. Simplify and unify
this.
In python2.4, any call to Popen() may attempt to wait on any active
process, and wait is not thread-safe. Make it thread-safe.
See http://bugs.python.org/issue1731717 for details.
When --time is specified, the interruption message of an interrupted test is
extended with the time the test has run
INTERRUPTED: /path/to/tests/test-example.t (after 513 seconds)
When the test run is aborted, a message is printed for each interrupted test.
This is helpful when a test is hanging.
example failure message:
INTERRUPTED: /path/to/tests/test-example.t
The message can appear before or after the line with the number of tests
This is especially important if extensions that use inotify are enabled,
because it's very easy to hit the inotify max_user_instances limit without
this.
os.listdir returns the files in any order. This has to be sorted.
But when given as argument, the user should be allowed to set any order.
This restores the behaviour before 9848a94e2a.
The test-profile test would fail if the user had HGPROF set to another
profiler in their environment. This fix makes the test independent of
that environment variable.
Reverts the previous attempt to fix this, which was not cross platoform.
When glob lines directly match on windows, "/" (and not "\") was output in the
path on the line. No glob matching is necessary in this case.
The test output will look like this (when 5 tests have passed and no 4 has an
unnecessary glob):
...
Info, unnecessary glob: info about some/thing (glob)
..
This happens when a path with "/" as only glob char is matched on a non windows
platform. (Currently one third of all glob matches.)
The slowdown on windows and the speedup on other os are neglectable.
This makes it possible to fix the seed by using for instance
PYTHONHASHSEED=7 ./run-tests.py ...
This can be very convenient when trying to debug problems that are influenced
by hash values. Try different seed values until you find one that triggers the
bad behaviour and then keep that while debugging.
The value 0 will restore default Python behavior and disable randomization.
In normal test mode stdin is closed and hg is thus not interactive. In --debug
mode stdin is inherited from the running console and to the tests, and hg could
thus wait in prompts when running on Windows.
See http://selenic.com/pipermail/mercurial-devel/2013-January/047548.html .
Instead set ui.interactive=False to make Mercurial non-interactive. Other
commands might still work differently in the --debug environment.
This should solve the problem with hg waiting for input but still make it
possible to add --debugger to hg in a test and run run-tests.py with --debug.
3951b91555f7 caused that some kind of interactive debugging no longer was
possible - such as running hg with --debugger in a test run with run-tests.py
--debug .
Python set and dict iteration order is in principle undefined but usually
'quite stable'. Setting PYTHONHASHSEED=random will make the iteration order
more random in Python 2.6.8 and 2.7.3 and where it has been backported. This
can thus help spot dependencies on undefined behaviour and prevent future
problems.
If interrupted while running with "--jobs N", run-tests asynchronously
spewed a bunch of output and backtraces from both the master and
slave processes, leaving the terminal full of goop. This patch makes
it behave more sensibly.
Before: a symlink for python in BINDIR was sometimes created, but it was never
updated when a different Python was used and it was never removed. An invalid
python could thus be left around and used when testing with --local.
Now: the symlink is removed when wrong and created when necessary.
The mechanism for finding the right name (python or python.exe) also had to be
simplified and made more explicit.
The older approach of trying to copy the python executable into the test
directory was doomed to fail.
There remains one weakness with this approach: if you've run "make local",
tests may pick up the wrong extension DLLs from inside the source tree. I
don't know why this happens.
A reasonable workaround for now is to test either using --local or with
a working directory that does not contain built DLLs.
Previously, we used os.spawnvp, which doesn't exist on Windows, and
isn't needed anyway (the command line begins with an absolute path).
We also need a slightly more convoluted way to wait for processes
without specifying an order on Windows, as it lacks os.wait.
Before a55b74d8de3a all crlf occurrences in test output on Windows were simply
changed to lf. In a55b74d8de3a it was replaced by more clever handling in the
.t test runner ... but the .py runner was forgotten and many .py tests were
failing on Windows.
The crlf/lf replacement is now reintroduced in the py test runner.
After d5471ad04cf6 all \r was stripped from output on Windows, and the places
where a \r explicitly was expected it was accepted that it was missing. Ugly
hack.
Instead we now accept that an extra \r might appear at the end of lines on
Windows. That is more to the point and less ugly.
str.splitlines could not be used in 2d839579ce70, but _now_ we would like to
have lines with other line endings than \n.
Some fine occurences of (esc) markup of \r is replaced with multiple lines
ending with '\r (no-eol) (esc)'. That is no win but also no significant loss.
This change makes it possible to drop filtercr.py - _that_ is a win.
MSYS replaces C:/... in arguments with C;... as it interprets the C:/ as a
colon separated POSIX path list. The colon is replaced with ; (path separator
on Windows) according to
http://www.mingw.org/wiki/Posix_path_conversion
So we must not replace \ with / for neither $TESTTMP nor $TESTDIR, but we
have to keep replacing \ with / for the Popen4 call of function hghave. If we
don't do the latter, test-run-tests.t will fail with
$ python run-tests.py --local test-run-tests.t
--- C:\Users\adi\hgrepos\hg-main\tests\test-run-tests.t
+++ C:\Users\adi\hgrepos\hg-main\tests\test-run-tests.t.err
@@ -70,6 +70,7 @@
tested
#else
$ echo skipped
+ skipped
#endif
#if false
An additional tweak in test-ssh.t is needed that globs away an encoded path,
as it can't be translated back to $TESTTMP, because the backslashes in the
output have been already encoded as %5C.
This patch makes test-ssh.t pass in MSYS on Windows.
This is just a short-term workaround for that issue. More work needs to be
done on scmutil.canonpath & friends.
$TMP on Windows is specified to be defined, and it has correct casing, so we
can use that as the default dir for tempfile.mkdtemp on Windows.
This makes it possible to have conditional sections like:
#if windows
$ echo foo
foo
#else
$ echo bar
bar
#endif
The directives and skipped sections are treated like comments, so don't
interleave them with commands and their output.
The parameters to #if are evaluated while preparing the test by passing them
over to hghave. Requirements can thus be negated with 'no-' prefix, and
multiple requirements must all be true to return true.
For test input lines of *.t files starting with ' >>> ', the code block for
' >>> '
609: if l.startswith(' >>> '): # python inlines
610: after.setdefault(pos, []).append(l)
was (unsurprisingly) executed, but because there was an "if" instead of an
"elif" on the condition "l.startswith(' ... ')", program execution proceeded
to line 636
635: elif l.startswith(' '): # results
636: # queue up a list of expected results
637: expected.setdefault(pos, []).append(l[2:])
due to the fact that if l starts with ' >>> ' it also starts with ' '.
The net effect was that python command lines in *.t files were (surprisingly)
also added to the "expected" dict.
This caused no externally observable bad behavior, as the "expected" dict was
not consulted for these lines.
Globbing is usually used for filenames, so on windows it is reasonable and very
convenient that glob patterns accepts '\' or '/' when the pattern specifies
'/'.
Normally changes in tests are reported like this in diffs:
$ cat foo
- a
+ b
Using -i mode lets us update tests when the new results are correct
and/or populate tests with their output.
But with the standard doctest framework, inline Python sections in
tests changes instead result in a big failure report that's unhelpful.
So here, we replace the doctest calls with a simple compile/eval loop.
This adds doctest like syntax to .t files, that can be interleaved with regular
shell code:
$ echo -n a > file
>>> print open('file').read()
a
>>> open('file', 'a').write('b')
$ cat file
ab
The syntax is exactly the same as regular doctests, so multiline statements
look like this:
>>> for i in range(3):
... print i
0
1
2
Each block has its own context, i.e.:
>>> x = 0
>>> print x
0
$ echo 'foo'
foo
>>> print x
will result in a NameError.
Errors are displayed in standard doctest format:
>>> print 'foo'
bar
--- /home/idan/dev/hg/default/tests/test-test.t
+++ /home/idan/dev/hg/default/tests/test-test.t.err
@@ -2,3 +2,16 @@
> >>> print 'foo'
> bar
> EOF
+ **********************************************************************
+ File "/tmp/tmps8X_0ohg-tst", line 1, in tmps8X_0ohg-tst
+ Failed example:
+ print 'foo'
+ Expected:
+ bar
+ Got:
+ foo
+ **********************************************************************
+ 1 items had failures:
+ 1 of 1 in tmps8X_0ohg-tst
+ ***Test Failed*** 1 failures.
+ [1]
As for the implementation, it's quite simple: when the test runner sees a line
starting with '>>>' it converts it, and all subsequent lines until the next
line that begins with '$' to a 'python -m heredoctest <<EOF' call with the
proper heredoc to follow. So if we have this test file:
>>> for c in 'abcd':
... print c
a
b
c
d
$ echo foo
foo
It gets converted to:
$ python -m heredoctest <<EOF
> >>> for c in 'abcd':
> ... print c
> a
> b
> c
> d
> EOF
$ echo foo
foo
And then processed like every other test file by converting it to a sh script.
It's desirable to run some tests all the time, for example
test-check-pyflakes.t and test-check-code-hg.py. This allows passing
--whitelist as a path to a file (flag can be specified more than once)
which contains a list of files to whitelist. Whitelisted tests are run
even if they're blacklisted or wouldn't match a --keyword test
run. For example, to do a quick test of usehttp2, one can now do
$ cat > test-whitelist <<EOF
> test-check-pyflakes.t
> test-check-code-hg.py
> EOF
$ (cd tests && ./run-tests.py --extra-config-opt 'ui.usehttp2=true'
> -k http -j 8 --whitelist test-whitelist)
and have all http-specific tests run as well as the two code linters.
hg.bat expects to live in pythonxx/scripts and the python interpreter
to be in pythonxx. run-tests.py file layout is a little different and
python location must be fixed.
