This patch starts the process of removing global variables from
run-tests.py. The goal of this is to make it easier to run tests
differently without having to concern yourself with global state.
As a safety precaution, we kill daemons in Test.cleanup(). This is
necessary for a subsequent patch that will reraise KeyboardInterrupt
before killdaemons() runs as part of run().
Reference output should be constant and doesn't need to be computed at
test execution time. We calculate it earlier.
This patch is the first in a mini series that will change how the
TestResult object works.
Test parsing is somewhat complicated. This patch extracts it into its
own function.
The impetus of this patch is folding tsttest() into the TTest class.
Subsequent patches will continue this work until tsttest() no longer
exists.
threadtmp is an implementation detail. We move the cleanup of this
directory to Test.cleanup() and make the variable internal. The
cleanup function will eventually disappear into unittest machinery.
Test.run() can now be executed multiple times on the same Test instance.
This feature is currently unused and there are no plans to implement it.
The main reason for this work was to refactor testtmp, replacements, and
env to be run-time specific as opposed to Test instance specific.
Some implementation details of test execution still live outside of
Test. These include determining what a result means and cleaning up
after the test.
To move to the world where more of this logic can live inside Test or a
derived object, the logic for test execution needs to be refactored.
Specifically, exception trapping and opportunities for result processing
need to be moved into Test.
This patch starts the process by establishing a TestResult class for
holding the results of a test execution. In order to actually use this
class, exception trapping and execution time recording needed to be
moved into Test.run().
testtmp is an implementation detail. It didn't need to be exposed to the
world.
threadtmp is derived from count. It is now created as part of the
constructor and mostly hidden from the outside world.
Environment variables are an implementation detail of how tests are
executed. This patch moves environment variable logic into Test and
completely hides it from the outside.
With this patch, a Test can be executed with two lines: init + run().
Tests are still single-use and take a more arguments to the constructor
than likely necessary. These will get addressed in subsequent patches.
createhgrc() is an implementation detail of how tests are run. It makes
sense to move it into Test.run().
Note that this will cause the test execution time to include the
creation of hgrc. The author does not believe this is a significant
change worth worrying about.
This patch starts the process of moving test-specific variables into the
Test class. The ultimate goal is to be able to instantiate a Test with
minimal arguments and to call run() on it without too much thinking.
This will make it much easier to run tests from other contexts. It will
also enable things like running a test multiple times.
Currently, the state for an individual test is scattered across a number
of functions and variables. This patch begins a process of isolating a
single test's state into instances of a class. It does this by
establishing a new Test base class and child classes for Python tests
and T tests. The class currently has a run() API that proxies into the
existing "runner" functions. Upcoming patches will move the logic for
each test type into the class.
Failing to start a server happens regularly, at least on windows buildbot.
Such a failure often has nothing to do with the test, but with the environment.
But half the test output can change because some data is missing. Therefore this
is worth an extended error message.
Detect the server failure in the diff output because it is most reliable
there. Checking the output only does not show if the server failure was
expected.
Old failure message when server start failed:
Failed test-serve.t: output changed
New message:
Failed test-serve.t: serve failed and output changed
Previously, test paths were assumed to be in the same directory and
wouldn't have a directory component. If a path with a directory
component was specified, it would be filtered out. This change allow
paths to contain directories. This in turn allows tests from other
directories to be executed.
Executing tests in other directories may break assumptions elsewhere in
the testing code. However, on initial glance, things appear to "just
work." This approach of running tests from other directories is
successfully being used at
https://hg.mozilla.org/hgcustom/version-control-tools/file/7085790ff3af/run-mercurial-tests.py
This patch moves the OptionParser population into its own function so
consumers may modify the OptionParser before arguments are evaluated.
This will allow consumers to add custom options, set different defaults,
etc.
Before, arguments were not passed into the optparse.OptionParser
instance and were coming from sys.argv. This patch enables consumers to
define the list of arguments to parse without having to adjust sys.argv.
Convenient when polishing patches and changing details of how they change test
output.
This will probably break in weird ways for revsets with special quoting ... but
it is good enough for run-tests.
Usage example:
yes | ./run-tests.py -li --changed qparent
This extension has always had correctness issues and has been
unmaintained for years. It is now removed in favor of the third-party
hgwatchman which is maintained and appears to be correct.
Users with inotify enabled in their config files will fall back to
standard status performance.
The state "warned" was reported too often. The problem fixed here is that
warnonly was only reset when a line did not match. When there was a line too
much, warnonly remained set.
Fix this by setting more states to warnonly.
More negative testing (testing on result "Failed") has been done this time.
The state "warned" was reported too often. The main problem was that
"False == 0" is true in python. Therefore use an empty string instead of 0
for reporting warn only for a line.
The other problem is fixed in the next patch.
Previously we'd always assume that --with-hg is a script in a user directory,
and would write out a 'python' symlink to the same location. That didn't work
if --with-hg was set to a system installation of hg, e.g. /usr/bin/hg.
Introduce a TMPBINDIR directory which is used to write out the python symlink.
heredoctest.py directory must be in python path to use heredoctest (>>>) in
out-of-tree extension tests like:
$ cd ext/tests
$ python /some/hg/install/tests/run-tests.py test-ext.t
When a glob is unnecessary, now there's a diff output and 'run-tests.py -i'
works for accepting the output.
On windows, some tests which have "passed" currently (with some info lines
printed) will now be reported as "warned". (I recommend to recognize "warned"
by buildbot before applying this patch.)
A test result is recognized as "warned" when the test runner returns the exit
code False. (False is similar to 0, which is reporting a command has run
sucessfully.)
The only difference in display is that the failure message while running writes
"Warning:" instead of "ERROR:". The diff output is the same as when the test
fails. Runing "run-tests.py -i" asks to accept the changed result also for
tests reported as "warned".
When running tests, a "warned" test would look like this:
..
--- xxxx\tests\test-something.t
+++ xxxx\tests\test-something.t.err
@@ -1279,7 +1279,7 @@
$ echo anything
$ hg commit -S -m whatever
committing subrepository s
- committing subrepository s/sbs
+ committing subrepository s/sbs (glob)
warning: something happened
committing subrepository t
$ echo something
Warning: xxxx\tests\test-sOMETHING.t output changed
~.s...s...s..
Reporting a test result as "warned" will be used in following patches.
While running, a test resulting in 'warned' is shown as '~'.
Test results with state warned are listed between the skipped and the failed
tests. Example:
Skipped test-revert-flags.t: missing feature: executable bit
Skipped test-inotify-lookup.t: missing feature: inotify extension support
Warned test-something.t: output changed
Failed test-largefiles.t: output changed
Failed test-subrepo.t: output changed
# Ran 11 tests, 2 skipped, 1 warned, 2 failed.
The test result "warned" will be used in later patches.
Extend the message with the test name and the approximate line number. (The
line number is the one of the command producing the output.)
Finding the line to fix is easier now.
old message:
......
Info, unnecessary glob: at a/b/c (glob)
..
new message:
......
Info, unnecessary glob in test-example.t (after line 9): at a/b/c (glob)
..
The test result is still pass as before.
When the line does not match because of \ instead of / (on windows), append
(glob) in the expected output.
This allows to rename test-bla.t.err to test-bla.t for getting a correct
output. This worked for other failures like missing (esc), but not here.
Output example (only +- lines of diff):
Before:
- path/with/local/sep
+ path\\with\\local/sep
Now:
- path/with/local/sep
+ path/with/local/sep (glob)
This has several advantages.
* Each match function can return some information to the caller runone (used in
the next patch).
* It is not checked that the line ends in " (glob)" when rematch() returns
false.
* And it looks more readable.
This does not happen when running normal. But when fiddling around with
the test infrastructure, this helps a lot.
Old traceback messge
Exception in thread Thread-7:
Traceback (most recent call last):
...
new traceback message
Exception in thread test-something.t:
Traceback (most recent call last):
...
Before no message was returned to the main thread. No result was registered
and no new thread was started.
This does not happen when running normal. But when fiddling around with
the test infrastructure, this helps a lot.
This extension saves shelved changes using a temporary draft commit,
and bundles the temporary commit and its draft ancestors, then
strips them.
This strategy makes it possible to use Mercurial's bundle and merge
machinery to resolve conflicts if necessary when unshelving, even
when the destination commit or its ancestors have been amended,
squashed, or evolved. (Once a change has been unshelved, its
associated unbundled commits are either rolled back or stripped.)
Storing the shelved change as a bundle also avoids the difficulty
that hidden commits would cause, of making it impossible to amend
the parent if it is a draft commits (a common scenario).
Although this extension shares its name and some functionality with
the third party hgshelve extension, it has little else in common.
Notably, the hgshelve extension shelves changes as unified diffs,
which makes conflict resolution a matter of finding .rej files and
conflict markers, and cleaning up the mess by hand.
We do not yet allow hunk-level choosing of changes to record.
Compared to the hgshelve extension, this is a small regression in
usability, but we hope to integrate that at a later point, once the
record machinery becomes more reusable and robust.
Recently this regexp was only appended when running a python test. When running
a tsttest there was a separate handling for each line type. Simplify and unify
this.
In python2.4, any call to Popen() may attempt to wait on any active
process, and wait is not thread-safe. Make it thread-safe.
See http://bugs.python.org/issue1731717 for details.
When --time is specified, the interruption message of an interrupted test is
extended with the time the test has run
INTERRUPTED: /path/to/tests/test-example.t (after 513 seconds)
When the test run is aborted, a message is printed for each interrupted test.
This is helpful when a test is hanging.
example failure message:
INTERRUPTED: /path/to/tests/test-example.t
The message can appear before or after the line with the number of tests
