As a safety precaution, we kill daemons in Test.cleanup(). This is
necessary for a subsequent patch that will reraise KeyboardInterrupt
before killdaemons() runs as part of run().
Reference output should be constant and doesn't need to be computed at
test execution time. We calculate it earlier.
This patch is the first in a mini series that will change how the
TestResult object works.
Test parsing is somewhat complicated. This patch extracts it into its
own function.
The impetus of this patch is folding tsttest() into the TTest class.
Subsequent patches will continue this work until tsttest() no longer
exists.
threadtmp is an implementation detail. We move the cleanup of this
directory to Test.cleanup() and make the variable internal. The
cleanup function will eventually disappear into unittest machinery.
Test.run() can now be executed multiple times on the same Test instance.
This feature is currently unused and there are no plans to implement it.
The main reason for this work was to refactor testtmp, replacements, and
env to be run-time specific as opposed to Test instance specific.
Some implementation details of test execution still live outside of
Test. These include determining what a result means and cleaning up
after the test.
To move to the world where more of this logic can live inside Test or a
derived object, the logic for test execution needs to be refactored.
Specifically, exception trapping and opportunities for result processing
need to be moved into Test.
This patch starts the process by establishing a TestResult class for
holding the results of a test execution. In order to actually use this
class, exception trapping and execution time recording needed to be
moved into Test.run().
testtmp is an implementation detail. It didn't need to be exposed to the
world.
threadtmp is derived from count. It is now created as part of the
constructor and mostly hidden from the outside world.
Environment variables are an implementation detail of how tests are
executed. This patch moves environment variable logic into Test and
completely hides it from the outside.
With this patch, a Test can be executed with two lines: init + run().
Tests are still single-use and take a more arguments to the constructor
than likely necessary. These will get addressed in subsequent patches.
createhgrc() is an implementation detail of how tests are run. It makes
sense to move it into Test.run().
Note that this will cause the test execution time to include the
creation of hgrc. The author does not believe this is a significant
change worth worrying about.
This patch starts the process of moving test-specific variables into the
Test class. The ultimate goal is to be able to instantiate a Test with
minimal arguments and to call run() on it without too much thinking.
This will make it much easier to run tests from other contexts. It will
also enable things like running a test multiple times.
Currently, the state for an individual test is scattered across a number
of functions and variables. This patch begins a process of isolating a
single test's state into instances of a class. It does this by
establishing a new Test base class and child classes for Python tests
and T tests. The class currently has a run() API that proxies into the
existing "runner" functions. Upcoming patches will move the logic for
each test type into the class.
Failing to start a server happens regularly, at least on windows buildbot.
Such a failure often has nothing to do with the test, but with the environment.
But half the test output can change because some data is missing. Therefore this
is worth an extended error message.
Detect the server failure in the diff output because it is most reliable
there. Checking the output only does not show if the server failure was
expected.
Old failure message when server start failed:
Failed test-serve.t: output changed
New message:
Failed test-serve.t: serve failed and output changed
Previously, test paths were assumed to be in the same directory and
wouldn't have a directory component. If a path with a directory
component was specified, it would be filtered out. This change allow
paths to contain directories. This in turn allows tests from other
directories to be executed.
Executing tests in other directories may break assumptions elsewhere in
the testing code. However, on initial glance, things appear to "just
work." This approach of running tests from other directories is
successfully being used at
https://hg.mozilla.org/hgcustom/version-control-tools/file/7085790ff3af/run-mercurial-tests.py
This patch moves the OptionParser population into its own function so
consumers may modify the OptionParser before arguments are evaluated.
This will allow consumers to add custom options, set different defaults,
etc.
Before, arguments were not passed into the optparse.OptionParser
instance and were coming from sys.argv. This patch enables consumers to
define the list of arguments to parse without having to adjust sys.argv.
Convenient when polishing patches and changing details of how they change test
output.
This will probably break in weird ways for revsets with special quoting ... but
it is good enough for run-tests.
Usage example:
yes | ./run-tests.py -li --changed qparent
This extension has always had correctness issues and has been
unmaintained for years. It is now removed in favor of the third-party
hgwatchman which is maintained and appears to be correct.
Users with inotify enabled in their config files will fall back to
standard status performance.
The state "warned" was reported too often. The problem fixed here is that
warnonly was only reset when a line did not match. When there was a line too
much, warnonly remained set.
Fix this by setting more states to warnonly.
More negative testing (testing on result "Failed") has been done this time.
The state "warned" was reported too often. The main problem was that
"False == 0" is true in python. Therefore use an empty string instead of 0
for reporting warn only for a line.
The other problem is fixed in the next patch.
Previously we'd always assume that --with-hg is a script in a user directory,
and would write out a 'python' symlink to the same location. That didn't work
if --with-hg was set to a system installation of hg, e.g. /usr/bin/hg.
Introduce a TMPBINDIR directory which is used to write out the python symlink.
heredoctest.py directory must be in python path to use heredoctest (>>>) in
out-of-tree extension tests like:
$ cd ext/tests
$ python /some/hg/install/tests/run-tests.py test-ext.t
When a glob is unnecessary, now there's a diff output and 'run-tests.py -i'
works for accepting the output.
On windows, some tests which have "passed" currently (with some info lines
printed) will now be reported as "warned". (I recommend to recognize "warned"
by buildbot before applying this patch.)
A test result is recognized as "warned" when the test runner returns the exit
code False. (False is similar to 0, which is reporting a command has run
sucessfully.)
The only difference in display is that the failure message while running writes
"Warning:" instead of "ERROR:". The diff output is the same as when the test
fails. Runing "run-tests.py -i" asks to accept the changed result also for
tests reported as "warned".
When running tests, a "warned" test would look like this:
..
--- xxxx\tests\test-something.t
+++ xxxx\tests\test-something.t.err
@@ -1279,7 +1279,7 @@
$ echo anything
$ hg commit -S -m whatever
committing subrepository s
- committing subrepository s/sbs
+ committing subrepository s/sbs (glob)
warning: something happened
committing subrepository t
$ echo something
Warning: xxxx\tests\test-sOMETHING.t output changed
~.s...s...s..
Reporting a test result as "warned" will be used in following patches.
While running, a test resulting in 'warned' is shown as '~'.
Test results with state warned are listed between the skipped and the failed
tests. Example:
Skipped test-revert-flags.t: missing feature: executable bit
Skipped test-inotify-lookup.t: missing feature: inotify extension support
Warned test-something.t: output changed
Failed test-largefiles.t: output changed
Failed test-subrepo.t: output changed
# Ran 11 tests, 2 skipped, 1 warned, 2 failed.
The test result "warned" will be used in later patches.
