The test runner has the ability to stop on first error.
Tests are executed in new Python threads. The test runner starts new
threads when it has capacity to do so. Before this patch, the "stop on
first error" logic would return immediately from the "run tests"
function, without waiting on test threads to complete. There was thus
a race between the test runner thread doing cleanup work and the test
thread performing activity. For example, the test thread could be in
the middle of executing a test shell script and the test runner
could remove the test's temporary directory. Depending on timing, this
could result in any number of output from the test runner.
This patch eliminates the race condition by having the test runner
explicitly wait for test threads to complete before continuing.
I discovered this issue as I modified the test harness in a subsequent
patch and was reliably able to tickle the race condition.
The xunit writer was passing a str to a minidom API. An implicit
.decode('ascii') was performed somewhere, causing UnicodeDecodeError
if test output contained non-ascii sequences.
This patch converts test output to utf-8 before passing it to minidom.
We use the "replace" strategy to ensure invalid utf-8 sequences get
munged into �.
Before this patch, "test-run-tests.t" doesn't test "run-tests.py" with
"--pure", even if outer "run-tests.py" is executed with it.
This patch uses not "HG_RUN_TESTS_PURE" but "HGTEST_RUN_TESTS_PURE",
because "HG_" prefixed environments are forcibly dropped in "_getenv()".
This is also useful to run "run-tests.py" successfully by
"run-tests.py --pure" on Windows without any compilation tools (like
VisualStudio).
Avoid spending too much time adding (glob) after running run-tests -i. This
doesn't handle all cases but it helps.
The run-tests tests add a bit of escaping of trailing (glob) in the output to
avoid interference from the outer test runner.
The regexp for matching the output lines contains a group for making multiline
substitute in a way that works with Python before 2.7.
test-largefiles-update.t, test-subrepo.t, test-tag.t, and
test-rename-dir-merge.t still warn about no result returned because of
unnecessary globs that test-check-code-hg.t wants, relating to output for
pushing to, pulling from and moving X to Y.
simplejson produces slightly different output from the built-in json
module, specifically:
* It uses 0.000 instead of 0.0000
* It likes to put a trailing space after a comma
This change works around both of those variations.
Without this change, --first causes currently-running tests to explode
in violent and surprising ways when their temporary directory gets
cleaned up. Now we just suppress failure messages from non-first
failures when running in --first mode.
This patch adds up a 'cuser' and 'csys'(cputime) info in report.json file
which generated when --json is enabled while testing.
Now the new format of report.json file is as below.
testreport ={
"test-success.t": {
"csys": "1.041",
"cuser": "1.041",
"result": "success",
"time": "2.041"
}
"test-failure.t": {
"csys": "1.041",
"cuser": "1.041",
"result": "failure",
"time": "4.430"
}
"test-skip.t": {
"csys": "1.041",
"cuser": "1.041",
"result": "skip",
"time": "3.754"
}
}
This patch added a new functionality '--json'. While testing, if '--json'
is enabled then test result data gets stored in newly created "report.json"
file in the following format.
testreport ={
"test-success.t": {
"result": "success",
"time": "2.041"
}
"test-failure.t": {
"result": "failure",
"time": "4.430"
}
"test-skip.t": {
"result": "skip"
"time": "3.754"
}
}
Otherwise, if '--json' is enabled but json module was not installed then it
will raise an error "json module not installed".
This "report.json" file will further accessed by html/javascript file for
graph usage.
The Jenkins CI system understands xunit reports natively, so this will
be helpful for anyone that wants to use Jenkins for testing hg or
extensions that use run-tests.py for their testing.
This preserves the current behavior that excludes ignored or skipped
tests from the number of tests run, except when tests are ignored due
to the --retest flag.
This adds a fair amount of overall instability in the enclosing .t. As
such, this is performed in its own commit, and a test for --keyword on
run-tests.t will be added in a followup change.
As our tests execute in child processes, this patch uses os.times()
module in replace of time.time() module to provide additional info like
user time and system time spent by child's processes along with real elapsed
time taken by a process.
There is one limitation of this patch. It can work only for Linux users and
not for Windows.
"os.times" module returns a 5-tuple of a floaing point numbers.
1) User time
2) System time
3) Child's user time
4) Child's system time
5) Ellapsed real time
On Windows, only the first two items are filled, the others are zero.
Therefore, below test cases does not break on Windows but instead gives the
zero value.
Otherwise diff may get separated from the corresponding prompt by
other threads. This required moving the interactive prompting from one
helper method to another.
After d5471ad04cf6 all \r was stripped from output on Windows, and the places
where a \r explicitly was expected it was accepted that it was missing. Ugly
hack.
Instead we now accept that an extra \r might appear at the end of lines on
Windows. That is more to the point and less ugly.
This makes it possible to have conditional sections like:
#if windows
$ echo foo
foo
#else
$ echo bar
bar
#endif
The directives and skipped sections are treated like comments, so don't
interleave them with commands and their output.
The parameters to #if are evaluated while preparing the test by passing them
over to hghave. Requirements can thus be negated with 'no-' prefix, and
multiple requirements must all be true to return true.
Normally changes in tests are reported like this in diffs:
$ cat foo
- a
+ b
Using -i mode lets us update tests when the new results are correct
and/or populate tests with their output.
But with the standard doctest framework, inline Python sections in
tests changes instead result in a big failure report that's unhelpful.
So here, we replace the doctest calls with a simple compile/eval loop.
