As far as I can tell, default values are evaluated far earlier on
Python 3.6 than 2.7, meaning we've got to be more careful about when
we read the defaults dictionary for the kwarg defaults. Sigh.
With this change, test-run-tests.t is significantly less of a mess
under 3.6.
Add `--bisect-repo` flag which accepts a different repo to bisect.
3rd party extensions may reuse `run-tests.py` from core to run tests. Test
failure could be caused by either a core hg change or the 3rd party
extension code itself. Having a way to specify which repo to bisect is
useful.
Differential Revision: https://phab.mercurial-scm.org/D578
This makes `run-tests.py -l test-run-tests.t` 23 seconds faster on my
laptop. Inside the test, `$ rt --known-good-rev=0 test-bisect.t` took 24.9
seconds before, and 1.2 seconds after.
Differential Revision: https://phab.mercurial-scm.org/D576
7340465bd added multiple test cases support. The latter has a problem -
output lines cannot be made conditional with `#if`:
```
# COUNTEREXAMPLE: DOES NOT WORK
#testcases A B
$ command-foo
common ouput
#if A
A's ouput
#else
B's ouput
#endif
common ouput
```
That's not trivial to fix (even if it works in test, `run-tests.py -i` may
be suboptimal because diff algorithm does not know how to skip the `#if`
lines, even if it does, it may have trouble figuring out whether a changed
line belongs to inside a `#if` block or outside).
Matching output lines conditionally is useful. 4eec2f04a added per-line
condition support for hghave. This patch extends that to also support test
case names.
Differential Revision: https://phab.mercurial-scm.org/D466
Some of the recent work on run-tests.py didn't work on Python 3. This
fixes the regressions.
Differential Revision: https://phab.mercurial-scm.org/D270
Essentially, these were acting as a verbose (?) flag, since they weren't being
dropped when required. Foozy has a nice description [1]. Basically, a couple
more places needed to check the features before treating it as optional.
I don't like how test-run-tests.py had to be hacked, but _hghave() can't be made
a static method. The test change was a change while developing `debugssl`,
prior to tightening up the cases where the message is printed, that this fix
would have caught.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101941.html
47985606a0ae (run-tests: check if stream is a tty before using color,
2017-07-18) made the check redundant but forgot to remove it.
Differential Revision: https://phab.mercurial-scm.org/D116
At first I updated the color field of the 'options' object (from the
CLI parser), but then I decided to put it directly on the test case
object itself to avoid mutating the shared object (even though all
tests would have the same value).
Differential Revision: https://phab.mercurial-scm.org/D114
Previous implementation (ccf66c9bf5af) checked only if sys.stderr was a tty
which was less general. Also makes sure that colors is never used if pygments is
not available, irrespective of --color flag value.
e80041832e introduced support to color the output of tests but used pygments
without checking whether it's installed or not. That breaks test-run-tests.t for
machines which don't have pygments installed. This patch conditionalize the
color test in test-run-tests.t and also add a check to make sure pygments is
installed before using that.
More Windows sadness. Maybe someone can figure out how to make win32 color
work, but I think we avoid importing stuff from the mercurial package in this
module. On the plus side, this conditionalizes away a test failure.
The output of run-tests has no formatting by default, which hampers readability.
This patch colors the diff output when pygments is available. To avoid coloring
even when pygments is available, use --color never.
Since os.environ may be overridden in run-tests.py, several important
variables such as PATH weren't restored.
I don't like the idea of using the system hg *by default* because the
executable and the configs are out of our control. But I don't mind as
long as the tests pass.
Update the code to correctly anchor the expression on the end of the name, to
require that the entire name match this expression. It was already anchored at
the start by using re.match(), but this does not anchor it at the end.
Update the syshgenv function to attempt to completely restore the original
environment, rather than only updating a few specific variables. run_tests.py
now generates a shell script that can be used to restore the original
environment, and syshgenv sources it.
This is a bit more complicated than the previous code, but should do a better
job of running the system hg in the correct environment.
I've tested it on Linux using python 2.x, but let me know if it causes issues
in other environments. I'm not terribly familiar with how the tests get run on
Windows, for instance, and how the environment needs to be updated there.
When running the tests, define ORIG_PATH and ORIG_PYTHONPATH environment
variables that contain the original contents of PATH and PYTHONPATH, before
they were modified by run-tests.py
This will make it possible for tests to refer to the original contents of these
variables if necessary. In particular, this is necessary for invoking the
correct version of hg for examining the local repository (the mercurial
repository itself, not the temporary test repositories). Various tests examine
the local repository to check the file lists and contents of commit messages.
The "#testcases" feature introduced by 250afd791085 has issues with "-i"
because "-i" uses "test.name.endswith('.t')" to test if a test is .t or not.
test.name could now be something like "test-foo.t (caseA)" so the above
endswith test is no longer valid.
This patch changes the test to use "self.path" which won't have the issue.
The .t file is both test input and reference output. They should always
match. However we have different code paths to read reference output
(Test.__init__ -> Test.readrefout) and test input (TTest._run) so they might
be inconsistent if somethings change the file between those two functions.
This patch assigns "lines" read by "_run" back to "_refout" if "_refout" is
not None (with --debug, see Test.readrefout) so reference output and test
input will always match.
The race condition is like:
1. run-tests.py reads test-a.t as reference output, content A
2. run-tests.py runs the test (which could be content B, another race
condition fixed by the next patch, but assume it's content A here)
3. something changes test-a.t to content C
4. run-tests.py compares test output (content D) with content A
5. with "-i", run-tests.py prompts diff(A, D), while the file has content
C instead of A at this time
This patch detects the above case and tell the user to rerun the test if
they want to apply test changes.
The only call site called addFailure before raising, which is
exactly what the failure exception handler does. So this
complexity is not needed.
We have test coverage of this "server failed to start" scenario
and nothing appeared to change.
The previous changeset removed the last caller of addWarn(). So,
we rip out that method and all the code related to tracking warned
tests in the results system.
There was even a comment saying we may want to fold warned tests into
the "failed" state, which is what the previous changeset did.
We would raise this if a test didn't return a result code. AFAICT
this can only occur if there is a logic error in the test harness
itself.
I don't think it is worth the code complexity to distinguish this
failure scenario from a regular test failure.
When hghave testing goes awry, the output order was changing on Windows.
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -920,10 +920,10 @@
> EOF
> done
$ rt -j 2
- ....
+ ....skipped: unknown feature: notarealhghavefeature\r (esc)
+
+
# Ran 5 tests, 0 skipped, 0 warned, 0 failed.
- skipped: unknown feature: notarealhghavefeature
-
$ cd ..
$ rm -rf broken
Since 'skipped: unknown feature: notarealhghavefeature\n\n' is printed to stdout
and the rest to stderr, it seems like maybe stdio isn't line buffered on
Windows. When a program exits, stdout is flushed before stderr[1].
[1] https://blogs.msdn.microsoft.com/oldnewthing/20060519-09/?p=31133
