The problem in commandserver was addressed by 766cfbe766dc, but it is tricky
to reuse ui.nontty option to disable echo back. Instead, this patch introduces
new option to enable echoing of prompt response.
Prompt echoing is changed to be off by default, which should avoid possible
breakage of output parsing in user scripts.
For now, this option is undocumented because it exists for internal use.
Diff generation didn't really fail, it recognized that an hg serve server has
failed to start, and thus skipped the diff generation intentionally.
The most common reason for a server to fail to start is that the port was
already in use, so output HGPORT as well, to help finding it (since pgrep -f
'hg serve' is not sufficient, if the command line is something like 'hg -R main
serve')
Without this change, --first causes currently-running tests to explode
in violent and surprising ways when their temporary directory gets
cleaned up. Now we just suppress failure messages from non-first
failures when running in --first mode.
This patch adds up a 'cuser' and 'csys'(cputime) info in report.json file
which generated when --json is enabled while testing.
Now the new format of report.json file is as below.
testreport ={
"test-success.t": {
"csys": "1.041",
"cuser": "1.041",
"result": "success",
"time": "2.041"
}
"test-failure.t": {
"csys": "1.041",
"cuser": "1.041",
"result": "failure",
"time": "4.430"
}
"test-skip.t": {
"csys": "1.041",
"cuser": "1.041",
"result": "skip",
"time": "3.754"
}
}
This patch added a new functionality '--json'. While testing, if '--json'
is enabled then test result data gets stored in newly created "report.json"
file in the following format.
testreport ={
"test-success.t": {
"result": "success",
"time": "2.041"
}
"test-failure.t": {
"result": "failure",
"time": "4.430"
}
"test-skip.t": {
"result": "skip"
"time": "3.754"
}
}
Otherwise, if '--json' is enabled but json module was not installed then it
will raise an error "json module not installed".
This "report.json" file will further accessed by html/javascript file for
graph usage.
Ideally, when using -j and -i together, when a prompt comes up, we'd
like all other output to wait (but testing to continue!). This gets
other output to wait by adding back a bunch of the locking that
formerly existed. We switch to a recursive lock to deal with the
restructuring due to unittest compatibility.
Running tests continue to run, but now the scheduler doesn't schedule
any new tasks while waiting at a prompt because no task slots become
available due to result output happening in the thread and
blocking on the iolock.
The Jenkins CI system understands xunit reports natively, so this will
be helpful for anyone that wants to use Jenkins for testing hg or
extensions that use run-tests.py for their testing.
This preserves the current behavior that excludes ignored or skipped
tests from the number of tests run, except when tests are ignored due
to the --retest flag.
As our tests execute in child processes, this patch uses os.times()
module in replace of time.time() module to provide additional info like
user time and system time spent by child's processes along with real elapsed
time taken by a process.
There is one limitation of this patch. It can work only for Linux users and
not for Windows.
"os.times" module returns a 5-tuple of a floaing point numbers.
1) User time
2) System time
3) Child's user time
4) Child's system time
5) Ellapsed real time
On Windows, only the first two items are filled, the others are zero.
Therefore, below test cases does not break on Windows but instead gives the
zero value.
Before this patch, 'detailed' is used as the default of '[ui]
mergemarkers'. This embeds non-ASCII characters in tags, branches,
bookmarks, author and/or commit descriptions into merged files in the
encoding specified by '--encoding' global option, 'HGENCODING' or
other locale setting environment variables.
But, if files to be merged use another encoding, this behavior breaks
consistency of encoding in merged files.
For example, ISO-2022-JP or EUC-JP are sometimes used as the file
encoding for Japanese characters, because of historical and/or
environmental reasons, even though UTF-8 or Shift-JIS are ordinarily
used as the terminal encoding.
This can't be resolved automatically, because Mercurial doesn't aware
encoding of managed files.
This patch uses 'basic' as the default of '[ui] mergemarkers' to avoid
embedding encoding sensitive characters for safety.
This patch puts '[ui] mergemarkers = detailed' into default hgrc file
for tests, to reduce changes for tests in this patch.
Otherwise diff may get separated from the corresponding prompt by
other threads. This required moving the interactive prompting from one
helper method to another.
This patch fixes a regression recently introduced by a refactoring.
Previously when failure occurs while testing with '--interactive' was enable,
it didn't prompt user by asking whether he wants to accept this failure
changes or not.
This was happening beacuse of the 'if' condition
if ret or not self._options.interactive or \
not os.path.exists(test.errpath):
Everytime failure occurs, this condition gets true and returns back even
when '--interactive' is enabled. This condition don't led the function to
execute further, which consist the '--interactive' functionality.
Now, on failure with '--interactive' enabled, it prompts user whether he wants
to accepts failure changes or not.
If yes then test gets passed and returns true, else test gets failed.
On every failure, results gets stored in "self.failures.append((test, reason))"
But if failure changes accepted by user then test must get "pop out" from
failed test list.
This patch fixes a regression recently introduced by a refactoring. (see
a2d588d82fa2 and about 200 previous changesets from Gregory Szorc)
While retesting, that is when '--retest' is enabled, only failure tests run
and others either skipped or ignored.
During retesting, "result.testsRun" holds the count of failure test that has
run. But as while printing output, we have subtracted the skipped and ignored
count from "result.testsRun". Therefore, to make the count remain
the same, we need to add skipped and ignored count before printing.
When '--retest' option is enabled then skipped test should not produce 'i' mark.
This fixes a regression introduced by a2d588d82fa2 and about 200 previous
changesets from Gregory Szorc.
This patch fixes a regression recently introduced by a refactoring (see
a2d588d82fa2 and about 200 previous changesets from Gregory Szorc). '!' mark
denotes that the test gets failed while testing.
This patch fixes a regression recently introduced by a refactoring (see
a2d588d82fa2 and about 200 previous changesets from Gregory Szorc). It produce
an error message everytime with a test filename which gets fail while testing
except at one condition when '--nodiff' option is enabled.
Earlier refactoring of run-tests.py accidentally broke --interactive
and external diff generation by not having .err files written before
they are consulted. This patch fixes that.
TestResult is the thing that captures all our test results. It's logical
for diff viewing to be handled there and not inside Test.
While writing this patch, it was discovered that Test.fail() was
printing redundant test result output. This has been removed.
Arguments controlling diffs have been removed from Test.__init__.
All options are now passed as arguments and we no longer need options.
This enables us to instantiate Test from "plain old data" types. Since
options must be given as arguments, it also makes people think harder
about adding things that may not belong in Test. This will help ensure a
proper separation of responsibility going forward.
Instead of making port numbers derived from count and a global initial
port, we now pass a start port to Test.__init__ and do the calculation
at a higher level.
Upcoming patches will move the execution of tests to separate processes.
To facilitate this, it makes sense to move logic out of Test.
Furthermore, test filtering is logically the domain of the test runner,
not the test itself.
This patch interrupts our mini series of factoring options into named
arguments because filtering consults many options and it is easier to
move this logic out of Test sooner so we don't have to introduce
arguments at all.
We no longer access any attributes on TestRunner besides options, so we
stop passing a TestRunner to Test.__init__ and now pass the options
data structure instead.
Subsequent patches will move accessed options attributes into named
arguments.
This patch starts a mini series of moving arguments to Test.__init__
from semi-complex data structures (such as the command options) to named
arguments. This will allow Test instances to be more easily instantiated
from other contexts. This improves the ability to run Mercurial tests in
new and different environments.
Previously, a Test's path came from the base directory of all tests and
a filename leaf. There is not a strong reason why an absolute test path
can not be specified.
This change isn't strictly necessary. But it does enable scenarios such
as more easily running tests from multiple, non-sibling directories.
TestResult has facilities for recording when tests start and stop. It
makes sense to move execution time recording into TestResult.
In addition, output generation is being moved into TestResult, a class
that has a concept of an output buffer (and a lock).
Now that we execute all tests via unittest, the MercurialTest wrapper is
not necessary. This patches moves the logic from MercurialTest into Test
and makes Test a child of unittest.TestCase.
Now that unittest mode is functionally equivalent to the default mode,
we switch the default execution mode to unittest and remove the choice
of different execution modes.
Unlike unittest's defaults, our result formatter does not print stack
traces. Here, we change TestResult.addFailure() to be compatible with
the existing/default execution mode.
rmtree() may fail under certain conditions. We ignore failures at the
individual test level because they can interfere with test execution.
Furthermore, we'll reattempt deletion in the high-level test runner
after all tests have finished.
Note that this patch is not a code refactor like most of the patches
before it. This change logically makes sense given the execution
behavior of the tests.
There is an execution mode on run-tests.py that stops after the first
failure. unittest mode was previously not obeying this option. This
patch fixes that.
testtmp is now a member variable of our test class. It's value is
computed during instance creation and the directory is managed via the
lifetime of the test object.
The unittest way of recording a skipped test is to raise an exception
(at least with modern unittest implementation). We change Test to raise
a SkipTest when operating in unittest mode.
This does prevent some "tear down" activities from running in unittest
mode. This will be fixed in subsequent patches. Since unittest mode
is experimental, this should be OK.
Simply wrapping TestCase.run() is not sufficient for robust results
reporting because unittest in Python 2.4 does not know about things
like skipped tests and reports them as success or failures instead of
skips.
We will reimplement TestCase.run() with knowledge and semantics present
in modern Python releases.
unittest does its own printing of progress indicators as part of
TestResult. So, there is no need to print them when running in unittest
mode.
This will fix the double output of progress indicators that had been
occurring in unittest mode.
We now have a custom unittest.TestSuite implementation that uses
_executetests() and thus knows how to execute tests concurrently.
Running tests in --unittest mode will use this TestSuite.
Since the TestSuite handles concurrency, the warnings around --jobs and
--loop have been removed.
Test instances (not paths) are passed into _executetests(). This means
the logic for instantiating Test instances has been moved outside of
_executetests(). This was done because an upcoming patch will reuse the
logic in _executetests().
As part of this change, test counts are no longer managed by
_executetests().
Tests executing in unittest mode behave a little differently. For
example, they report their results to a TestResult rather than just
printing. This patch paves the way for better integration of Test into
the unittest framework.
Previously, our unittest wrapper didn't report results properly. We now
properly report failures.
We had to rename the local variable to prevent "t" from being
overwritten in the local scope.
The unittest package in Python's standard library provides an almost
universal mechanism for defining and running tests. This patch starts
the process of making run-tests.py talk to it.
The main benefit of speaking unittest is that this will enable
Mercurial's tests to be more easily consumed by other testing tools,
like nose. This is useful for 3rd party extensions having their own
test infrastructure, for example.
Running tests in unittest mode will not result in completely sane
behavior until the unittest mode is made the default execution mode.
Expect things like double printing of output until support stabilizes.
One must choose between ``"y yes".split()`` and ``('y', 'yes')``. I choose the
later.
The feature still crash when you answer "yes" to use it. But at least, the
prompt itself works.
The `--interactive` flag workis by overwriting the original test file by its
`.err` version. So we need to write it before calling `self.fail`. Otherwise the
`.err` file does not exists and `--interactive` is ignored.
We can move that block code around because it is dedicated to write changed
output and we moves it in the try-except dedicated to handling changed output.
Note that the flog is still badly broken after this change. But at least it crash
instead of being ignored.
This patch starts the process of removing global variables from
run-tests.py. The goal of this is to make it easier to run tests
differently without having to concern yourself with global state.
As a safety precaution, we kill daemons in Test.cleanup(). This is
necessary for a subsequent patch that will reraise KeyboardInterrupt
before killdaemons() runs as part of run().
Reference output should be constant and doesn't need to be computed at
test execution time. We calculate it earlier.
This patch is the first in a mini series that will change how the
TestResult object works.
Test parsing is somewhat complicated. This patch extracts it into its
own function.
The impetus of this patch is folding tsttest() into the TTest class.
Subsequent patches will continue this work until tsttest() no longer
exists.
threadtmp is an implementation detail. We move the cleanup of this
directory to Test.cleanup() and make the variable internal. The
cleanup function will eventually disappear into unittest machinery.
Test.run() can now be executed multiple times on the same Test instance.
This feature is currently unused and there are no plans to implement it.
The main reason for this work was to refactor testtmp, replacements, and
env to be run-time specific as opposed to Test instance specific.
