Computing the set of directories in the dirstate is expensive. It turns out
that it isn't necessary for operations like 'hg status' at all.
Why? Consider the file 'foo/bar' on disk, which is represented in the dirstate
as 'FOO/BAR'.
On 'hg status', we'd walk down the directory tree, coming across 'foo' first.
Before: we'd normalize 'foo' to 'FOO', then add 'FOO' to our visited stack.
We'd then visit 'FOO', finding the file 'bar'. We'd normalize 'FOO/bar' to
'FOO/BAR', then add it to the results dict.
After: we wouldn't normalize 'foo' at all. We'd add it to our visited stack,
then visit 'foo', finding the file 'bar'. We'd normalize 'foo/bar' to
'FOO/BAR', then add it to the results dict.
So whether we normalize intermediate directories or not actually makes no
difference in most cases.
The only case where normalization matters at all is if a file is replaced with
a directory with the same case-folded name. In that case we can do a relatively
cheap file normalization instead and still get away with not computing the set
of directories.
This is a nice boost in status performance. On OS X with case-insensitive HFS+,
for a large repo with over 200,000 files, this brings down 'hg status' from
4.00 seconds to 3.62.
Computing the set of directories in the dirstate can be pretty expensive. For
'hg status' without arguments, it turns out we actually never need to figure
out the right case for directories in the foldmap. (An upcoming patch explains
why.)
This patch splits up the directory and file maps into separate ones, allowing
for the subsequent optimization in status.
When 'term' is None because it isn't in the environment, don't iterate over it.
Unfortunately, unsetting $TERM or exporting it as '' doesn't work in the tests,
so there's no way to simulate cmd.exe in the test suite.
Caches should be transparent. If a cache is damaged, it should
silently be rebuilt, much like if it were invalid. No one seems to
have ever hit this in the wild.
For manifest v2, revlog.revdiff() usually does not provide enough
information to produce a manifest. As a simple workaround, implement
readdelta() by reading both the old and the new manifest and use
manifest.diff() to find the difference. This is several times slower
than the current readdelta() for v1 manifests, but there seems to be
no other simple option, and this is still much faster than returning
the full manifest (at least for verify).
We may add support for the fastdelta optimization for manifest v2 at a
later point, but let's disable it for now, so we don't have to
implement it right away.
With tree manifests, hashes will change anyway, so now is a good time
to also take up the old plans of a new manifest format. While there
should be little or no reason to use tree manifests with the current
manifest format (v1) once the new format (v2) is supported, we'll try
to keep the two dimensions (flat/tree and v1/v2) separate.
In preparation for adding a the new format, let's add configuration
for it and propagate that configuration to the manifest revlog
subclass. The new configuration ("experimental.manifestv2") says in
what format to write the manifest data. We may later add other
configuration to choose how to hash it, either keeping the v1 hash for
BC or hashing the v2 content.
See http://mercurial.selenic.com/wiki/ManifestV2Plan for more details.
By extracting a method that generates (path, node, flags) tuples, we
can reuse the code for parsing a manifest without doing it via a
_lazymanifest like treemanifest currently does. It also prepares for
parsing the new manifest format.
Note that this makes parsing into treemanifest slower, since the
parsing is now always done in pure Python. Since treemanifests will be
expected (or even forced) to be used only with the new manifest
format, parsing via _lazymanifest was not an option anyway.
The overwhelmingly common case is running commands like 'hg diff' with no
arguments. Therefore the only file that'll be listed is the root directory.
Normalizing that's just a waste of time.
This means that for a plain 'hg diff' we'll never need to construct the
foldmap, saving us a significant chunk of time.
On case-insensitive HFS+ on OS X, for a large repository with over 200,000
files, this brings down 'hg diff' from 2.97 seconds to 2.36.
This will be useful to execute actions after the tree is parsed and
before the revset returns a match. Finding symbols in the parse tree
will later allow hashes of hidden revisions to work on the command
line without the --hidden flag.
Now that command running is part of Test, we no longer need to pass
a list of replacements down through various call layers.
The impetus for this change is to fetch replacements after
command execution, not before. This will allow replacements to be
defined as part of test execution.
Upcoming patches will change how the replacements system works
to make it more flexible. To prepare for this, eliminate the one-off
use of replacements to perform newline normalization on Windows.
Future patches will change how replacements work. Since the logic in
run() is strongly tied to the operation of individual tests and since
there is potential to make the implementation simpler by giving the
function access to Test attributes, move it into Test.
The test runner has the ability to stop on first error.
Tests are executed in new Python threads. The test runner starts new
threads when it has capacity to do so. Before this patch, the "stop on
first error" logic would return immediately from the "run tests"
function, without waiting on test threads to complete. There was thus
a race between the test runner thread doing cleanup work and the test
thread performing activity. For example, the test thread could be in
the middle of executing a test shell script and the test runner
could remove the test's temporary directory. Depending on timing, this
could result in any number of output from the test runner.
This patch eliminates the race condition by having the test runner
explicitly wait for test threads to complete before continuing.
I discovered this issue as I modified the test harness in a subsequent
patch and was reliably able to tickle the race condition.
As part of testing code coverage output, I noticed some files were
being reported twice: there was an entry for the file in the install
location and for the file in the source tree. I'm not sure why this
is. But it resulted in under-reporting of coverage data since some
lines weren't getting covered in both locations.
I also noticed that files in the source directory and outside the
"mercurial" and "hgext" packages were getting included in the
coverage report. Cosmetically, this seemed odd to me. It's not
difficult to filter paths from the report. But I figure this data
can be useful (we could start reporting run-tests.py coverage,
for example).
This patch switches the coverage API to report code coverage from
the source directory. It registers a path alias so that data from
the install location is merged into data from the source directory.
We now get merged results for files that were being reported in
multiple locations.
Since code coverage reporting now relies on the profiled install
now being in sync with the source tree, an additional check to
disallow code coverage when --with-hg is specified has been added.
This should have been present before, as --local was previously
disallowed for the same reasons.
Merging the paths raises our aggregate line coverage from ~60 to
81%.
Before this patch, every Python process during a code coverage run was
writing coverage data to the same file. I'm not sure if the coverage
package even tries to obtain a lock on the file. But what I do know is
there was some last write wins leading to loss of code coverage data, at
least with -j > 1.
This patch changes the code coverage mechanism to be multiple process
safe. The mechanism for initializing code coverage via sitecustomize.py
has been tweaked so each Python process will produce a separate coverage
data file on disk. Unless two processes generate the same random value,
there are no race conditions writing to the same file. At the end of the
test run, we combine all written files into an aggregate report.
On my machine, running the full test suite produces a little over
20,000 coverage files consuming ~350 MB. As you can imagine, it takes
several seconds to load and merge these coverage files. But when it is
done, you have an accurate picture of the aggregate code coverage for the
entire test suite, which is ~60% line coverage.
Before, we were invoking the "coverage" program provided by the
"coverage" module. This patch changes the code to go through the
Python API. This makes the next patch a little bit easier to reason
about.
A side effect of this patch is that writing code coverage reports
will be slightly faster, as we won't have to redundantly load
coverage data.
The only difference for the first two was to add the globs, but the third line
of output on Windows was '..\dir2\copy'. I'm not sure why 'copy' is output on
Windows instead of '*'.
The xunit writer was passing a str to a minidom API. An implicit
.decode('ascii') was performed somewhere, causing UnicodeDecodeError
if test output contained non-ascii sequences.
This patch converts test output to utf-8 before passing it to minidom.
We use the "replace" strategy to ensure invalid utf-8 sequences get
munged into �.
It seems better to leave the actual output in place instead of globbing
everything but 'abort:', in case it starts aborting for other reasons.
It isn't clear the purpose for reversing the file name position, but that
originates in windows.posixfile.
manifest.intersectfiles() is just a utility used by manifest.matches(), and
a future commit removes intersectfiles for treemanifest for optimization
purposes.
This commit makes the intersectfiles methods on manifestdict and treemanifest
internal, and converts its test to a more generic testMatches(), which has the
exact same coverage.
With record's curses interface toggling and untoggling a newly added
file would lead to a confusing UI (the header was marked as partial
and the hunks as unselected). Tested additionally using the curses
interface with newly added, removed and modified files in a test repo.
Breadth-first allows finding the shortest cycle including the starting
module. This lets us terminate our search early when we've discovered
shorter paths already. This gives a tremendous speed-up to the
cycle-finding portion of the test, dropping total runtime from 39s to
3s.
