To speed up the computation of draft(), secret(), divergent(), obsolete() and
unstable() we need to have a fast way of getting the list of revisions that
are in draft(), secret() or the union of both: not public().
This patch extends the work on phase computation in C and make the phase
computation code also return a list of set for each non public phase.
Using these sets we can quickly obtain all the revisions of a given phase.
We do not return a set for the public phase as we expect it to be roughly the
size of the repo. Also, it can be computed easily by substracting the entries in the
non public phases from all the revs in the repo.
If v = -INT_MAX - 1, -v would exceed INT_MAX. I don't think this would cause
problems such as issue4627, but we can't blame it as a compiler bug because
signed integer overflow is undefined in C.
On a repo with over 300,000 files, this speeds up perffilefoldmap:
before: wall 0.178421 comb 0.180000 user 0.160000 sys 0.020000 (best of 55)
after: wall 0.164462 comb 0.160000 user 0.140000 sys 0.020000 (best of 59)
This is a hot path on case-insensitive filesystems -- it's guaranteed to be
called every time 'hg status' is run.
This is significantly faster than the equivalent Python code: see the following
patch for numbers.
We're going to reuse this in upcoming patches.
The change to Py_ssize_t is necessary because parsers.c doesn't define
PY_SSIZE_T_CLEAN. That macro changes the behavior of PyArg_ParseTuple but not
PyBytes_GET_SIZE.
Previously, the phase computation would grow much slower as the oldest draft
commit in the repository grew older (which is very common in repos with evolve
on) and the number of commits increase.
By rewriting the computation in C we can speed it up from 700ms to 7ms on
a large repository whose oldest draft commit is a year old.
This lets us iterate manifests in order, but do a _lot_ less work in
the common case when we only care about a few manifest entries.
Many thanks to Mike Edgar for reviewing this in advance of it going
out to the list, which caught many things I missed.
This version of the patch includes C89 fixes from Sean Farley and
many correctness/efficiency cleanups from Martin von
Zweigbergk. Thanks to both!
This moves perfloadmarkers from
! result: 63866
! wall 0.239217 comb 0.250000 user 0.240000 sys 0.010000 (best of 42)
to
! result: 63866
! wall 0.218795 comb 0.210000 user 0.210000 sys 0.000000 (best of 46)
This lets us do most of the interesting work of parsing obsolete
markers in C, which should provide significant time savings.
Thanks to Martin von Zweigbergk for some cleanups on this code.
Currently, the revlog index C implementation assumes its node tree will be
initialized before a new element is inserted by revnum. For example, revlog.py
executes 'self.index.insert(-1, e)' in _addrevision(). This is only safe
because the node tree has been initialized by a "node in self.nodemap"
check made in addrevision().
(For context, this was discovered while developing an experimental revlog
mixin which stores "elided nodes" via a separate code path from
_addrevision(); that new code path segfaults without this patch.)
All extensions that have this function do support filtering. The existing
headrevs function may support filtering but we cannot reliably detect whether
it does.
'n' was introduced in Mercurial in 5d1adb6683fa and broke Python 2.4 support in
mysterious ways that only showed failure in test-glog.t. Py_BuildValue failed
because of the unknown format and a TypeError was thrown ... but it never
showed up on the Python side and it happily continued processing with wrong
data.
Quoting https://docs.python.org/2/c-api/arg.html :
n (integer) [Py_ssize_t]
Convert a Python integer or long integer to a C Py_ssize_t.
New in version 2.5.
k (integer) [unsigned long]
Convert a Python integer or long integer to a C unsigned long without
overflow checking.
This will use unsigned long instead of Py_ssize_t. That is not a good solution,
but good is not an option when we have to support Python 2.4.
We need a way to efficiently lowercase ASCII strings. For example, 'hg status'
needs to build up the fold map -- a map from a canonical case (for OS X,
lowercase) to the actual case of each file and directory in the dirstate.
The current way we do that is to try decoding to ASCII and then calling
lower() on the string, labeled 'orig' below:
str.decode('ascii')
return str.lower()
This is pretty inefficient, and it turns out we can do much better.
I also tested out a condition-based approach, labeled 'cond' below:
(c >= 'A' && c <= 'Z') ? (c + ('a' - 'A')) : c
'cond' turned out to be slower in all cases. A 256-byte lookup table with
invalid values for everything past 127 performed similarly, but this was less
verbose.
On OS X 10.9 with LLVM version 6.0 (clang-600.0.51), the asciilower function
was run against two corpuses.
Corpus 1 (list of files from real-world repo, > 100k files):
orig: wall 0.428567 comb 0.430000 user 0.430000 sys 0.000000 (best of 24)
cond: wall 0.077204 comb 0.070000 user 0.070000 sys 0.000000 (best of 100)
lookup: wall 0.060714 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
Corpus 2 (mozilla-central, 113k files):
orig: wall 0.238406 comb 0.240000 user 0.240000 sys 0.000000 (best of 42)
cond: wall 0.040779 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
lookup: wall 0.037623 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
On a Linux server-class machine with GCC 4.4.6 20120305 (Red Hat 4.4.6-4):
Corpus 1 (real-world repo, > 100k files):
orig: wall 0.260899 comb 0.260000 user 0.260000 sys 0.000000 (best of 38)
cond: wall 0.054818 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
lookup: wall 0.048489 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
Corpus 2 (mozilla-central, 113k files):
orig: wall 0.153082 comb 0.150000 user 0.150000 sys 0.000000 (best of 65)
cond: wall 0.031007 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
lookup: wall 0.028793 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)
SSE instructions might help even more, but I didn't experiment with those.
Since d0ec15e840e5, we were getting this strange message with Py2.4:
TypeError: argument 1 must be impossible<bad format char>, not int
..because we were using the 'n' type specifier introduced in 2.5.
It turns out that offset is actually a revision number index, which
ought to be an int anyway. So we store it in an int, use the 'i'
specifier, rely on Py_ParseTuple for range checking, and rename it to
avoid type confusion.
Previously, if there were filtered revs the repository could not use the C fast
path for computing the head revs in the changelog. This slowed down many
operations in large repositories.
This adds the ability to filter revs to the C fast path. This speeds up histedit
on repositories with filtered revs by 30% (13s to 9s). This could be improved
further by sorting the filtered revs and walking the sorted list while we walk
the changelog, but even this initial version that just calls __contains__ is
still massively faster.
The new C api is compatible for both new and old python clients, and the new
python client can call both new and old C apis.
Now using Py_ssize_t instead of long to denote offset in file whose length is
already measured using Py_ssize_t. Length and offset are now consistent. Based
on warning from Microsoft Visual C++ 2008.
The passed variable is a Py_ssize_t, not a long, and consequently should use
PyInt_FromSsize_t rather than PyInt-FromLong. Fixed based on warning from
Microsoft Visual C++ 2008.
Previously, while unpacking the dirstate we'd create 3-4 new CPython objects
for most dirstate values:
- the state is a single character string, which is pooled by CPython
- the mode is a new object if it isn't 0 due to being in the lookup set
- the size is a new object if it is greater than 255
- the mtime is a new object if it isn't -1 due to being in the lookup set
- the tuple to contain them all
In some cases such as regular hg status, we actually look at all the objects.
In other cases like hg add, hg status for a subdirectory, or hg status with the
third-party hgwatchman enabled, we look at almost none of the objects.
This patch eliminates most object creation in these cases by defining a custom
C struct that is exposed to Python with an interface similar to a tuple. Only
when tuple elements are actually requested are the respective objects created.
The gains, where they're expected, are significant. The following tests are run
against a working copy with over 270,000 files.
parse_dirstate becomes significantly faster:
$ hg perfdirstate
before: wall 0.186437 comb 0.180000 user 0.160000 sys 0.020000 (best of 35)
after: wall 0.093158 comb 0.100000 user 0.090000 sys 0.010000 (best of 95)
and as a result, several commands benefit:
$ time hg status # with hgwatchman enabled
before: 0.42s user 0.14s system 99% cpu 0.563 total
after: 0.34s user 0.12s system 99% cpu 0.471 total
$ time hg add new-file
before: 0.85s user 0.18s system 99% cpu 1.033 total
after: 0.76s user 0.17s system 99% cpu 0.931 total
There is a slight regression in regular status performance, but this is fixed
in an upcoming patch.
For files written out in the last second, Mercurial used to invalidate all the
stat data (state, size, mode, mtime) while persisting to disk. This included
invalidating the data in the dirstate dict as well.
In commit c7a2ac9361a7, this was found to be unnecessary, and Mercurial
switched to invalidating only the mtime. However, in the C version of
pack_dirstate the value set in the dict was still the fully invalidated one.
Switch to invalidating just the mtime in the dict as well.
This change causes an informative ImportError to be raised when importing
the parsers extension module if the minor version of the currently-running
Python interpreter doesn't match that of the Python used when compiling
the extension module.
This change also exposes a parsers.versionerrortext constant in the
C implementation of the module. Its presence can be used to determine
whether this behavior is present in a version of the module. The value
of the constant is the leading text of the ImportError raised and is set
to "Python minor version mismatch".
Here is an example of what the new error looks like:
Traceback (most recent call last):
File "test.py", line 1, in <module>
import mercurial.parsers
ImportError: Python minor version mismatch: The Mercurial extension
modules were compiled with Python 2.7.6, but Mercurial is currently using
Python with sys.hexversion=33883888: Python 2.5.6
(r256:88840, Nov 18 2012, 05:37:10)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))]
at: /opt/local/Library/Frameworks/Python.framework/Versions/2.5/Resources/
Python.app/Contents/MacOS/Python
The reason for raising an error in this scenario is that Python's C API
is known not to be compatible from minor version to minor version, even
if sys.api_version is the same. See for example this Python bug report
about incompatibilities between 2.5 and 2.6+:
http://bugs.python.org/issue8118
These incompatibilities can cause Mercurial to break in mysterious,
unforeseen ways. For example, when Mercurial compiled with Python 2.7 was
run with 2.5, the following crash occurred when running "hg status":
http://bz.selenic.com/show_bug.cgi?id=4110
After this crash was fixed, running with Python 2.5 no longer crashes, but
the following puzzling behavior still occurs:
$ hg status
...
File ".../mercurial/changelog.py", line 123, in __init__
revlog.revlog.__init__(self, opener, "00changelog.i")
File ".../mercurial/revlog.py", line 251, in __init__
d = self._io.parseindex(i, self._inline)
File ".../mercurial/revlog.py", line 158, in parseindex
index, cache = parsers.parse_index2(data, inline)
TypeError: data is not a string
which can be reproduced more simply with:
import mercurial.parsers as parsers
parsers.parse_index2("", True)
Both the crash and the TypeError occurred because the Python C API's
PyString_Check() returns the wrong value when the C header files from
Python 2.7 are run with Python 2.5. This is an example of an
incompatibility of the sort mentioned in the Python bug report above.
Failing fast with an informative error message results in a better user
experience in cases like the above. The information in the ImportError
also simplifies troubleshooting for those on Mercurial mailing lists, the
bug tracker, etc.
This patch only adds the version check to parsers.c, which is sufficient
to affect command-line commands like "hg status" and "hg summary".
An idea for a future improvement is to move the version-checking C code
to a more central location, and have it run when importing all
Mercurial extension modules and not just parsers.c.
