The Python version of the dirs type stores only the number of direct
children associated with each directory. That means that while adding
a directory, it only has to walk backwards until it runs into a
directory that is already in its map. The C version walks all the way
to the top-most directory. By copying the Python version's clever
trick to the C code, we can speed it up quite a bit.
On the Firefox repo, perfdirs now runs in 0.031390, from 0.056518
before the undoing Sid's optimization in the previous change, and
0.061835 before previous his optimization. More practically, it speeds
up 'hg status nonexistent' on the Firefox repo from 0.176s to 0.155s.
It's unclear why the C version did not have the same cleverness
implemented from the start, especially given that they were both
written by the same person (Bryan O'Sullivan) very close in time:
36b515ad5d16 (scmutil: add a dirs class, 2013-04-10)
1b2603494a5d (scmutil: rewrite dirs in C, use if available, 2013-04-10)
This backs out the changes below. The next patch will implement a
faster algorithm based on backward-walking in finddirs().
88c2b37a1220 (dirs._addpath: reinstate use of Py_CLEAR, 2015-04-07)
4f6b7d37d06a (dirs._addpath: don't mutate Python strings after exposing them (issue4589), 2015-04-06)
3f1ebc88eb7b (dirs.addpath: rework algorithm to search forward, 2015-03-27)
I changed this to an explicit Py_DECREF + set to null in 4f6b7d37d06a. This was
a silly misunderstanding on my part -- for some reason I thought Py_CLEAR set
its argument to null only if its refcount reached 0. Turns out that's not
actually the case -- Py_CLEAR is just Py_DECREF + set to null with some
additional precautions around destructors that aren't relevant here.
The real bug that 4f6b7d37d06a fixed was the fact that we were mutating the
string after setting it in the Python dictionary.
One of the rules of Python strings is that they're immutable. dirs._addpath
breaks this assumption for performance, which is fine as long as it is done
safely -- once a string is no longer internal-only it shouldn't be mutated.
Unfortunately, we weren't being safe here -- we were mutating 'key' even after
adding it to a dictionary.
This only really affects other C code that reads strings, so it's somewhat hard
to write a test for this without poking into the internal representation of the
string via ctypes or similar. There is currently no C code that reads the
output of the string, but there will likely be some soon as the bug indicates.
There's no significant difference in performance.
This improves performance because it uses strchr rather than a loop.
For LLVM/clang version "Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM
3.5svn)" on OS X, for a repo with over 200,000 files, this improves perfdirs
from 0.248 seconds to 0.230 (7.3%)
For gcc 4.4.6 on Linux, for a test repo with over 500,000 files, this improves
perfdirs from 0.704 seconds to 0.658 (6.5%).
Previously, while unpacking the dirstate we'd create 3-4 new CPython objects
for most dirstate values:
- the state is a single character string, which is pooled by CPython
- the mode is a new object if it isn't 0 due to being in the lookup set
- the size is a new object if it is greater than 255
- the mtime is a new object if it isn't -1 due to being in the lookup set
- the tuple to contain them all
In some cases such as regular hg status, we actually look at all the objects.
In other cases like hg add, hg status for a subdirectory, or hg status with the
third-party hgwatchman enabled, we look at almost none of the objects.
This patch eliminates most object creation in these cases by defining a custom
C struct that is exposed to Python with an interface similar to a tuple. Only
when tuple elements are actually requested are the respective objects created.
The gains, where they're expected, are significant. The following tests are run
against a working copy with over 270,000 files.
parse_dirstate becomes significantly faster:
$ hg perfdirstate
before: wall 0.186437 comb 0.180000 user 0.160000 sys 0.020000 (best of 35)
after: wall 0.093158 comb 0.100000 user 0.090000 sys 0.010000 (best of 95)
and as a result, several commands benefit:
$ time hg status # with hgwatchman enabled
before: 0.42s user 0.14s system 99% cpu 0.563 total
after: 0.34s user 0.12s system 99% cpu 0.471 total
$ time hg add new-file
before: 0.85s user 0.18s system 99% cpu 1.033 total
after: 0.76s user 0.17s system 99% cpu 0.931 total
There is a slight regression in regular status performance, but this is fixed
in an upcoming patch.
perfdirs results for a working dir with 170,000 files:
Python 638 msec
C 244
C+int 192
C+int+str 168
In the large repo above, the nearly 0.5 second time improvement is
visible in commands like "hg add" and "hg update".
hg add
Python 1100 msec
C+int+str 600
hg update (with nothing to do)
Python 2800 msec
C+int+str 2240
This is over twice as fast as the Python dirs code. Upcoming changes
will nearly double its speed again.
perfdirs results for a working dir with 170,000 files:
Python 638 msec
C 244
