Change the C implementation of phasecache.loadphaserevs to provide only
the sets for draft and secret phase as well as the number of revisions
seen.
Change the pure Python implementation of the same functino to compute
the sets instead of the list of phases for each revision.
Change phasecache.phase to check the phase sets and assume public if the
revision is in neither draft nor secret set. This is computationally
slightly more expensive.
Change phasecache.getrevset for public() based queries to compute the
set of non-matching revisions and return the result as filtered
fullreposet. A shortcut is taken when no draft or secret revision
exists.
Bump the module version for the changed interface contract.
Overall, this saves around 16 Bytes per revision whenever the phasecache
is used, for the test case in issue5691 it is around 3MB. getrevset()
for a large repository is around 13% slower here, that seems an
acceptable trade off. Performance impact for phase() should be similar.
Differential Revision: https://phab.mercurial-scm.org/D1606
# skip-blame because parsers.c is mechanically rewritten by
clang-format with no semantic change.
Differential Revision: https://phab.mercurial-scm.org/D1170
This gives clang-format the right notion about formatting these struct
initializers, therefore allowing us to automatically format several
additional files.
# skip-blame because this is just a content-free comment addition
Differential Revision: https://phab.mercurial-scm.org/D1169
We want a slightly weird format here so that it's easier to read, but
in order to preserve that we need to disable clang-format.
Differential Revision: https://phab.mercurial-scm.org/D1168
This seems to be the prevailing style, even though it is a bit more
verbose for very simple switch statements.
Differential Revision: https://phab.mercurial-scm.org/D909
We mostly abide by this style.
In one case, a blank line was inserted to prevent a local
`#include "file"` from coming before a `#include <file>`.
Differential Revision: https://phab.mercurial-scm.org/D908
The minimum input size to exploit is ~682MB (= INT_MAX / len('\\u0000') * 2)
on 32bit system, which isn't easy to achieve using Python str in 2GB process
address space, but probably doable.
This extracts charencode.c from parsers.c, which seems big enough for me
to hesitate to add new JSON functions. Still charencode.o is linked to
parsers.so to avoid duplication of binary codes.
find_deepest is used to find the "best" ancestors given a list. In the main
loop it keeps an invariant called 'ninteresting' which is supposed to contain
the number of non-zero entries in the 'interesting' array. This invariant is
incorrectly maintained, however, which leads the the algorithm returning an
empty result for certain graphs. This has been fixed.
Also, the 'interesting' array is supposed to fit 2^ancestors values, but is
incorrectly allocated to twice that size. This has been fixed as well.
The tests in test-ancestor.py compare the Python and C versions of the code,
and report the error correctly, since the Python version works correct. Even
so, I have added an additional test against the expected result, in the event
that both algorithms have an identical error in the future.
This fixes issue5623.
* Scope of "value" is reduced
* index_baserev() is documented
* Error is no longer redundantly set for -2 return values
* Error values are compared <= -2 instead of == -2 to protect
against odd failure scenarios
I've seen revlog._deltachain() appear in a number of performance
profiles. I suspect there are 2 reasons for this:
1. Delta chain resolution performs many index lookups, thus triggering
population of index tuples. Creating possibly tens of thousands of
PyObject will have overhead.
2. Delta chain resolution is a tight loop.
By moving delta chain resolution to C, we can defer instantiation
of full index entry tuples and make the loop faster courtesy of
not running in Python.
We can measure the impact to delta chain resolution via
`hg perflogrevision` using the mozilla-central repo with a recent
manifest having delta chain length of 33726:
$ hg perfrevlogrevision -m 364895
! full
! wall 0.367585 comb 0.370000 user 0.340000 sys 0.030000 (best of 27)
! wall 0.357581 comb 0.360000 user 0.350000 sys 0.010000 (best of 28)
! deltachain
! wall 0.010644 comb 0.010000 user 0.010000 sys 0.000000 (best of 270)
! wall 0.000292 comb 0.000000 user 0.000000 sys 0.000000 (best of 8729)
$ hg perfrevlogrevision --cache -m 364895
! deltachain
! wall 0.003904 comb 0.000000 user 0.000000 sys 0.000000 (best of 712)
! wall 0.000284 comb 0.000000 user 0.000000 sys 0.000000 (best of 9926)
The first test measures savings from both not instantiating index
entries and moving to C. The second test (which doesn't clear the
index caches) essentially isolates the benefits of moving from Python
to C. It still shows a 13.7x speedup (versus 36.4x). And there are
multiple milliseconds of savings within the critical path for resolving
revision data. I think that justifies the existence of C code.
A more striking example of the benefits of this change can be
demonstrated by timing `hg debugdeltachain -m` for the mozilla-central
repo:
$ time hg debugdeltachain -m > /dev/null
before: 1057.4s
after: 503.3s
PyPy2.7 5.8.0: 220.0s
It's worth noting that the C code isn't as optimal as it could be.
We're still instantiating a new PyObject for every revision. A future
optimization would be to reuse the PyObject on the cached index tuple.
We could potentially also get wins by using a memory array of raw
integers. There is also room for a delta chain cache on revlog
instances. Of course, the best optimization is to implement revlog
reading outside of Python so Python doesn't need to be concerned
about the relatively expensive index entries and operations on them.
parsers.c is ~3000 lines and ~2/3 of it is related to the revlog
index type.
We already have separate C source files for directory utilities
and manifest parsing. I think the quite unwieldy revlog/index
parsing code should be self-contained as well.
I performed the extraction as a file copy then removed content
from both sides in order to preserve file history and blame.
As part of this, I also had to move the hexdigit table and
function to a shared header since it is used by both parsers.c
and revlog.c
# no-check-commit
I'm going to restructure cext/pure modules and get rid of our hgimporter
hack. C extension modules will be moved to cext/ directory so old and new
compiled modules can coexist in development tree. This is necessary to
run 'hg bisect' without recompiling.
New extension modules will be loaded by an importer function:
base85 = policy.importmod('base85') # select pure.base85 or cext.base85
This will also allow us to split cffi from pure modules, which is currently
difficult because pure modules can't be imported by name.
