This is part of a refactoring that moves some phase query optimization from
revset.py to phases.py. See previous patches for the motivation.
Now we have APIs in phasecache to get the non-public set efficiently, let's
use it directly instead of going through the "not public()" revset language
in "obsolete()" computation.
This patch was meaured using:
for i in 'public()' 'not public()' 'draft()' 'not draft()'; do
hg perfrevset "$i"; hg perfrevset "$i" --hidden;
done
and no noticeable (> 1%) performance difference was observed.
Since one of the original patches was accepted already and people on the
mailing list still have suggestions as to how this should be improved, I'm
implementing those suggestions in the following patches (this and the ones that
might follow).
Previously, if you ran obsolete.createmarkers with a bunch of markers that did
not have successors (like when you do a prune), it encountered a n^2 computation
behavior because the loop would read the changelog (to get ctx.parents()), then
add a marker, in a loop. Adding a marker invalidated the computehidden cache,
and reading the changelog recomputed it.
This resulted in pruning 150 commits taking 150+ seconds in a large repo.
The fix is to break the reading part of the loop to be separate from the writing
part.
This patch makes _computeobsoleteset much faster by looping
over the draft and secrets as opposed to looping over the
successors.
This works because "number of draft and secret" is typically
way smaller(<100) than the number of successor in the repo (~90k in
my checkout of core mercurial as of today). And also because
it is very fast to compute "not public()".
I timed the code with the following setup:
"""
from mercurial import hg, ui, obsolete
ui = ui.ui()
repo = hg.repository(ui, "~/hg")
l = repo.obsstore.successors # This caches the result
"""
With about 90k successors.
k=obsolete._computeobsoleteset(repo) before this patch:
10 loops, best of 3: 33.9 ms per loop
k=obsolete._computeobsoleteset(repo) after this patch:
10000 loops, best of 3: 83.3 µs per loop
We want to gracefully handle the read only case in some case (current target:
advisory obsmarkers parts in bundle2). So we expose the attribute in a clean
way.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
This will allow us to use cached revisions without parsing obsstore.
Because _version isn't determined at __init__, the debugobsconvert command no
longer works correctly. I'll fix it by a separate patch for the evolve
extension.
The next patch will make _all variable propertycached to avoid costly parsing
of obsstore. This means we can't call _addmarkers() to initialize _all.
Because all cached markers depend on _all, it isn't necessary to update these
caches when _all is initially loaded.
We do not return the first successors we found, we returns the latest (non
obsolete (mostly)) one following the obsolete link transitively. We update the
documentation to make this clean.
The phrase "cannot edit immutable changeset" is kind of tautological.
Of course unchangeable things can't be changed. We instead mention
"public" and provide a hint so that we can point to the actual
problem. Even in cases where some operation other than edition cannot
be performed, "public" gives the root cause that results in the
"immutable" effect.
There is a precedent for saying "public" instead of "immutable", for
example, in `hg commit --amend`.
Because bundle2 allows a more precise exchange of obsmarkers during pull, it
sends them in a different order (previously unstable because of sets.) As
a result, they are added to the repository in a different order. To stabilize
the order and ensure tests are unchanged when moving from bundle1 to bundle2 we
sort markers when exchanging them.
In the long run, the obsstore will probably not use a linear storage.
Speed up the computation of the unstable revset by using the not public()
revset. In another series of patches, we optimize the not public() revset and
together it leads to a 50-100x speedup on the computation of unstable() for
our big repos.
Speed up the computation of the bumped revset by using the not public() revset.
In another patch series, we optimized the not public() revset. Together,
this cuts the cost of the computation of bumped() from 17% of the total time of
smartlog on our big repo to under 0.1% of the total time.
As for other currently in place cycle detection, arbitrarily cut the first
obsolescence link that create a cycle avoiding the infinite loop. This will have
to be made more deterministic in the future but we do not really care right now.
The precursors and children dictionaries are not used by many
commands. By making them lazily populated,
'hg log -r @~10::@ >/dev/null' is sped up from 0.564s to 0.440s on my
hg.hg repo with 73k markers.
Also make successors lazily populated, mostly for consistency with the
others.
We will soon delay populating precursors until we have to. To prepare
for that, make _checkinvalidmarkers() scan for a nullid precursor in
the list of new markers instead of the (currently) cheaper 'if nullid
in precursors' check.
In preparation for making the successors, precursors, and children
dictionaries lazily populated, break up _load() into one function for
adding markers to each dictionary.
This moves perfloadmarkers from
! result: 63866
! wall 0.239217 comb 0.250000 user 0.240000 sys 0.010000 (best of 42)
to
! result: 63866
! wall 0.218795 comb 0.210000 user 0.210000 sys 0.000000 (best of 46)
This moves perfloadmarkers on my linux workstation (63494 markers from
mpm, crew, and myself) performance from
! wall 0.357657 comb 0.360000 user 0.350000 sys 0.010000 (best of 28)
to
! wall 0.222345 comb 0.220000 user 0.210000 sys 0.010000 (best of 41)
which is a pretty good improvement.
On my BSD machine, which is ancient and slow, before:
! wall 3.584964 comb 3.578125 user 3.539062 sys 0.039062 (best of 3)
after:
! wall 2.267974 comb 2.265625 user 2.195312 sys 0.070312 (best of 5)
I feel like we could do better by moving the whole generator function
into C, but I didn't want to tackle that right away.
Some evolve user ignored the invalid markers for about two years and still have
some of them in some repository. This lead to plain abort whenever mercurial try
to open such repo. We need reinstall some way to clean this up in the evolve
extension. For this purpose, we need the checker code wrap-able independently.
This is scheduled for stable as this issue is blocking some evolve user.
