Getting the data necessary for the cache key using the changelog/revlog method
adds a significant overhead. Given how simple the underlying implementation is
and often this code is ran, it makes sense to violate layering and directly
compute the data.
Testing `hg log` on Mozilla-central, this reduce the time spent on changelog
cache validation by an extra half:
before: 12.2s of 69s
after: 6.1s of 62s
Total speed up from this patch and it's parent is 3x
(With stupid python profiler overhead)
The global speedup without profiler overhead is still there,
Before: 51s
After: 39s (-23%)
As explained in the comment, we were computing the key of the cache value every
time because of some obscure MQ test failure. I've dropped that code and ran the
test again that failure is gone. I assume some transaction cleanup got rid of
it.
So we are dropping that code. This provide a significant speedup.
Testing `hg log` on Mozilla-central this reduce the time spent on changelog
cache validation by a third:
before: 19.5s of 80s
after: 12.2s of 69s
(With stupid python profiler overhead)
Since all children are processed before their parents, we can apply the following algorithm:
For each rev (descending order):
* If I'm still hidden, no children will block me,
* If I'm not hidden, I must remove my parent from the hidden set,
This allows us to dynamically change the set of 'hidden' revisions, dropping the
need for the 'actuallyhidden' dictionary and the 'blocked' boolean in the queue.
As before, we start iterating from all heads and stop at the first public
changesets. This ensures the hidden computation is 'O(not public())' instead of
'O(len(min(not public()):))'.
Since we want to process all non-public changesets from top to bottom, a heap
seems more appropriate. This will ensure any revision is processed after all
its children, opening the way to code simplification.
Previously we would compute the repoview's static blockers by finding all the
children of hidden commits that were not hidden. This was O(number of commits
since first hidden change) since 'children' requires walking every commit from
tip until the first hidden change.
The new algorithm walks all heads down until it sees a public commit. This makes
the computation O(number of draft) commits, which is much faster in large
repositories with a large number of commits and a low number of drafts.
On a large repo with 1000+ obsolete markers and the earliest draft commit around
tip~200000, this improves computehidden perf by 200x (2s to 0.01s).
Starting with b1e85ff3a7fc, when a clone reached the checkout stage,
the cached changelog in the filtered view was still seeing the
_delayed flag, even though the changelog had already been finalized.
Monkey patching repoview does not really work and making it really work will
be really hard. So we better have it broken without complexity than broken with
extra complexity.
It doesn't seem to be a common idiom for repo instances, but the status() method
is replaced in largefiles' purge() override. Since __setattr__ is implemented
in repoview to setattr() on the unfiltered repo, the replacement method wouldn't
get called unless it was invoked with the unfiltered repo, because the filtered
repo remains unchanged.
Since this doesn't seem to be commonly used, I didn't bother to filter out
methods that perhaps shouldn't be replaced, such as changelog().
We have been running hiddencache verification since 3.1.1 and so
far not received a bug report concerning it. Therefore we remove
the verification code and make the hiddencache authoritive. That
way we get the intended speedup.
Incidentally, this avoids the changelog cache being invalidated each time
it's accessed on a repoview.
On a filtering experiment on a repository the size of mozilla-central,
this makes a significant difference:
Before, running hg log -l 10 --time with about 8k changesets filtered out:
time: real 1.490 secs (user 1.450+0.000 sys 0.040+0.000)
After:
time: real 0.540 secs (user 0.530+0.000 sys 0.010+0.000)
Use the introduced caching infrastructure to cache hidden
changesets. We crosscheck if the content of the cache unless
experimental.verifyhiddencache is set to False. This will be removed
in the future. Without crosschecking the caches speed ups hg status and
other commands:
without caching:
$ time hg status
hg status 0.72s user 0.20s system 100% cpu 0.917 total
with caching
$ time hg status
hg status 0.49s user 0.15s system 100% cpu 0.645 total
Add a caching infrastructure to cache hidden changesets. The cache tries to read
the cache lazily and falls back to recomputing if no wlock can be obtain.
To validate the cache we store a sha of the obstore content and repo heads in
the beginning of the cache which we check every request.
Split up _gethiddenblockers into two categories: (1) "static' blockers
that solely rely on the contents of obstore and are visible children of
hidden changsets. (2) "dynamic" blockers, appearing by having wd parents,
bookmarks or tags pointing to hidden changesets.
We assume that (1) doesn't change often and can be easily cached with a good
invalidation strategy. (2) change often, but barely produce blockers, so we
can recompute them if necessary.
No functionality has changed, since we've only extracted the code into its own
function. Now extensions can wrap _gethiddenblockers to provide their own
blocker without polluting bookmarks or local tags.
For repos with a large number of heads (including hidden heads), a stale tag
cache would cause computehidden to be drastically slower because of a the call
to repo.tags() (which would build the tag cache).
We actually don't need the tag cache for computehidden because we filter out
global tags. This patch replaces the call to repo.tags with readlocaltags so
as to avoid the tag cache.
Previously, only bookmarks would be considered for blocking a changeset from
being hidden. Now, we also consider non-global tags. This is helpful if we have
local tags that might be hard to find once they are hidden, or tag that are
added by extensions (e.g. hggit or remotebranches).
Changeset aad678a92970 moved `subsettable` from `mercurial/repoview.py` to
`mercurial/branchmap.py`. This mean that `filtertable` and `subsettable` are no
longer next to each other. So we add a comment to remind people to update both.
This is a step towards breaking an import cycle between revset and
repoview. Import cycles happened to work in Python 2 with implicit
relative imports, but breaks on Python 3 when we start using explicit
relative imports via 2to3 rewrite rules.
Creating a new changelog object for each access is costly and prevents efficient
caching changelog side. This introduced a x5 performance regression in log
because chunk read from disk were never reused. We were jumping from about 100
disk read to about 20 000.
This changeset introduce a simple cache mechanism that help the last changelog
object created by a repoview. The changelog is reused until the changelog or the
filtering changes.
The cache invalidation is much more complicated than it should be. But MQ test
show a strange cache desync. I was unable to track down the source of this
desync in descent time so I'm not sure if the issue is in MQ or core. However
given the proximity to the 2.5 freeze, I'm choosing the inelegant but safe route
that makes the cache mechanism safer.
If for some reason the phase roots contains nullid, the set of filtered revs
will contains -1. That confuse Mercurial a lot. In particular this corrupt the
branchcache.
Standard code path does not result in nullid phase root. It can only result from
altered `.hg/store/phaseroots` or buggy extension. However better safe than
sorry.
Now that changelog filtering is in place, it's become evident that
naming the filters according to the set of revs _not_ included in the
filtered changelog is confusing. This is especially evident in the
collaborative branch cache scheme.
This changes the names of the filters to reflect the revs that _are_
included:
hidden -> visible
unserved -> served
mutable -> immutable
impactable -> base
repoview.filteredrevs is renamed to filterrevs, so that callers read a
bit more sensibly, e.g.:
filterrevs('visible') # filter revs according to what's visible
In their current state, revset calls can be very costly, as we test
predicates on the entire repository. The "mutable" filter is used
during branch cache loading operation. We need to make it fast.
This change drops revset calls in favor of direct testing of the
phase of a changeset.
Performance test on my Mercurial checkout
- 19857 total changesets,
- 1646 mutable revision
Before:
! mutable
! wall 0.032405
After:
! mutable
! wall 0.001469
Performance test on a Mozilla central checkout:
- 117293 total changesets,
- 1 mutable changeset,
Before:
! mutable
! wall 0.188636
After:
! mutable
! wall 0.000022
In their current state, revset calls can be very costly, as we test
predicates on the entire repository. The "unserved" filter is used
in multiple applications, and in particular in some branch cache
loading operations. We need to make it fast.
This change drops revset calls in favor of direct testing of the
phase of a changeset.
Performance test on my Mercurial checkout
- 19857 total changesets,
- 1584 obsolete changesets,
- 13310 obsolescence markers.
Before:
! unserved
! wall 0.030477
After:
! unserved
! wall 0.011844
Performance test on a Mozilla central checkout:
- 117293 total changesets,
- 1 obsolete changeset,
- 1 obsolescence marker.
Before:
! unserved
! wall 0.111259
After:
! unserved
! wall 0.000084
In their current state, revset calls can be very costlys, as we
test predicates on the entire repository. The hidden filter is very
widely used, and needs to be very fast.
This change drops revset calls in favor of direct revision manipulation.
Performance test on my Mercurial checkout
- 19857 total changesets,
- 1584 obsolete changesets,
- 13310 obsolescence markers.
Before:
! hidden
! wall 0.077553
After this changes:
! hidden
! wall 0.011230
Performance test on a Mozilla central checkout:
- 117293 total changesets,
- 1 obsolete changeset,
- 1 obsolescence marker.
Before:
! hidden
! wall 0.389472
After:
! hidden
! wall 0.000079
