Before this patch, "hg perftags" command doesn't measure performance
of "repo.tags()" correctly, because it doesn't clear tags cache
correctly.
a043ed82a5cd replaced repo._tags with repo._tagscache, but didn't
change the code path to clear tags cache in perftags() at that time.
BTW, full history of "tags cache" is:
- b8d757d45f24 (or 0.6) introduced repo.tagscache as the first "tags cache"
- 4cbf51c74e8c (or 1.4) replaced repo.tagscache with repo._tags
- a043ed82a5cd (or 2.0) replaced repo._tags with repo._tagscache
- 04c204f1ed65 (or 2.5) made repo._tagscache filteredpropertycache
To make perftags clear tags cache correctly, and to increase
"historical portability" of perftags, this patch examines existence of
attributes in repo object, and guess appropriate procedure to clear
tags cache.
To avoid examining existence of attributes at each repetition, this
patch makes repocleartagscachefunc() return the function, which
actually clears tags cache.
mozilla-central repo (85 tags on 308365 revs) with each Mercurial
version between before and after this patch.
==== ========= =========
ver before after
==== ========= =========
1.9 0.476062 0.466464
------- *1 -------
2.0 0.346309 0.458327
2.1 0.343106 0.454489
------- *2 -------
2.2 0.069790 0.071263
2.3 0.067829 0.069340
2.4 0.068075 0.069573
------- *3 -------
2.5 0.021896 0.022406
2.6 0.021900 0.022374
2.7 0.021883 0.022379
2.8 0.021949 0.022327
2.9 0.021877 0.022330
3.0 0.021860 0.022314
3.1 0.021869 0.022669
3.2 0.021831 0.022668
3.3 0.021809 0.022691
3.4 0.021861 0.022916
3.5 0.019335 0.020749
3.6 0.019319 0.020866
3.7 0.018781 0.020251
------- *4 -------
3.8 0.068262 0.072558
3.9 0.069682 0.073773
==== ========= =========
(*1) repo._tags was replaced with repo._tagscache at this point
"repo._tags = None" in perftags "before" this patch doesn't clear
tags cache for Mercurial 2.0 or later. This causes significant
gap of "before" between 1.9 and 2.0 .
(*2) I'm not sure about significant gap at this point, but release
note of 2.2 described "a number of significant performance
improvements for large repositories"
(*3) filtered changelog was cached in repoview as repoview.changelog
at this point (by 131b01a4654d)
This avoids calculation of filtered changelog at each repetition
of t().
(*4) calculation of filtered changelog was included into wall time at
this point (by adf01efe43a5), again
See below for detail about this significant gap:
https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-April/083410.html
Before this patch, using ui.configint() prevents perf.py from
measuring performance with Mercurial earlier than 1.9 (or
12e7e9fbf243), because ui.configint() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
This patch replaces ui.configint() invocations by newly introduced
getint().
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of ui.configint() in perf.py.
BTW, this patch doesn't choose adding configint() method at runtime by
replacing ui.__class__ like below, even though this is the recommended
way to modern Mercurial extensions.
def uisetup(ui):
if not util.safehasattr(ui, 'configint'):
class uiwrap(ui.__class__):
def configint(self, section, name, ....):
....
ui.__class__ = uiwrap
Because changes to ui.__class__ by uisetup() of loaded extension have
been propagated since 1.6.1 (or 07a6e7bd0cc1), the recommended way
above doesn't work as expected with Mercurial earlier than it.
Before this patch, referring ui.ferr prevents perf.py from measuring
performance with Mercurial earlier than 1.9 (or bac01d164cc1), because
ui.ferr isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 1.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
Before this patch, using ui.formatter() prevents perf.py from
measuring performance with Mercurial earlier than 2.2 (or
045c8375c770), because ui.formatter() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 2.2 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
This patch defines formatter class locally, and use it instead of the
value returned by ui.formatter(), if perf.py is used with Mercurial
earlier than 2.2.
In this case, we don't need to think about -T/--template option for
formatter, because previous patch made -T/--template disabled for
perf.py with Mercurial earlier than 3.2 (or 44a82ed65df7).
Before this patch, using svfs prevents perf.py from measuring
performance of Mercurial earlier than 2.3 (or 12df7401e8cd), because
svfs isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 2.3 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
To get appropriate vfs-like object to access files under .hg/store,
this patch adds getsvfs() (and also getvfs(), for future use).
To avoid examining existence of attribute at each repetition while
measuring performance, getsvfs() is invoked outside the function to be
called repeatedly.
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of repo.(vfs|svfs|opener|sopener) in perf.py.
Mercurial 2.5 (or 0eb7dcc721cb) introduced "perfbranchmap" command,
and tried to avoid actual writing branch cache out by replacing
write() of branchcache class in branchmap.py with no-op function
(probably, for elimination of noisy and heavy file I/O factor).
But its implementation isn't correct, because 0eb7dcc721cb replaced
not branchmap.branchcache.write() but branchmap.write(). The latter
doesn't exist, even at that change.
To avoid actual writing branch cache out correctly, this patch
replaces branchmap.branchcache.write() with no-op function.
To detect mistake of replacement or change of API in the future
quickly, this patch uses safeattrsetter() instead of direct attribute
assignment. For similarity between replacements, this patch also
changes replacement of branchmap.read().
In this patch, replacement of read()/write() can run safely outside
"try" block, because two safeattrsetter() invocations ensure that
replacement doesn't cause exception.
FYI, the table below compares "base" filter wall time of perfbranchmap
on recent mozilla-central repo with each Mercurial version between
before and after this patch.
==== ========= =========
ver before after
==== ========= =========
2.5 18.492334 18.232455
2.6 18.733858 18.156702
2.7 18.245598 18.349210
2.8 18.289070 18.528422
2.9 17.572742 16.989655
3.0 17.406953 17.615012
3.1 17.228419 17.689805
3.2 17.862961 17.718367
3.3 2.632110 2.707960
3.4 3.285683 3.272060
3.5 3.370141 3.352176
3.6 3.366939 3.242455
3.7 3.300778 3.367328
3.8 3.300132 3.267298
3.9 3.418996 3.370265
==== ========= =========
IMHO, there is no serious overlooking performance regression.
Before this patch, using branchmap.subsettable prevents perfbranchmap
from measuring performance of Mercurial earlier than 2.9 (or
aad678a92970), because aad678a92970 moved subsettable from repoview.py
to branchmap.py, even though there are some code paths for Mercurial
earlier than 2.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
To get subsettable from appropriate module, this patch examines
existence of subsettable in branchmap and repoview.
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of subsettable attribute in perf.py.
Referring not-existing attribute immediately causes failure, but
assigning a value to such attribute doesn't.
For example, perf.py has code paths below, which assign a value to
not-existing attribute. This causes incorrect performance measurement,
but these code paths are executed successfully.
- "repo._tags = None" in perftags()
recent Mercurial has tags cache information in repo._tagscache
- "branchmap.write = lambda repo: None" in perfbranchmap()
branchmap cache is written out by branchcache.write() in branchmap.py
"util.safehasattr() before assignment" can avoid this issue, but might
increase mistake at "copy & paste" attribute name or so.
To centralize (1) examining existence of, (2) assigning a value to,
and (3) restoring an old value to the attribute, this patch introduces
safeattrsetter(). This is used to replace direct attribute assignment
in subsequent patches.
Encapsulation of restoring is needed to completely remove direct
attribute assignment from perf.py, even though restoring isn't needed
so often.
This command can be used for testing the performance of producing the
changelog portion of a changegroup.
We could use additional perf* commands for testing other parts of
changegroup. Those can be written another time, when they are needed.
(And those may want to refactor the changegroup generation API so code
can be reused.) Speaking of code reuse, yes, this command does reinvent
a small wheel. I didn't want to scope bloat to change the changegroup
API because that will invite bikeshedding.
demandimport of early Mercurial loads an imported module immediately,
if a module is imported absolutely by "from a import b" style. Recent
perf.py satisfies this condition, because it does:
- have "from __future__ import absolute_import" line
- use "from a import b" style for modules in "mercurial" package
Before this patch, importing modules below prevents perf.py from being
loaded by earlier Mercurial, because these aren't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9.
- branchmap 2.5 (or e3354ba12eef)
- repoview 2.5 (or 7d207cb7e38a)
- obsolete 2.3 (or b50a017cd9ac)
- scmutil 1.9 (or 065064cdde5f)
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with Mercurial earlier than 1.8 (or
1299f0c14572).
After this patch, "mercurial.error" is the only blocker in "from
mercurial import" statement for loading perf.py with Mercurial earlier
than 1.2. This patch ignores it, because just importing it separately
isn't enough.
Before this patch, using cmdutil.command() for "@command" annotation
prevents perf.py from being loaded by Mercurial earlier than 1.9 (or
d4096ee63f8e), because cmdutil.command() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
In addition to it, "norepo" option of command annotation has been
available since 3.1 (or 9cbb59f03d57), and this is another blocker for
loading perf.py with earlier Mercurial.
============ ============ ======
command of
hg version cmdutil norepo
============ ============ ======
3.1 or later o o
1.9 or later o x
earlier x x
============ ============ ======
This patch defines "command()" for annotation locally as below:
- define wrapper of existing cmdutil.command(), if cmdutil.command()
doesn't support "norepo"
(for Mercurial earlier than 3.1)
- define full command() locally with minimum function, if
cmdutil.command() isn't available at runtime
(for Mercurial earlier than 1.9)
This patch also defines parsealiases() locally without examining
whether it is available or not, because it is small enough to define
locally.
Before this patch, referring commands.formatteropts prevents perf.py
from being loaded by Mercurial earlier than 3.2 (or 44a82ed65df7),
because it isn't available in such Mercurial, even though formatting
itself has been available since 2.2 (or 045c8375c770).
In addition to it, there are some code paths for Mercurial earlier
than 3.2. For example, setting "_prereadsize" attribute in perfindex()
and perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
This patch uses empty option list as formatteropts, if it isn't
available in commands module at runtime.
Disabling -T/--template option for earlier Mercurial should be
reasonable, because:
- since 0c03b2a1206a, -T/--template for formatter has been available
- since 44a82ed65df7, commands.formatteropts has been available
- the latter revision is direct child of the former
Before this patch, referring commands.debugrevlogopts prevents perf.py
from being loaded by Mercurial earlier than 3.7 (or 1a89336e03aa),
because it isn't available in such Mercurial, even though
cmdutil.openrevlog(), a user of these options, has been available
since 1.9 (or f32fd7cab084).
In addition to it, there are some code paths for Mercurial earlier
than 3.7. For example, setting "_prereadsize" attribute in perfindex()
and perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
But just "using locally defined revlog option list" might cause
unexpected behavior at runtime. If --dir option is specified to
cmdutil.openrevlog() of Mercurial earlier than 3.5 (or e3ab0b30c05e),
it is silently ignored without any warning or so.
============ ============ ===== ===============
debugrevlogopts
hg version openrevlog() --dir of commands
============ ============ ===== ===============
3.7 or later o o o
3.5 or later o o x
1.9 or later o x x
earlier x x x
============ ============ ===== ===============
Therefore, this patch does:
- use locally defined option list, if commands.debugrevlogopts isn't
available (for Mercurial earlier than 3.7)
- wrap cmdutil.openrevlog(), if it is ambiguous whether
cmdutil.openrevlog() can recognize --dir option correctly
(for Mercurial earlier than 3.5)
This wrapper function aborts execution, if:
- --dir option is specified, and
- localrepository doesn't have "dirlog" attribute, which indicates
that localrepository has a function for '--dir'
BTW, extensions.wrapfunction() has been available since 1.1 (or
56ba0b824b91), and this seems old enough for "historical
portability" of perf.py, which has been available since 1.1 (or
bca5e7427e89).
Before this patch, using util.safehasattr() prevents perf.py from
being loaded by Mercurial earlier than 1.9.3 (or 36f152380d7d),
because util.safehasattr() isn't available in such Mercurial, even
though there are some code paths for Mercurial earlier than 1.9.3.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with Mercurial earlier than 1.8 (or
1299f0c14572).
This patch is a preparation for using util.safehasattr() safely in
subsequent patches.
This patch defines util.safehasattr() forcibly without examining
whether it is available or not, because:
- examining existence of "safehasattr" safely itself needs similar logic
- safehasattr() is small enough to define locally
A subsequent patch will refactor _chunks() and the calculation of the
offset will no longer occur in that function. Prepare by returning the
offset from _chunkraw().
We have a convention of using -c|-m|FILE elsewhere for reading from
revlogs. Use it for `hg perfrevlog`.
While I was here, I also added a docstring to document what this
command does, as "perfrevlog" is ambiguous.
As part of investigating performance improvements to revlog reading,
I needed a mechanism to measure every part of revlog reading so I knew
where time was spent and how effective optimizations were.
This patch implements a perf command for benchmarking the various
stages of reading a single revlog revision.
When executed against a manifest revision at the end of a 30,000+
long delta chain in mozilla-central, the command demonstrates that
~80% of time is spent in zlib decompression.
The old code only partially cleared the caches. Now that we have a
comprehensive method for wiping all caches, let's call it.
This appears to introduce a marginal regression in `hg perfmanifest`
on mozilla-central. This is good because the new result is more
accurate since caches aren't being used.
The /dev/null redirect was causing the following error:
The system cannot find the path specified.
Adjusting HGRCPATH as part of the command line causes the system to try to
execute 'HGRCPATH'.
Once we get a matcher down into manifestmerge, we can make narrowhg
work more easily and potentially let manifest.match().diff() do less
work in manifestmerge.
Previously, perfrevset called repo.revs(), which only returns integer
revisions. Many revset consumers call repo.set(), which returns
changectx instances. Or they obtain a context manually later.
Since obtaining changectx instances when evaluating revsets is common,
this patch adds support for benchmarking this use case.
While we added an if conditional for every benchmark loop, it
doesn't appear to matter since revset evaluation dwarfs the cost
of a single if.
Now, 'dirstate.write(tr)' delays writing in-memory changes out, if a
transaction is running.
This may cause treating this revision as "the first bad one" at
bisecting in some cases using external hook process inside transaction
scope, because some external hooks and editor process are still
invoked without HG_PENDING and pending changes aren't visible to them.
'dirstate.write()' callers below in localrepo.py explicitly use 'None'
as 'tr', because they can assume that no transaction is running:
- just before starting transaction
- at closing transaction, or
- at unlocking wlock
This change touches every module in which repository.sopener was being used, and
changes it for the equivalent repository.svfs.
It should now be possible to remove localrepo.sopener.
