Commit Graph

156 Commits

Author SHA1 Message Date
Pierre-Yves David
667fac2e9e perf: have a generic "clearstorecache" function
There are multiple places where we will want to purge some store cache. So we
promote the existing _clearobsstore function to a something reusable.
2017-06-07 17:31:30 +01:00
Pierre-Yves David
3324c6b39d perfbranchmap: add an option to purge the revbranch cache
The perf extension needs to be able to reflect this reality too. (eg: 4s vs 60s
on a Million-ish revisions repository).
2017-06-05 16:24:01 +01:00
Gregory Szorc
21a83ae27a perf: benchmark command for revlog indexes
We didn't have explicit microbenchmark coverage for loading revlog
indexes. That seems like a useful thing to have, so let's add it.

We currently measure the low-level nodemap APIs. There is room to
hook in at the actual revlog layer. This could be done as a follow-up.

The hackiest thing about this patch is specifying revlog paths.
Other commands have arguments that allow resolution of changelog,
manifest, and filelog. I needed to hook in at a lower level of
the revlog API than what the existing helper functions to resolve
revlogs allowed. I was too lazy to write some new APIs. This could
be done as a follow-up easily enough.

Example output for `hg perfrevlogindex 00changelog.i` on my
Firefox repo (404418 revisions):

! revlog constructor
! wall 0.003106 comb 0.000000 user 0.000000 sys 0.000000 (best of 912)
! read
! wall 0.003077 comb 0.000000 user 0.000000 sys 0.000000 (best of 924)
! create index object
! wall 0.000000 comb 0.000000 user 0.000000 sys 0.000000 (best of 1803994)
! retrieve index entry for rev 0
! wall 0.000193 comb 0.000000 user 0.000000 sys 0.000000 (best of 14037)
! look up missing node
! wall 0.003313 comb 0.000000 user 0.000000 sys 0.000000 (best of 865)
! look up node at rev 0
! wall 0.003295 comb 0.010000 user 0.010000 sys 0.000000 (best of 858)
! look up node at 1/4 len
! wall 0.002598 comb 0.010000 user 0.010000 sys 0.000000 (best of 1103)
! look up node at 1/2 len
! wall 0.001909 comb 0.000000 user 0.000000 sys 0.000000 (best of 1507)
! look up node at 3/4 len
! wall 0.001213 comb 0.000000 user 0.000000 sys 0.000000 (best of 2275)
! look up node at tip
! wall 0.000453 comb 0.000000 user 0.000000 sys 0.000000 (best of 5697)
! look up all nodes (forward)
! wall 0.094615 comb 0.100000 user 0.100000 sys 0.000000 (best of 100)
! look up all nodes (reverse)
! wall 0.045889 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
! retrieve all index entries (forward)
! wall 0.078398 comb 0.080000 user 0.060000 sys 0.020000 (best of 100)
! retrieve all index entries (reverse)
! wall 0.079376 comb 0.080000 user 0.070000 sys 0.010000 (best of 100)
2017-05-28 11:13:10 -07:00
Gregory Szorc
845664d2aa perf: rename perfrevlog to perfrevlogrevisions
We have a couple of commands beginning with "perfrevlog." The
actual "perfrevlog" command actually measures resolving the fulltext
of multiple revisions. So let's rename it to reflect what it actually
does.
2017-05-28 10:56:28 -07:00
Pierre-Yves David
41d67dfdab perfphases: add 'perfphases' command
This commands benchmark the time spend computing the data in a repository.
2017-05-23 02:23:07 +02:00
Pierre-Yves David
55566b12d1 perf: allow to clear the obsstore in 'perfvolatilesets'
Loading the obsstore can become a large part of the time necessary to compute
the important volatile set. We add a flag purging all known obsstore related
data.

For example, computing the 'bumped' set currently requires reading the full
obsstore, so timing greatly differ with or without that flag:

Without:
! bumped
! wall 0.005047 comb 0.000000 user 0.000000 sys 0.000000 (best of 446)

With:
! bumped
! wall 0.512367 comb 0.510000 user 0.480000 sys 0.030000 (best of 15)
2017-05-21 13:32:07 +02:00
Yuya Nishihara
6c2103bc71 commands: move templates of common command options to cmdutil (API)
The goal is to get rid of the debugcommands -> commands dependency.

Since globalopts is the property of the commands, it's kept in the commands
module.
2017-05-14 16:19:47 +09:00
Yuya Nishihara
3e663dde68 registrar: move cmdutil.command to registrar module (API)
cmdutil.command wasn't a member of the registrar framework only for a
historical reason. Let's make that happen. This patch keeps cmdutil.command
as an alias for extension compatibility.
2016-01-09 23:07:20 +09:00
Gregory Szorc
34c0f2716e perf: always pass node to revlog.revision()
I removed this in ff97c183abcf thinking it wasn't necessary. In fact,
we need to always pass a node so the code is compatible with revisions
before f4a6c9197dbd.

The new code uses a variable to avoid check-style complaining
about "r.revision(r.node(" patterns.
2017-05-15 19:37:11 -07:00
Gregory Szorc
f7646dc551 perf: move revlog construction and length calculation out of benchmark
We don't need to measure the time it takes to open the revlog or
calculate its length.

This is more consistent with what other perf* functions do.

While I was here, I also renamed the revlog variable from "r" to
"rl" - again in the name of consistency.
2017-05-06 11:16:59 -07:00
Gregory Szorc
86020b42fd perf: clear revlog caches on every iteration
cmdutil.openrevlog() may return a cached revlog instance. This /may/
be a recent "regression" due to refactoring of the manifest API. I'm
not sure.

Either way, this perf command was broken for at least manifests because
subsequent invocations of the perf function would get cache hits from
previous invocations, invalidating results. In the extreme case,
testing the last revision in the revlog resulted in near-instantanous
execution of subsequent runs (since the fulltext is cached). A time
of ~1us would be reported in this case.
2017-05-06 11:15:56 -07:00
Gregory Szorc
6d8d6e3d41 perf: don't convert rev to node before calling revlog.revision() 2017-05-06 11:12:23 -07:00
Gregory Szorc
5d6e940365 revlog: rename _chunkraw to _getsegmentforrevs()
This completes our rename of internal revlog methods to
distinguish between low-level raw revlog data "segments" and
higher-level, per-revision "chunks."

perf.py has been updated to consult both names so it will work
against older Mercurial versions.
2017-05-06 12:12:53 -07:00
Gregory Szorc
e83c5e1628 perf: store reference to revlog._chunkraw in a local variable
To prepare for renaming revlog._chunkraw, we stuff a reference to this
metho in a local variable. This does 2 things. First, it moves the
attribute lookup outside of a loop, which more accurately measures
the time of the code being invoked. Second, it allows us to alias
to different methods depending on their presence (perf.py needs to
support running against old Mercurial versions).

Removing an attribute lookup from a tigh loop appears to shift the
numbers slightly with mozilla-central:

$ hg perfrevlogchunks -c

! read
! wall 0.354789 comb 0.340000 user 0.330000 sys 0.010000 (best of 28)
! wall 0.335932 comb 0.330000 user 0.290000 sys 0.040000 (best of 30)
! read w/ reused fd
! wall 0.342326 comb 0.340000 user 0.320000 sys 0.020000 (best of 29)
! wall 0.332857 comb 0.340000 user 0.290000 sys 0.050000 (best of 30)
! read batch
! wall 0.023623 comb 0.020000 user 0.000000 sys 0.020000 (best of 124)
! wall 0.023666 comb 0.020000 user 0.000000 sys 0.020000 (best of 125)
! read batch w/ reused fd
! wall 0.023828 comb 0.020000 user 0.000000 sys 0.020000 (best of 124)
! wall 0.023556 comb 0.020000 user 0.000000 sys 0.020000 (best of 126)
2017-05-06 12:02:31 -07:00
Gregory Szorc
d84946776c perf: move gettimer() call
This is more consistent with other perf* functions.
2017-05-06 11:01:02 -07:00
Gregory Szorc
c89bd8ad72 perf: don't clobber startrev variable
Previously, the "startrev" argument would be ignored due to
"startrev = 0" in the benchmark function. This meant that
`hg perfrevlog` always started at revision 0.

Rename the local variable to "beginrev" so the variable does the
right thing.
2017-05-06 10:59:38 -07:00
Yuya Nishihara
d9d64e114f bdiff: proxy through mdiff module
See the previous commit for why.

mdiff seems a good place to host bdiff functions. bdiff.bdiff was already
aliased as textdiff, so we use it.
2017-04-26 22:03:37 +09:00
Philippe Pepiot
e5957f96bd perf: add historical portability for util.timer
util.timer has been introduced in ae5d60bb and used in perf.py since 22fbca1d.
For historical portability, forcibly define util.timer in perf.py
2017-04-06 14:41:42 +02:00
Gregory Szorc
5ca0f908bf py3: add __bool__ to every class defining __nonzero__
__nonzero__ was renamed to __bool__ in Python 3. This patch simply
aliases __bool__ to __nonzero__ for every class implementing
__nonzero__.
2017-03-13 12:40:14 -07:00
Gregory Szorc
ddf0e60af7 perf: perform a garbage collection before each iteration
Currently, no explicit garbage collection is performed when running
the microbenchmarks in `hg perf`. I think this is wrong because
garbage collection can have a significant impact on execution times.
And, if gc is triggered via the default heuristics, it will
fire effectively randomly during subsequent benchmark iterations
due to variable amount of garbage left over from previous runs.

Running a gc before invoking the measured function will help ensure
state is more consistent across all iterations.
2017-03-13 18:16:42 -07:00
Simon Farnsworth
01a98361c5 contrib: add a write microbenchmark to perf.py
I'm adding some performance logging to ui.write - this benchmark lets us
confirm that the cost of that logging is acceptably low.

At this point, the microbenchmark on Linux over SSH shows:

! wall 3.213560 comb 0.410000 user 0.350000 sys 0.060000 (best of 4)

while on the Mac locally, it shows:

! wall 0.342325 comb 0.180000 user 0.110000 sys 0.070000 (best of 20)
2017-02-15 13:07:26 -08:00
Simon Farnsworth
e0b70e4f7f mercurial: switch to util.timer for all interval timings
util.timer is now the best available interval timer, at the expense of not
having a known epoch. Let's use it whenever the epoch is irrelevant.
2017-02-15 13:17:39 -08:00
Gregory Szorc
5c3d4574aa perf: split obtaining chunks from decompression
Previously, the code was similar to what revlog._chunks() was doing,
which took a raw data segment and delta chain, obtained buffers for
the raw revlog chunks within, and decompressed them.

This commit splits the "get raw chunks" action from "decompress." The
goal of this change is to more accurately measurely decompression
performance.

On a ~50k deltachain for a manifest in mozilla-central:

! full
! wall 0.430548 comb 0.440000 user 0.410000 sys 0.030000 (best of 24)
! deltachain
! wall 0.016053 comb 0.010000 user 0.010000 sys 0.000000 (best of 181)
! read
! wall 0.008078 comb 0.010000 user 0.000000 sys 0.010000 (best of 362)
! rawchunks
! wall 0.033785 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
! decompress
! wall 0.327126 comb 0.320000 user 0.320000 sys 0.000000 (best of 31)
! patch
! wall 0.032391 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)
! hash
! wall 0.012587 comb 0.010000 user 0.010000 sys 0.000000 (best of 233)
2017-02-04 08:47:46 -08:00
Gregory Szorc
b2e18c67d3 perf: support multiple compression engines in perfrevlogchunks
Now that the revlog has a reference to a compressor, it is
possible to swap in other compression engines. So, teach
`hg perfrevlogchunks` to do that.

The default behavior of `hg perfrevlogchunks` is now to measure the
compression performance of all compression engines implementing the
revlog compressor API. This effectively adds the no-op "none"
compressor and zstd (when available) into the default set.

While we can't yet plug alternate compressors into revlogs, this
command gives us a preview of the performance. On the mozilla-unified
repository:

$ hg perfrevlogchunks -c
! compress w/ none
! wall 0.115159 comb 0.110000 user 0.110000 sys 0.000000 (best of 86)
! compress w/ zlib
! wall 5.681406 comb 5.680000 user 5.680000 sys 0.000000 (best of 3)
! compress w/ zstd
! wall 2.624781 comb 2.620000 user 2.620000 sys 0.000000 (best of 4)

$ hg perfrevlogchunks -m
! compress w/ none
! wall 0.124486 comb 0.120000 user 0.120000 sys 0.000000 (best of 79)
! compress w/ zlib
! wall 10.144701 comb 10.150000 user 10.150000 sys 0.000000 (best of 3)
! compress w/ zstd
! wall 4.383118 comb 4.390000 user 4.390000 sys 0.000000 (best of 3)

Those numbers for zstd look promising. But they aren't the full story.
For that, we'll need to look at decompression times and storage sizes.
Stay tuned...
2017-01-02 12:02:08 -08:00
Gregory Szorc
1a6670d670 revlog: move decompress() from module to revlog class (API)
Upcoming patches will convert revlogs to use the compression engine
APIs to perform all things compression. The yet-to-be-introduced
APIs support a persistent "compressor" object so the same object
can be reused for multiple compression operations, leading to
better performance. In addition, compression engines like zstd
may wish to tweak compression engine state based on the revlog
(e.g. per-revlog compression dictionaries).

A global and shared decompress() function will shortly no longer
make much sense. So, we move decompress() to be a method of the
revlog class. It joins compress() there.

On the mozilla-unified repo, we can measure the impact of this change
on reading performance:

$ hg perfrevlogchunks -c
! chunk
! wall 1.932573 comb 1.930000 user 1.900000 sys 0.030000 (best of 6)
! wall 1.955183 comb 1.960000 user 1.930000 sys 0.030000 (best of 6)
! chunk batch
! wall 1.787879 comb 1.780000 user 1.770000 sys 0.010000 (best of 6
! wall 1.774444 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)

"chunk" appeared to become slower but "chunk batch" got faster. Upon
further examination by running both sets multiple times, the numbers
appear to converge across all runs. This tells me that there is no
perceived performance impact to this refactor.
2017-01-02 13:00:16 -08:00
Remi Chaintron
cc88d4a3c4 revlog: merge hash checking subfunctions
This patch factors the behavior of both methods into 'checkhash'.
2016-12-13 14:21:36 +00:00
Gregory Szorc
67e45331a8 perf: add command for measuring revlog chunk operations
Upcoming commits will teach revlogs to leverage the new compression
engine API so that new compression formats can more easily be
leveraged in revlogs. We want to be sure this refactoring doesn't
regress performance. So this commit introduces "perfrevchunks" to
explicitly test performance of reading, decompressing, and
recompressing revlog chunks.

Here is output when run on the mozilla-unified repo:

$ hg perfrevlogchunks -c
! read
! wall 0.346603 comb 0.350000 user 0.340000 sys 0.010000 (best of 28)
! read w/ reused fd
! wall 0.337707 comb 0.340000 user 0.320000 sys 0.020000 (best of 30)
! read batch
! wall 0.013206 comb 0.020000 user 0.000000 sys 0.020000 (best of 221)
! read batch w/ reused fd
! wall 0.013259 comb 0.030000 user 0.010000 sys 0.020000 (best of 222)
! chunk
! wall 1.909939 comb 1.910000 user 1.900000 sys 0.010000 (best of 6)
! chunk batch
! wall 1.750677 comb 1.760000 user 1.740000 sys 0.020000 (best of 6)
! compress
! wall 5.668004 comb 5.670000 user 5.670000 sys 0.000000 (best of 3)

$ hg perfrevlogchunks -m
! read
! wall 0.365834 comb 0.370000 user 0.350000 sys 0.020000 (best of 26)
! read w/ reused fd
! wall 0.350160 comb 0.350000 user 0.320000 sys 0.030000 (best of 28)
! read batch
! wall 0.024777 comb 0.020000 user 0.000000 sys 0.020000 (best of 119)
! read batch w/ reused fd
! wall 0.024895 comb 0.030000 user 0.000000 sys 0.030000 (best of 118)
! chunk
! wall 2.514061 comb 2.520000 user 2.480000 sys 0.040000 (best of 4)
! chunk batch
! wall 2.380788 comb 2.380000 user 2.360000 sys 0.020000 (best of 5)
! compress
! wall 9.815297 comb 9.820000 user 9.820000 sys 0.000000 (best of 3)

We already see some interesting data, such as how much slower
non-batched chunk reading is and that zlib compression appears to be
>2x slower than decompression.

I didn't have the data when I wrote this commit message, but I ran this
on Mozilla's NFS-based Mercurial server and the time for reading with a
reused file descriptor was faster. So I think it is worth testing both
with and without file descriptor reuse so we can make informed
decisions about recycling file descriptors.
2016-11-17 20:17:51 -08:00
Gregory Szorc
1c6b8908b4 perf: unbust perfbdiff --alldata
This broke in c7236da49964 due to a refactored manifest API.

The fix is a bit hacky - perfbdiff doesn't yet support tree manifests
for example. But it gets the job done.

A test has been added for --alldata so this doesn't happen again.
2016-11-17 08:52:52 -08:00
Philippe Pepiot
b115cec2e6 perf: omit copying ui and redirect to ferr if buffer API is in use
This allow to get the output of contrib/perf.py commands using the
ui.pushbuffer() API.
2016-11-15 16:10:57 +01:00
Durham Goode
240c640350 manifest: move clearcaches to manifestlog
This is part of removing all functionality from manifest.manifest so we can
delete the class entirely.
2016-11-10 02:13:19 -08:00
Durham Goode
64058b3c19 manifest: remove usages of manifest.read
Now that the two manifestctx implementations have working read() functions,
let's remove the existing uses of manifest.read and drop the function.
2016-11-10 02:13:19 -08:00
Gregory Szorc
426cc425cb perf: support measuring bdiff for all changeset related data
The --all argument changes the behavior of `perfbdiff` to pull
in fulltext revision pairs for all changes related to a changeset.
The p1 and p2 manifests will be bdiffed against current. Every file
that changed between p1 and current will have its file revisions
loaded and bdiffed.

This mode of operation effectively measured the bdiff time required
for `hg commit`.
2016-11-06 10:46:55 -08:00
Gregory Szorc
590ec2f6d1 perf: support bdiffing multiple revisions in a single revlog
This is useful for testing bdiff performance on several revision
pairs at a time.
2016-11-06 11:01:25 -08:00
Gregory Szorc
69b3988dc7 perf: prepare to handle multiple pairs in perfbdiff
Before, we only supported benchmarking a single pair of texts
with bdiff. We want to enable feeding larger corpora into this
benchmark. So rewrite the code to support that.
2016-11-06 09:51:14 -08:00
Gregory Szorc
51504da4ad perf: add perfbdiff
bdiff shows up a lot in profiling. I think it would be useful to have
a perf command that runs bdiff over and over so we can find hot spots.
2016-11-05 23:41:52 -07:00
Durham Goode
871d515e3d manifest: make manifestlog a storecache
The old @property on manifestlog was broken. It meant that we would always
recreate the manifestlog instance, which meant the cache was never hit. Since
we'll eventually remove repo.manifest and make manifestlog the only property,
let's go ahead and make manifestlog the @storecache property, have manifestlog
own the manifest instance, and have repo.manifest refer to it via manifestlog.

This means all accesses go through repo.manifestlog, which is now invalidated
correctly.
2016-10-18 17:33:39 -07:00
FUJIWARA Katsunori
fc0b0c4b06 perf: make perftags clear tags cache correctly
Before this patch, "hg perftags" command doesn't measure performance
of "repo.tags()" correctly, because it doesn't clear tags cache
correctly.

a043ed82a5cd replaced repo._tags with repo._tagscache, but didn't
change the code path to clear tags cache in perftags() at that time.
BTW, full history of "tags cache" is:

  - b8d757d45f24 (or 0.6) introduced repo.tagscache as the first "tags cache"
  - 4cbf51c74e8c (or 1.4) replaced repo.tagscache with repo._tags
  - a043ed82a5cd (or 2.0) replaced repo._tags with repo._tagscache
  - 04c204f1ed65 (or 2.5) made repo._tagscache filteredpropertycache

To make perftags clear tags cache correctly, and to increase
"historical portability" of perftags, this patch examines existence of
attributes in repo object, and guess appropriate procedure to clear
tags cache.

To avoid examining existence of attributes at each repetition, this
patch makes repocleartagscachefunc() return the function, which
actually clears tags cache.

mozilla-central repo (85 tags on 308365 revs) with each Mercurial
version between before and after this patch.

  ==== ========= =========
  ver  before    after
  ==== ========= =========
  1.9  0.476062  0.466464
       ------- *1 -------
  2.0  0.346309  0.458327
  2.1  0.343106  0.454489
       ------- *2 -------
  2.2  0.069790  0.071263
  2.3  0.067829  0.069340
  2.4  0.068075  0.069573
       ------- *3 -------
  2.5  0.021896  0.022406
  2.6  0.021900  0.022374
  2.7  0.021883  0.022379
  2.8  0.021949  0.022327
  2.9  0.021877  0.022330
  3.0  0.021860  0.022314
  3.1  0.021869  0.022669
  3.2  0.021831  0.022668
  3.3  0.021809  0.022691
  3.4  0.021861  0.022916
  3.5  0.019335  0.020749
  3.6  0.019319  0.020866
  3.7  0.018781  0.020251
       ------- *4 -------
  3.8  0.068262  0.072558
  3.9  0.069682  0.073773
  ==== ========= =========

(*1) repo._tags was replaced with repo._tagscache at this point

     "repo._tags = None" in perftags "before" this patch doesn't clear
     tags cache for Mercurial 2.0 or later. This causes significant
     gap of "before" between 1.9 and 2.0 .

(*2) I'm not sure about significant gap at this point, but release
     note of 2.2 described "a number of significant performance
     improvements for large repositories"

(*3) filtered changelog was cached in repoview as repoview.changelog
     at this point (by 131b01a4654d)

     This avoids calculation of filtered changelog at each repetition
     of t().

(*4) calculation of filtered changelog was included into wall time at
     this point (by adf01efe43a5), again

     See below for detail about this significant gap:

     https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-April/083410.html
2016-10-09 01:03:20 +09:00
FUJIWARA Katsunori
37e273ea35 perf: replace ui.configint() by getint() for Mercurial earlier than 1.9
Before this patch, using ui.configint() prevents perf.py from
measuring performance with Mercurial earlier than 1.9 (or
12e7e9fbf243), because ui.configint() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9 in perf.py.

For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).

This patch replaces ui.configint() invocations by newly introduced
getint().

This patch also adds check-perf-code.py an extra check entry to detect
direct usage of ui.configint() in perf.py.

BTW, this patch doesn't choose adding configint() method at runtime by
replacing ui.__class__ like below, even though this is the recommended
way to modern Mercurial extensions.

    def uisetup(ui):
        if not util.safehasattr(ui, 'configint'):
            class uiwrap(ui.__class__):
                def configint(self, section, name, ....):
                    ....
            ui.__class__ = uiwrap

Because changes to ui.__class__ by uisetup() of loaded extension have
been propagated since 1.6.1 (or 07a6e7bd0cc1), the recommended way
above doesn't work as expected with Mercurial earlier than it.
2016-10-09 01:03:19 +09:00
FUJIWARA Katsunori
87f398d18d perf: omit copying from ui.ferr to ui.fout for Mercurial earlier than 1.9
Before this patch, referring ui.ferr prevents perf.py from measuring
performance with Mercurial earlier than 1.9 (or bac01d164cc1), because
ui.ferr isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 1.9 in perf.py.

For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
2016-10-09 01:03:19 +09:00
FUJIWARA Katsunori
31be956823 perf: define formatter locally for Mercurial earlier than 2.2
Before this patch, using ui.formatter() prevents perf.py from
measuring performance with Mercurial earlier than 2.2 (or
045c8375c770), because ui.formatter() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 2.2 in perf.py.

For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).

This patch defines formatter class locally, and use it instead of the
value returned by ui.formatter(), if perf.py is used with Mercurial
earlier than 2.2.

In this case, we don't need to think about -T/--template option for
formatter, because previous patch made -T/--template disabled for
perf.py with Mercurial earlier than 3.2 (or 44a82ed65df7).
2016-10-09 01:03:18 +09:00
FUJIWARA Katsunori
dd96bd471b perf: add functions to get vfs-like object for Mercurial earlier than 2.3
Before this patch, using svfs prevents perf.py from measuring
performance of Mercurial earlier than 2.3 (or 12df7401e8cd), because
svfs isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 2.3 in perf.py.

For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).

To get appropriate vfs-like object to access files under .hg/store,
this patch adds getsvfs() (and also getvfs(), for future use).

To avoid examining existence of attribute at each repetition while
measuring performance, getsvfs() is invoked outside the function to be
called repeatedly.

This patch also adds check-perf-code.py an extra check entry to detect
direct usage of repo.(vfs|svfs|opener|sopener) in perf.py.
2016-10-09 01:03:18 +09:00
FUJIWARA Katsunori
956b59a347 perf: avoid actual writing branch cache out correctly
Mercurial 2.5 (or 0eb7dcc721cb) introduced "perfbranchmap" command,
and tried to avoid actual writing branch cache out by replacing
write() of branchcache class in branchmap.py with no-op function
(probably, for elimination of noisy and heavy file I/O factor).

But its implementation isn't correct, because 0eb7dcc721cb replaced
not branchmap.branchcache.write() but branchmap.write(). The latter
doesn't exist, even at that change.

To avoid actual writing branch cache out correctly, this patch
replaces branchmap.branchcache.write() with no-op function.

To detect mistake of replacement or change of API in the future
quickly, this patch uses safeattrsetter() instead of direct attribute
assignment. For similarity between replacements, this patch also
changes replacement of branchmap.read().

In this patch, replacement of read()/write() can run safely outside
"try" block, because two safeattrsetter() invocations ensure that
replacement doesn't cause exception.

FYI, the table below compares "base" filter wall time of perfbranchmap
on recent mozilla-central repo with each Mercurial version between
before and after this patch.

  ==== ========= =========
  ver  before    after
  ==== ========= =========
  2.5  18.492334 18.232455
  2.6  18.733858 18.156702
  2.7  18.245598 18.349210
  2.8  18.289070 18.528422
  2.9  17.572742 16.989655
  3.0  17.406953 17.615012
  3.1  17.228419 17.689805
  3.2  17.862961 17.718367
  3.3   2.632110  2.707960
  3.4   3.285683  3.272060
  3.5   3.370141  3.352176
  3.6   3.366939  3.242455
  3.7   3.300778  3.367328
  3.8   3.300132  3.267298
  3.9   3.418996  3.370265
  ==== ========= =========

IMHO, there is no serious overlooking performance regression.
2016-10-09 01:03:17 +09:00
FUJIWARA Katsunori
9d79ac8dbc perf: get subsettable from appropriate module for Mercurial earlier than 2.9
Before this patch, using branchmap.subsettable prevents perfbranchmap
from measuring performance of Mercurial earlier than 2.9 (or
aad678a92970), because aad678a92970 moved subsettable from repoview.py
to branchmap.py, even though there are some code paths for Mercurial
earlier than 2.9 in perf.py.

For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).

To get subsettable from appropriate module, this patch examines
existence of subsettable in branchmap and repoview.

This patch also adds check-perf-code.py an extra check entry to detect
direct usage of subsettable attribute in perf.py.
2016-10-09 01:03:17 +09:00
FUJIWARA Katsunori
0f1711011f perf: introduce safeattrsetter to replace direct attribute assignment
Referring not-existing attribute immediately causes failure, but
assigning a value to such attribute doesn't.

For example, perf.py has code paths below, which assign a value to
not-existing attribute. This causes incorrect performance measurement,
but these code paths are executed successfully.

  - "repo._tags = None" in perftags()
    recent Mercurial has tags cache information in repo._tagscache

  - "branchmap.write = lambda repo: None" in perfbranchmap()
    branchmap cache is written out by branchcache.write() in branchmap.py

"util.safehasattr() before assignment" can avoid this issue, but might
increase mistake at "copy & paste" attribute name or so.

To centralize (1) examining existence of, (2) assigning a value to,
and (3) restoring an old value to the attribute, this patch introduces
safeattrsetter(). This is used to replace direct attribute assignment
in subsequent patches.

Encapsulation of restoring is needed to completely remove direct
attribute assignment from perf.py, even though restoring isn't needed
so often.
2016-10-09 01:03:16 +09:00
Pierre-Yves David
21997e4a02 perf: release lock after transaction in perffncachewrite
The transaction should be closed within the lock.
2016-08-11 14:51:19 +02:00
Gregory Szorc
55983533a4 perf: add perfchangegroupchangelog command
This command can be used for testing the performance of producing the
changelog portion of a changegroup.

We could use additional perf* commands for testing other parts of
changegroup. Those can be written another time, when they are needed.
(And those may want to refactor the changegroup generation API so code
can be reused.) Speaking of code reuse, yes, this command does reinvent
a small wheel. I didn't want to scope bloat to change the changegroup
API because that will invite bikeshedding.
2016-09-24 12:22:30 -07:00
Gregory Szorc
46efca30f4 perf: add --reverse to perfrevlog
It can be useful to know how fast we can read revisions from a revlog
in reverse. This operation tends to occur in `hg log` commands,
for example.
2016-09-24 10:44:37 -07:00
FUJIWARA Katsunori
8bd5aefb64 perf: import newer modules separately for earlier Mercurial
demandimport of early Mercurial loads an imported module immediately,
if a module is imported absolutely by "from a import b" style. Recent
perf.py satisfies this condition, because it does:

  - have "from __future__ import absolute_import" line
  - use "from a import b" style for modules in "mercurial" package

Before this patch, importing modules below prevents perf.py from being
loaded by earlier Mercurial, because these aren't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9.

  - branchmap 2.5 (or e3354ba12eef)
  - repoview  2.5 (or 7d207cb7e38a)
  - obsolete  2.3 (or b50a017cd9ac)
  - scmutil   1.9 (or 065064cdde5f)

For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with Mercurial earlier than 1.8 (or
1299f0c14572).

After this patch, "mercurial.error" is the only blocker in "from
mercurial import" statement for loading perf.py with Mercurial earlier
than 1.2. This patch ignores it, because just importing it separately
isn't enough.
2016-05-20 09:47:35 +09:00
FUJIWARA Katsunori
6fdee58a92 perf: define command annotation locally for Mercurial earlier than 3.1
Before this patch, using cmdutil.command() for "@command" annotation
prevents perf.py from being loaded by Mercurial earlier than 1.9 (or
d4096ee63f8e), because cmdutil.command() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9.

For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).

In addition to it, "norepo" option of command annotation has been
available since 3.1 (or 9cbb59f03d57), and this is another blocker for
loading perf.py with earlier Mercurial.

  ============ ============ ======
               command of
  hg version   cmdutil      norepo
  ============ ============ ======
  3.1 or later      o         o
  1.9 or later      o         x
  earlier           x         x
  ============ ============ ======

This patch defines "command()" for annotation locally as below:

  - define wrapper of existing cmdutil.command(), if cmdutil.command()
    doesn't support "norepo"
    (for Mercurial earlier than 3.1)

  - define full command() locally with minimum function, if
    cmdutil.command() isn't available at runtime
    (for Mercurial earlier than 1.9)

This patch also defines parsealiases() locally without examining
whether it is available or not, because it is small enough to define
locally.
2016-07-05 07:25:51 +09:00
FUJIWARA Katsunori
a838d26c49 perf: avoid using formatteropts for Mercurial earlier than 3.2
Before this patch, referring commands.formatteropts prevents perf.py
from being loaded by Mercurial earlier than 3.2 (or 44a82ed65df7),
because it isn't available in such Mercurial, even though formatting
itself has been available since 2.2 (or 045c8375c770).

In addition to it, there are some code paths for Mercurial earlier
than 3.2. For example, setting "_prereadsize" attribute in perfindex()
and perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).

This patch uses empty option list as formatteropts, if it isn't
available in commands module at runtime.

Disabling -T/--template option for earlier Mercurial should be
reasonable, because:

  - since 0c03b2a1206a, -T/--template for formatter has been available
  - since 44a82ed65df7, commands.formatteropts has been available
  - the latter revision is direct child of the former
2016-07-05 07:25:51 +09:00