The last commit removed the last use of the bundle2 part seek() API
in the generic bundle2 part iteration code. This means we can now
switch to using unseekable bundle2 parts by default and have the
special consumers that actually need the behavior request it.
This commit changes unbundle20.iterparts() to expose non-seekable
unbundlepart instances by default. If seekable parts are needed,
callers can pass "seekable=True." The bundlerepo class needs
seekable parts, so it does this.
The interrupt handler is also changed to use a regular unbundlepart.
So, by default, all consumers except bundlerepo will see unseekable
parts.
Because the behavior of the iterparts() benchmark changed, we add
a variation to test seekable parts vs unseekable parts. And because
parts no longer have seek() unless "seekable=True," we update the
"part seek" benchmark.
Speaking of benchmarks, this change has the following impact to
`hg perfbundleread` on an uncompressed bundle of the Firefox repo
(6,070,036,163 bytes):
! read(8k)
! wall 0.722709 comb 0.720000 user 0.150000 sys 0.570000 (best of 14)
! read(16k)
! wall 0.602208 comb 0.590000 user 0.080000 sys 0.510000 (best of 17)
! read(32k)
! wall 0.554018 comb 0.560000 user 0.050000 sys 0.510000 (best of 18)
! read(128k)
! wall 0.520086 comb 0.530000 user 0.020000 sys 0.510000 (best of 20)
! bundle2 forwardchunks()
! wall 2.996329 comb 3.000000 user 2.300000 sys 0.700000 (best of 4)
! bundle2 iterparts()
! wall 8.070791 comb 8.060000 user 7.180000 sys 0.880000 (best of 3)
! wall 6.983756 comb 6.980000 user 6.220000 sys 0.760000 (best of 3)
! bundle2 iterparts() seekable
! wall 8.132131 comb 8.110000 user 7.160000 sys 0.950000 (best of 3)
! bundle2 part seek()
! wall 10.370142 comb 10.350000 user 7.430000 sys 2.920000 (best of 3)
! wall 10.860942 comb 10.840000 user 7.790000 sys 3.050000 (best of 3)
! bundle2 part read(8k)
! wall 8.599892 comb 8.580000 user 7.720000 sys 0.860000 (best of 3)
! wall 7.258035 comb 7.260000 user 6.470000 sys 0.790000 (best of 3)
! bundle2 part read(16k)
! wall 8.265361 comb 8.250000 user 7.360000 sys 0.890000 (best of 3)
! wall 7.099891 comb 7.080000 user 6.310000 sys 0.770000 (best of 3)
! bundle2 part read(32k)
! wall 8.290308 comb 8.280000 user 7.330000 sys 0.950000 (best of 3)
! wall 6.964685 comb 6.950000 user 6.130000 sys 0.820000 (best of 3)
! bundle2 part read(128k)
! wall 8.204900 comb 8.150000 user 7.210000 sys 0.940000 (best of 3)
! wall 6.852867 comb 6.850000 user 6.060000 sys 0.790000 (best of 3)
The significant speedup is due to not incurring the overhead to track
payload offset data. Of course, this overhead is proportional to
bundle2 part size. So a multiple gigabyte changegroup part is on the
extreme side of the spectrum for real-world impact.
In addition to the CPU efficiency wins, not tracking offset data
also means not using memory to hold that data. Using a bundle based on
the example BSD repository in issue 5691, this change has a drastic
impact to memory usage during `hg unbundle` (`hg clone` would behave
similarly). Before, memory usage incrementally increased for the
duration of bundle processing. In other words, as we advanced through
the changegroup and bundle2 part, we kept allocating more memory to
hold offset data. After this change, we still increase memory during
changegroup application. But the rate of increase is significantly
slower. (A bulk of the remaining gradual increase appears to be the
storing of revlog sizes in the transaction object to facilitate
rollback.)
The RSS at the end of filelog application is as follows:
Before: ~752 MB
After: ~567 MB
So, we were storing ~185 MB of offset data that we never even used.
Talk about wasteful!
.. api::
bundle2 parts are no longer seekable by default.
.. perf::
bundle2 read I/O throughput significantly increased.
.. perf::
Significant memory use reductions when reading from bundle2 bundles.
On the BSD repository, peak RSS during changegroup application
decreased by ~185 MB from ~752 MB to ~567 MB.
Differential Revision: https://phab.mercurial-scm.org/D1390
Upcoming commits will be refactoring bundle2 I/O code.
This commit establishes a `hg perfbundleread` command that measures
how long it takes to read a bundle using various mechanisms.
As a baseline, here's output from an uncompressed bundle1
bundle of my Firefox repo (7,098,622,890 bytes):
! read(8k)
! wall 0.763481 comb 0.760000 user 0.160000 sys 0.600000 (best of 6)
! read(16k)
! wall 0.644512 comb 0.640000 user 0.110000 sys 0.530000 (best of 16)
! read(32k)
! wall 0.581172 comb 0.590000 user 0.060000 sys 0.530000 (best of 18)
! read(128k)
! wall 0.535183 comb 0.530000 user 0.010000 sys 0.520000 (best of 19)
! cg1 deltaiter()
! wall 0.873500 comb 0.880000 user 0.840000 sys 0.040000 (best of 12)
! cg1 getchunks()
! wall 6.283797 comb 6.270000 user 5.570000 sys 0.700000 (best of 3)
! cg1 read(8k)
! wall 1.097173 comb 1.100000 user 0.400000 sys 0.700000 (best of 10)
! cg1 read(16k)
! wall 0.810750 comb 0.800000 user 0.200000 sys 0.600000 (best of 13)
! cg1 read(32k)
! wall 0.671215 comb 0.670000 user 0.110000 sys 0.560000 (best of 15)
! cg1 read(128k)
! wall 0.597857 comb 0.600000 user 0.020000 sys 0.580000 (best of 15)
And from an uncompressed bundle2 bundle (6,070,036,163 bytes):
! read(8k)
! wall 0.676997 comb 0.680000 user 0.160000 sys 0.520000 (best of 15)
! read(16k)
! wall 0.592706 comb 0.590000 user 0.080000 sys 0.510000 (best of 17)
! read(32k)
! wall 0.529395 comb 0.530000 user 0.050000 sys 0.480000 (best of 16)
! read(128k)
! wall 0.491270 comb 0.490000 user 0.010000 sys 0.480000 (best of 19)
! bundle2 forwardchunks()
! wall 2.997131 comb 2.990000 user 2.270000 sys 0.720000 (best of 4)
! bundle2 iterparts()
! wall 12.247197 comb 10.670000 user 8.170000 sys 2.500000 (best of 3)
! bundle2 part seek()
! wall 11.761675 comb 10.500000 user 8.240000 sys 2.260000 (best of 3)
! bundle2 part read(8k)
! wall 9.116163 comb 9.110000 user 8.240000 sys 0.870000 (best of 3)
! bundle2 part read(16k)
! wall 8.984362 comb 8.970000 user 8.110000 sys 0.860000 (best of 3)
! bundle2 part read(32k)
! wall 8.758364 comb 8.740000 user 7.860000 sys 0.880000 (best of 3)
! bundle2 part read(128k)
! wall 8.749040 comb 8.730000 user 7.830000 sys 0.900000 (best of 3)
We already see some interesting data. Notably that bundle2 has
significant overhead compared to bundle1. This matters for e.g. stream
clone bundles, which can be applied at >1Gbps.
Differential Revision: https://phab.mercurial-scm.org/D1385
A bit of useless trivia found while researching this: OpenBSD's grep's -C has a
default value (of 2) and disallows space before the argument (while -A and -B
allow).
The set-like object returned by dirstate.dirs may be difficult for other
implementations of the dirstate to provide, and is unnecessary as it is
only ever used for __contains__. Instead, provide an explicit method for
testing for a directory.
.. api::
dirstate no longer provides a `dirs()` method. To test for the existence of
a directory in the dirstate, use `dirstate.hasdir(dirname)`.
Differential Revision: https://phab.mercurial-scm.org/D1345
This is a short topic to explain how command-line flags can be specified.
Some users have been confused by hg offerring different flag syntax than some
other libraries, so it'd be nice to point them to this rather than explaining
it every time.
Differential Revision: https://phab.mercurial-scm.org/D1270
Reverts 5b2411bab704 and replaces it with a more general solution.
- works for both rpm and deb
- sidesteps eventual problems with local extensions that have nothing to do with
the build process (hg-git, for example, fails with version 4.4 because
ccf72242b638 removed peerrepository, and hg-git still uses it as of 0.8.9)
# skip-blame because parsers.c is mechanically rewritten by
clang-format with no semantic change.
Differential Revision: https://phab.mercurial-scm.org/D1170
For now all .c and .h files are blacklisted. As they become
clang-formatted, we'll remove them from the blacklist,and then this
test will produce output if there are diffs.
Differential Revision: https://phab.mercurial-scm.org/D1133
We're moving towards a clang-format world, and clang-format is able to
wrap argument lists with spaces reliably, while still enforcing tabs
globally. Let's let clang-format do its job, and not do as much
C-style enforcement with regular expressions.
Differential Revision: https://phab.mercurial-scm.org/D1130
When bootstrapping a deb/rpm build, packagelib.sh starts performing a local
build for the sole purpose of parsing the output of "hg version".
Then it "hg archive"s the source code, and builds everything again.
For that initial step, we are perfectly good in using a pure python mercurial,
without compiling the c modules (base85, bdiff, zstdlib, ...).
On my personal system, this cuts down 22 seconds for a package build (the
bootstrapping build goes from ~30 to ~8 seconds).
Now that dirstatemap is the source of truth for the list of directories, let's
move _dirfoldmap on to it.
This pattern of moving cached variables onto the dirstate map makes it easier to
invalidate them, as seen by how the cache invalidation functions are slowly
shrinking to just be recreating the dirstatemap instance.
Differential Revision: https://phab.mercurial-scm.org/D983
Now that dirs is source of truthed on the dirstatemap, let's get rid of the
_dirs propertycache on the dirstate.
Differential Revision: https://phab.mercurial-scm.org/D982
Now that the filefoldmap is source of truthed on the dirstatemap, let's get rid
of the property cache on the dirstate.
Differential Revision: https://phab.mercurial-scm.org/D981
As part of separating dirstate business logic from storage, let's move the
nonnormal and otherparent storage to the dirstatemap class. This will allow
alternative dirstate storage to persist these sets instead of recomputing them.
Differential Revision: https://phab.mercurial-scm.org/D979
This rule is no longer useful because chg daemon may be killed and respawned
per config/environment hash. We can't reliably run a daemon in foreground.
Callbacks registered with 'atexit' are sometimes not called (like when
hg aborts and calls os._exit). On the other hand, callbacks registered with
'ui.atexit' are called in most cases. Therefore, encouraging the use of
'ui.atexit' by failing the test 'test-check-code.t' appropriately.
Test Plan:
Ran the test 'test-check-code.t' after importing 'atexit' in one of
the py files and confirmed that the test fails.
Differential Revision: https://phab.mercurial-scm.org/D961
unittest has a `maxDiff` parameter which has to be set to `None` in order for
large enough failure diffs to be displayed. Add a comment to disable the
camelcase check for `self.maxDiff = None` lines.
Differential Revision: https://phab.mercurial-scm.org/D895
This location is used by debian (and ubuntu) to store completions provided by
other deb packages. The default fpath appears to have this before any of the
zsh-provided instances of the completions, so this should take precedence.
Differential Revision: https://phab.mercurial-scm.org/D882
These have annoyed me for a long time, and I'm tired of commenting on
them in reviews. I'm sorry for how complicated the regular expression
is, but I was too lazy to go crack open pylint's code and add the
check there.
The arguments are especially non-obvious because the order is
different from dirstate.status().
Differential Revision: https://phab.mercurial-scm.org/D846
If the call to dirstate.walk() failed, we would try to fall back to
older versions. These were removed in 869e4a22ee0d (walk: remove
cmdutil.walk, 2008-05-12) and 4f67ccefb6ef (dirstate: fold statwalk
and walk, 2008-06-26). We don't care about testing performance of
versions that old versions at this point, so let's clean up.
Differential Revision: https://phab.mercurial-scm.org/D845
I've poked at this on and off several times, and I can't figure out
what regressed. Let's kick this out of the whitelist for now so that
we can get the *rest* of our progress covered by the buildbots.
Not sure why, but I got `phabsend` hang on work network pretty frequently.
The traceback indicates it hangs at `_sslobj.do_handshake()`:
File "mercurial/sslutil.py", line 404, in wrapsocket
sslsocket = sslcontext.wrap_socket(sock, server_hostname=serverhostname)
File "/usr/lib/python2.7/ssl.py", line 363, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 611, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 840, in do_handshake
self._sslobj.do_handshake()
I had tried adding `timeout` in various places but they seem not effective.
It seems easier to just allow shelling out to `curl` with retry and timeout
flags.
This could also be helpful for people with an older Python installed without
modern security (SNI).
Differential Revision: https://phab.mercurial-scm.org/D605
Previously, the `--confirm` text could have colors but the main `phabsend`
does not. This patch adjusts the main command so it also has colors.
A default color table was added so the colors are visible by default.
Differential Revision: https://phab.mercurial-scm.org/D515
Some platforms (ex. FreeBSD) do not have `bash` by default. Therefore it
should not be used in test scripts.
Differential Revision: https://phab.mercurial-scm.org/D609
This would make the checker more friendly for 3rd-party code. For example,
In remotefilelog/x.py, it may have:
from . import shallowutils
That could trigger "relative import of stdlib module" if
"remotefilelog" was installed in the system. If the module being checked
conflicts with the system module, it makes sense to not treat that module as
system module. This patch makes it so.
Differential Revision: https://phab.mercurial-scm.org/D552
