The last commit removed the last use of the bundle2 part seek() API
in the generic bundle2 part iteration code. This means we can now
switch to using unseekable bundle2 parts by default and have the
special consumers that actually need the behavior request it.
This commit changes unbundle20.iterparts() to expose non-seekable
unbundlepart instances by default. If seekable parts are needed,
callers can pass "seekable=True." The bundlerepo class needs
seekable parts, so it does this.
The interrupt handler is also changed to use a regular unbundlepart.
So, by default, all consumers except bundlerepo will see unseekable
parts.
Because the behavior of the iterparts() benchmark changed, we add
a variation to test seekable parts vs unseekable parts. And because
parts no longer have seek() unless "seekable=True," we update the
"part seek" benchmark.
Speaking of benchmarks, this change has the following impact to
`hg perfbundleread` on an uncompressed bundle of the Firefox repo
(6,070,036,163 bytes):
! read(8k)
! wall 0.722709 comb 0.720000 user 0.150000 sys 0.570000 (best of 14)
! read(16k)
! wall 0.602208 comb 0.590000 user 0.080000 sys 0.510000 (best of 17)
! read(32k)
! wall 0.554018 comb 0.560000 user 0.050000 sys 0.510000 (best of 18)
! read(128k)
! wall 0.520086 comb 0.530000 user 0.020000 sys 0.510000 (best of 20)
! bundle2 forwardchunks()
! wall 2.996329 comb 3.000000 user 2.300000 sys 0.700000 (best of 4)
! bundle2 iterparts()
! wall 8.070791 comb 8.060000 user 7.180000 sys 0.880000 (best of 3)
! wall 6.983756 comb 6.980000 user 6.220000 sys 0.760000 (best of 3)
! bundle2 iterparts() seekable
! wall 8.132131 comb 8.110000 user 7.160000 sys 0.950000 (best of 3)
! bundle2 part seek()
! wall 10.370142 comb 10.350000 user 7.430000 sys 2.920000 (best of 3)
! wall 10.860942 comb 10.840000 user 7.790000 sys 3.050000 (best of 3)
! bundle2 part read(8k)
! wall 8.599892 comb 8.580000 user 7.720000 sys 0.860000 (best of 3)
! wall 7.258035 comb 7.260000 user 6.470000 sys 0.790000 (best of 3)
! bundle2 part read(16k)
! wall 8.265361 comb 8.250000 user 7.360000 sys 0.890000 (best of 3)
! wall 7.099891 comb 7.080000 user 6.310000 sys 0.770000 (best of 3)
! bundle2 part read(32k)
! wall 8.290308 comb 8.280000 user 7.330000 sys 0.950000 (best of 3)
! wall 6.964685 comb 6.950000 user 6.130000 sys 0.820000 (best of 3)
! bundle2 part read(128k)
! wall 8.204900 comb 8.150000 user 7.210000 sys 0.940000 (best of 3)
! wall 6.852867 comb 6.850000 user 6.060000 sys 0.790000 (best of 3)
The significant speedup is due to not incurring the overhead to track
payload offset data. Of course, this overhead is proportional to
bundle2 part size. So a multiple gigabyte changegroup part is on the
extreme side of the spectrum for real-world impact.
In addition to the CPU efficiency wins, not tracking offset data
also means not using memory to hold that data. Using a bundle based on
the example BSD repository in issue 5691, this change has a drastic
impact to memory usage during `hg unbundle` (`hg clone` would behave
similarly). Before, memory usage incrementally increased for the
duration of bundle processing. In other words, as we advanced through
the changegroup and bundle2 part, we kept allocating more memory to
hold offset data. After this change, we still increase memory during
changegroup application. But the rate of increase is significantly
slower. (A bulk of the remaining gradual increase appears to be the
storing of revlog sizes in the transaction object to facilitate
rollback.)
The RSS at the end of filelog application is as follows:
Before: ~752 MB
After: ~567 MB
So, we were storing ~185 MB of offset data that we never even used.
Talk about wasteful!
.. api::
bundle2 parts are no longer seekable by default.
.. perf::
bundle2 read I/O throughput significantly increased.
.. perf::
Significant memory use reductions when reading from bundle2 bundles.
On the BSD repository, peak RSS during changegroup application
decreased by ~185 MB from ~752 MB to ~567 MB.
Differential Revision: https://phab.mercurial-scm.org/D1390
Upcoming commits will be refactoring bundle2 I/O code.
This commit establishes a `hg perfbundleread` command that measures
how long it takes to read a bundle using various mechanisms.
As a baseline, here's output from an uncompressed bundle1
bundle of my Firefox repo (7,098,622,890 bytes):
! read(8k)
! wall 0.763481 comb 0.760000 user 0.160000 sys 0.600000 (best of 6)
! read(16k)
! wall 0.644512 comb 0.640000 user 0.110000 sys 0.530000 (best of 16)
! read(32k)
! wall 0.581172 comb 0.590000 user 0.060000 sys 0.530000 (best of 18)
! read(128k)
! wall 0.535183 comb 0.530000 user 0.010000 sys 0.520000 (best of 19)
! cg1 deltaiter()
! wall 0.873500 comb 0.880000 user 0.840000 sys 0.040000 (best of 12)
! cg1 getchunks()
! wall 6.283797 comb 6.270000 user 5.570000 sys 0.700000 (best of 3)
! cg1 read(8k)
! wall 1.097173 comb 1.100000 user 0.400000 sys 0.700000 (best of 10)
! cg1 read(16k)
! wall 0.810750 comb 0.800000 user 0.200000 sys 0.600000 (best of 13)
! cg1 read(32k)
! wall 0.671215 comb 0.670000 user 0.110000 sys 0.560000 (best of 15)
! cg1 read(128k)
! wall 0.597857 comb 0.600000 user 0.020000 sys 0.580000 (best of 15)
And from an uncompressed bundle2 bundle (6,070,036,163 bytes):
! read(8k)
! wall 0.676997 comb 0.680000 user 0.160000 sys 0.520000 (best of 15)
! read(16k)
! wall 0.592706 comb 0.590000 user 0.080000 sys 0.510000 (best of 17)
! read(32k)
! wall 0.529395 comb 0.530000 user 0.050000 sys 0.480000 (best of 16)
! read(128k)
! wall 0.491270 comb 0.490000 user 0.010000 sys 0.480000 (best of 19)
! bundle2 forwardchunks()
! wall 2.997131 comb 2.990000 user 2.270000 sys 0.720000 (best of 4)
! bundle2 iterparts()
! wall 12.247197 comb 10.670000 user 8.170000 sys 2.500000 (best of 3)
! bundle2 part seek()
! wall 11.761675 comb 10.500000 user 8.240000 sys 2.260000 (best of 3)
! bundle2 part read(8k)
! wall 9.116163 comb 9.110000 user 8.240000 sys 0.870000 (best of 3)
! bundle2 part read(16k)
! wall 8.984362 comb 8.970000 user 8.110000 sys 0.860000 (best of 3)
! bundle2 part read(32k)
! wall 8.758364 comb 8.740000 user 7.860000 sys 0.880000 (best of 3)
! bundle2 part read(128k)
! wall 8.749040 comb 8.730000 user 7.830000 sys 0.900000 (best of 3)
We already see some interesting data. Notably that bundle2 has
significant overhead compared to bundle1. This matters for e.g. stream
clone bundles, which can be applied at >1Gbps.
Differential Revision: https://phab.mercurial-scm.org/D1385
A bit of useless trivia found while researching this: OpenBSD's grep's -C has a
default value (of 2) and disallows space before the argument (while -A and -B
allow).
The set-like object returned by dirstate.dirs may be difficult for other
implementations of the dirstate to provide, and is unnecessary as it is
only ever used for __contains__. Instead, provide an explicit method for
testing for a directory.
.. api::
dirstate no longer provides a `dirs()` method. To test for the existence of
a directory in the dirstate, use `dirstate.hasdir(dirname)`.
Differential Revision: https://phab.mercurial-scm.org/D1345
This is a short topic to explain how command-line flags can be specified.
Some users have been confused by hg offerring different flag syntax than some
other libraries, so it'd be nice to point them to this rather than explaining
it every time.
Differential Revision: https://phab.mercurial-scm.org/D1270
Reverts 5b2411bab704 and replaces it with a more general solution.
- works for both rpm and deb
- sidesteps eventual problems with local extensions that have nothing to do with
the build process (hg-git, for example, fails with version 4.4 because
ccf72242b638 removed peerrepository, and hg-git still uses it as of 0.8.9)
# skip-blame because parsers.c is mechanically rewritten by
clang-format with no semantic change.
Differential Revision: https://phab.mercurial-scm.org/D1170
For now all .c and .h files are blacklisted. As they become
clang-formatted, we'll remove them from the blacklist,and then this
test will produce output if there are diffs.
Differential Revision: https://phab.mercurial-scm.org/D1133
We're moving towards a clang-format world, and clang-format is able to
wrap argument lists with spaces reliably, while still enforcing tabs
globally. Let's let clang-format do its job, and not do as much
C-style enforcement with regular expressions.
Differential Revision: https://phab.mercurial-scm.org/D1130
When bootstrapping a deb/rpm build, packagelib.sh starts performing a local
build for the sole purpose of parsing the output of "hg version".
Then it "hg archive"s the source code, and builds everything again.
For that initial step, we are perfectly good in using a pure python mercurial,
without compiling the c modules (base85, bdiff, zstdlib, ...).
On my personal system, this cuts down 22 seconds for a package build (the
bootstrapping build goes from ~30 to ~8 seconds).
Now that dirstatemap is the source of truth for the list of directories, let's
move _dirfoldmap on to it.
This pattern of moving cached variables onto the dirstate map makes it easier to
invalidate them, as seen by how the cache invalidation functions are slowly
shrinking to just be recreating the dirstatemap instance.
Differential Revision: https://phab.mercurial-scm.org/D983
Now that dirs is source of truthed on the dirstatemap, let's get rid of the
_dirs propertycache on the dirstate.
Differential Revision: https://phab.mercurial-scm.org/D982
Now that the filefoldmap is source of truthed on the dirstatemap, let's get rid
of the property cache on the dirstate.
Differential Revision: https://phab.mercurial-scm.org/D981
As part of separating dirstate business logic from storage, let's move the
nonnormal and otherparent storage to the dirstatemap class. This will allow
alternative dirstate storage to persist these sets instead of recomputing them.
Differential Revision: https://phab.mercurial-scm.org/D979
This rule is no longer useful because chg daemon may be killed and respawned
per config/environment hash. We can't reliably run a daemon in foreground.
Callbacks registered with 'atexit' are sometimes not called (like when
hg aborts and calls os._exit). On the other hand, callbacks registered with
'ui.atexit' are called in most cases. Therefore, encouraging the use of
'ui.atexit' by failing the test 'test-check-code.t' appropriately.
Test Plan:
Ran the test 'test-check-code.t' after importing 'atexit' in one of
the py files and confirmed that the test fails.
Differential Revision: https://phab.mercurial-scm.org/D961
unittest has a `maxDiff` parameter which has to be set to `None` in order for
large enough failure diffs to be displayed. Add a comment to disable the
camelcase check for `self.maxDiff = None` lines.
Differential Revision: https://phab.mercurial-scm.org/D895
This location is used by debian (and ubuntu) to store completions provided by
other deb packages. The default fpath appears to have this before any of the
zsh-provided instances of the completions, so this should take precedence.
Differential Revision: https://phab.mercurial-scm.org/D882
These have annoyed me for a long time, and I'm tired of commenting on
them in reviews. I'm sorry for how complicated the regular expression
is, but I was too lazy to go crack open pylint's code and add the
check there.
The arguments are especially non-obvious because the order is
different from dirstate.status().
Differential Revision: https://phab.mercurial-scm.org/D846
If the call to dirstate.walk() failed, we would try to fall back to
older versions. These were removed in 869e4a22ee0d (walk: remove
cmdutil.walk, 2008-05-12) and 4f67ccefb6ef (dirstate: fold statwalk
and walk, 2008-06-26). We don't care about testing performance of
versions that old versions at this point, so let's clean up.
Differential Revision: https://phab.mercurial-scm.org/D845
I've poked at this on and off several times, and I can't figure out
what regressed. Let's kick this out of the whitelist for now so that
we can get the *rest* of our progress covered by the buildbots.
Not sure why, but I got `phabsend` hang on work network pretty frequently.
The traceback indicates it hangs at `_sslobj.do_handshake()`:
File "mercurial/sslutil.py", line 404, in wrapsocket
sslsocket = sslcontext.wrap_socket(sock, server_hostname=serverhostname)
File "/usr/lib/python2.7/ssl.py", line 363, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 611, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 840, in do_handshake
self._sslobj.do_handshake()
I had tried adding `timeout` in various places but they seem not effective.
It seems easier to just allow shelling out to `curl` with retry and timeout
flags.
This could also be helpful for people with an older Python installed without
modern security (SNI).
Differential Revision: https://phab.mercurial-scm.org/D605
Previously, the `--confirm` text could have colors but the main `phabsend`
does not. This patch adjusts the main command so it also has colors.
A default color table was added so the colors are visible by default.
Differential Revision: https://phab.mercurial-scm.org/D515
Some platforms (ex. FreeBSD) do not have `bash` by default. Therefore it
should not be used in test scripts.
Differential Revision: https://phab.mercurial-scm.org/D609
This would make the checker more friendly for 3rd-party code. For example,
In remotefilelog/x.py, it may have:
from . import shallowutils
That could trigger "relative import of stdlib module" if
"remotefilelog" was installed in the system. If the module being checked
conflicts with the system module, it makes sense to not treat that module as
system module. This patch makes it so.
Differential Revision: https://phab.mercurial-scm.org/D552
Often people running `phabsend --confirm` just want to check whether a
commit will trigger a creation of new Differential Revision, or update an
existing one. This patch implements that. The `--confirm` message was
changed to use node instead of revision number to be consistent with what
`phabsend` outputs.
An example output looks like:
D487 - a80f447973a0 test-extension: enable demandimport explicitly
D494 - cf440ea6e47e test-casecollision-merge: fix the test
NEW - 0a6b97147128 phabsend: polish the docstring a bit
Send the above changes to https://phab.mercurial-scm.org/ (yn)?
Differential Revision: https://phab.mercurial-scm.org/D514
Sometimes people have multiple Phabricator endpoints set in multiple repos.
It seems better for `--confirm` to prompt about the Phabricator endpoint
patches being sent to.
Differential Revision: https://phab.mercurial-scm.org/D513
Previously phabsend has an optimization that will skip uploading a diff if
the patch (with context line number = 1) remains unchanged. That could be
confusing:
Aug 24 15:52:28 <martinvonz> phillco: something is wrong with phabricator'your patches/
Aug 24 15:52:45 <martinvonz> ... with phabricator's view of your patches again
Aug 24 15:53:38 <martinvonz> if i phabread D388 and then D399, i get a version of filemerge.py with "a, b, c" somewhere on line 344, which is not what phabricator shows for D399
Aug 24 15:53:51 <martinvonz> junw: maybe that's more for you ^
Fix that by checking context with 32767 lines, which is the same as what
will be actually sent.
Differential Revision: https://phab.mercurial-scm.org/D512
The local tag feature was intended to make `phabsend` closer to `email`
workflow. But its experience is not great in multiple ways:
- after rebase, obsoleted changesets are still visible because of tags
- without obsstore, the association information will get lost
- even with obsstore, things could go wrong with graft, export+import
- no easy way to tell which Differential Revision a commit is associated
Therefore make `--amend` the default. People wanting the old behavior can
use `--no-amend`.
Differential Revision: https://phab.mercurial-scm.org/D511
run-tests.py and import-checker.py take wholly different approaches to
.t-tests, and the result was that they had different notions about
when a Python heredoc had ended, leading to conflicts. This resolves
part of the issue, and the rest I can deal with by tweaking the one
offending test file.
`simplemerge()` will soon require context-like objects to work. Create a simple
context-like object that wraps the requested files and can be passed to the new
API.
Differential Revision: https://phab.mercurial-scm.org/D378
I didn't do anything to ensure correctness here, just enough to avoid
tracebacks in the import checker, which uses the native ast module to
try and parse all our Python files.
Changing status (accept, etc) on the webpage is not very convenient -
currently there is no way to accept (or abandon etc.) a stack using a single
click or command.
This patch adds a `phabupdate` command that could be used to change status
in batch. It also supports `--comment` which will write a comment on the
last revision, which is similar to what we do using emails.
Differential Revision: https://phab.mercurial-scm.org/D127
This patch adds status words (ex. `abandoned`, `accepted`, `needsreview`,
`needsrevision`, `closed`) to the revision query language so people can
select revision in a more flexible way.
Test Plan:
Try something like `phabread ':2 & accepted'`, `phabread ':105 - closed` and
make sure they have desired outputs.
Differential Revision: https://phab.mercurial-scm.org/D126
Previously, `phabread` can only be used to read a single patch, or a single
stack of patches. In the future, we want to have more complex queries like
filtering with status (open, accepted, closed, etc), or maybe more complex
like filtering by reviewers etc. The command line flag approach won't scale
with that.
Besides, we might want to have other commands to update Differential
Revision status in batch, like accepting a stack using a single command.
Therefore, this patch adds a small language. It has basic set operations:
`&`, `+`, `-` and an ancestor operator to support `--stack`.
Test Plan:
Try querying this Phabricator instance:
hg phabread 1+2 # 1, 2
hg phabread D2+D1 # 2, 1
hg phabread ':118-115+:2-1' # 114, 116, 117, 118, 2
hg phabread '((:118-(D115+117)))&:117' # 114, 116
hg phabread ':2&:117' --debug # differential.query is called only once
Make sure the output is expected and prefetch works.
Differential Revision: https://phab.mercurial-scm.org/D125
Previously, `readpatch` and `querydrev` take a same `params` and `stack`
parameters. This patch changes `readpatch` so it takes the output of
`querydrev`, not the input of `querydrev`. This makes the code more
flexible and cleaner.
Differential Revision: https://phab.mercurial-scm.org/D124
Previously `hg phabsend` was imitating `hg email` and won't mutate
changesets. That works fine with reviewer-push workflow, reviewers run
`phabread`, `import`.
However, it does not work well with author-push workflow. Namely, the author
needs to run extra commands to get the right commit message, and remove the
local tag after push.
This patch solves those issues by adding the `--amend` option, so local
changesets will have the right commit message, and tags become unnecessary.
Test Plan:
Given the following DAG:
o 17
o 16
| o 15
| @ 14
|/
o 13
o 12
Run `hg phabsend '(13::)-17' --amend`, check the new DAG looks like:
o 21
| o 20
| @ 19
|/
o 18
| o 17
| x 16
| x 13
|/
o 12
And commit messages are updated to contain the `Differential Revision` lines.
Use `phabread` to make sure Phabricator has the amended node recorded.
Also check `phabsend .` followed by a `phabsend . --amend`, the commit
message will be updated and the tag will be removed.
Differential Revision: https://phab.mercurial-scm.org/D122
Without this flag, compilation fails with:
hgclient.c: In function 'hgc_open':
hgclient.c:466: error: 'O_DIRECTORY' undeclared (first use in this function)
hgclient.c:466: error: (Each undeclared identifier is reported only once
hgclient.c:466: error: for each function it appears in.)
Differential Revision: https://phab.mercurial-scm.org/D260
CentOS 5 has reached end-of-life. A working build for it is still
possible using 'vault.centos.org'.
Differential Revision: https://phab.mercurial-scm.org/D261
This is a quick fix to make `hg phabread D189` work.
It seems we might want to replace all `r''` to `u''`, and add more
`encoding.*to*` to be more explicit when interacting with `json` module.
Differential Revision: https://phab.mercurial-scm.org/D192
If you're shipping prerelease or rc packages using Munki, you'll
eventually discover that Munki's version comparison logic is not as
good as pip's. In theory we should be able to fix Munki, but it seems
entirely reasonable to produce version strings that sort reasonably
under these conditions. Since the requried logic not brief, add a new
script and some tests of that logic.
A followup change will wire this into the Makefile.
5ba3f753c9b1 introduced a use of 'hg revert --no-backup' which is
currently broken. Rather than try and do a fix on the (complicated)
revert code during the freeze, let's just give up on this test under
Python 3 until later. Once we do fix things, the test ratchet script
will re-enable it.
