Bundle2 compression is more complex than the bundle1 one. Therefore it
is handled by the bundler itself. Moreover, on-disk bundle2 will
probably have a large number of flavors so simply adding a new "format"
for it does not seems the way to go.
This will be used in the next changeset to compress bundle2 strip backup.
The bundle10 format (plain changegroup-01) does not support general delta and
result into expensive delta re-computation when stripping. If the repository is
general delta, we store backups as bundle20 containing a changegroup-02 payload.
We remove the experimental feature related to strip backup bundle format because
this achieve the same goal in a leaner way. Removing the experimental option is
fine, that is why it experimental in the first place.
Compression of these bundles are coming in later changesets.
This allows us to hide {branches} template keyword in a better way. Currently
it is achieved by copying keywords table and deleting 'branches' from it.
Future patches will require ui module to be passed to a help function, so
let's use it where appropriate.
Additional parens are necessary to silence a check-code warning.
Because _phaserevs and _phasesets cache revision numbers, they must be
invalidated if there are new commits or stripped revisions. We could do
that by calling _phasecache.invalidate(), but it wasn't simple to be
integrated with the filecache mechanism.
So for now, phasecache will be recreated after repo.invalidate() if
00changelog.i was modified before.
Because _phaserevs and _phasesets cache revision numbers, they must be
invalidated if there are new commits or stripped revisions. We could do
that by calling _phasecache.invalidate(), but it wasn't simple to be
integrated with the filecache mechanism.
So for now, phasecache will be recreated after repo.invalidate() if
00changelog.i was modified before.
This avoids some visual flickering of the progress bar in convert and
probably some other operations. Previously, a line of output would
erase the progress bar, and then it would wait `progress.refresh`
seconds (default of 0.1) before redrawing the progress bar. Now if
we've ever painted a progress bar, we schedule the progress bar for
immediate repainting on the next progress call, which helps lend the
illusion that the progress bar is "always" there. In practice, it's
merely there more of the time, but it ends up being a lot easier to
read during convert.
This changeset adds support for a 'compression' parameter in bundle2 streams.
When set, it controls the compression algorithm used for the payload part of the
bundle2.
There is currently no usage of this except in tests.
The old lazy-copy code formed a chain of copied manifests with each
copy. Under typical operation, the stack never got more than a couple
of manifests deep and was fine. Under conditions like hgsubversion or
convert, the stack could get hundreds of manifests deep, and
eventually overflow the recursion limit for Python. I was able to
consistently reproduce this by converting an hgsubversion clone of
svn's history to treemanifests.
This may result in fewer manifests staying in memory during operations
like convert when treemanifests are in use, and should make those
operations faster since there will be significantly fewer noop
function calls going on.
A previous attempt (never mailed) of mine to fix this problem tried to
simply have all treemanifests only have a loadfunc - that caused
somewhat weird problems because the gettext() callable passed into
read() wasn't idempotent, so the easy solution is to have a loadfunc
and a copyfunc.
When looking up a base revision, we were ignoring the contents that were already
available in the manifest's _mancache. This patch allows us to use that data
instead of reading from the revlog.
This is useful in our pushrebase extension (which allows rebasing on the server
side during a push) because it allows us to prefetch the bundle base manifest
before aquiring the repo lock (1 second saving), which means doing less work inside the lock,
which means a 20% higher commit rate.
We are about to allow compressing the core of the bundle2. So we extract the
generation of this bits in its own parts to make this compression phases easier
in a later changesets.
As a comment in the code have been asking for, it is now possible to register
piece of code that handle parameters for the stream. I've been wondering is such
function should be class methods or not. I eventually went for externally
decorated methods to stick with the culture of extensibility from an extensions
that apply to bundle2.
The original name explicitly mention "Part", however it is also used outside of
parts related feature. We rename from 'UnsupportedPartError' to
'BundleUnknownFeatureError' to fix this.
For "space saving", bundle1 "strip" the first two bytes of the BZ stream since
they always are 'BZ'. So the current code boostrap the uncompressor with 'BZ'.
This hack is impractical in more generic case so we move it in a dedicated
"decompression".
This is adapted from 635285e0a942, that was added to paper for 3.5 release.
It adds another way to refer to branches, tags and bookmarks in urls: by name.
It's still possible to navigate to a specific changeset hash, but now you can
get more descriptive urls straight from /summary page, for example.
branchentry template (and so the whole branches table on /summary and
/branches) lost the column that had a plain changeset hash, because tags and
bookmarks don't have this column and also because there is already a way to
address branch by its changeset hash (changeset link just next to it). Maybe we
can instead bring this column with a plain changeset hash to tags and
bookmarks, but this, more terse, new look feels fine.
This is adapted from 635285e0a942, that was added to paper for 3.5 release.
It adds another way to refer to branches, tags and bookmarks in urls: by name.
It's still possible to navigate to a specific changeset hash, but now you can
get more descriptive urls straight from /summary page, for example.
branchentry template (and so the whole branches table on /summary and
/branches) lost the column that had a plain changeset hash, because tags and
bookmarks don't have this column and also because there is already a way to
address branch by its changeset hash (changeset link just next to it). Maybe we
can instead bring this column with a plain changeset hash to tags and
bookmarks, but this, more terse, new look feels fine.
This is part of a series that will allow locks to be inherited by subprocesses
in limited circumstances. This patch enables the logic introduced in previous
patches.
The earlier test worked only because the held count went up to 2, so the first
release brought it down to 1. Making a copy of the lock fixes that issue.
The current behavior of revlogs is to flush the data file when writing
data to it. Tracing system calls revealed that changegroup processing
incurred numerous write(2) calls for values much smaller than the
default buffer size (Python defaults to 4096, but it can be adjusted
based on detected block size at run time by CPython).
The reason we flush revlogs is so readers have all data available.
For example, the current code in revlog.py will re-open the revlog
file (instead of seeking an existing file handle) to read the text
of a revision. This happens when starting a new delta chain when
adding several revisions from changegroups, for example. Yes, this
is likely sub-optimal (we should probably be sharing file descriptors
between readers and writers to avoid the flushing and associated
overhead of re-opening files).
While flushing revlogs is necessary, it appears all callers are
diligent about flushing files before a read is performed (see
buildtext() in _addrevision()), making the flush in
_writeentry() redundant and unncessary. So, we remove it. In practice,
this means we incur a write(2) a) when the buffer is full (typically
4096 bytes) b) when a new delta chain is created rather than after
every added revision. This applies to every revlog, but by volume
it mostly impacts filelogs.
Removing the redundant flush from _writeentry() significantly
reduces the number of write(2) calls during changegroup processing on
my Linux machine. When applying a changegroup of the hg repo based on
my local repo, the total number of write(2) calls during application
of the mercurial/localrepo.py revlogs dropped from 1,320 to 217 with
this patch applied. Total I/O related system calls dropped from 1,577
to 474.
When unbundling a mozilla-central gzipped bundle (264,403 changesets
with 1,492,215 changes to 222,507 files), total write(2) calls
dropped from 1,252,881 to 827,106 and total system calls dropped from
3,601,259 to 3,178,636 - a reduction of 425,775!
While the system call reduction is significant, it appears
to have no impact on wall time on my Linux and Windows machines. Still,
fewer syscalls is fewer syscalls. Surely this can't hurt. If nothing
else, it makes examining remaining system call usage simpler and opens
the door to experimenting with the performance impact of different
buffer sizes.
_addrevision() may need to read from revlogs as part of computing
deltas. Previously, we would flush existing file handles and open
a new, short-lived file handle to perform the reading.
If we have an existing file handle, it seems logical to reuse it
for reading instead of opening a new file handle. This patch
makes that the new behavior.
After this patch, revlog files are only reopened when adding
revisions if the revlog is switched from inline to non-inline.
On Linux when unbundling a bundle of the mozilla-central repo, this
patch has the following impact on system call counts:
Call Before After Delta
write 827,639 673,390 -154,249
open 700,103 684,089 -16,014
read 74,489 74,489 0
fstat 493,924 461,896 -32,028
close 249,131 233,117 -16,014
stat 242,001 242,001 0
lstat 18,676 18,676 0
lseek 20,268 20,268 0
ioctl 14,652 13,173 -1,479
TOTAL 3,180,758 2,930,679 -250,079
It's worth noting that many of the open() calls fail due to missing
files. That's why there are many more open() calls than close().
Despite the significant system call reduction, this change does not
seem to have a significant performance impact on Linux.
On Windows 10 (not a VM, on a SSD), this patch appears to reduce
unbundle time for mozilla-central from ~960s to ~920s. This isn't
as significant as I was hoping. But a decrease it is nonetheless.
Still, Windows unbundle performance is still >2x slower than Linux.
Despite the lack of significant gains, fewer system calls is fewer
system calls. If nothing else, this will narrow the focus of potential
areas to optimize in the future.
An upcoming patch will teach revlogs to use the existing file
handle to read revision data instead of opening a new file handle
just for quick reads. For this to work, files must be opened for
reading as well.
This patch is merely cosmetic: there are no behavior changes.
Currently, the low-level revlog reading code always opens a new file
handle. In some key scenarios, the revlog is already opened and an
existing file handle could be used to read. This patch paves the
road to that by teaching various revlog reading functions to accept
an optional existing file handle to read from.
fopen() and fdopen() have a unique-to-Windows requirement that
transitions between read and write operations in files opened
in modes r+, w+, and a+ perform a file positioning call
(fsetpos, fseek, or rewind) in between. While the MSDN docs don't
say what will happen if this is not done, observations reveal
that Python raises an IOError with errno 0. Furthermore, I
/think/ this behavior isn't deterministic. But I can reproduce
it reliably with subsequent patches applied that open revlogs
in a+ mode and perform both reads and writes.
This patch introduces a proxy class for file handles opened
in r+, w+, and a+ mode on Windows. The class intercepts calls
and audits whether a file positioning function has been called
between read and write operations. If not, a dummy, no-op seek
to the current file position is performed. This appears to be
sufficient to "trick" Windows into allowing transitions between
read and writes without raising errors.
Subversion 1.9 shows more verbose messages than 1.8 and the tests fail
because of them. These outputs are not important in our tests, so let's
suppress them by -q or grep -v.
