This allows revlog-style files to be written out with checkambig=True
easily.
Because avoiding file stat ambiguity is needed only for filecache-ed
manifest and changelog, this patch does:
- use False for default value of checkambig
- focus only on writing changes of index file out
This patch also adds optional argument checkambig to _divert/_delay
for changelog, to safely accept checkambig specified in revlog
layer. But this argument can be fully ignored, because:
- changes are written into other than index file, if name != target
- changes are never written into index file, otherwise
(into pending file by _divert, or into in-memory buffer by _delay)
This is a part of ExactCacheValidationPlan.
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
Profiling using statprof revealed a hotspot during changegroup
application calculating delta chain bases on generaldelta repos.
Essentially, revlog._addrevision() was performing a lot of redundant
work tracing the delta chain as part of determining when the chain
distance was acceptable. This was most pronounced when adding
revisions to manifests, which can have delta chains thousands of
revisions long.
There was a delta chain base cache on revlogs before, but it only
captured a single revision. This was acceptable before generaldelta,
when _addrevision would build deltas from the previous revision and
thus we'd pretty much guarantee a cache hit when resolving the delta
chain base on a subsequent _addrevision call. However, it isn't
suitable for generaldelta because parent revisions aren't necessarily
the last processed revision.
This patch converts the delta chain base cache to an LRU dict cache.
The cache can hold multiple entries, so generaldelta repos have a
higher chance of getting a cache hit.
The impact of this change when processing changegroup additions is
significant. On a generaldelta conversion of the "mozilla-unified"
repo (which contains heads of the main Firefox repositories in
chronological order - this means there are lots of transitions between
heads in revlog order), this change has the following impact when
performing an `hg unbundle` of an uncompressed bundle of the repo:
before: 5:42 CPU time
after: 4:34 CPU time
Most of this time is saved when applying the changelog and manifest
revlogs:
before: 2:30 CPU time
after: 1:17 CPU time
That nearly a 50% reduction in CPU time applying changesets and
manifests!
Applying a gzipped bundle of the same repo (effectively simulating a
`hg clone` over HTTP) showed a similar speedup:
before: 5:53 CPU time
after: 4:46 CPU time
Wall time improvements were basically the same as CPU time.
I didn't measure explicitly, but it feels like most of the time
is saved when processing manifests. This makes sense, as large
manifests tend to have very long delta chains and thus benefit the
most from this cache.
So, this change effectively makes changegroup application (which is
used by `hg unbundle`, `hg clone`, `hg pull`, `hg unshelve`, and
various other commands) significantly faster when delta chains are
long (which can happen on repos with large numbers of files and thus
large manifests).
In theory, this change can result in more memory utilization. However,
we're caching a dict of ints. At most we have 200 ints + Python object
overhead per revlog. And, the cache is really only populated when
performing read-heavy operations, such as adding changegroups or
scanning an individual revlog. For memory bloat to be an issue, we'd
need to scan/read several revisions from several revlogs all while
having active references to several revlogs. I don't think there are
many operations that do this, so I don't think memory bloat from the
cache will be an issue.
Before fd1bb7c, if the C index.partialmatch raises RevlogError, the Python
code raises "ambiguous identifier" error immediately, which is efficient.
fd1bb7c took hidden revisions into consideration and forced the slow path
enumerating the changelog to double-check hidden revisions. But it's not
necessary if we know the revlog has no hidden revisions.
This patch adds back the fast path for unfiltered revlogs.
Previously, we likely called _chunkraw() multiple times in order to
ensure it didn't change out from under us. I'm pretty certain this code
had its origins in the days where we attempted to have thread safety of
localrepository and thus revlog instances.
revlog instances are already not thread safe for writing. And, as of
Mercurial 3.6, hgweb uses a separate localrepository instance per
request, so there should only be a single thread reading a revlog at
a time. We more or less decided that attempting to make classes like
revlog thread safe is a lost cause.
So, this patch removes thread safety from _chunks. As a result, we make
one less call into _chunkraw() when the initial read isn't serviced
by the cache. This translates to savings of 4 function calls overall
and possibly prevents the creation of an additional buffer view into the
cache. I doubt this translates into any real world performance wins
because decompression will almost certainly dwarf time spent in
_chunks(). But it does make the code simpler, so it is an improvement.
A subsequent patch will refactor _chunks() and the calculation of the
offset will no longer occur in that function. Prepare by returning the
offset from _chunkraw().
Revlog chunks can be stored uncompressed. If the first byte of the
raw data is \0, we store the data as is. Else we prefix it with 'u'.
Before, we performed a string slice to strip out the 'u' prefix.
With this patch, we use a buffer to avoid an extra memory copy and
associated garbage collection overhead.
I was unable to verify any performance impact of this patch. For both
mozilla-central and the hg repos, the number of manifest revisions
with 'u' prefixes is very small - under 1%. So this change likely
isn't called enough to have an impact on manifest reading. However,
the reasoning behind this change is solid, so it should be safe.
This code is already written in multiple locations.
While this code needs to be fast and extracting it to its own function
adds overhead, code paths reading delta chains typically read,
decompress, and do binary patching on revlog data from the delta chain.
This other work (especially zlib decompression) almost certainly
accounts for a lot more time than the overhead of introducing a Python
function call. So I'm not worried about the performance impact of this
change.
clearcaches() was added several years ago in 1e47437a1ca7 as part
of implementing a perf command. Since revlog instances have many caches
and since the spirit of this mostly unused method is to facilitate
performance testing, I think it's appropriate for all the revlog's
caches to get cleared when it is called.
Revlogs were recently refactored to open file handles in "a+" and use a
persistent file handle for reading and writing. This drastically
reduced the number of file handles being opened.
Unfortunately, it appears that some versions of Solaris lose the file
offset when performing a write after the handle has been seeked.
The simplest workaround is to seek to EOF on files opened in a+ mode
before writing to them, which is what this patch does.
Ideally, this code would exist in the vfs layer. However, this would
require creating a proxy class for file objects in order to provide a
custom implementation of write(). This would add overhead. Since
revlogs are the only files we open in a+ mode, the one-off workaround
in revlog.py should be sufficient.
This patch appears to have little to no impact on performance on my
Linux machine.
In the most complex case, we try using the incoming delta base, then
we try both parents, and then we try the previous revlog entry. If
none of these result in a good delta, we natually use the null
revision as base. However, we sometimes consider the nullrev before we
have exhausted our other options. Specifically, when both parents are
null, we use the nullrev as delta base if it produces a good delta
(according to _isgooddelta()), and we fail to try the previous revlog
entry as delta base. After e60126c6093d (addrevision: use general
delta when the incoming base delta is bad, 2015-12-01), it can also
happen for non-merge commits when the incoming delta is not good.
The Firefox repo (from many months back) shrinks a tiny bit with this
patch: from 1.855GB to 1.830GB (1.4%). The hg repo itself shrinks even
less: by less than 0.1%. There may be repos that get larger instead.
This undoes the unexplained test change in e60126c6093d.
We always want to call _isgooddelta() before accepting a delta. We
mostly call the function right after building the delta, but not
always. Instead, we have an extra call at the end of the big code
block. Let's make it consistent, so we call _isgooddelta() right after
builddelta() and exactly once per delta. That also lets us rely on
"delta is None" to mean we didn't find a good delta.
The tested delta revisions are added to the 'tested' set. These are
the same revisions we pass to builddelta(). However, in one case, we
add builddelta(rev)[3] to the set intead of adding 'rev' itself. In
that particular case, that element is the same as the function's input
revision (because self._generaldelta is true), so the effect is the
same. Still, let's just add the function's input revision to avoid
confusing future readers.
We unify the delta selection process to be a simple three options process:
- try to use the incoming delta (if lazydeltabase is on)
- try to find a suitable parents to delta against (if gd is on)
- try to delta against the tipmost revision
The first of this option that yield a valid delta will be used.
The test change in 'test-generaldelta.t' show this behavior as we use a delta
against the parent instead of a full delta when the incoming delta is not
suitable.
This as some impact on 'test-bundle.t' because a delta somewhere changes. It
does not seems to change the test semantic and have been ignored.
The old code have multiple explicit tests and code duplications. This makes it
hard to improve the code. We rewrite the logic in a more generic way, not
changing anything of the computed result.
The final goal here is to eventually be able to:
- factor out the default fallback case "try against 'prev'" in a single place
- allow 'lazydeltabase' case to use the smarter general delta code path when
the incoming base does not provide us with a good delta.
Before this change, the 'lazydeltabase' would blindly build a delta using the
base provided by the incoming bundle and try to use it. If that base was far
down the revlog, the delta would be seen as "no good" and we would fall back to
a full text revision.
We now check if the delta is good and fallback to a computing a delta again the
tipmost revision otherwise (as we would do without general delta).
Later changesets will improve the logic to compute the fallback delta using the
general delta logic.
We would like to be able to exit the delta generation block without a valid
delta (for a more flexible control flow). So we make sure we do not expand the
"delta" content unless we actually have a delta.
We can do it one level lower because 'delta' is initialised at None anyway. Not
adding a level to the assignment prevent a line length issue.
The new convention is that bundles contain changegroups. bundle1
happens to *only* be a changegroup, but bundle2 is a more featureful
container that isn't something you can pass to addgroup().
The current behavior of revlogs is to flush the data file when writing
data to it. Tracing system calls revealed that changegroup processing
incurred numerous write(2) calls for values much smaller than the
default buffer size (Python defaults to 4096, but it can be adjusted
based on detected block size at run time by CPython).
The reason we flush revlogs is so readers have all data available.
For example, the current code in revlog.py will re-open the revlog
file (instead of seeking an existing file handle) to read the text
of a revision. This happens when starting a new delta chain when
adding several revisions from changegroups, for example. Yes, this
is likely sub-optimal (we should probably be sharing file descriptors
between readers and writers to avoid the flushing and associated
overhead of re-opening files).
While flushing revlogs is necessary, it appears all callers are
diligent about flushing files before a read is performed (see
buildtext() in _addrevision()), making the flush in
_writeentry() redundant and unncessary. So, we remove it. In practice,
this means we incur a write(2) a) when the buffer is full (typically
4096 bytes) b) when a new delta chain is created rather than after
every added revision. This applies to every revlog, but by volume
it mostly impacts filelogs.
Removing the redundant flush from _writeentry() significantly
reduces the number of write(2) calls during changegroup processing on
my Linux machine. When applying a changegroup of the hg repo based on
my local repo, the total number of write(2) calls during application
of the mercurial/localrepo.py revlogs dropped from 1,320 to 217 with
this patch applied. Total I/O related system calls dropped from 1,577
to 474.
When unbundling a mozilla-central gzipped bundle (264,403 changesets
with 1,492,215 changes to 222,507 files), total write(2) calls
dropped from 1,252,881 to 827,106 and total system calls dropped from
3,601,259 to 3,178,636 - a reduction of 425,775!
While the system call reduction is significant, it appears
to have no impact on wall time on my Linux and Windows machines. Still,
fewer syscalls is fewer syscalls. Surely this can't hurt. If nothing
else, it makes examining remaining system call usage simpler and opens
the door to experimenting with the performance impact of different
buffer sizes.
_addrevision() may need to read from revlogs as part of computing
deltas. Previously, we would flush existing file handles and open
a new, short-lived file handle to perform the reading.
If we have an existing file handle, it seems logical to reuse it
for reading instead of opening a new file handle. This patch
makes that the new behavior.
After this patch, revlog files are only reopened when adding
revisions if the revlog is switched from inline to non-inline.
On Linux when unbundling a bundle of the mozilla-central repo, this
patch has the following impact on system call counts:
Call Before After Delta
write 827,639 673,390 -154,249
open 700,103 684,089 -16,014
read 74,489 74,489 0
fstat 493,924 461,896 -32,028
close 249,131 233,117 -16,014
stat 242,001 242,001 0
lstat 18,676 18,676 0
lseek 20,268 20,268 0
ioctl 14,652 13,173 -1,479
TOTAL 3,180,758 2,930,679 -250,079
It's worth noting that many of the open() calls fail due to missing
files. That's why there are many more open() calls than close().
Despite the significant system call reduction, this change does not
seem to have a significant performance impact on Linux.
On Windows 10 (not a VM, on a SSD), this patch appears to reduce
unbundle time for mozilla-central from ~960s to ~920s. This isn't
as significant as I was hoping. But a decrease it is nonetheless.
Still, Windows unbundle performance is still >2x slower than Linux.
Despite the lack of significant gains, fewer system calls is fewer
system calls. If nothing else, this will narrow the focus of potential
areas to optimize in the future.
An upcoming patch will teach revlogs to use the existing file
handle to read revision data instead of opening a new file handle
just for quick reads. For this to work, files must be opened for
reading as well.
This patch is merely cosmetic: there are no behavior changes.
Currently, the low-level revlog reading code always opens a new file
handle. In some key scenarios, the revlog is already opened and an
existing file handle could be used to read. This patch paves the
road to that by teaching various revlog reading functions to accept
an optional existing file handle to read from.
revlog instances can cache the full text of a single revision. Typically
the most recently read revision is cached.
When adding a delta group via addgroup() and _addrevision(), the
full text isn't always computed: sometimes only the passed in delta is
sufficient for adding a new revision to the revlog.
When writing the changelog from a delta group, the just-added full
text revision is always read immediately after it is written because
the changegroup code needs to extract the set of files from the entry.
In other words, revision() is *always* being called and caching the full
text of the just-added revision is guaranteed to result in a cache hit,
making the cache worthwhile.
This patch adds support to _addrevision() for always building and
caching the full text. This option is currently only active when
processing changelog entries from a changegroup.
While the total number of revision() calls is the same, the location
matters: buildtext() calls into revision() on the base revision when
building the full text of the just-added revision. Since the previous
revision's _addrevision() built the full text and the the previous
revision is likely the base revision, this means that the base
revision's full text is likely cached and can be used to compute the
current full text from just a delta. No extra I/O required.
The end result is the changelog isn't opened and read after adding every
revision from a changegroup.
On my 2013 MacBook Pro running OS X 10.10.5 from an SSD and Python 2.7,
this patch impacted the time taken to apply ~262,000 changesets from a
mozilla-central gzip bundle:
before: ~43s
after: ~32s
~25% reduction in changelog processing times. Not bad.
The purpose of this code was to provide thread safety. With the
conversion of hgweb to use separate localrepository instances per
request/thread, we should no longer have any consumers that need to
access revlog instances from multiple threads. Remove the code.
This adds an option for delta'ing against both p1 and p2 when applying merge
revisions and picking whichever is smallest.
Some before and after stats on manifest.d size:
internal large repo:
before: 1.2 GB
after: 930 MB
mozilla-central:
before: 261 MB
after: 92 MB
The old generaldelta heuristic was "if p1 (or p2) was closer than the last full text,
use it, otherwise use prev". This was problematic when a repo contained multiple
branches that were very different. If commits to branch A were pushed, and the
last full text was branch B, it would generate a fulltext. Then if branch B was
pushed, it would generate another fulltext. The problem is that the last
fulltext (and delta'ing against `prev` in general) has no correlation with the
contents of the incoming revision, and therefore will always have degenerate
cases.
According to the blame, that algorithm was chosen to minimize the chain length.
Since there is already code that protects against that (the delta-vs-fulltext
code), and since it has been improved since the original generaldelta algorithm
went in (2011), I believe the chain length criteria will still be preserved.
The new algorithm always diffs against p1 (or p2 if it's closer), unless the
resulting delta will fail the delta-vs-fulltext check, in which case we delta
against prev.
Some before and after stats on manifest.d size.
internal large repo
old heuristic - 2.0 GB
new heuristic - 1.2 GB
mozilla-central
old heuristic - 242 MB
new heuristic - 261 MB
The regression in mozilla central is due to the new heuristic choosing p2r as
the delta when it's closer to the tip. Switching the algorithm to always prefer
p1r brings the size back down (242 MB). This is result of the way in which
mozilla does merges and pushes, and the result could easily swing the other
direction in other repos (depending on if they merge X into Y or Y into X), but
will never be as degenerate as before.
I future patch will address the regression by introducing an optional, even more
aggressive delta heuristic which will knock the mozilla manifest size down
dramatically.
This moves the textlen calculation to be above the delta chooser. Since textlen
is needed for calling isgooddelta, we need it above the delta chooser so future
patches can call isgooddelta.
This moves the delta vs fulltext comparison to its own function. This will allow
us to reuse the function in future patches for more efficient delta choices. As
a side effect, this will also allow extensions to modify our delta criteria.
This option will make repositories created as general delta by default but will
not make Mercurial aggressively recompute deltas for all incoming bundle.
Instead, the delta contained in the bundle will be used. This will allow us to
start having general delta repositories created everywhere without triggering
massive recomputation costs for all new clients cloning from old servers.
A subsequent patch will add a feature that performs iterative
computation as changesets are added from a changegroup. To facilitate
this type of processing in a generic manner, we add a mechanism for
calling a function whenever a revision is added via revlog.addgroup().
There are potential performance concerns with this callback, as using it
will flush the revlog after every revision is added.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Before we were relying on _pack to error out when trying to pass an
integer that was too large for the "i" format specifier. Now we check
this earlier so we can form a better error message.
The error message unfortunately must exclude the filename at this
level of the call stack. The problem is that this name is not
available here, and the error can be triggered by a large manifest or
by a large file itself. Although perhaps we could provide the name of
a revlog index file (from the revlog object, instead of the revlogio
object), this seems like too much leakage of internal data structures.
It's not ideal already that an error message even mentions revlogs,
but this does seem unavoidable here.
