Now that the nodemap is lazily created, we use linear scanning back
from tip for typical node to rev mapping. Given that nodemap creation
is O(n log n) and revisions searched for are usually very close to
tip, this is often a significant performance win for a small number of
searches.
When we do end up building a nodemap for bulk lookups, the scanning
function is replaced with a hash lookup.
We were not returning the correct result if nullrev was in revs, as we
are checking parent(currentrev) != nullrev before yielding currentrev
test-convert-hg-startrev was wrong: if we start converting from rev -1 and
onwards, all the descendants of -1 (full repo) should be converted.
When parentdelta is enabled, we choose the delta that has the minimum
distance to its base. Otherwise, base may be sufficiently far away to
require a full version, resulting in greatly reduced compression.
While reading changegroup if a node with missing parents is encountered,
we add a punched entry in the index with null parents for the missing
parent node.
Rename some variables, making the name more obvious (in particular "cache" was
actually two different variable.
Move code around, moving the index preloading before the deltachain computation,
without that index preloading was useless (everything was read in deltachain).
Refactor revlog._addrevision() and put the correct base rev in the
parent-delta case: base(rev) should always be equal to the first full snapshot
that is needed by the delta chain, in both parent-delta and tip-delta case.
Before this fix, the base rev was in most case wrong (and in the case where
p1 == nullid, this triggered the bug from issue2337). This means that
repositories converted to parent-delta earlier are corrupted and needs to be
reconverted.
index flag to identify a revision as punched, i.e. it contains no data.
REVIDX_PUNCHED_FLAG = 2, is used to mark a revision as punched.
REVIDX_KNOWN_FLAGS is the accumulation of all index flags.
This is similar to the __builtin__.cmp behaviour, but still not
straightforward, as the dailylife meaning of a comparison usually is
"find out if they are different".
Previously, it only returned revisions that were in the revlog when it
was originally opened; revisions added since then were invisible.
This broke revlog._partialmatch() and therefore repo.lookup().
(Credit to Benoit Boissinot for simplifying my original test script
and for the actual fix.)
This lets us change the threshold at which a *.d file will be split
out, which should make it much easier to construct test cases that
probe revlogs with a separate data file.
(issue2137)
This lets us change the threshold at which a *.d file will be split
out, which should make it much easier to construct test cases that
probe revlogs with a separate data file.
(issue2137)
If a buffer of an mutable object is passed to revlog.addrevision(), the revlog
will happily store it in its cache. Later when the revlog reuses the cached
entry, if the manifest modified the object in-between, all kind of bugs
appears.
We fix it by:
- passing immutable objects to addrevision() if they are already available
- only storing the text in the cache if it's of str type
Then we can remove the conversion of the cache entry to str() during
retrieval. That was probably just there hiding the bug for the common cases
but not really fixing it.
- chunk to _chunk
- _prime to _chunkraw
- _chunkclear for cache clearing
- _chunk calls _chunkraw
- clean up _prime a bit
- simplify users in revision and checkinlinesize
- drop file descriptor passing (we're better off opening fds lazily
The built-in None object is a singleton and it is therefore safe to
compare memory addresses with is. It is also faster, how much depends
on the object being compared. For a simple type like str I get:
| s = "foo" | s = None
----------+-----------+----------
s == None | 0.25 usec | 0.21 usec
s is None | 0.17 usec | 0.17 usec
Uses a transaction instance from the local repository to journal the
truncation of revlog files, such that if a strip only partially completes,
hg recover will be able to finish the truncate of all the files.
The potential unbundling of changes that have been backed up to be restored
later will, in case of an error, have to be unbundled manually. The
difference is that it will be possible to recover the repository state so
the unbundle can actually succeed.
Because we often compute sha1(nullid), it's interesting to copy a precomputed
hash of nullid instead of computing everytime the same hash. Similarly, when
one of the parents is null, we can avoid a < comparison (sort).
Overall, this change adds a string equality comparison on each hash() call,
but when p2 is null, we drop one string < comparison, and copy a hash instead
of computing it. Since it is common to have revisions with only one parent,
this change makes hash() 25% faster when cloning a big repository.
They are unnecessary. I did leave them in localrepo.py where there is
something like:
_junk = foo()
_junk = None
to free memory early. I don't know if just `foo()` will free the return
value as early.
- create error.py for exception classes to reduce demandloading
- move revlog exceptions to it
- change users to import error and drop revlog import if possible
changegroup() has a problem when nodes which does not descend from a node
in <bases> are added to remote after the discovery phase.
If that happens, changegroup() won't send the correct set of nodes, ie.
some nodes will be missing.
To correct it we have to find the set of nodes that both remote and self
have (called <common>), and send all the nodes not in <common>.
This fix has some overhead, in the worst case it will re-send a whole branch.
A proper fix to avoid this overhead might be to change the protocol so that
the <common> nodes are sent (instead of the <bases> of the missing nodes).
Previously, an unknown node id would lead to the following error:
abort: 00changelog.i@343445453433: no node!
All other unknown revision would instead display as:
abort: unknown revision '343445453'!
The former error message has been suppressed in favor of the latter.
This patch adds two methods to revlog:
- ancestors: given a list of revisions returns their ancestors
- descendants: given a list of revisions return their descendants
If there's no inline data, revlog.revision opens the data file every
time it's called. This is useful if we're going to call chunk many
times, but, if we're going to call it only once, it's better to let
chunk open the file - if we're lucky, all the data we're going to need
is already cached and we won't need to even look at the file.
When we remove revision N from the repository, all revisions >= N are
affected: either it's a descendant from N and will also be removed, or
it's not a descendant of N and will be renumbered.
As a consequence, we have to (at least temporarily) remove all filelog
and manifest revisions that have a linkrev >= N, readding some of them
later.
Unfortunately, it's possible to have a revlog with two revisions
r1 and r2 such that r1 < r2, but linkrev(r1) > linkrev(r2). If we try
to strip revision linkrev(r1) from the repository, we'll also lose
revision r2 when we truncate this revlog.
We already use changegroupsubset to create a temporary changegroup
containing the revisions that have to be restored, but that function is
unable to detect that we also wanted to save the r2 in the case above.
So we manually calculate these extra nodes and pass it to changegroupsubset.
This should fix issue764.
Python's zlib apparently makes an internal copy of strings passed to
compress(). To avoid this, compress strings 1M at a time, then join
them at the end if the result would be smaller than the original.
For initial commits of large but compressible files, this cuts peak
memory usage nearly in half.
- use a buffer to extract the delta from a chunk
- avoid concatenating to a compressed delta
- use a buffer to directly extra full text from a trivial delta
- delete chunk and delta objects after use
- handle chunk headers separately rather than prepending them to
(potentially large) chunks
- break large chunks into 1M pieces for compression
- don't prepend file metadata onto (potentially large) file data
To avoid extra memory usage and performance issues with large files,
generate a trivial delta header for deltas against the null revision
rather than calling the usual delta generator.
We append the delta header to meta rather than prepending it to data
to avoid a large allocate and copy.
We want to store version information about the revlog in the first
entry of its index. The code in packentry was using some heuristics
to detect whether this was the first entry, but these heuristics could
fail in some cases (e.g. rev 0 was empty; rev 1 descends directly from
the nullid and is stored as a delta).
We now give the revision number to packentry to avoid heuristics.
This function is fairly performance sensitive, so we make a couple
ugly tweaks:
- keep all entries packed so we needn't test entry types
- fold index lookup/load into unpack call to eliminate
local variable setting
- remove unused defaults for p1, p2, and text
- reduce some if/else
- use better variable names
- remove some extra variables
- remove some obsolete corner tests
- simply first entry handling for revlogng
- simply inline vs outofline writeout
We expand our index by one entry so that index[nullrev] points to a
unique entry, the null revision. This naturally eliminates numerous
extra tests in the performance-sensitive index access functions, most
of which are now trivial again.
Adding new entries is now done with insert(-1, e) rather than
append(e).
This way can use one additional bit, and when encountering invalid revlogs
with the first bit set don't produce python warnings or strange error messages.
This should fix issue255.
It looks like the problem there happens when addgroup calls addrevision
to add a full revision, and addrevision decides to split the index file
into a .i/.d pair. Since addgroup has an open file handle for the
index file, the renaming of the new .i file to its final name fails on
windows.
manifest.add gives revlog.addrevision a buffer object, which may
be cached and used for a second call in the same session (as mq does
when pushing multiple patches). The other option would be to cast the
buffer to str when caching it.
Instead of converting each node from the filenode to a hex form,
convert the arg to a bin form.
For a revlog with 26711 entries, doing 100 lookup:
before: ~18s
after : ~13s
- add comments
- do a clean separation of the different cases
- don't use a list of each possible node when
doing the lookup, just keep the previous entry
Make depth calculation non-recursive
Add simple shortcut for linear ancestry
Convert context to use ancestor function
make memoized parents function
Convert revlog to use ancestor function
On the kernel repo:
$ hg heads -q
before after
RevlogNG 1.11 0.52
Revlogv0 0.80 0.69
Since the current code for tags has to find all the heads of the repo,
this also helps there:
$ hg tags
before after
RevlogNG 2.35 1.76
Revlogv0 2.04 1.90
This allows one to walk the revision graph using only revision numbers,
which can be faster than using revision hashes, especially for
RevlogNG, where the parents of a revision are stored as revision
numbers.
parseindex could fail if read returns too little data in the right
moment (e.g. when there's still leftover data from the previous
iteration and read returns less than "s" bytes).
During a clone, an inline index is not converted to a split index
file until the very end. When the conversion happens, the index
can be very large, and the inline index loading functions always load
the entire index file into ram.
This changes the revlog code to read the index in smaller chunks.
This changes revlog specify revlogng by default. Inlined
data files are also used unless a flags option is found in the .hgrc.
Some example hgrc files:
[revlog]
# use the original revlog format
format=0
[revlog]
# use revlogng. Because no flags are included, inlined data files
# also be selected
format=1
[revlog]
# use revlogng but do not inline the data files with the index
flags=
[revlog]
# the new default
format=1
flags=inline
revlog.py wasn't trying to detect the version of a revlog file that
doesn't exist on the filesystem (as is the case with old-http).
Additionally, there was an off-by-one error in httprangereader.read
(ranges in HTTP Range headers are inclusive), making it get more data
than what was asked for. This made a struct.unpack complain that
"unpack str size does not match format".
Finally, with the two fixes above, test-static-http fails, since
BaseHTTPServer doesn't understand ranges and returns too much data.
Work around that by reading only the specified amount.
it used to cache open files. this made revlogng break because it wants
to rename files when splitting .i into .i/.d, but cannot rename or unlink
open files on windows.
new code is bit slower, but safe on linux and windows. proper fix for
too many open/close of changelog/manifest belongs in different place.
can get 10% speed improvement back.
The appendfile code was not passing default version info to the
changelog or manifest classes, and so they were always being created
as version 0.
revlog.checkinlinesize had to be corrected to seek to the end
of the index file when no index file was passed (only clone does this)
revlog.ancestors can be expensive on big repos. This cuts down the overall
time for hg update by ~19% by short cutting revlog.ancestors when one of the
revisions is reachable from another.
Storing the tuple returned by struct.unpack significantly increases
the memory required to store the entire index in ram. This patch
uses struct.unpack on demand instead.
This tunes for large repositories. It does not read the whole
index file in one big chunk, but tries to buffer reads in more
reasonable chunks instead.
Search speeds are improved in two ways. When trying to find a
specific sha hash, it searches from the end of the file backward.
More recent entries are more likely to be relevant, especially the
tip.
Also, this can load only the mapping of nodes to revlog index number.
Loading the map uses less cpu (no struct.unpack) and much less
memory than loading both the map and the index.
This cuts down the time for hg tip on the 80,000 changeset
kernel repo from 1.8s to 3.69s. Most commands the pull a single
rev out of a big index get roughly the same benefit. Commands
that read the whole index are not slower.
This uses code from Matt to calculate the size change that
would result from applying a delta to keep an accurate running
total of the text size during revlog.addgroup
The revlog.checkinlinesize() uses an atomic opener to replace the
index file after converting it from inline to traditional .i and .d
files. If this operation is interrupted, the atomic file class can
overwrite a valid file with a partially written one.
This patch introduces an atomic opener that does not automatically
replace the destination file with the tempfile. This way
an interrupted checkinlinesize() call turns into a noop.
The appendfile class needs a few changes to make it work with interleaved
index files. It needs to support the tell() method, opening in a+ mode,
and it needs to delay the checkinlinesize call until after the
append file is written.
Given that open(file, "a+") doesn't always seek to the end of the file,
this adds seek operations to appendfile that understand whence args
This patch allows you to optionally inline data bytes with the
revlog index file. It saves considerable space and checkout
time by reducing the number of inodes, wasted partial blocks and
system calls.
To use the inline data add this to your .hgrc
[revlog]
# inline data only works with revlogng
format=1
# inline is the only valid flag right now.
flags=inline
revlogng results in smaller indexes, can address larger data files, and
supports flags and version numbers.
By default the original revlog format is used. To use the new format,
use the following .hgrc field:
[revlog]
# format choices are 0 (classic revlog format) and 1 revlogng
format=1
The empty changegroup can be caused by remote servers dying soon after
findincoming, and further code in pull assumes (correctly) that there are
new changesets.
Incoming ssh needs to detect the end of the changegroup, otherwise it would
block trying to read from the ssh pipe. This is done by parsing the
changegroup chunks.
bundlerepo.getchunk() already is identical to
localrepo.addchangegroup.getchunk(), which is followed by getgroup which
looks much like what you can re-use in bundlerepository.__init__() and in
write_bundle(). bundlerevlog.__init__.genchunk() looks very similar, too,
as do some while loops in localrepo.py.
Applied patch from Benoit Boissinot to move duplicate/related code
to mercurial/changegroup.py and use this to fix incoming ssh.
revlog.group cached every chunk from the revlog, the behaviour was
needed to minimize the roundtrip with old-http.
We now cache the revlog data ~4MB at a time.
The memory used server side when pulling goes down to 35Mo maximum
whereas without the patch more than 160Mo was used when cloning the linux kernel
repository.
The time used by cloning is higher mainly because of the check in revlog.revision.
before
110.25user 20.90system 2:52.00elapsed 76%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+708707minor)pagefaults 0swaps
after
117.56user 18.86system 2:50.43elapsed 80%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+748366minor)pagefaults 0swaps
Make it actually work (undefined variable 'rev'; allow to pass a rev parameter).
repo.branchlookup() doesn't need a copy of heads because it doesn't modify it.
Use None as default argument to heads() instead of nullid.
Doc string PEPification.
and replaces it with calls to nodesbetween.
nodesbetween calculates all the changesets needed to have a complete
revision graph between a given set of base nodes and a given set of
head nodes.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Fix out of range regression
From: Filip Brcic <brcha@users.sourceforge.net>
The old revlog.py issued "index out of range" error when cloning the repository
Now I have reverted the parts of revlog.py to the old state when prev was
initialized as -1 and later assigned self.tip() only if that is possible.
Previously prev was always initialized as self.tip() and that is where the
out of range error was.
manifest hash: c94c9aee8b6d382ef52c3981f306a6e7e5f4c4d1
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
iD8DBQFCzzIxywK+sNU5EO8RAtlcAJ0TX9FXuC2c3YHuYXNwqZhdzPWUlgCggq+a
yJzUKDKH/gvnD3Tx3jcmCn8=
=euPi
-----END PGP SIGNATURE-----
