Before this bundle repository were unable to work with compressed
bundle2. We use the same approach as with bundle1, we extract the
changegroup in uncompressed form into a temporary file.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
Incoming was using bundle1 in all cases, as bundle1 is restricted to
changegroup1 and does not support general delta, this can lead to significant
CPU overhead if the server is using general delta storage. We now properly
request and store a bundle2 to disk.
If the server include any output or error in the bundle, they will be stored on
disk and replayed when the bundle is read. As 'hg incoming' is going to read the
bundle right away, we call that 'good' enough and go back to the bigger plan of
having general delta on by default.
This was tracked as 4864
When looking up a base revision, we were ignoring the contents that were already
available in the manifest's _mancache. This patch allows us to use that data
instead of reading from the revlog.
This is useful in our pushrebase extension (which allows rebasing on the server
side during a push) because it allows us to prefetch the bundle base manifest
before aquiring the repo lock (1 second saving), which means doing less work inside the lock,
which means a 20% higher commit rate.
Revision d6a8b4e28635 (filelog: add file function to open other
filelogs, 2011-05-10) added a _file() method to revlog, which also
required a 'repo' parameter to be added to bundlefilelog's
constructor. The _file() method was then removed in 55fa3c487b0f
(filelog: remove unused _file method, 2015-01-22), which made the
constructor parameter unused, so let's remove that too.
This avoids the following error that happened if base revision of bundle file
was hidden. bundlerevlog needs it to construct revision texts from bundle
content as revlog.revision() does.
File "mercurial/context.py", line 485, in _changeset
return self._repo.changelog.read(self.rev())
File "mercurial/changelog.py", line 319, in read
text = self.revision(node)
File "mercurial/bundlerepo.py", line 124, in revision
text = self.baserevision(iterrev)
File "mercurial/bundlerepo.py", line 160, in baserevision
return changelog.changelog.revision(self, nodeorrev)
File "mercurial/revlog.py", line 1041, in revision
node = self.node(rev)
File "mercurial/changelog.py", line 211, in node
raise error.FilteredIndexError(rev)
mercurial.error.FilteredIndexError: 1
Since Python 2.7.9, "os.path.join(path, '')" doesn't add "os.sep" at
the end of UNC path (see issue4557 for detail).
This makes bundlerepo incorrectly work, if:
1. cwd is the root of UNC share (e.g. "\host\share"), and
2. mainreporoot is near cwd (e.g. "\host\sharefoo\repo")
- host of UNC path is same as one of cwd
- share of UNC path starts with one of cwd
3. "repopath" isn't specified in bundle URI
(e.g. "bundle:bundlefile" or just "bundlefile")
For example:
$ hg --cwd \host\share -R \host\sharefoo\repo incoming bundle
In this case:
- os.path.join(r"\host\share", "") returns r"\host\share",
- r"\host\sharefoo\repo".startswith(r"\host\share") returns True, then
- r"foo\repo" is treated as repopath of bundlerepo instead of
r"\host\sharefoo\repo"
This causes failure of combining "\host\sharefoo\repo" and bundle
file: in addition to it, "\host\share\foo\repo" may be combined with
bundle file, if it accidentally exists.
This patch uses "pathutil.normasprefix()" to ensure "os.sep" at the
end of cwd safely, even with some problematic encodings, which use
0x5c (= "os.sep" on Windows) as the tail byte of some multi-byte
characters.
BTW, normalization before "pathutil.normasprefix()" isn't needed in
this case, because "os.getcwd()" always returns normalized one.
It is finally time to freeze the bundle2 format! To do so we:
- rename HG2Y to HG20,
- drop "b2x:" prefix from all part names,
- rename capability to "bundle2-exp" to "bundle2"
- rename the hook flag from 'bundle2-exp' to 'bundle2'
Although Python supports `X = Y if COND else Z`, this was only
introduced in Python 2.5. Since we have to support Python 2.4, it was
a very common thing to write instead `X = COND and Y or Z`, which is a
bit obscure at a glance. It requires some intricate knowledge of
Python to understand how to parse these one-liners.
We change instead all of these one-liners to 4-liners. This was
executed with the following perlism:
find -name "*.py" -exec perl -pi -e 's,(\s*)([\.\w]+) = \(?(\S+)\s+and\s+(\S*)\)?\s+or\s+(\S*)$,$1if $3:\n$1 $2 = $4\n$1else:\n$1 $2 = $5,' {} \;
I tweaked the following cases from the automatic Perl output:
prev = (parents and parents[0]) or nullid
port = (use_ssl and 443 or 80)
cwd = (pats and repo.getcwd()) or ''
rename = fctx and webutil.renamelink(fctx) or []
ctx = fctx and fctx or ctx
self.base = (mapfile and os.path.dirname(mapfile)) or ''
I also added some newlines wherever they seemd appropriate for readability
There are probably a few ersatz ternary operators still in the code
somewhere, lurking away from the power of a simple regex.
For bundlerepo to work with bundle2 files, we need to find the part that
contains the bundle's changegroup data and work with that instead of the
entire bundle. Future work can add separate processing for other bundle2
parts.
Bundlerepo should always close() the object that it receives from
exchange.readbundle(). When bundle2 support is added in a later diff,
self.bundle will be overwritten to be the changegroup part instead of the
entire bundle unpacker.
The next diff will add support for writing bundle2 files to writebundle, but
the bundle2 generator wants access to a ui object. This changes the signature
and callsites to pass one in.
This change touches every module in which repository.sopener was being used, and
changes it for the equivalent repository.svfs.
It should now be possible to remove localrepo.sopener.
This patch makes bundrepo retract the phase boundary for new commits to 'draft'
status, which is consistent with the behavior of 'hg unbundle'. The old
behavior was for commits to appear with the same phase as their nearest
ancestor in the base repository.
This affects several classes of operation:
* Inspecting a bundle with the -B flag
* Treating a bundle file as a peer (old: everything public, new: everything draft)
* Incoming command (neither old or new behavior is sensible -- fixed in next patch)
This patch makes bundlerepo use a subclass of phasecache that will allow phase
boundaries to be moved around, but will never write them to the underlying
repository.
Pull would send a getbundle command where common heads were sent both as common
and head, even though there is no reason to request a common head.
The request was thus twice as big as necessary and more likely to hit HTTP
header size limits.
Instead, don't request heads that already are common.
This is fixed in bundlerepo.getremotechanges . It could perhaps also have been
fixed in discovery.findcommonincoming but that would have a bigger impact.
According to foozy:
This patch should be backed out, because "bundlename" and "bundle" in
this case are not relative paths to the root of repositories.
The former is specified via "hg incoming --bundle BUNDLENAME"
(relative path to cwd, or absolute one), and the latter is generated
in "changegroup.writebundle" by "tempfile.mkstemp" for internal
temporary usage (absolute path).
To be exact, the latter hunk in this patch can be applied, because
"os.join" for two absolute paths can generate correct result. But the
former hunk can't, because it may unexpected result, if specified path
is relative to cwd and cwd != root.
Some extensions set configuration settings that showed up in 'hg showconfig
--debug' with 'none' as source. That was confusing.
Instead, they will now tell which extension they come from.
This change tries to be consistent and specify a source everywhere - also where
it perhaps is less relevant.
This makes possible to use bundlerevlog class with subclasses of revlog
that override revlog's 'revision' method. In particular this change is necessary
to implement manifest compression, as it allows extension to replace manifest
class and override 'revision' method there.
A repo should not get the configuration from an other repo, so create it with
the global configuration in repo.baseui.
This is done too when recreating a repo. The repo configuration is reread
anyway. And now deleted repo configuration does not persist.
Bundle revisions had some info in their fake revlog intries and some info in a
dict with all the bundle revisions. This dict was used to get the stored data
and to distinguish repo revisions from bundle revisions.
Real repo revisions and bundle revisions will now be distinguished by comparing
with the tip revision of the original repo. This reintroduces something similar
to disktiprev which was unused and removed in 822a1e6e3d7b and let that replace
the O(reposize) dict.
In its current state discovery may return (remotely) filtered elements
in "common". This has usually no impact as "missing" is kept clear of
filtered elements. However when the "remote" repo is a local repo (disk
accessible, and directly created in memory) the incoming code takes a
shortcut and directly uses the "remote" repo to generate the incoming
output. When some common elements are filtered this led to a crash. We
now ensure we use an unfiltered repository to generate the incoming
output. This does not change the behavior as missing is clear of
filtered revision.
Now that we have proper low level filtering, incoming code needs a
deeper cleanup but it is already planned.
The old check is no longer needed:
The checked paths will be passed on to filelog.__init__ which will prefix the
path with 'data/'. A leading '/' will thus not do any harm.
And: The paths will be used by an opener which will use a pathauditor.
And finally: The old check did not consider Windows paths and was thus
insufficient.
We do not want anything computed with the bundle overlay to be written back in
the repo. Such write will likely contains invalid data.
The short terms goal of this change is to drop use of `_cacheabletip` in bundle
repo.
Instead of preventing any cache write we allow writing cache for all content of
the original repo.
The motivation for this change is to drop the custom _writebranchcache of
bundlerepo to help extraction of the branchmap logic out of localrepo.
Problem:
getremotechanges would return the 'other' repo if nothing was incoming and
there thus wasn't any bundle to base the repo on. The 'other' could be a http
peer which only implement the functionality available over the http protocol.
Transplant could thus fail with
TypeError: argument of type 'httppeer' is not iterable
Solution:
Return the local repo instead of the remote peer if there is no reason to place
a bundlerepo on top of the local repo.
All filecache usage on repo is for logic that should be unfiltered. The
caches should be common to all filtered instances, and computation must
be done unfiltered. A dedicated storecache subclass is created for
this purpose.
bundle() revset expression returns all changes that are present
in the bundle file (no matter whether they are in the repo or not).
Bundle file should be specified via -R option.
