Ubuntu's current LTS release (Trusty Tahr) comes with Mercurial 2.8.2,
which still works fine with current code. The older LTS comes with a
version that's broken, but I don't have to worry about that anymore.
hg pull calls listkeys for bookmarks. This would previously cause a pack with
all refs to be fetched. For Mercurial mirrors of Git repositories where only
some refs were mirrored, this would cause problems in a bunch of ways:
- A larger pack would be fetched than necessary.
- The final refs written out to the Git repo would only be the set of refs we
were actually interested in. If a GC was subsequently run, unreferenced
objects would be deleted. Those objects might be referred to on subsequent
fetches, which could cause hg-git to crash.
We replace all that logic with a simple null fetch. The tests introduced in the
previous patch ensure no regressions.
hg perfrevset 'max(fromgit())' on a repo with around 60,000 commits:
before: ! wall 1.055093 comb 1.050000 user 1.050000 sys 0.000000 (best of 10)
after: ! wall 0.148586 comb 0.140000 user 0.140000 sys 0.000000 (best of 62)
In reality, perfrevset doesn't clear the Git-to-Mercurial map, which means that
a call like `hg log -r 'max(fromgit())'` speeds up from around 1.5 seconds to
0.6.
For a repository with around 60,000 commits, perfrevset for gitnode becomes:
before: ! wall 1.130716 comb 1.130000 user 1.130000 sys 0.000000 (best of 9)
after: ! wall 0.178828 comb 0.180000 user 0.180000 sys 0.000000 (best of 54)
In reality, perfrevset doesn't clear the Git-to-Mercurial map, which means that
a call like `hg log -r 'gitnode(...)'` speeds up from around 1.5 seconds to
0.6.
Mercurial commit a6d634896102 added support for hg cat across subrepositories.
That caused these tests to break. Fix this by using hg manifest instead.
Previously, whenever a tree that wasn't the root ('') was stored, we'd prepend
a '/' to it. Then, when we'd try retrieving the entry, we'd do so without the
leading '/'. This caused data loss because existing tree entries were dropped
on the floor. Fix that by only adding '/' if we're adding to a non-empty
initial path.
This wasn't detected in tests because most of them deal only with files in the
root and not ones in subdirectories.
Previously, we'd spin up the Mercurial incremental exporter from the null
commit and build up state from there. This meant that for the first exported
commit, we'd have to read all the files in that commit and compute Git blobs
and trees based on that.
The current Mercurial to Git conversion scheme makes most sense with
Mercurial's current default storage format, where manifests are diffed against
the numerically previous revision. At some point in the future, the default
will switch to generaldelta, where manifests would be diffed against one of
their parents. In that world it might make more sense to have a stateless
exporter that diffed each commit against its generaldelta parent and calculated
dirty trees based on that instead. However, more experiments need to be done to
see what export scheme is best.
For a repo with around 50,000 files, this brings down an incremental 'hg
gexport' of one commit from 18 seconds with a hot file cache (and tens of
minutes with a cold one) to around 2 seconds with a hot file cache.
The usage of getattr was unsafe. Use hgutil.safehasattr instead.
util.safehasattr has been around since Mercurial 2.0.
This also fixes the formerly disabled test in test-pull.t.
Previously we'd attempt to import every single reachable commit in the Git
object store.
The test adds another branch to the Git repo and doesn't import it until much
later. Previously we'd import it when we ran `hg -R hgrepo pull -r beta`. Now
we won't.
The return value as implemented in git_handler.fetch was pretty bogus. It used
to return the number of values that changed in the 'refs/heads/' namespace,
regardless of whether multiple values in there point to the same Mercurial
commit, or whether particular heads were even imported. Fix all of that by
using the actual heads in the changelog, just like vanilla Mercurial.
The test output changes demonstrate examples where the code was buggy.
Since Mercurial is commit-oriented, the 'no changes found' message really
should rely on what new commits are in the repo, not on new heads. This also
makes an upcoming patch much simpler.
Since everything around this code is completely broken anyway, writing a test
for this that doesn't trigger other bugs is close to impossible. An upcoming
patch will include tests.
The test output change is for an empty clone -- the output is precisely how
vanilla Mercurial treats an empty clone.
The theme of this and upcoming patches is that relying on self.git.object_store
to figure out which commits/tags/bookmarks to import is not great. This breaks
if the git repo is manually put in place (as might be done in a server-based
replication scenario), or if a partial fetch pulled too many commits in for
whatever reason. Indeed we were just about always pulling an entire pack in,
because listkeys for bookmarks currently calls fetch_pack without any
filtering. (This is probably a bug and should be fixed, but this series doesn't
do that.)
Instead, rely on whether we actually imported the commit into Mercurial to
determine whether to import the tag. This is clean, straightforward, and
clearly correct.
There is a whole series of bugs in this code that any test case for this would
hit -- an upcoming patch will include a test for all these bugs at once.
object_store.add_object doesn't check to see if the object is already in a
pack, so it is still written out in that case. Do the check ourselves before
calling add_object.
Since the Git to Mercurial conversion process is incremental, it's at risk of
missing files, or recording files the wrong way, or recording the wrong commit
metadata. Add a command called 'gverify' that can verify the contents of a
particular Mercurial rev against the corresponding Git commit.
Currently, this is limited to checking file names, flags and contents, but this
can be made as robust as desired. Further additions will probably require
refactoring git_handler.py a bit though.
This function is pretty fast: on a Linux machine with a warm cache, verifying a
repository with around 50,000 files takes just 20 seconds. There is scope for
further improvement through parallelization, but conducting tree walks in
parallel is non-trivial with the current worker infrastructure in Mercurial.
This allows other functions to be able to use the `git` property without
needing to care about initializing it.
An upcoming patch will remove the `init_if_missing` function.
Previously, we'd try to access commit.parents[0] and fail. Now, check for
commit.parents being empty and return what Mercurial thinks is a repository
root in that case.
Previously we'd just test if gitrev was falsy, which it is if the rev returned
is 0, even though it shouldn't be. With this patch, test against None
explicitly.
This unmasks another bug: see next patch for a fix and a test.
Previously we'd recompute the repo tags each time we'd consider importing a Git
tag. This is O(n^2) in the number of tags and produced noticeable slowdowns in
repos with large numbers of tags.
To fix this, compute the tags just once. This is correct because the only case
where we'd have issues is if multiple new Git tags with the same name were
introduced, which can't happen because Git tags cannot share names.
For a repository with over 200 tags, this causes a no-op hg pull to be sped up
by around 0.5 seconds.
A new property called _tagscache was introduced in Mercurial 2.0, so the cache
wasn't actually working.
The contract for tags() also changed at some point -- it stopped returning
nodes that weren't in the repo. This will need to be accounted for if we
start using the tags cache again. However, it isn't very clear whether the
Mercurial tags cache is actually worth doing, since we already have a
separate in-memory cache for Git tags in the handler.
