Since the Git to Mercurial conversion process is incremental, it's at risk of
missing files, or recording files the wrong way, or recording the wrong commit
metadata. Add a command called 'gverify' that can verify the contents of a
particular Mercurial rev against the corresponding Git commit.
Currently, this is limited to checking file names, flags and contents, but this
can be made as robust as desired. Further additions will probably require
refactoring git_handler.py a bit though.
This function is pretty fast: on a Linux machine with a warm cache, verifying a
repository with around 50,000 files takes just 20 seconds. There is scope for
further improvement through parallelization, but conducting tree walks in
parallel is non-trivial with the current worker infrastructure in Mercurial.
This allows other functions to be able to use the `git` property without
needing to care about initializing it.
An upcoming patch will remove the `init_if_missing` function.
Previously, we'd try to access commit.parents[0] and fail. Now, check for
commit.parents being empty and return what Mercurial thinks is a repository
root in that case.
Previously we'd just test if gitrev was falsy, which it is if the rev returned
is 0, even though it shouldn't be. With this patch, test against None
explicitly.
This unmasks another bug: see next patch for a fix and a test.
Previously we'd recompute the repo tags each time we'd consider importing a Git
tag. This is O(n^2) in the number of tags and produced noticeable slowdowns in
repos with large numbers of tags.
To fix this, compute the tags just once. This is correct because the only case
where we'd have issues is if multiple new Git tags with the same name were
introduced, which can't happen because Git tags cannot share names.
For a repository with over 200 tags, this causes a no-op hg pull to be sped up
by around 0.5 seconds.
A new property called _tagscache was introduced in Mercurial 2.0, so the cache
wasn't actually working.
The contract for tags() also changed at some point -- it stopped returning
nodes that weren't in the repo. This will need to be accounted for if we
start using the tags cache again. However, it isn't very clear whether the
Mercurial tags cache is actually worth doing, since we already have a
separate in-memory cache for Git tags in the handler.
Previously, the correctness of _handle_subrepos was based on the order the
files were processed in. For example, consider the case where a subrepo at
location 'loc' is replaced with a file at 'loc', while another subrepo exists.
This would cause .hgsubstate and .hgsub to be modified and the file added.
If .hgsubstate was seen _before_ 'loc' in the modified/added loop, then
_handle_subrepos would run and remove 'loc' correctly, before 'loc' was added
back later. If, however, .hgsubstate was seen _after_ 'loc', then
_handle_subrepos would run after 'loc' was added and would remove 'loc'.
With this patch, _handle_subrepos merely computes the changes that need to be
applied. The changes are then applied, making sure removed files and subrepos
are processed before added ones.
This was detected by setting a random PYTHONHASHSEED (in this case, 3910358828)
and running the test suite against it. An upcoming patch will randomize the
PYTHONHASHSEED in run-tests.py, just like is done in Mercurial.
Since a fresh GitHandler is no longer created for every commit, this speeds up
the {gitnode} template massively.
For a repo with over 50,000 commits, the command
hg log -l 10 --template '{gitnode}\n'
speeds up from 2.4 seconds to 0.3.
Previously we'd load the git and hg maps twice on separate git handler objects.
This avoids that.
For a repo with over 50,000 commits, this brings a no-op hg pull down from 2.45
seconds to 2.37.
Currently we call hgrepo.tags() separately for each tag. (This should be fixed
at some point.) This avoids initializing a separate git handler for each tag.
For a repository with over 150 tags, this brings down a no-op hg pull by 0.05
seconds.
Any commit in _map_git is already known, so there's no point walking further
down the DAG.
For a repo with over 50,000 commits, this brings down a no-op hg pull from 38
seconds to 2.5.
getnewgitcommits() does a weird traversal where a particular commit SHA is
visited as many times as the number of parents it has, effectively doubling
object reads in the standard case with one parent. This patch makes the
convert_list a cache for objects, so that a particular Git object is read just
once.
On a mostly linear repository with over 50,000 commits, this brings a no-op hg
pull down from 70 seconds to 38, which is close to half the time, as expected.
Note that even a no-op hg pull currently does a full DAG traversal -- an
upcoming patch will fix this.
For a repo with over 50,000 commits, this brings down the computation of
'export' from 1.25 seconds to 0.25 seconds.
To scale this to hundreds of thousands of commits, one solution might be to
maintain the mapping in a DAG data structure mirroring the changelog, over
which findcommonmissing can be used.
Before this patch, in the git to hg conversion, .hgsubstate once created is
never deleted, even if no submodules are any longer present. This is broken
state, as shown by the test for which the SHA changes. Fix that by looking at
the diff instead of just what submodules are present.
Since 'gitlinks' now contains *changed* gitlinks, not *all* gitlinks, it no
longer makes sense to gate gitmodules checks on that.
This patch simply demonstrates that the test was broken; an upcoming patch will
introduce more tests.
Bonus: this also makes the import process faster because we no longer need to
walk the entire tree to collect gitlinks.
This will cause the SHAs of repos that have submodules added and then removed
to change.
