mercurial_source.getchanges() seems to care about files whose nodeid
has changed even if their contents has not (i.e. it has been
reverted/backed out). The method uses ctx1.status(ctx2) to find
differencing files. However, that method is currently broken and
reports reverted changes as modified. In order to fix that method, we
first need to rewrite getchanges() using manifest.diff(), which does
report reverted files as modified (because it's about differences in
the manifest, so about nodeids).
The next commit will rewrite the way we find changes between two
manifests. By making the cache not care about the difference between
added and modified files, we don't require the rewritten code to care
about that difference either. Also extract the call to ctx.status() to
simplify the next commit.
Once we get a matcher down into manifestmerge, we can make narrowhg
work more easily and potentially let manifest.match().diff() do less
work in manifestmerge.
Before this patch, convert was using repo._bookmarks.write, a deprecated API
for saving bookmarks.
This patch changes the use of repo._bookmarks.write to
repo._bookmarks.recordchange.
Instead of reporting
spliced in ['82544090e14fe18091e04f1fb0f0d7991cbe6e7e'] as parents of 369fd983d9e13330e9f12d9fce820deae84ea223
report
spliced in 82544090e14fe18091e04f1fb0f0d7991cbe6e7e as parents of 369fd983d9e13330e9f12d9fce820deae84ea223
cvsps computes the parent revisions of log entries by walking the cvs log
sorted by (rcs, revision) and by iteratively maintaining a 'versions'
dictionary which maps a (rcs, branch) pair onto the last revision seen for that
pair. When log caching is on and a log cache exists, cvsps fails to set the
parent revisions of new log entries because it does not iterate over the log
cache in the parents computation. A complication is that a file rcs can change
(move to/from the attic), with respect to its value in the log cache, if the
file is removed/added back. This patch adds an iteration over the log cache to
update the rcs of cached log entries, if changed, and to properly populate the
'versions' dictionary.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
In some languages that have no caps, "DEPRECATED" and "deprecated" can be
translated to the same byte sequence. So it is too wild to exclude messages
by _("DEPRECATED").
Multiple --rev args on convert is a new feature, and was initially disabled for
all sources. It has since been enabled on git sources, and this patch enables it
on mercurial sources.
Recently we fixed converting merges to correctly sync changes from p2. We missed
the case of deletes though (so p2 deleted a file that p1 had not yet deleted,
and the file does not belong to the source).
The fix is to detect when p2 doesn't have the file, so we just sync it as a
delete to p1 in the merge.
Updated the test, and verified it failed before the fix.
This adds an option to not pull in gitsubmodules during a convert. This is
useful when converting large git repositories where gitsubmodules were allowed
historically, but are no longer wanted.
When converting a merge commit using a filemap convert (i.e. when moving
contents from the root of the repo into subdir1/), convert would silently drop
the entire contents of the target repo's p2. This was because when it built the
target commit, it did so by taking the target p1 and adding only the files that
changed in the source repo's merge commit.
This breaks in the case where the target repo has files that are unrelated to
the source repo (like in the case where you use convert to import a repo as a
subdirectory of another).
The fix is to use Mercurial's merge logic to detect which files in p2 we should
carry over to the merge. It follows three rules:
1) if the file belongs to the source, don't try to merge it. Rely on the list of
files provided to putcommit to be correct.
2) if the file requires merging or user input (change vs deleted), throw an
exception. We don't have enough info to do this.
3) if p2 has the newest, non-merge-requiring version of the file, take it
I've also added a test to cover this issue.
This is an implementation of the new targetfilebelongstosource() function for
the filemapper. It simply checks if the given file name is prefixed by any of
the rename destinations.
It is not a perfect implementation since it doesn't account for the filemap
specifying includes or excludes, but that makes the problem much harder, and
this implementation should suffice for most cases.
This adds a base implementation of a function that tests if a given file from a
target repo came from the source repo. This will be used later to detect which
files did not come from the source repo during a merge, so we can merge those
files correctly instead of dropping them.
There was a bug in the git convert code where if you copied a file and modified
the copy source in the same commit, and if the copy dest was alphabetically
earlier than the copy source, the converted version would use the copy dest
contents for both the source and the target.
The root of the bug is that the git diff-tree output is formatted like so:
:<mode> <mode> <oldhash> <newhash> <state> <src> <dest>
:100644 100644 c1ab79a15... 3dfc779ab... C069 oldname newname
:100644 100644 c1ab79a15... 03e2188a6... M oldname
The old code would always take the 'oldname' field as the name of the file being
processed, then it would try to do an extra convert for the newname. This works
for renames because it does a delete for the oldname and a create for the
newname.
For copies though, it ends up associating the copied content (3dfc779ab above)
with the oldname. It only happened when the dest was alphabetically before
because that meant the copy got processed before the modification.
The fix is the treat copy lines as affecting only the newname, and not marking
the oldname as processed.
On Windows Perforce command line client uses default system locale to encode
output. Using 'latin_1' causes locale-specific characters to be replaced with
question marks. With this patch we will use default locale by default whilst
allowing to specify it explicity with 'convert.p4.encoding' config option.
This is a potentially breaking change for any scripts relying on output treated
as in 'latin_1' encoding.
Also because hgext.convert.convcmd overwrites detected default system locale
with UTF-8 we had to introduce an import cycle in hgext.convert.p4 to retrieve
originally detected encoding from hgext.convert.convcmd.
As it turned out, even when getting relatively small files, concatenating
string data every time when new chunk is received is very inefficient.
Maintaining a string list of data chunks and concatenating everything in one go
at the end seems much more efficient - in my testing it made getting 40 MB file
7 times faster, whilst converting of a particularly big changelist with some big
files went down from 20 hours to 3 hours.
The conversion from git to hg was reading the remote branch list directly from
the origin server. If the origin's branch had moved forward since the last git
fetch, it would return a git hash which didn't exist locally, and therefore the
branch was not converted.
This changes it to rely on the local repo's refs/remotes list of branches
instead, so it's completely cut off from the server.
A fix for issue2653 with f5abbf51a76e introduced a discrepancy how default
branch should be denoted when converting with branchmap from different SCM.
E.g. for Git and Mercurial you need to use 'default' whilst for Perforce and
SVN you had to use 'None'. This changeset unifies 'default' for such purposes
whilst falling back to 'None' when no 'default' mapping specified.
Previously all git remotes were created as "remote/foo". This patch adds a
configuration option for deciding what the prefix should be. This is useful if
you want the bookmarks to be "origin/foo" like they are in git, or if you're
integrating with the remotenames extension and don't want the local remote/foo
bookmarks to overlap with the remote foo bookmarks.
When converting a particularly large Perforce changelist (especially with some
big files), it is very likely to run into an intermittent network issue (e.g.
WSAECONNRESET or WSAETIMEDOUT) getting one of the files, which will result in
the entire changelist converting being aborted. Which can be quite unfortunate
since you might have waited hours getting all other files. To mitigate this
let's attempt to get the file one more time, escalating original exception
if that attempt fails.
This creates the convert.hg.sourcename config option which will embed a user
defined name into each commit created by the convert. This is useful when using
the convert extension to merge several repositories together and we want to
record where each commit came from.
Previously convert could only take one '--rev'. This change allows the user to
specify multiple --rev entries. For instance, this could allow converting
multiple branches (but not all branches) at once from git.
In this first patch, we disable support for this for all sources. Future
patches will enable it for select sources (like git).
Mercurial uses tags of null to mark deletions, but convert was
silently discarding these because it had no mapping for them. Thus, it
was resurrecting deleted tags.
We've had a couple reports of subversion tracebacks that trigger when
the parents list is empty, and thus block showing what the commit
failure was on the next two lines.
When converting merge commits, convert checks if any of the parents are
ancestors of any of the other parents. To do this, it builds an ancestor list
for every commit in the repository. On large repos this can take a long time
(30min+). Let's add an option for disabling this check to preserve performance.
The downside of this is that it may create unnecessary parent connections when
enabled (which is unfortunate, but not incorrect).
To verify, I ran the convert tests with the flag enabled, and verified the graph
changes were all just to add new parents that were ancestors of existing
parents.
In some cases we do not want to convert tags from the source repo to be tags in
the target repo (for instance, in a large repository, hgtags cause scaling
issues so we want to avoid them). This adds a config option to disable
converting tags.
Previously convert would throw an exception if it encountered a git commit with
a .gitmodules file that was malformed (i.e. was missing, but had submodule
files, or was malformed).
Instead of breaking the convert entirely, let's print error messages and move
on.
Convert had a bug where it relied on repo.tip() to be the newly committed
commit. This was not the case if the commit already existed in the repository
(since repo.commitctx() did nothing, the tip() referenced some random other
commit and the revmap got corrupted).
This fixes it by using the node returned by repo.commitctx().
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Python 2.6 introduced a new octal syntax: "0oXXX", replacing "0XXX". The
old syntax is not recognized in Python 3 and will result in a parse
error.
Mass rewrite all instances of the old octal syntax to the new syntax.
This patch was generated by `2to3 -f numliterals -w -n .` and the diff
was selectively recorded to exclude changes to "<N>l" syntax conversion,
which will be handled separately.
If the conversion doesn't change the hash, and the cset is public in the source,
it should be public in the destination. (This can happen if file remapping is
done that doesn't affect the initial commits.) This also propagates the secret
phase from the source, regardless of the hash, because presumably the content is
what is secret. Otherwise, the destination commit stays in the draft phase.
Maybe any draft cset with an unchanged hash should be changed to public, because
it has effectively been shared, but convert pretty strongly implies throwing
away (or at least readonly archiving) the source repo.
The change in the rollback output is because the name of the outer transaction
is now 'convert', which seems more accurate. Unfortunately, the memctx won't
indicate the hash prior to committing, so the proper phase can't be applied with
the commit.
The repo is already write locked in mercurial_sink.before().
This will be needed in the next patch to determine if phases need to be
adjusted. The insertion of the source revision in 'extras{}' is still
controlled by the config property.
Conversion of a merge starts with p1 and re-adds the files that were changed in
the merge or came unmodified from p2. Files that are unmodified from p1 will
thus not be touched and take no time. Files that are unmodified from p2 would be
retrieved and rehashed. They would end up getting the same hash as in p2 and end
up reusing the filelog entry and look like the p1 case ... but it was slow.
Instead, make getchanges also return 'files that are unmodified from p2' so the
sink can reuse the existing p2 entry instead of calling getfile.
Reuse of filelog entries can make a big difference when files are big and with
long revlong chains so they take time to retrieve and hash, or when using an
expensive custom getfile function (think
http://mercurial.selenic.com/wiki/ConvertExtension#Customization with a code
reformatter).
This in combination with changes to reuse filectx entries in
localrepo._filecommit make 'unchanged from p2' almost as fast as 'unchanged
from p1'.
This is so far only implemented for the combination of hg source and hg sink.
This is a refactoring/optimization. It is covered by existing tests and show no
changes - which is a good thing.
For merges, we walk the files N-1 times, where N is the number of
parents. This means that for an octopus merge with 3 parents and 2
changed files, we actually fetch 6 files. This corrects the progress
output of the convert command when such commits are encountered.
