Mercurial uses tags of null to mark deletions, but convert was
silently discarding these because it had no mapping for them. Thus, it
was resurrecting deleted tags.
We've had a couple reports of subversion tracebacks that trigger when
the parents list is empty, and thus block showing what the commit
failure was on the next two lines.
When converting merge commits, convert checks if any of the parents are
ancestors of any of the other parents. To do this, it builds an ancestor list
for every commit in the repository. On large repos this can take a long time
(30min+). Let's add an option for disabling this check to preserve performance.
The downside of this is that it may create unnecessary parent connections when
enabled (which is unfortunate, but not incorrect).
To verify, I ran the convert tests with the flag enabled, and verified the graph
changes were all just to add new parents that were ancestors of existing
parents.
In some cases we do not want to convert tags from the source repo to be tags in
the target repo (for instance, in a large repository, hgtags cause scaling
issues so we want to avoid them). This adds a config option to disable
converting tags.
Previously convert would throw an exception if it encountered a git commit with
a .gitmodules file that was malformed (i.e. was missing, but had submodule
files, or was malformed).
Instead of breaking the convert entirely, let's print error messages and move
on.
Convert had a bug where it relied on repo.tip() to be the newly committed
commit. This was not the case if the commit already existed in the repository
(since repo.commitctx() did nothing, the tip() referenced some random other
commit and the revmap got corrupted).
This fixes it by using the node returned by repo.commitctx().
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Python 2.6 introduced a new octal syntax: "0oXXX", replacing "0XXX". The
old syntax is not recognized in Python 3 and will result in a parse
error.
Mass rewrite all instances of the old octal syntax to the new syntax.
This patch was generated by `2to3 -f numliterals -w -n .` and the diff
was selectively recorded to exclude changes to "<N>l" syntax conversion,
which will be handled separately.
If the conversion doesn't change the hash, and the cset is public in the source,
it should be public in the destination. (This can happen if file remapping is
done that doesn't affect the initial commits.) This also propagates the secret
phase from the source, regardless of the hash, because presumably the content is
what is secret. Otherwise, the destination commit stays in the draft phase.
Maybe any draft cset with an unchanged hash should be changed to public, because
it has effectively been shared, but convert pretty strongly implies throwing
away (or at least readonly archiving) the source repo.
The change in the rollback output is because the name of the outer transaction
is now 'convert', which seems more accurate. Unfortunately, the memctx won't
indicate the hash prior to committing, so the proper phase can't be applied with
the commit.
The repo is already write locked in mercurial_sink.before().
This will be needed in the next patch to determine if phases need to be
adjusted. The insertion of the source revision in 'extras{}' is still
controlled by the config property.
Conversion of a merge starts with p1 and re-adds the files that were changed in
the merge or came unmodified from p2. Files that are unmodified from p1 will
thus not be touched and take no time. Files that are unmodified from p2 would be
retrieved and rehashed. They would end up getting the same hash as in p2 and end
up reusing the filelog entry and look like the p1 case ... but it was slow.
Instead, make getchanges also return 'files that are unmodified from p2' so the
sink can reuse the existing p2 entry instead of calling getfile.
Reuse of filelog entries can make a big difference when files are big and with
long revlong chains so they take time to retrieve and hash, or when using an
expensive custom getfile function (think
http://mercurial.selenic.com/wiki/ConvertExtension#Customization with a code
reformatter).
This in combination with changes to reuse filectx entries in
localrepo._filecommit make 'unchanged from p2' almost as fast as 'unchanged
from p1'.
This is so far only implemented for the combination of hg source and hg sink.
This is a refactoring/optimization. It is covered by existing tests and show no
changes - which is a good thing.
For merges, we walk the files N-1 times, where N is the number of
parents. This means that for an octopus merge with 3 parents and 2
changed files, we actually fetch 6 files. This corrects the progress
output of the convert command when such commits are encountered.
Although Python supports `X = Y if COND else Z`, this was only
introduced in Python 2.5. Since we have to support Python 2.4, it was
a very common thing to write instead `X = COND and Y or Z`, which is a
bit obscure at a glance. It requires some intricate knowledge of
Python to understand how to parse these one-liners.
We change instead all of these one-liners to 4-liners. This was
executed with the following perlism:
find -name "*.py" -exec perl -pi -e 's,(\s*)([\.\w]+) = \(?(\S+)\s+and\s+(\S*)\)?\s+or\s+(\S*)$,$1if $3:\n$1 $2 = $4\n$1else:\n$1 $2 = $5,' {} \;
I tweaked the following cases from the automatic Perl output:
prev = (parents and parents[0]) or nullid
port = (use_ssl and 443 or 80)
cwd = (pats and repo.getcwd()) or ''
rename = fctx and webutil.renamelink(fctx) or []
ctx = fctx and fctx or ctx
self.base = (mapfile and os.path.dirname(mapfile)) or ''
I also added some newlines wherever they seemd appropriate for readability
There are probably a few ersatz ternary operators still in the code
somewhere, lurking away from the power of a simple regex.
After adding some sneaky debug printing[0], I determined that this
test flaked when a CVS commit containing two files starts too close to
the end of a second, thus putting file "a" in one second and "b/c" in
the following second. The secondary sort key meant that these changes
sorted in a different order when the timestamps were different than
they did when they matched. As far as I can tell, CVS walks through
the files in a stable order, so by sorting on the filenames in cvsps
we'll get stable output. It's fine for us to switch from sorting on
the branchpoint as a secondary key because this was already the point
when we didn't care, and we're just trying to break ties in a stable
way. It's unclear to be if having the branchpoint present matters
anymore, but it doesn't really hurt to leave it.
With this change in place, I was able to run test-convert-cvs over 650
times in a row without a failure. test-convert-cvcs-synthetic.t
appears to still be flaky, but I don't think it's *worse* than it was
before - just not better (I observed one flaky failure in 200 runs on
that test).
0: The helpful debug hack ended up being this, in case it's useful to
future flaky test assassins:
--- a/hgext/convert/cvsps.py
+++ b/hgext/convert/cvsps.py
@@ -854,6 +854,8 @@ def debugcvsps(ui, *args, **opts):
ui.write(('Branch: %s\n' % (cs.branch or 'HEAD')))
ui.write(('Tag%s: %s \n' % (['', 's'][len(cs.tags) > 1],
','.join(cs.tags) or '(none)')))
+ if cs.comment == 'ci1' and (cs.id == 6) == bool(cs.branchpoints):
+ ui.write('raw timestamp %r\n' % (cs.date,))
if cs.branchpoints:
ui.write(('Branchpoints: %s \n') %
', '.join(sorted(cs.branchpoints)))
Monotone used '1' for close while core Mercurial use 1. Now, for consistency,
use the same value everywhere. It will be stored as a string anyway and the
change will not make any real difference.
(The actual value of 'close' doesn't matter as long as extras has such a key.)
Convert will try to find references to revisions in commit messages and replace
them with references to the converted revision. It will take any string that
looks like a hash (and thus also decimal numbers) and look it up in the source
repo. If it finds anything, it will use that in the commit message instead.
It would do that for all hex digit sequences of 6 to 40 characters. That was
usually no problem for small repos where it was unlikely that there would be a
matching 6 'digit' hash prefix. It was also no problem on repos with less than
100000 changesets where numbers with 6 or more digits not would match any
revision number. With more than 100000 revisions random numbers in commit
messages would be replaced with a "random" hash. For example, 'handle 100000
requests' would be changed to to 'handle 9117c6 requests'. Convert could thus
not really be used on real repositories with more than 100000 changesets.
The default hash length shown by Mercurial is 12 'digits'. It is unexpected and
unwanted that convert by default tries to replace revision references that use
less than that amount of 'digits'.
To fix this, don't match strings that are less than the default hash size of 12
characters.
This is in line with the documentation on the base class method, and is related
to issue4496 (but doesn't fix the reporter's problem of not mangling other data
that matches a revision pattern). Now instead of aborting when there is an
ambiguous source rev, it simply won't update the commit comment. A warning
message might be nice, but a None return masks whether the problem was no
matching revision, or more than one.
The only other caller of this is the logic that converts tags, but those are
never ambiguous since they are always 40 characters.
A test isn't feasible because there simply aren't enough commits in the test
suite repos to have an ambiguous identifier that is at least 6 characters long,
and it would be too easy for the ambiguity to disappear when unrelated changes
are made. Instead, I simply ran 'hg --traceback log -r c' on the hg repo, and
handled the error it threw.
We have our own fast-path logic to see if something passes a sniff
test for being a Subversion repository, but it's possible for a user
to svnsync a repo using svn 1.8 and then use svn 1.7 bindings (as in
the bug) to try and convert the repo. If we at least tell the user the
version of libsvn that we used, they might get enough of a hint to
check on their own for format incompatibilities between their
svn{admin,sync} and the libsvn used by hg.
This was implied in issue3486, which specifically asked for subrepo support in
lfconvert. Now that lfconvert uses the convert extension internally when going
to normal files, the issue is half fixed. But now even non largefile repos
benefit when other transformations are needed.
Supporting a full subrepo tree conversion from a single command doesn't seem
reasonable, given the number of options that can be provided, and the
transformations that would need to occur when entering a subrepo (consider
'filemap' paths). Instead, this allows the user to incrementally convert each
hg subrepo from bottom up like so:
# so convert knows the dest type when it sees a non empty dest dir
$ hg init converted
$ hg convert orig/sub1 converted/sub1
$ hg convert orig/sub2 converted/sub2
$ hg convert orig converted
This allows different options to be applied to different subrepos more readily.
It assumes the shamap is in the default location in each converted subrepo for
simplicity. It also allows for a subrepo to be cloned into place, in case _it_
doesn't need a conversion. I was able to convert away from using
largefiles/bfiles in several subrepos with this mechanism.
Extension authors (notably at companies using hg) have been
cargo-culting the `testedwith = 'internal'` bit from hg's own
extensions, which then defeats our "file bugs over here" logic in
dispatch. Let's be more aggressive about trying to give extension
authors a hint about what testedwith should say.
This default mirrors the default for 'git diff'. Other commands have slightly
different defaults -- for example, the move/copy detection for 'git blame'
assumes that a hunk is moved if more than 40 alphanumeric characters are the
same, or copied if more than 20 alphanumeric characters are the same. 50% seems
to be the most common default, though.
Git is fairly unique among VCSes in that it doesn't record copies and renames,
instead choosing to detect them on the fly. Since Mercurial expects copies and
renames to be recorded, it can be valuable to preserve this history while
converting a Git repository to Mercurial. This patch adds a new convert option,
called 'convert.git.similarity', which determines how similar files must be to
be treated as renames or copies.
This produces slightly bad results when branches are in play, but
overall I think it's probably worthwhile. We might be able to do
better with branches somehow, but I haven't given it any thought.
This makes it possible to estimate how long the "scanning source"
phase will take, if the specified source repo type supports a quick
"how many changes" check.
Convert will normally only process files that were changed in a source
revision, apply the filemap, and record it has a change in the target
repository. (If it ends up not really changing anything, nothing changes.)
That means that _if_ the filemap is changed before continuing an incremental
convert, the change will only kick in when the files it affects are modified in
a source revision and thus processed.
With --full, convert will make a full conversion every time and process
all files in the source repo and remove target repo files that shouldn't be
there. Filemap changes will thus kick in on the first converted revision, no
matter what is changed.
This flag should in most cases not make any difference but will make convert
significantly slower.
Other names has been considered for this feature, such as "resync", "sync",
"checkunmodified", "all" or "allfiles", but I found that they were less obvious
and required more explanation than "full" and were harder to describe
consistently.
Since it was introduced in 670e8681d92a, tidy_dirs has been comparing
the result of os.listdir with a string - which never can be true.
Convert apparently works anyway and there is no test coverage of it.
It also seems like it could make a bigger difference on older svn versions but
is less relevant with more recent versions.
Instead of trying to fix the code, we take the low risk option and remove it.
The internal API used IOError to indicate that a file should be marked as
removed.
There is some correlation between IOError (especially with ENOENT) and files
that should be removed, but using IOErrors to represent file removal internally
required some hacks.
Instead, use the value None to indicate that the file not is present.
Before, spurious IO errors could cause commits that silently removed files.
They will now be reported like all other IO errors so the root cause can be
fixed.
Perforce has the concept of "+Sn" files where only the last revisions of the
file is stored. In p4d 2012.1 old purged revisions were not included in the
"manifest". With 2012.2 they started being included and convert's getfile
failed to recognize the "purged" flag and saw it as an empty file. That made
test-convert-p4-filetypes.t fail.
There is no point in storing an empty file as placeholder for a purged file so
we restore the old behaviour by checking the flag and letting getfile consider
purged files deleted.
(It is questionable whether it makes sense to convert not-yet-purged +S files
to mercurial ... but that is another question.)
test-convert-cvs.t has been a little flaky for a while now. Add an
extra tiebreaker in cscmp so that all the cases in the test will sort
reliably.
Without this patch, test-convert-cvs.t failed after 346 runs. With
this patch, I stopped trying to get it to fail after 615 runs. While
not conclusive, that makes me pretty optimistic that this is a working
fix.
In all the remaining cases the comprehension variable is used for the same
thing as a previous loop variable.
This will mute some pyflakes "list comprehension redefines" warnings.
This patch fixes argument mismatch at formatting the abort message,
introduced by dcf325a55d85: the last '%s' doesn't have corresponded
argument.
This patch uses "unexpected size" in the abort message, to distinguish
the reason of failure from "unexpected type" failure checked in the
prior code path below:
if info[1] != type:
raise util.Abort(_('cannot read %r object at %s') % (type, rev))
Before this patch, all operations applied on ".gitmodules" at git
source revisions are treated as modification, even if they are
actually removal of it.
If removal of ".gitmodules" is treated as modification unexpectedly,
"hg convert" is aborted by the exception raised in
"retrievegitmodules()" for ".gitmodules" at the git source revision
removing it, because that revision doesn't have any information of
".gitmodules".
This patch detects removal of ".gitmodules" at git source revisions
correctly.
If ".gitmodules" is removed at the git source revision, this patch
records "hex(nullid)" as the contents hash value for ".hgsub" and
".hgsubstate" at the destination revision.
This patch makes "getfile()" raise IOError also for ".hgstatus" and
".hgsubstate" if the contents hash value is "hex(nullid)", and this
tells removal of ".hgstatus" and ".hgsubstate" at the destination
revision to "localrepository.commitctx()" correctly.
For files other than ".hgstatus" and ".hgsubstate", checking the
contents hash value in "getfile()" may be redundant, because
"catfile()" for them also does so.
But this patch chooses writing it only once at the beginning of
"getfile()", to avoid writing same code twice both for ".hgsub" and
".hgsubstate" separately.
This change allows the origin() and destination() revsets to yield the same
results in the new and old repos after a conversion. Previously, nothing would
be listed for queries in the new repo.
Like the SHA1 updates to the commit messages, this is only operational when the
'convert.hg.saverev=True' option is specified. If the old reference cannot be
found, it is left as-is. It seems slightly better to leave stale evidence of
the graft/transplant/rebase than to eliminate it entirely.
This patch changes the calling signature of memfilectx's __init__ to fall in
line with the other file contexts.
Calling code and tests have been updated accordingly.
Rollback or strip could leave a Mercurial repo with a shamap with revisions no
longer in the repository.
To ensure reliable conversions we now check that the commit actually exists and
consider it non-existing if it doesn't exist.
Mercurial has stable revision identifiers and rollback and strip. Revisions
referenced in the shamap are thus not necessarily still present but we can
easily check for it.
Subversion do not have stable identifiers and no rollback or strip(?). We must
thus assume that all revisions referenced from a shamap still must be present.
This method is similar to hascommitforsplicemap but different ...
The name 'hascommit' sounds like something generic ... but it might
also throw exceptions in specific cases and it is thus (apparently)
only useful for splicemap.
We would formerly exec git cat-file once for every commit, plus once for
every tree and file we wnated to read. This switches to using git
cat-file's batch mode, which is much, much, much faster.
Using this new code, converting the git git repo to hg ran in 106
minutes on my machine. Using the stock mercurial, it required 1239
minutes. I believe this to be typical of the speedups we will see
form this patch.
Closemap solves a very specific use case. It would be better to have a more
generic solution than to have to maintain this forever.
Closemap has not been released yet and removing it now will not break any
backward compatibility contract.
There is no test coverage for closemap but it seems like the same can be
achieved with a simple and much more powerful custom extension:
import hgext.convert.hg
class source(hgext.convert.hg.mercurial_source):
def getcommit(self, rev):
c = super(source, self).getcommit(rev)
if rev in ['''
d643f67092ff123f6a192d52f12e7d123dae229f
3a6a38229d418ba09cb7784c01453a93b4d363f8
facceca31c18f7ef800977055dbcbd7fcb5c5cb2
''']:
c.extra = c.extra.copy()
c.extra['close'] = '1'
return c
hgext.convert.hg.mercurial_source = source
Tagmap solves a very specific use case. It would be better to have a more
generic solution than to have to maintain this forever.
Tagmap has not been released yet and removing it now will not break any
backward compatibility contract.
There is no test coverage for tagmap but it seems like the same can be achieved
with a (relatively) simple and much more powerful custom extension:
import hgext.convert.hg
def f(tag):
return tag.replace('some', 'other')
class source(hgext.convert.hg.mercurial_source):
def gettags(self):
return dict((f(tag), node)
for tag, node in in super(source, self).gettags().items())
def getfile(self, name, rev):
data, flags = super(source, self).getfile(name, rev)
if name == '.hgtags':
data = ''.join(l[:41] + f(l[41:]) + '\n' for l in data.splitlines())
return data, flags
hgext.convert.hg.mercurial_source = source
The fix for issue2653 broke the ability to map the default branch of a source
repository to a non-default named branch in the destination repository. Leave
the default behaviour as is, but allow the branch name "None" to be used to map
to a non-default named branch in the destination repository.
Subversion issues involving svn log such as 1e493b49245f can be tricky to
debug when it is run in an 'hg debugsvnlog' sub process. Debugging is simpler
when convert only uses one process.
With this change convert will invoke the svn log directly when setting
[convert]
svn.debugsvnlog = False
This is intentionally not documented.
Mercurial tags can be local (tag -l, stored in .hg/localtags) or global (normal
tags, tracked in .hgtags) ... or extensions can add other kind of tags.
Convert would take all tags (except "tip"), not just the ones from .hgtags, and
put them into .hgtags.
Instead, convert only the global tags that come from .hgtags.
Previously, there was no way to rewrite tags on the fly while converting. Now,
we add similar logic to branchmap to provide a way to map old tags to new tags.
Currently, this is not enabled since there is not yet a command-line option.
Previously, when converting from a mercurial repo there would be an extraneous
commit at the end of the convert process that would rewrite tags. Now, we check
if there are any new tags before doing this rewriting.
Previously, the hg sink for puttags would just use one head for getting the old
tags which would sometimes lead to tags disappearing. Now, we iterate over all
heads and merge the results.
Upcoming patches will add new map files so we change the calling sequence of
checkrevformat so that error messages will let the user know which file has the
wrong rev format.
This is a simple find-and-replace strategy for matching anything in the
old description of a converted commit and, if that matched sha1 exists
in the mapping, replacing it with the new sha1.
In particular, this is helpful for descriptions that contain tags with
messages such as, "Added tag 1.0 for commit abcde1234567" which will now
be automatically converted.
Tests have been updated accordingly.
get_log started calling back with orig_paths=None on Fedora 20 with
subversion-1.8.3. That broke test-convert-svn-source.t .
There used to be some handling of that situation until d17c619e40d5 apparently
broke it. This patch restores what seems to be the most obvious handling of the
situation.
The reverse mapping was introduced in 51f9f23e6ccc to make roundtrip
conversions possible ... but it did not work when using filemap.
Roundtrips with filemaps will of course only work flawlessly if inverse
mappings are used.
Especially, if a lossy convert mapping is used in one direction, then only
linear lines of development can be converted in the other direction. With this
constraint convert will do the right thing by assuming that excluded files
haven't been changed.)
A test case with general coverage of hg-hg roundtrips with filemap is added.
(There a cases where adding records of converted revisions to the shamap in the
source repository doesn't work - especially when converting the same repo to
several other repos and back. It would arguably be better if convert only
updated the shamaps in the target repo but read shamaps from both the source
and and target repo ... but that is a different story. Making the stuff we have
work consistently is step forward no matter what.)
The existing knobs for controlling which revisions to convert were often
insufficient. Revsets is a shiny hammer that provides a better solution.
Revsets has been introduced in --rev handling in a lot of other places while
being more or less backwards compatible. Doing the same here would be a much
more elegant ... but that would unfortunately not work in this case. "--rev 7"
used to mean revision 0 to 7 - it would be an unacceptable change if it
suddenly just meant revision 7.
Instead we introduce a new configuration setting. It will only work for
Mercurial repositories so adding a new commandline option for it would not be a
nice solution.
There is no way to use the fancy deprecation markup for configuration settings
so we just remove the documentation of hg.startrev.
destc is not a string and can thus not be os.path.join'ed. Convert would crash
if we ended up there ... but we wouldn't because both the sinks (hg and
subversion) sinks implement .revmapfile and "never" throws exceptions.
The 'copynode' was looked up in self.keep as if it was a changeset node. It is
however a filelog node, and self.keep would thus fail if it actually looked at
its parameter ... which it only did if a startrev was specified.
Instead we now don't check the copy node - we don't have to. It must have been
copied from one of the parents, and we already check whether one of the parents
have the copy source.
We could perhaps use linkrev to see if the corresponding changeset was
converted ... but that would sometimes be wrong.
The existing test of this was wrong - now it is better, but it seems like it
exposes a 'log' issue.
The cvsps.py:getrepopath suffers from a string parsing bug (it returns
"user@server/path/to/repository" if the CVSROOT is given like this:
":pserver:user@server/path/to/repository" ), which gives returnes the wrong
value becouse cvsps.py fails to strip the prefix from filenames.
With this patch for the same input we get the correct repo path that is:
"/path/to/repository"
Implemented error handling on splicemap file when source is
subversion (This checks are similar to when source is hg or git).
The revision string is expected to be of svn:<uuid><path>@<number>
format.
the test case has been enhanced to check this format.
Implemented similar error handling that is done for hg in an earlier revision.
These are:
a. add checking for splicemap file format
b. add checking for each revision string formats
1. Introduced 2 levels of error handling for splicemap files
a. Check the splicemap file for rules which are same across different
types of source repos. This is done through enhancing parsesplicemap
function
b. Check revision string formats. Each repo may have their own format.
This is done usign checkrevformat function
c. Implemented the above two for hg
If you actively work with branches, sometimes you need to close old branches
which last commited hundreds revisions ago. After close you will see long
lines in graph visually spoiling history. This sort only moves closed
revisions as close as possible to parents and does not increase storage size
as datesort do.
Since git v1.7.8.2-327-g926f1dd (the change was first released in git
1.7.10), git does not return non-zero when "git ls-remote --tags ..."
is run and the repository is damaged. This causes the "damaged
repository with missing commit" test in test-convert-git.t to
unexpectedly succeed.
Fix by aborting if git outputs any lines beginning with "error:",
which required adding some subprocess use in convert/git.py.
Simplify core logic by no longer attempting to work around missing
class attributes. Instead always generate the attributes and ignore
the cache if the attributes are missing
The default for the time zone offset in a converted changeset has
always been 0 (UTC). With this patch, the converted changeset is
modified so that the local offset from UTC is specified as the time
zone offset.
The option is specified as the boolean convert.localtimezone (default
False). Example usage:
hg convert -s cvs --config convert.localtimezone=True example-cvs example-hg
IMPORTANT: the patch only applies to conversions from cvs or svn.
The documentation for the option only appears in those two sections
in the convert help text.
Previously, convert aborted upon encountering a git submodule. This patch
changes it so that it now succeeds. It modifies convert_git to manually generate
'.hgsub' and '.hgsubstate' files for each git revision, so as to convert git sub
modules to non-mercurial subrepositories.
Bookmarks persistence still showed a fair amount of its legacy as a
monkeypatching extension. This encapsulates all bookmarks
serialization and parsing in a single class, and offers a single
location where other bookmarks storage engines can be substituted
in. As a result, many files no longer import the bookmarks module,
which strikes me as an encapsulation win.
This doesn't do anything to the current bookmark state yet, but I'm
hoping put that in the bmstore class as well.
convert doesn't normalise double slashes in paths. Path normalization
is applied when a path is loaded into filemap and when a file lookup
request is issued to filemap.
Avoid mixing popen and subprocess calls, it simplifies the command line
generation and quoting issues with redirections.
In practice, it fixes the subversion sink on Windows and probably helps
with monotone and darcs sources.
Some help topics use "-" for the top level underlining section mark,
but "-" is used also for the top level categorization in generated
documents: "hg.1.html", for example.
So, TOC in such documents contain "sections in each topics", too.
This patch changes underlining section mark in some help topics to
unify section level in generated documents.
After this patching, levels of each section marks are:
level0
""""""
level1
======
level2
------
level3
......
level4
######
And use of section markers in each documents are:
- mercurial/help/*.txt can use level1 or more
(now these use level1 and level2)
- help for core commands can use level2 or more
(now these use no section marker)
- descriptions of extensions can use level2 or more
(now hgext/acl uses level2)
- help for commands defined in extension can use level4 or more
(now "convert" of hgext/convert uses level4)
"Level0" is used as top level categorization only in "doc/hg.1.txt"
and the intermediate file generated by "doc/gendoc.py", so end users
don't see it in "hg help" outoput and so on.
When the source repository had a revision renaming "$new -> $old",
but the filemap a "$old -> $new" rename, the converted revision could
use either $new (deleting the file) or $old (keeping the file) when
getting the file data, depending on the lexicographical order of
those names. So the resulting revision would leave some files
untouched (as expected), but delete others arbitrarely.
When running convert with a filemap, merge parents which are ancestors
of other parents are ignored. This is hardly a problem when parents
belong to the same branch, but the result could be confusing when named
branches are involved. With:
-o-a1-a2-a3... <- A
\ \
b1-b2-b3...-m- <- B
If all b* revisions are discarded, it is useful to preserve 'm' even if
it is empty after filtering to record the branch switch.
This patch makes filemap preserve "ancestor parents" if there is no
"non-ancestor parent" on the same branch than the merge revision.
Remarks:
- I am not completely convinced by the reasons given above and those
detailed by Matt in this thread:
http://selenic.com/pipermail/mercurial-devel/2012-May/040627.html
The properties we try to preserve are not clearly defined. That said,
I know this patch already helped someone on IRC and the tests output
look reasonable.
- This is a new version of the original "convert: filemap must preserve
fast-forward merges" patch. It has exactly the same output for 2
parents merges, the additional complexity is here to handle more than
two parents.
'hg debugsvnlog' failed with a crash when using the uninitialized transport in
get_log_child if the import of the svn libraries had failed.
'convert' should never get as far as launching 'hg debugsvnlog' if the svn
libraries are missing, but by launching a subprocess there is risk that the
environment is mangled so the second import fails.
It is in principle also possible to launch the command manually.
Subversion can handle ':' quoted as '%3A' but urllib.url2pathname can't and
Mercurial thus rejected some valid subversions URLs.
This particular case will now be handled by some preprocessing before handing
it over to urllib.url2pathname.
This is tested by a0c992a723f9 when test-convert-svn-source.t and
test-convert-svn-move.t can be run on Windows.
Calling propset/propdel with subversion 1.6 on FAT gave
abort: svn exited with status 256
and made test-convert-hg-svn.t and test-convert-svn-sink.t fail. 1.7 worked.
This is a rework of 5ba59c098f03 but ignores the executable bit when it isn't
supported instead of using an approximation.
on some non "en" locale environments, "hg convert" is aborted, because
"util.parsedate()" fails.
it fails in "memctx.__init__()" called by "putcommit()" of "convert".
in "hg convert", datetimes gotten from source repository
are usually formatted by "util.datestr()" with default format "%a %b
%d %H:%M:%S %Y %1%2".
but on some environments, "%a" and "%b" may cause locale sensitive
string, and such string may cause parse error in "util.parsedate()".
this path uses "%Y-%m-%d %H:%M:%S %1%2" as intermediate representation
format for datetimes, because it consists only of locale insensitive
elements.
datetimes in above format are only used for passing them from
conversion logic to memctx object, so it doesn't have to be formatted
by locale sensitive one.
this patch just avoids locale sensitivity problem of "datestr()" and
"parsedate()" combintion.
"svn add file" now fails if "file" is already tracked. To filter them we have
to mirror the svn manifest in the sink.
Tested with svn 1.6.12 and 1.7.4.
Subversion conversion works by picking trunk and branches heads, computing a
revision graph from them and converting the selected commits. By design we fail
to convert empty revisions so we have to be careful when discovering the
revision graph. In this particular issue, the source svn repository was a
partial mirror made by svnsync. The funny part is svnsync preserves all
revisions including empty ones. Also, we trusted ra.stat(path,
stop).created_rev to give us the latest revision with changes in path history
up to stop. This assumption broke at least when path is '', that is the
repository root, which always returned 'stop' revision despited being empty.
The workaround is to first trust ra.stat() but if the returned revision appear
empty, search the whole path history from stop to r1 until some changes are
found.
Do not blindly filter out non ending ^{} tags. The new logic
is:
- if both "tag" and "tag^{}" exist, "tag^{}" is what we want
- if only "tag" exists, "tag" is fine
This improves the error message when convert encounters a git
submodule. Now, instead of a git-cat-file error, we'll directly report
the lack of support for git submodules.
Splicemap lines are documented in hg help convert like:
key parent1, parent2
but parsed like:
key, parents = line.strip().rsplit(' ', 1)
parents = parents.replace(',', ' ').split()
The rsplit() call was introduced to handle spaces in keys for the generic
mapfile format. Spaces can appear in svn identifiers since they contain path
components. This logic makes less sense with splicemap since svn identifiers
can also appear on the right side, even if it is a bit less likely. Given the
parsing is theorically broken, I would rather follow what is documented already
and is correct in the main case where all identifiers are hg hashes. Also,
using svn identifiers in a splicemap sounds difficult as they are not easily
accessible.
When sorting revisions before converting them, we have to edit the revision
graph using splicemap entries. Otherwise, a spliced revision may be converted
before its synthetic parents. Invalid splicemap revisions are now detected
before starting the conversion.
Parsing the splicemap as a mapfile was a pain because map does not let us
override its parsing code and splicemap entries are not key/values. Besides we
had no need for mapfiles extra features. Just parse the splicemap and return a
dictionary.
some problematic encodings use backslash as part of multi-byte characters.
util.pconvert() can treat strings in such encodings correctly, if
win32mbcs is enabled, but str.replace() can not.
Instead of opening the target bzr checkout as a single branch, we try to open
it as a repository. This has the following effects:
- All branches are now converted
- bzr branch names are preserved. Previously, the selected branch was always
converted as 'default'. Branches without a name or 'trunk' are mapped to
'default branch.
- Lightweight checkouts are no longer supported. Maybe they can be, I did not
try to fix that at all.
Implementation notes:
- This was a quick fix, I have no knowledge of bzr API besides browsing 2.0.3
sources.
- The fix was only tested on OSX against bzr 2.4.2.
- Tags discovery does not handle collisions. I have no idea how tags work in
bzr so maybe such collisions are not possible.
Before this patch, metadata and file names were interpreted like:
- unicode objects were converted to UTF-8
- non unicode objects were left unchanged
Looking at the code and bzr being known for transcoding filenames, we expect
everything to be returned as unicode objects, and we want to encode them in
UTF-8, like the subversion source does. To do that, we just remove the custom
implementation of .recode().
The GPLv3 FAQ suggests to upgrade by
[...] replace all your existing v2 license notices (usually at the
top of each file) with the new recommended text available on the GNU
licenses howto. It's more future-proof because it no longer includes
the FSF's postal mailing address.
This removes the postal address, but leaves the version number at 2+.
- catch all exceptions
- pickle a stringified version of the exception
- use a normal abort
Hopefully this will result in less mysterious convert exceptions
A convert run with a branchmap made with
echo default namedbranch > branchmap
on Windows fails silently and surprisingly; it actually
adds a space after 'namedbranch', so it ends up mapping
"default namedbranch" to "".
This also affects splicemaps, since the same parser is used
for both.
As of svn 1.7, many svn calls expect "canonical" paths. In theory, we should
call svn.core.*canonicalize() on all paths before passing them to the API.
Instead, we assume the base url is canonical and copy the behaviour of svn URL
encoding function so we can extend it safely with new components.
With renames like:
a -> b
a/c -> a/c
We were ignoring or duplicating the second one instead of leaving files
unchanged or moving them to their proper destination only.
To avoid this, we process the files in reverse lexicographic order, from most
to least specific change, and ignore files already processed.
v2:
- Add a test
- Change "reverse=1" into "reverse=True"
I have not tried to produce the bug but here is idea: b2b0622d9e96 stopped
passing the modified files list to commit. This makes commit more fragile since
we better not touch unrelated files by mistake. But putcommit() still applies
file changes before exiting upon ignored revisions. So in theory, we could
apply changes from a skipped branch then commit them as part of another
revision.
This patch makes the sink apply the changes after possibly skipping the
revision. The real fix would be to use svn commit --targets option to pass the
file names in an argument file. Unfortunately, it seems to be bugged in svn
1.7.1:
http://svn.haxx.se/dev/archive-2011-11/0211.shtml
