This produces slightly bad results when branches are in play, but
overall I think it's probably worthwhile. We might be able to do
better with branches somehow, but I haven't given it any thought.
This makes it possible to estimate how long the "scanning source"
phase will take, if the specified source repo type supports a quick
"how many changes" check.
This wraps all the locations of dirstate.setparent with the appropriate
begin/endparentchange calls. This will prevent exceptions during those calls
from causing incoherent dirstates (issue4353).
No Mercurial DAG has ever (!) been drawn in a way where 'up to' would apply.
Instead, describe deliberatey vague and informal instead of seemingly precise
but not describing the essentials:
rebase the tree around the specified changeset without ancestors of dest
A common mistake can be to type 'hg rebase -i' to discover interactive history
editing. We add a -i/--interactive flag as discussed in the sprint and deprecate
it right away, but hint people using it to use histedit instead.
Convert will normally only process files that were changed in a source
revision, apply the filemap, and record it has a change in the target
repository. (If it ends up not really changing anything, nothing changes.)
That means that _if_ the filemap is changed before continuing an incremental
convert, the change will only kick in when the files it affects are modified in
a source revision and thus processed.
With --full, convert will make a full conversion every time and process
all files in the source repo and remove target repo files that shouldn't be
there. Filemap changes will thus kick in on the first converted revision, no
matter what is changed.
This flag should in most cases not make any difference but will make convert
significantly slower.
Other names has been considered for this feature, such as "resync", "sync",
"checkunmodified", "all" or "allfiles", but I found that they were less obvious
and required more explanation than "full" and were harder to describe
consistently.
Since it was introduced in 670e8681d92a, tidy_dirs has been comparing
the result of os.listdir with a string - which never can be true.
Convert apparently works anyway and there is no test coverage of it.
It also seems like it could make a bigger difference on older svn versions but
is less relevant with more recent versions.
Instead of trying to fix the code, we take the low risk option and remove it.
The internal API used IOError to indicate that a file should be marked as
removed.
There is some correlation between IOError (especially with ENOENT) and files
that should be removed, but using IOErrors to represent file removal internally
required some hacks.
Instead, use the value None to indicate that the file not is present.
Before, spurious IO errors could cause commits that silently removed files.
They will now be reported like all other IO errors so the root cause can be
fixed.
Perforce has the concept of "+Sn" files where only the last revisions of the
file is stored. In p4d 2012.1 old purged revisions were not included in the
"manifest". With 2012.2 they started being included and convert's getfile
failed to recognize the "purged" flag and saw it as an empty file. That made
test-convert-p4-filetypes.t fail.
There is no point in storing an empty file as placeholder for a purged file so
we restore the old behaviour by checking the flag and letting getfile consider
purged files deleted.
(It is questionable whether it makes sense to convert not-yet-purged +S files
to mercurial ... but that is another question.)
After previous patches, largefiles in the working directory are
ensured to be updated before "repo.commit" invocation for automated
committing below:
- by "overrides.mergeupdate" via "merge.update" for rebase
- by "overrides.scmutilmarktouched" via "patch.patch" for transplant
This patch removes redundant "lfcommands.updatelfiles" invocation in
"Case 0" code path of "lfilesrepo.commit" for automated committing,
and revises detailed comment.
Before this patch, largefiles in the working directory aren't updated
correctly, if transplant is aborted by conflict. This prevents users
from viewing appropriate largefiles while resolving conflicts.
While transplant, largefiles in the working directory are updated only
at successful committing in the special code path of
"lfilesrepo.commit()".
To update largefiles even if transplant is aborted by conflict, this
patch wraps "scmutil.marktouched", which is invoked from "patch.patch"
with "files" list of added/modified/deleted files.
This patch invokes "updatelfiles" with:
- "printmessage=False", to suppress "getting changed largefiles ..."
messages while automated committing by transplant
- "normallookup=True", because "patch.patch" doesn't update dirstate
for modified files
in such case, "normallookup=False" may cause marking modified
largefiles as "clean" unexpectedly
Before this patch, largefiles in the working directory aren't updated
correctly, if rebase is aborted by conflict. This prevents users from
viewing appropriate largefiles while resolving conflicts.
While rebase, largefiles in the working directory are updated only at
successful committing in the special code path of
"lfilesrepo.commit()".
To update largefiles even if rebase is aborted by conflict, this patch
centralizes the logic of updating largefiles in the working directory
into the "mergeupdate" wrapping "merge.update".
This is a temporary way to fix with less changes. For fundamental
resolution of this kind of problems in the future, largefiles in the
working directory should be updated with other (normal) files
simultaneously while "merge.update" execution: maybe by hooking
"applyupdates".
"Action list based updating" introduced by hooking "applyupdates" will
also improve performance of updating, because it automatically
decreases target files to be checked.
Just after this patch, there are some improper things in "Case 0" code
path of "lfilesrepo.commit()":
- "updatelfiles" invocation is redundant for rebase
- detailed comment doesn't meet to rebase behavior
These will be resolved after the subsequent patch for transplant,
because this code path is shared with transplant.
Even though replacing "merge.update" in rebase extension by "hg.merge"
can also avoid this problem, this patch chooses centralizing the logic
into "mergeupdate", because:
- "merge.update" invocation in rebase extension can't be directly
replaced by "hg.merge", because:
- rebase requires some extra arguments, which "hg.merge" doesn't
take (e.g. "ancestor")
- rebase doesn't require statistics information forcibly displayed
in "hg.merge"
- introducing "mergeupdate" can resolve also problem of some other
code paths directly using "merge.update"
largefiles in the working directory aren't updated regardless of
the result of commands below, before this patch:
- backout (for revisions other than the parent revision of the
working directory without "--merge")
- graft
- histedit (for revisions other than the parent of the working
directory
When "partial" is specified, "merge.update" doesn't update dirstate
entries for standins, even though standins themselves are updated.
In this case, "normallookup" should be used to mark largefiles as
"possibly dirty" forcibly, because applying "normal" on lfdirstate
treats them as "clean" unexpectedly.
This is reason why "normallookup=partial" is specified for
"lfcommands.updatelfiles".
This patch doesn't test "hg rebase --continue", because it doesn't
work correctly if largefiles in the working directory are modified
manually while resolving conflicts. This will be fixed in the next
step of refactoring for largefiles.
All changes of tests/*.t files other than test-largefiles-update.t in
this patch come from invoking "updatelfiles" not after but before
statistics output of "hg.update", "hg.clean" and "hg.merge".
Code paths below expect "hg.updaterepo" (or "hg.update" using it) to
execute linear merging:
- "update" in commands
- "postincoming" in commands, used for:
- "hg pull --update"
- "hg unbundle --update"
- "hgsubrepo.get" in subrepo
For linear merging with largefiles, standins should be updated
according to (possibly dirty) largefiles before "merge.update"
invocation to detect conflicts correctly.
Before this patch, only the "update" command can execute linear merging
correctly, because largefiles extension takes care of only it.
This patch moves "updatestandin" invocation from "overrideupdate" ("hg
update" wrapper) to "_hgupdaterepo" ("hg.updaterepo" wrapper) to
execute linear merging in "hg.updaterepo" correctly.
This is also a preparation to centralize the logic of updating
largefiles in the working directory into the function wrapping
"merge.update" in the subsequent patch.
Before this patch, standinds not known to the restored dirstate at
rollback still exist after rollback of the parent of the working
directory, and they become orphans unexpectedly.
This patch unlinks standins not known to the restored dirstate.
This patch saves names of standins matched against not
"repo.dirstate[f] == 'a'" but "repo.dirstate[f] != 'r'" before
rollback, because branch merging marks files newly added to
dirstate as not "a" but "n".
Such standins will also become orphan after rollback, because they are
not known to the restored dirstate.
Before this patch, standins are restored from the NEW parent of the
working directory at "hg rollback", and this causes:
- standins removed in the rollback-ed revision are restored, and
become orphan, because they are already marked as "R" in the
restored dirstate and expected to be unlinked
- standins added in the rollback-ed revision are left as they were
before rollback, because they are not included in the new parent
(this may not be so serious)
This patch replaces the "merge.update" invocation with a specific
implementation to restore standins according to restored dirstate.
This is also the preparation to centralize the logic of updating
largefiles into the function wrapping "merge.update" in the subsequent
patch.
After that patch, "merge.update" will also update largefiles in the
working directory and be redundant for restoring standins only.
Before this patch, "hg rollback" can't restore standins correclty, if:
- old parent of the working directory is rollback-ed, and
- new parent of the working directory is not branch-tip
"overriderollback" uses "merge.update" as a kind of "revert" utility
to restore only standins with "node=None", and this makes
"merge.update" choose "branch-tip" revision as the updating target
unexpectedly.
Then, "merge.update" restores standins from the branch-tip revision
regardless of the parent of the working directory after rollback and
this may cause unexpected behavior.
This patch invokes "merge.update" with "node='.'" to restore standins
from the parent revision of the working directory.
In fact, this "merge.update" invocation will be replaced in the
subsequent patch to fix another problem, but this change is usefull to
inform reason why such complicated case should be tested.
For efficiency, this patch omits restoring standins and updating
lfdirstate, if the parent of the working directory is not rollbacked.
This patch adds the test not to confirm whether restoring is skipped
or not, but to detect unexpected regression in the future: it is
difficult to distinguish between skipping and perfectly restoring.
This makes hg log --follow --patch work, since in cmdutil._makelogrevset we
use the non-follow matcher for hg log --follow --patch with no file arguments.
This has actually been broken since at least Mercurial 2.8 -- hg log --patch
with largefiles only used to work when no largefiles existed. Rev 658ce4a0a0a9
exposed this bug for all cases.
lfstatus should only be True for operations where we want standins to be
printed out. We explicitly do not want that for historical operations like log.
Other historical operations like hg diff -r A -r B don't print out standins
either.
This is required to fix issue4334, but doesn't fix anything by itself. That's
why there aren't any tests accompanying this patch.
When the authorship of the changeset folded in does not match that of
the base changeset, we currently use the configured ui.username
instead. This is especially surprising when the user is not the author
of either of the changesets. In such cases, the resulting authorship
(the user's) is clearly incorrect. Even when the user is folding in a
patch they authored themselves, it's not clear whether they should
take over the authorship. Let's instead keep it simple and always
preserve the base changeset's authorship. This is also how
"git rebase -i" handles folding/squashing.
This is a quick fix for some consoles on windows (consoles that are not mingw
based) so that the debugcolor command doesn't throw a KeyError when effects
aren't supported (e.g. italic).
This patch fixes argument mismatch at formatting the abort message,
introduced by dcf325a55d85: the last '%s' doesn't have corresponded
argument.
This patch uses "unexpected size" in the abort message, to distinguish
the reason of failure from "unexpected type" failure checked in the
prior code path below:
if info[1] != type:
raise util.Abort(_('cannot read %r object at %s') % (type, rev))
This patch changes help text for "--edit" option of commands below:
- fetch
- qnew
- qrefresh
- qfold
- commit
- tag
This unification reduces translation cost, too.
This patch chooses not "further edit commit message already specified"
(of "hg commit") but "invoke editor on commit messages" as unified
help text for "--edit" option, because the latter is much older than
the former.
This patch changes help text for "--message" option of commands below
for unification.
- sign (of gpg)
- tag
This unification reduces translation cost, too.
This patch doesn't change the description for "--message" of "hg
rebase" below, because this should contain "collapse" word to explain
its purpose (only for "--collapse") clearly.
use text as collapse commit message
Before this patch, largefiles gotten from revisions other than the
parent of the working directory at "hg revert" become "clean"
unexpectedly in steps below:
1. "repo.status()" is invoked (for status check before reverting)
1-1 "dirstate" entry for standinfile SF is "normal"-ed
(1-2 "lfdirstate" entry of largefile LF (for SF) is "normal"-ed)
2. "cmdutil.revert()" is invoked
2-1 standinfile SF is updated in the working directory
2-2 "dirstate" entry for SF is NOT updated
3. "lfcommands.updatelfiles()" is invoked (by "overrides.overriderevert()")
3-1 largefile LF (for SF) is updated in the working directory
3-2 "dirstate" returns "n" and valid timestamp for SF (by 1-1 and 2-2)
3-3 "lfdirstate" entry for LF is "normal"-ed
3-4 "lfdirstate" is written into ".hg/largefiles/dirstate", and
timestamp of LF is stored into "lfdirstate" file (by 3-3)
(ASSUMPTION: timestamp of LF differs from one of "lfdirstate" file)
Then, "hs status" treats LF as "clean", even though LF is updated by
"other" revision (by 3-1), because "lfilesrepo.status()" always treats
"normal"-ed files (by 3-3 and 3-4) as "clean".
When largefiles are reverted, they should be "normallookup"-ed
forcibly.
This patch uses "normallookup" on "lfdirstate" while reverting, by
passing "True" to newly added argument "normallookup".
Forcible "normallookup"-ing is not so expensive, because list of
target largefiles is explicitly specified in this case.
This patch uses "[debug] dirstate.delaywrite" feature in the test, to
ensure that timestamp of the largefile gotten from "other" revision is
stored into ".hg/largefiles/dirstate" (for ASSUMPTION at 3-4)
Before this patch, largefiles gotten from "other" revision (with
conflict) at "hg merge" become "clean" unexpectedly in steps below:
1. "repo.status()" is invoked (for status check before merging)
1-1 "dirstate" entry for standinfile SF is "normal"-ed
1-2 "lfdirstate" entry of largefile LF (for SF) is "normal"-ed
2. "merge.update()" is invoked
2-1 SF is updated in the working directory
(ASSUMPTION: user choice "other" at conflict)
2-2 "dirstate" entry for SF is "merge"-ed
3. "lfcommands.updatelfiles()" is invoked (by "overrides.hgmerge()")
3-1 largefile LF (for SF) is updated in the working directory
3-2 "dirstate" returns "m" for SF (by 2-2)
3-3 "lfdirstate" entry for LF is left as it is
3-4 "lfdirstate" is written into ".hg/largefiles/dirstate", and
timestamp of LF is stored into "lfdirstate" file (by 1-2)
(ASSUMPTION: timestamp of LF differs from one of "lfdirstate" file)
Then, "hs status" treats LF as "clean", even though LF is updated by
"other" revision (by 3-1), because "lfilesrepo.status()" always treats
"normal"-ed files (by 1-2 and 3-4) as "clean".
When state of standinfile in "dirstate" is "m", largefile should be
"normallookup"-ed.
This patch invokes "normallookup" on "lfdirstate" for merged files.
This patch uses "[debug] dirstate.delaywrite" feature in the test, to
ensure that timestamp of the largefile gotten from "other" revision is
stored into ".hg/largefiles/dirstate". (for ASSUMPTION at 3-4)
Before this patch, largefiles gotten from "other" revision (without
conflict) at "hg merge" become "clean" unexpectedly in steps below:
1. "merge.update()" is invoked
1-1 standinfile SF is updated in the working directory
1-2 "dirstate" entry for SF is "normallookup"-ed
2. "lfcommands.updatelfiles()" is invoked (by "overrides.hgmerge()")
2-1 largefile LF (for SF) is updated in the working directory
2-2 "dirstate" returns "n" for SF (by 1-2)
2-3 "lfdirstate" entry for LF is "normal"-ed
2-4 "lfdirstate" is written into ".hg/largefiles/dirstate", and
timestamp of LF is stored into "lfdirstate" file
(ASSUMPTION: timestamp of LF differs from one of "lfdirstate" file)
Then, "hs status" treats LF as "clean", even though LF is updated by
"other" revision (by 2-1), because "lfilesrepo.status()" always treats
"normal"-ed files (by 2-3 and 2-4) as "clean".
When timestamp is not set (= negative value) for standinfile in
"dirstate", largefile should be "normallookup"-ed regardless of
rebasing or not, because "n" state in "dirstate" doesn't ensure
"clean"-ness of a standinfile at that time.
This patch uses "normallookup" instead of "normal", if "mtime" of
standin is unset
This is a temporary way to fix with less changes. For fundamental
resolution of this kind of problems in the future, "lfdirstate" should
be updated with "dirstate" simultaneously while "merge.update"
execution: maybe by hooking "recordupdates"
It is also why this patch (temporarily) uses internal field "_map" of
"dirstate" directly.
This patch uses "[debug] dirstate.delaywrite" feature in the test, to
ensure that timestamp of the largefile gotten from "other" revision is
stored into ".hg/largefiles/dirstate". (for ASSUMPTION at 2-4)
This patch newly adds "test-largefiles-update.t", to avoid increasing
cost to run other tests for largefiles by subsequent patches
(especially, "[debug] dirstate.delaywrite" causes so).
Previously, the directory '.hg/largefiles' would always be created if it didn't
exist when the lfdirstate was opened. If there were no standin files, no
dirstate file would be created in the directory. The end result was that
enabling the largefiles extension globally, but not explicitly adding a
largefile would result in the repository eventually sprouting this directory.
Creation of this directory effectively changes readonly operations like summary
and status into operations that require write access. Without write access,
commands that would succeed without the extension loaded would abort with a
surprising error when the extension is loaded, but not actively used:
$ hg sum -R /tmp/thg --config extensions.largefiles=
parent: 16541:00dc703d5aed
repowidget: specify incoming bundle by plain file path to avoid url parsing
branch: default
abort: Permission denied: '/tmp/thg/.hg/largefiles'
This change is simpler than changing the callers of openlfdirstate() to use the
'create' parameter that was introduced in 74522122b97d, and probably how that
should have been implemented in the first place.
test-convert-cvs.t has been a little flaky for a while now. Add an
extra tiebreaker in cscmp so that all the cases in the test will sort
reliably.
Without this patch, test-convert-cvs.t failed after 346 runs. With
this patch, I stopped trying to get it to fail after 615 runs. While
not conclusive, that makes me pretty optimistic that this is a working
fix.
If match.traversedir is not None, we're forced to do full walks. However when
we aren't purging directories we don't need to set match.traversedir to
anything.
This speeds up non-full walks such as the one hgwatchman makes possible. For
mozilla-central with hgwatchman enabled, 'hg purge --files' goes from 0.88
seconds to 0.22.
"editform" argument for "getcommiteditor" is decided according to the
format below:
EXTENSION[.COMMAND][.ROUTE]
- EXTENSION: name of extension
- COMMAND: name of command, if there are two or more commands in EXTENSION
- ROUTE: name of route, if there are two or more routes in COMMAND
This patch newly adds "normal" and "merge" as ROUTE, to distinguish
merge commits from other.
This patch adds 4 test patterns to test combination of "merge"(x2) and
"--continue"(x2).
"editform" argument for "getcommiteditor" is decided according to the
format below:
EXTENSION[.COMMAND][.ROUTE]
- EXTENSION: name of extension
- COMMAND: name of command, if there are two or more commands in EXTENSION
- ROUTE: name of route, if there are two or more routes in COMMAND
This patch newly adds "merge" as ROUTE, to distinguish merge commits
from other.
This patch passes bool as "ctxorbool" to "mergeeditform", because
working context has always 2 parents at this point. Dropping the
second parent of non-merging commits is executed in "concludenode".
Unlike other patches in this series (e.g. for "hg commit"), this patch
doesn't add "normal.normal"/"normal.merge" style ROUTEs, because there
is no "merge" case in "collapse" ROUTE.
Before this patch, if both "--message" and "--collapse" are specified
for "hg rebase", "rebaes.normal" is used as "editform" unexpectedly.
Unlike patches before and after in this series for improvement, this
is bug fix patch.
In all the remaining cases the comprehension variable is used for the same
thing as a previous loop variable.
This will mute some pyflakes "list comprehension redefines" warnings.
_ is usually used for i18n markup but we also used it for I-don't-care
variables.
Instead, name don't-care variables in a slightly descriptive way but use the _
prefix to designate unused variable.
This will mute some pyflakes "import '_' ... shadowed by loop variable"
warnings.
Before this patch, linear merging of modified largefiles causes
an unexpected result, if (1) largefile collides with same-name normal one
in the target revision and (2) "local" largefile is chosen, even
though branch merging between such revisions works correctly.
Expected result of such linear merging is marking the largefile as
(re-)"added", but the actual result is marking it as "modified".
The standin of modified "local largefile" is not changed by linear
merging, and updating/merging update lfdirstate entries only for
largefiles of which standins are changed.
This patch adds the code path to update lfdirstate only for largefiles
of which standins are not changed.
In this case, "synclfdirstate" should be invoked with True as
"normallookup" argument always to force using "normallookup" on
dirstate for "n" files, because "normal" may mark target files as
"clean" unexpectedly.
To reduce cost of "lfile not in filelist", this patch converts
"filelist" to a "set" object: "filelist" is used only in (1) the newly
added code path and (2) the next line of "filelist = set(filelist)".
This is a temporary way to fix with less changes. For fundamental
resolution of this kind of problems in the future, "lfdirstate" should
be updated with "dirstate" simultaneously during "merge.update"
execution: maybe by hooking "recordupdates" (+ total refactoring
around lfdirstate handling)
Before this patch, linear merging of modified or newly added largefile
causes unexpected result, if (1) largefile collides with same name
normal one in the target revision and (2) "local" largefile is chosen,
even though branch merging between such revisions doesn't.
Expected result of such linear merging is:
(1) (not yet recorded) largefile is kept in the working directory
(2) largefile is marked as (re-)"added"
(3) colliding normal file is marked as "removed"
But actual result is:
(1) largefile in the working directory is unlinked
(2) largefile is marked as "normal" (so treated as "missing")
(3) the dirstate entry for colliding normal file is just dropped
(1) is very serious, because there is no way to restore temporarily
modified largefiles.
(3) prevents the next commit from adding the manifest with correct
"removal of (normal) file" information for newly created changeset.
The root cause of this problem is putting "lfile" into "actions['r']"
in linear-merging case. At liner merging, "actions['r']" causes:
- unlinking "target file" in the working directory, but "lfile" as
"target file" is also largefile itself in this case
- dropping the dirstate entry for target file
"actions['f']" (= "forget") does only the latter, and this is reason
why this patch doesn't choose putting "lfile" into it instead of
"actions['r']".
This patch newly introduces action "lfmr" (LargeFiles: Mark as
Removed) to mark colliding normal file as "removed" without unlinking
it.
This patch uses "hg debugdirstate" instead of "hg status" in test,
because:
- choosing "local largefile" hides "removed" status of "remote
normal file" in "hg status" output, and
- "hg status" for "large2" in this case has another problem fixed in
the subsequent patch
