To detect change of a file without redundant comparison of file
content, dirstate recognizes a file as certainly clean, if:
(1) it is already known as "normal",
(2) dirstate entry for it has valid (= not "-1") timestamp, and
(3) mode, size and timestamp of it on the filesystem are as same as
ones expected in dirstate
This works as expected in many cases, but doesn't in the corner case
that changing a file keeps mode, size and timestamp of it on the
filesystem.
The timetable below shows steps in one of typical such situations:
---- ----------------------------------- ----------------
timestamp of "f"
----------------
dirstate file-
time action mem file system
---- ----------------------------------- ---- ----- -----
N -1 ***
- make file "f" clean N
- execute 'hg foobar'
- instantiate 'dirstate' -1 -1
- 'dirstate.normal("f")' N -1
(e.g. via dirty check)
- change "f", but keep size N
N+1
- release wlock
- 'dirstate.write()' N N
- 'hg status' shows "f" as "clean" N N N
---- ----------------------------------- ---- ----- -----
The most important point is that 'dirstate.write()' is executed at N+1
or later. This causes writing dirstate timestamp N of "f" out
successfully. If it is executed at N, 'parsers.pack_dirstate()'
replaces timestamp N with "-1" before actual writing dirstate out.
Occasional test failure for unexpected file status is typical example
of this corner case. Batch execution with small working directory is
finished in no time, and rarely satisfies condition (2) above.
This issue can occur in cases below;
- 'hg revert --rev REV' for revisions other than the parent
- failure of 'merge.update()' before 'merge.recordupdates()'
The root cause of this issue is that files are changed without
flushing in-memory dirstate changes via 'repo.commit()' (even though
omitting 'dirstate.normallookup()' on changed files also causes this
issue).
To detect changes of files correctly, this patch writes in-memory
dirstate changes out explicitly after marking files as clean in
'workingctx._checklookup()', which is invoked via 'repo.status()'.
After this change, timetable is changed as below:
---- ----------------------------------- ----------------
timestamp of "f"
----------------
dirstate file-
time action mem file system
---- ----------------------------------- ---- ----- -----
N -1 ***
- make file "f" clean N
- execute 'hg foobar'
- instantiate 'dirstate' -1 -1
- 'dirstate.normal("f")' N -1
(e.g. via dirty check)
----------------------------------- ---- ----- -----
- 'dirsttate.write()' -1 -1
----------------------------------- ---- ----- -----
- change "f", but keep size N
N+1
- release wlock
- 'dirstate.write()' -1 -1
- 'hg status' -1 -1 N
---- ----------------------------------- ---- ----- -----
To reproduce this issue in tests certainly, this patch emulates some
timing critical actions as below:
- timestamp of "f" in '.hg/dirstate' is -1 at the beginning
'hg debugrebuildstate' before command invocation ensures it.
- make file "f" clean at N
- change "f" at N
'touch -t 200001010000' before and after command invocation
changes mtime of "f" to "2000-01-01 00:00" (= N).
- invoke 'dirstate.write()' via 'repo.status()' at N
'fakedirstatewritetime.py' forces 'pack_dirstate()' to use
"2000-01-01 00:00" as "now", only if 'pack_dirstate()' is invoked
via 'workingctx._checklookup()'.
- invoke 'dirstate.write()' via releasing wlock at N+1 (or "not at N")
'pack_dirstate()' via releasing wlock uses actual timestamp at
runtime as "now", and it should be different from the "2000-01-01
00:00" of "f".
BTW, this patch also changes 'test-largefiles-misc.t', because adding
'dirstate.write()' makes recent dirstate changes visible to external
process.
Previously, if a subrepo was added in ctx2 and then compared to another without
it (ctx1), the subrepo for ctx2 was returned amongst all of the ctx1 based
subrepos, since no subrepo exists in ctx1 to replace it in the 'subpaths' dict.
The two callers of this, basectx.status() and cmdutil.diffordiffstat(), both
compare the yielded subrepo against ctx2, and thus saw no changes when ctx2's
subrepo was returned. The tests here previously didn't mention 's/a' for the
'p1()' case.
This appears to have been a known issue, because some diffordiffstat() comments
mention that the subpath disappeared, and "the best we can do is ignore it". I
originally ran into the issue with some custom convert code to flatten a tree of
subrepos causing hg.putcommit() to abort, but this new behavior seems like the
correct status and diff behavior regardless. (The abort in convert isn't
something users will see, because convert doesn't currently support subrepos in
the official repo.)
The phase of the pending commit depends on the parent of the working directory
and on the phases.newcommit configuration.
First, this information rather depend on the commit line which describe the
pending commit.
Then, we only want to be advertised when the pending phase is going to be higher
than the default new commit phase.
So the format will change from
$ hg summary
parent: 2:ab91dfabc5ad
foo
parent: 3:24f1031ad244 tip
bar
branch: default
commit: 1 modified, 1 unknown, 1 unresolved (merge)
update: (current)
phases: 1 secret (secret)
to
parent: 2:ab91dfabc5ad
foo
parent: 3:24f1031ad244 tip
bar
branch: default
commit: 1 modified, 1 unknown, 1 unresolved (merge) (secret)
update: (current)
phases: 1 secret
We are doing some strange special casing of phase push when:
- the source is a subrepo
- the destination is publishing
- some changeset are still draft on the destination
In that case we do not push phases information (to publish the draft changesets)
because it could break simple cycle of 'clone/pull/push' of subrepos. We have to
detect this case earlier to have bundle2 respecting it.
We change the test to check the behavior for both bundle1 and bundle2.
When the progress extension is not enabled, each call to 'ui.progress' used to
issue a debug message. This results is a very verbose output and often redundant
in tests. Dropping it makes tests less volatile to factor they do not meant to
test.
We had to alter the sed trick in 'test-rename-merge2.t'. Sed is used to drop all
output from a certain point and hidding the progress output remove its anchor.
So we anchor on something else.
The number of draft and secret changesets are currently not summarized.
This is an important information because the number of drafts give some rough
idea of the number of outgoing changesets in typical workflows, without needing
to probe a remote repository. And a non-zero number of secrets means that
those changeset will not be pushed.
If the repository is "dirty" - some draft or secret changesets exists - then
summary will display a line like:
phases: X draft, Y secret (public)
The phase in parenthesis corresponds to the highest phase of the parents of
the working directory, i.e. the current phase.
By default, the line is not printed if the repository is "clean" - all
changesets are public - but if verbose is activated, it will display:
phases: (public)
On the other hand, nothing will be printed if quiet is in action.
A few tests have been added in test-phases.t to cover the -v and -q cases.
When passing a --rev, 'hg incoming -S' previously suffered from the same output
truncation or abort that was fixed for 'hg outgoing -S' in the previous patch,
for the same reasons.
Unlike push, subrepos are currently only pulled when the outer repo is updated,
not when the outer repo is pulled. That makes matching 'hg pull' behavior
impossible. Listing all incoming csets in the subrepo seems like the most
useful behavior, and is consistent with 'hg outgoing -S'.
The previous behavior didn't reflect what would actually be pushed- push will
ignore --rev and --branch in the subrepo and push everything. Therefore,
'push -r {rev}' would not list everything, unless {rev} was also the revision of
the subrepo's tip. Worse, if a hash was passed in, the command would abort
because that hash would either not be in the outer repo or not in the subrepo.
The '' that is used to represent the state of a not-yet-committed
subrepo cannot be written to the file, because the code that parses
the file splits on ' ' and expects two parts.
Given that the .hgsubstate file is automatically rewritten on commit, it seems
a little strange that the file is written out during a merge.
This patch newly adds "dirtyreason()" to centralize composing dirty
reason message like "uncommitted changes in subrepository 'xxxx'".
There are 3 similar messages below, and this patch is a part of
preparations for unifying them into (1), too.
1. uncommitted changes in subrepository 'XXXX'
2. uncommitted changes in subrepository XXXX
3. uncommitted changes in subrepo XXXX
This patch chooses adding new method "dirtyreason()" instead of making
"dirty()" return "reason string", because:
- some of existing "dirty()" implementation is too complicated to do
so simply, and
- ill-mannered 3rd party subrepo classes, of which "dirty()" doesn't
return "reason string", cause meaningless message (even though it
is rare case)
Paths into the subrepo are not yet supported.
The need to use the workingctx in the subrepo will likely be used more in the
future, with the proposed working directory revset symbol. It is also needed
with archive, if that code is to be reused to support 'extdiff -S'.
Unfortunately, it doesn't seem possible to put the smarts in subrepo.subrepo(),
as it breaks various status and diff tests.
I opted not to pass the desired revision into the subrepo method explicitly,
because the only ones that do pass an explicit revision are methods like status
and diff, which actually operate on two contexts- the subrepo state and the
explicitly passed revision.
There are currently no tests for change/remove conflicts of subrepos,
and it's pretty broken. Add some tests demonstrating some of the
breakages and fix the most obvious one (a KeyError when trying to look
up a subrepo in the wrong context).
Before this patch, failure of updating subrepos may cause inconsistent
".hgsubstate". For example:
1. dirstate entry for ".hgsubstate" of the parent repo is filled
with valid size/date (via "hg state" or so)
2. "hg update" is invoked at the parent repo
3. ".hgsubstate" of the parent repo is updated on the filesystem as
a part of "g"(et) action in "merge.applyupdates"
4. it is assumed that size/date of ".hgsubstate" on the filesystem
aren't changed from ones at (1)
this is not so difficult condition, because just changing hash
ids (every ids are same in length) in ".hgsubstate" doesn't
change the file size of it
5. "subrepo.submerge()" is invoked to update subrepos
6. failure of updating in one of subrepos raises exception
(e.g. "untracked file differs")
7. "hg update" is aborted without updating dirstate of the parent repo
dirstate entry for ".hgsubstate" still holds size/date at (1)
Then, ".hgsubstate" of the parent repo is treated as "CLEAN"
unexpectedly, because updating ".hgsubstate" at (3) doesn't change
size/date of it on the filesystem: see assumption at (4).
This inconsistent ".hgsubstate" status causes unexpected behavior, for
example:
- "hg revert" forgets to revert ".hgsubstate"
- "hg update" misunderstands that (not yet updated) subrepos diverge
(then, it shows the prompt to confirm user's decision)
To avoid inconsistent ".hgsubstate" status above, this patch marks
".hgsubstate" as possibly dirty before "submerge" invocation.
"normallookup"-ed (= dirty) dirstate should be written out, even if
processing is aborted by failure.
This patch marks ".hgsubstate" as possibly dirty before "submerge",
also when it is removed or merged while merging, for safety. This
should prevent Mercurial from misunderstanding inconsistent
".hgsubstate" as clean.
To satisfy conditions at (1) and (4) above, this patch uses "hg status
--config debug.dirstate.delaywrite=2" (to fill valid size/date into
dirstate) and "touch" (to fix date of the file).
Since the point of --dry-run is to show what will happen, the output with and
without it should agree. And since revert wasn't being called on subrepos with
--dry-run before, revert in the subrepo had to be defanged in this case.
test-largefiles-update.t, test-subrepo.t, test-tag.t, and
test-rename-dir-merge.t still warn about no result returned because of
unnecessary globs that test-check-code-hg.t wants, relating to output for
pushing to, pulling from and moving X to Y.
'subrepos' template keyword newly added by this patch shows updated
subrepositories.
For the compatibility with the list of subrepositories shown in the
editor at commit:
- 'subrepos' is empty, at revisions removing '.hgsub' itself
- 'subrepos' is calculated between the revision and the first parent
of it, at merge revisions
To avoid silent regression, this patch also confirms "hg diff" of
".hgsubstate" and parents for each target revisions in the test.
This test is added for changes in the subsequent patch.
This test doesn't use "(glob)" for expected output, because "[paths]"
is configured at subrepo creation by "_abssource()" using
"posixpath.join()" to join path components.
We already have a ":" after the user name to denote the start of the
description. The current usage of quotes around the description is
problematic as the truncation to 80 chars is likely to drop the
closing quote. This may confuse syntax coloration in some editors.
Adds a conflict marker formatter that can produce custom conflict marker
descriptions. It can be set via ui.mergemarkertemplate. The old behavior
can be used still by setting ui.mergemarkers=basic.
The default format is similar to:
{node|short} {tag} {branch} {bookmarks} - {author}: "{desc|firstline}"
And renders as:
contextblahblah
<<<<<<< local: c7fdd7ce4652 - durham: "Fix broken stuff in my feature branch"
line from my changes
=======
line from the other changes
>>>>>>> other: a3e55d7f4d38 master - sid0: "This is a commit to master th...
morecontextblahblah
Preparing for action list split-up, making sure the final change don't have any
test changes.
The patch moves debug statements around without really changing anything.
Arguably, it temporarily makes the code worse. The only justification is that
it makes it easier to review the test changes ... and in the end the big change
will not change test output at all.
The changes to test output are due to changes in the ordering of debug output.
That is mainly because we now do the debug logging for files when we actually
process them. Files are also processed in a slightly different but still
correct order. It is now primarily ordered by action type, secondarily by
filename.
The patch introduces some redundancy. Some of it will be removed again, some of
it will in the end help code readability and efficiency. It is possible that we
later on could introduce a "process this action list and do some logging and
progress reporting and apply this function".
The "preserving X for resolve" debug statements will only have single space
indentation. It will no longer have a leading single space indented "f: msg ->
m" message. Having this message double indented would thus no longer make
sense.
The bid actions will temporarily be sorted using a custom sort key that happens
to match the sort order the simplified code will have in the end.
The cat command with an explicit path into a subrepo is now handled by invoking
cat on the file, from that subrepo. The previous behavior was to complain that
the file didn't exist in the revision (of the top most repo). Now when the file
is actually missing, the revision of the subrepo is named instead (though it is
probably desirable to continue naming the top level repo).
The documented output formatters %d and %p reflect the path from the top level
repo, since the purpose of this is to give the illusion of a unified repository.
Support for the undocumented (for cat) formatters %H, %R, %h, %m and %r was
added long ago (I tested back as far as 0.5), but unfortunately these will
reflect the subrepo node instead of the parent context.
The previous implementation was a bit loose with the return value, i.e. it would
return 0 if _any_ file requested was cat'd successfully. This maintains that
behavior.
Before this patch, "localrepository.commit()" omits ".hgsubstate" from
"modified" (changes[0]) and "removed" (changes[2]) file list before
checking subrepositories, but leaves one in "added" (changes[1]) as it
is.
Then, "localrepository.commit()" adds ".hgsubstate" into "modified" or
"removed" list forcibly, according to subrepository statuses.
If "added" contains ".hgsubstate", the committed context will contain
two ".hgsubstate" in its "files": one from "added" (not omitted one),
and another from "modified" or "removed" (newly added one).
How many times ".hgsubstate" appears in "files" changes node hash,
even though revision content is same, because node hash calculation
uses the specified "files" directly (without duplication check or so).
This means that node hash of committed revision changes according to
existence of ".hgsubstate" in "added" at "localrepository.commit()".
".hgsubstate" is treated as "added", not only in accidental cases, but
also in the case of "qpush" for the patch adding ".hgsubstate".
This patch omits ".hgsubstate" also from "added" files before checking
subrepositories. This patch also omits ".hgsubstate" exclusion in
"qnew"/"qrefresh" introduced by changeset bbb8109a634f, because this
patch makes them meaningless.
"hg parents --template '{files}\n'" newly added to "test-mq-subrepo.t"
enhances checking unexpected multiple appearances of ".hgsubstate" in
"files" of created/refreshed MQ revisions.
Before this patch, `hg commit --secret` was not getting propagated
correctly, and subrepos were not getting the commit in the secret
phase. The problem is that subrepos get their ui from the base repo's
baseui object and ignore the ui object passed on to them. This sets
and restores both ui objects with the appropriate option.
In some case Backout silently succeeded to back out but left all the change
uncommitted. This may be confusing for user so this changeset add a note
reminding to commit. Other backout case already actively informs the user about
created commit.
Before the changeset the backout process was:
1) go to <target>
2) revert to <target> parent
3) update back to changeset we came from
The two update steps can takes a very long time to move back and forth unrelated
file change between <target> and current working directory.
The new process is just merging current working directory with the parent of
<target> using <target> as ancestor. This give the very same result but skip
the two updates. On big repo with a lot of files and changes that save a lots of
time (x20 for one week window).
The "merge" version (hg backout --merge) is still done with upgrades. We could
imagine using in memory commit to speed it up but this is another fish.
Before this patch, phase of newly created commit is determined by
"phases.new-commit" configuration regardless of phase of state in each
subrepositories.
For example, this may cause the "public" revision in the parent
repository referring the "secret" one in subrepository.
This patch checks phase of state in each subrepositories before
committing in the parent, and aborts or changes phase of newly created
commit if subrepositories have more restricted phase than the parent.
This patch uses "follow" as default value of "phases.checksubrepos"
configuration, because it can keep consistency between phases of the
parent and subrepositories without breaking existing tool chains.
"use push -f to force" in the hint at abortion of "hg push" may cause
novice users to execute "push -f" easily without understanding about
problems of multiple branch heads in the repository.
This patch hides description about "-f" in the hint, and leads into
seeing "hg help push" for details about pushing new heads.
When a subrepo has changed on the local and remote revisions, prompt the user
whether it wants to merge those subrepo revisions, keep the local revision or
keep the remote revision.
Up until now mercurial would always perform a merge on a subrepo that had
changed on the local and the remote revisions. This is often inconvenient. For
example:
- You may want to perform the actual subrepo merge after you have merged the
parent subrepo files.
- Some subrepos may be considered "read only", in the sense that you are not
supposed to add new revisions to them. In those cases "merging a subrepo" means
choosing which _existing_ revision you want to use on the merged revision. This
is often the case for subrepos that contain binary dependencies (such as DLLs,
etc).
This new prompt makes mercurial better cope with those common scenarios.
Notes:
- The default behavior (which is the one that is used when ui is not
interactive) remains unchanged (i.e. merge is the default action).
- This prompt will be shown even if the ui --tool flag is set.
- I don't know of a way to test the "keep local" and "keep remote" options (i.e.
to force the test to choose those options).
# HG changeset patch
# User Angel Ezquerra <angel.ezquerra@gmail.com>
# Date 1378420708 -7200
# Fri Sep 06 00:38:28 2013 +0200
# Node ID 2fb9cb0c7b26303ac3178b7739975e663075857d
# Parent 796d34e1b749b79834321ef1181ed8433a5515d9
merge: let the user choose to merge, keep local or keep remote subrepo revisions
When a subrepo has changed on the local and remote revisions, prompt the user
whether it wants to merge those subrepo revisions, keep the local revision or
keep the remote revision.
Up until now mercurial would always perform a merge on a subrepo that had
changed on the local and the remote revisions. This is often inconvenient. For
example:
- You may want to perform the actual subrepo merge after you have merged the
parent subrepo files.
- Some subrepos may be considered "read only", in the sense that you are not
supposed to add new revisions to them. In those cases "merging a subrepo" means
choosing which _existing_ revision you want to use on the merged revision. This
is often the case for subrepos that contain binary dependencies (such as DLLs,
etc).
This new prompt makes mercurial better cope with those common scenarios.
Notes:
- The default behavior (which is the one that is used when ui is not
interactive) remains unchanged (i.e. merge is the default action).
- This prompt will be shown even if the ui --tool flag is set.
- I don't know of a way to test the "keep local" and "keep remote" options (i.e.
to force the test to choose those options).
Publishing server may contains draft changeset when they are created locally. As
publishing is the default, it is actually fairly common. Because of this
"inconsistency" phases synchronization may be done even to publishing server.
This may cause severe issues for subrepo. It is possible to reference read-only
repository as subrepo. Push in a super repo recursively push subrepo. Those
pushes to potential read only repo are not optional, they are "suffered" not
"choosed". This does not break because as the repo is untouched the push is
supposed to be empty. If the reference repo locally contains draft changesets, a
courtesy push is triggered to turn them public. As the repo is read only, the
push fails (after possible prompt asking for credential). Failure of the
sub-push aborts the whole subrepo push. This force the user to define a custom
default-push for such subrepo.
This changeset introduce a prevention of this error client side by skipping the
courtesy phase synchronisation in problematic situation. The phases
synchronisation is skipped when four conditions are gathered:
- this is a subrepo push, (normal push to read-only repo)
- and remote support phase
- and remote is publishing
- and no changesets was pushed (if we pushed changesets, repo is not read only)
The internal config option used in this version is not definitive. It is here to
demonstrate a working fix to the issue.
In the future we probably wants to track subrepo changes and avoid pushing to
untouched one. That will prevent any attempt to push to read-only or unreachable
subrepo.
Another fix to prevent courtesy push from older clients to push to newer server
is also still needed.
Show messages at a point where the actions have been sorted, thus preparing for
backout of 14f4258e3526.
This makes manifestmerge more of a silent operation, just like 'copies' is.
Indent 'preserving' messages to make them subordinate to the action logging so
they fit in the new context. (The 'preserving' messages are quite redundant and
could also be removed completely.)
Locating the share source when no default path is available is now handled in
subrepo._abssource(), so unconditionally setting a default path (and the
associated problems) can be avoided.
The test change reflects the fact that a default path is no longer set on the
resulting share.
This is an alternative fix for issue3518, enabling sharing of repositories with
subrepos, without unconditionally setting the default path in the resulting
repo's hgrc file. Better test coverage is added here, but won't prove this code
is working until f48752441ca0 is backed out.
The problem with the original fix is, if a default path is not available to be
copied over from the share source, the default path on the resulting repo is set
to the source location. Since that's where the actual repository is stored, the
path is essentially self-referential, so push, pull, incoming and outgoing
effectively operate on itself. While incoming and outgoing make it look like
nothing was changed, push currently hangs (see issue3657). In this case where
there is not a real default path, these operations should abort with
"default(-push) not found", like the source repo would. Note this problem with
the original fix affected repos without subrepos too.
This change appends the subrepo path to subrepo errors. That is, when there
is an error performing an operation a subrepo, rather than displaying a message
such as:
pushing subrepo MYSUBREPO to PATH
searching for changes
abort: push creates new remote head HEADHASH!
hint: did you forget to merge? use push -f to force
mercurial will show:
pushing subrepo MYSUBREPO to PATH
searching for changes
abort: push creates new remote head HEADHASH! (in subrepo MYSUBREPO)
hint: did you forget to merge? use push -f to force
The rationale for this change is that the current error messages make it hard
for TortoiseHg (and similar tools) to tell the user which subrepo caused the
push failure.
The "(in subrepo MYSUBREPO)" message has been added to those subrepo methods
were it made sense (by using a decorator). We avoid appending "(in subrepo XXX)"
multiple times when subrepos are nexted by throwing a "SubrepoAbort" exception
after the extra message is appended. The decorator will then "ignore" (i.e. just
re-raise) the exception and never add the message again.
A small drawback of this method is that part of the exception trace is lost when
the exception is catched and re-raised by the annotatesubrepoerror decorator.
Also, because the state() function already printed the subrepo path when it
threw an error, that error has been changed to avoid duplicating the subrepo
path in the error message.
Note that I have also updated several subrepo related tests to reflect these
changes.
