To detect change of a file without redundant comparison of file
content, dirstate recognizes a file as certainly clean, if:
(1) it is already known as "normal",
(2) dirstate entry for it has valid (= not "-1") timestamp, and
(3) mode, size and timestamp of it on the filesystem are as same as
ones expected in dirstate
This works as expected in many cases, but doesn't in the corner case
that changing a file keeps mode, size and timestamp of it on the
filesystem.
The timetable below shows steps in one of typical such situations:
---- ----------------------------------- ----------------
timestamp of "f"
----------------
dirstate file-
time action mem file system
---- ----------------------------------- ---- ----- -----
N -1 ***
- make file "f" clean N
- execute 'hg foobar'
- instantiate 'dirstate' -1 -1
- 'dirstate.normal("f")' N -1
(e.g. via dirty check)
- change "f", but keep size N
N+1
- release wlock
- 'dirstate.write()' N N
- 'hg status' shows "f" as "clean" N N N
---- ----------------------------------- ---- ----- -----
The most important point is that 'dirstate.write()' is executed at N+1
or later. This causes writing dirstate timestamp N of "f" out
successfully. If it is executed at N, 'parsers.pack_dirstate()'
replaces timestamp N with "-1" before actual writing dirstate out.
Occasional test failure for unexpected file status is typical example
of this corner case. Batch execution with small working directory is
finished in no time, and rarely satisfies condition (2) above.
This issue can occur in cases below;
- 'hg revert --rev REV' for revisions other than the parent
- failure of 'merge.update()' before 'merge.recordupdates()'
The root cause of this issue is that files are changed without
flushing in-memory dirstate changes via 'repo.commit()' (even though
omitting 'dirstate.normallookup()' on changed files also causes this
issue).
To detect changes of files correctly, this patch writes in-memory
dirstate changes out explicitly after marking files as clean in
'workingctx._checklookup()', which is invoked via 'repo.status()'.
After this change, timetable is changed as below:
---- ----------------------------------- ----------------
timestamp of "f"
----------------
dirstate file-
time action mem file system
---- ----------------------------------- ---- ----- -----
N -1 ***
- make file "f" clean N
- execute 'hg foobar'
- instantiate 'dirstate' -1 -1
- 'dirstate.normal("f")' N -1
(e.g. via dirty check)
----------------------------------- ---- ----- -----
- 'dirsttate.write()' -1 -1
----------------------------------- ---- ----- -----
- change "f", but keep size N
N+1
- release wlock
- 'dirstate.write()' -1 -1
- 'hg status' -1 -1 N
---- ----------------------------------- ---- ----- -----
To reproduce this issue in tests certainly, this patch emulates some
timing critical actions as below:
- timestamp of "f" in '.hg/dirstate' is -1 at the beginning
'hg debugrebuildstate' before command invocation ensures it.
- make file "f" clean at N
- change "f" at N
'touch -t 200001010000' before and after command invocation
changes mtime of "f" to "2000-01-01 00:00" (= N).
- invoke 'dirstate.write()' via 'repo.status()' at N
'fakedirstatewritetime.py' forces 'pack_dirstate()' to use
"2000-01-01 00:00" as "now", only if 'pack_dirstate()' is invoked
via 'workingctx._checklookup()'.
- invoke 'dirstate.write()' via releasing wlock at N+1 (or "not at N")
'pack_dirstate()' via releasing wlock uses actual timestamp at
runtime as "now", and it should be different from the "2000-01-01
00:00" of "f".
BTW, this patch also changes 'test-largefiles-misc.t', because adding
'dirstate.write()' makes recent dirstate changes visible to external
process.
The phase of the pending commit depends on the parent of the working directory
and on the phases.newcommit configuration.
First, this information rather depend on the commit line which describe the
pending commit.
Then, we only want to be advertised when the pending phase is going to be higher
than the default new commit phase.
So the format will change from
$ hg summary
parent: 2:ab91dfabc5ad
foo
parent: 3:24f1031ad244 tip
bar
branch: default
commit: 1 modified, 1 unknown, 1 unresolved (merge)
update: (current)
phases: 1 secret (secret)
to
parent: 2:ab91dfabc5ad
foo
parent: 3:24f1031ad244 tip
bar
branch: default
commit: 1 modified, 1 unknown, 1 unresolved (merge) (secret)
update: (current)
phases: 1 secret
The number of draft and secret changesets are currently not summarized.
This is an important information because the number of drafts give some rough
idea of the number of outgoing changesets in typical workflows, without needing
to probe a remote repository. And a non-zero number of secrets means that
those changeset will not be pushed.
If the repository is "dirty" - some draft or secret changesets exists - then
summary will display a line like:
phases: X draft, Y secret (public)
The phase in parenthesis corresponds to the highest phase of the parents of
the working directory, i.e. the current phase.
By default, the line is not printed if the repository is "clean" - all
changesets are public - but if verbose is activated, it will display:
phases: (public)
On the other hand, nothing will be printed if quiet is in action.
A few tests have been added in test-phases.t to cover the -v and -q cases.
'hg revert -I foo' currently fails with
abort: no files or directories specified
(use --all to revert all files, or 'hg update 1' to update)
It doesn't seem intentional that -I/-X without other paths or
--all/--interactive should fail, and it doesn't seem that harmful to
allow it either, so let's just do that.
When calculating status involving the working copy and a revision
other than the parent of the working copy, the files that are not in
the working context manifest ('mf2' in the basectx._buildstatus())
will be reported as removed (note that deleted files _are_ in the
working context manifest). However, if the file is reported as deleted
in the dirstate, it will get that status too (as shown by failing
tests).
Fix by removing deleted files from the 'removed' list after the main
loop in _buildstatus().
Add a parameter to generate-working-copy-states.py that indicates
how many changesets are wanted. This number impacts all the
subcommands. The current 'filelist' subcommand becomes 'filelist 2',
the current 'base' and 'parent' subcommands become 'state 2 1' and
'state 2 2' respectively, while 'wc' becomes 'state 2 wc'.
See earlier patch for motivation.
With the recent change in naming of the generated files, it becomes
much easier to generate the files by iterating over all the possible
states than over the state transitions.
When a file is missing in the 'parent' version and is tracked but
missing in the working directory, which happens by the 'missing' or
'removed' types, and the 'clean' type in the working directory, the
file does not exist in the working directory (unlike it would had the
'deleted' type been used). Thus, the *_missing_missing_tracked are not
actually tracked and they end up testing the same state as
*_missing_missing_untracked. To make them tracked, add a temporary
file, just like we do for the delete case. For simplicity's sake,
let's make sure the gen-revert-cases.py script always puts a file in
the working directory, whether or not it's going to be deleted.
Future patches will change how the output of 'gen-revert-cases.py
filelist' is generated, so now we want the order to depend on just the
filename again.
This is the main patch in a series. See motivation in earlier patch.
In this patch, we actually change the names of the generated
files. For example, the file that is currently called missing_clean
becomes missing_missing_missing-tracked and it's clearer that it
should be tracked. It turns out that since the state was not
previously clear, it ended up testing an untracked state, which was
the same as for missing_clean. We'll fix this in a later patch.
Let's also change the content from (base,parent,wc) to
(content1,content2,content3) to make them all the same length so they
line up when displayed.
The next patch will change the names of the files produced by the
script in test-revert. In order to reduce the size and increase the
clarity of the next patch, make the order produced by the internal
'gen-revert-cases.py filelist' command independent of the filenames.
By putting the file content rather than keys in the 'combination'
list, we restrict the knowledge of 'ctxcontent' and 'wccontent' to the
loop generating the combinations. That will make it easier to replace
the generation code.
The 'wccontent' variable has eight different states, four of them
tracked, and the other four untracked (at least when the file existed
in the parent revision). Among these eight states, 'removed' sticks
out by lacking the 'untracked-' prefix despite resulting in an
untracked state. To make the symmetry clearer, and to prepare for
future patches, rename 'removed' to 'untracked-deleted', which is
exactly what it is.
Note that, unlike 'remove', 'deleted' is configured in
gen-revert-cases.py to have content in the working directory and that
that content is instead expected to be removed in the test script.
However, no changes are needed to the test script, since it already
contains 'hg forget *untracked*' and 'rm *deleted*', which together
have the same effect as 'hg remove'.
See additional motivation in earlier patch.
The tests for removed_deleted and removed_removed test the same state
as removed_clean and removed_untracked-clean, respectively. Drop the
duplicate tests.
See additional motivation in earlier patch.
The tests for added_revert and added_untracked-revert test the same
state as added_deleted and added_removed, respectively. Drop the
duplicate tests.
See additional motivation in earlier patch.
This is the first step in a series that aims to put the state, not the
state transitions, in the filenames of the files generated by the
gen-revert-cases.py script. The possible state of a file in a revision
and in the working copy is only whether it exists and what its content
is (the tests don't care check flags). In the dirstate, the only state
is whether it's tracked or not. With the new naming, the file that is
currently called modified_untracked-clean now becomes
content1_content2_content2-untracked, for example.
By putting these states in the filename, it becomes easier to see that
we're not missing or duplicating any state, and to check that the
state is what we think it is. For example, the file that is currently
called missing_clean becomes missing_missing_missing-tracked and it's
clearer that it should be tracked.
Putting the content in the filename will also make the tests of file
content (e.g. "cat ../content-parent.txt") very obvious.
When we put the state in the filename, the filenames clearly need to
be unique. However, it turns out that some states are currently tested
multiple times. The 'revert' transition in the script means to take
the content from the grandparent. If the parent is the same as the
grandparent, there is no change compared to the parent, which is
exactly what 'clean' means. Avoid testing the same state twice.
It seems like the last known misbehvior that the comment was referring
to was dealt with in 708ed580cbea (revert: properly back up added
files with local modification, 2014-08-31).
These files were previously not backed up because the backup mechanism was not
smart enough. This leads to data lose for the user since uncommitted contents
were discarded.
We now properly move the modified version to <filename>.orig before deleting it.
We have to use a small hack to do a different action if "--no-backup" is
specified. This is needed because the backup process is actually a move (not a
copy) so the file is already missing when we backup. The internet kitten is a
bit disapointed about that, but such is life.
This patch concludes the "lets refactor revert" phases. We can now open the
"Lets find stupid bug with renames and merge" phases.
I'm sure that now that the code is clearer we could do it in another simpler
way, but I consider the current improvement good enough for now.
There are multiple possible cases for added files. But it's all handled by magic
much lower in the stack. We document them, simplify the codes and move on.
Using status information against the target ensures we are catching all
files with modifications that need reverting.
We still need to distinguish fresh modifications for backup purpose.
test-largefile is affected because it reverted a file that needs no content
change.
Tracking clean files is the simplest way to be able to reports files that need
no changes. So we explicitly retrieve them.
This fixes a couple of test outputs where the lack of changes was not reported.
When a file was marked as removed in the working copy and did not existed in the
target of the revert, we did not issued any message pointing that no change was
needed to the file (implicitly saying that revert had changed the file).
We now properly issue a message in this situation. Tests change in and handful
of case where the message was documented as missing.
Unlike untracked, the file is also missing from the working directory.
This test highlights a small misbehavior in output when reverting to another
revision.
We now also test reverting file to another revision's content. However
this differs from previously introduced test by using the explicit path
of each "case file" when calling revert. This should result in the
same result regarding file content and backup creation, but the output
of the `hg revert` call should differ.
We now also test reverting file to the working directory parent
content. However this differs from the previously introduced test by using
the explicit path of each "case file" when calling revert. This should
result in the same result regarding file content and backup creation,
but the output of the `hg revert` call should differ.
Now that we can automatically generate states, we need to actually run
revert on them and check the result. While running such tests we are
checking multiple elements. The output of the `hg revert` command, the
resulting content of file, and the creation of backup file.
The first practical test is using the simple case `hg revert --all`, reverting
all files to working directory parent content.
The text version is just a list of existing files with their content. We use a
small custom script for that.
This is going to be very useful for comparing revert results with
revert target content.
There are multiple comments explaining the expected output of commands. This is
an old relic of the pre-unified test era. We remove them for uselessness.
We highlight the behavior tested by each sections. (This is a gratuitous
improvement before significant upgrade of the test and massive refactoring of
the revert code)
revert was always using p1 as parent. This created some minor misbehavior when
reverting against p2. See test change for an example of that.
This is also a useful cleanup for coming refactoring to revert.
