state == 'a' implies check
I fail to see what the point of this check parameter is. Near as I can see,
the only _addpath call where it was set to True was in add(), but there, state
is 'a'.
This is a follow-up to 24a646d9943a.
Factor update code common to all callers of _addpath into _addpath.
By centralizing the update code here, it provides one place to put
updates to new data structures - in a future patch. It also removes
a few lines of duplicate code.
This is about 9 times faster than the Python dirstate packing code.
The relatively small speedup is due to the poor locality and memory
access patterns caused by traversing dicts and other boxed Python
values.
The fix introduced in 3509b9cf8f86 was only partially successful. It is correct
to turn dirstate 'm' merge records into normal/dirty ones but copy records are
lost in the process. To adjust them as well, we need to look in the first
parent manifest to know which files were added and preserve only related
records. But the dirstate does not have access to changesets, the logic has to
moved at another level, in localrepo.
The original issue was something like:
$ hg init repo
$ cd repo
$ mkdir D
$ echo a > D/a
$ hg ci -Am adda
adding D/a
$ mv D temp
$ mv temp d
$ echo b > d/b
$ hg add d/b
adding D/b
$ hg ci -m addb
$ hg mv d/b d/c
moving D/b to d/c
$ hg st
A d/c
R D/b
Here we expected:
A D/c
R D/b
the logic being we try to preserve case of path components already known in the
dirstate. This is fixed by the current patch.
Note the following stories are not still not supported:
Changing directory case
$ hg mv D d
moving D/a to D/D/a
moving D/b to D/D/b
$ hg st
A D/D/a
A D/D/b
R D/a
R D/b
or:
$ hg mv D/* d
D/a: not overwriting - file exists
D/b: not overwriting - file exists
And if they were, there are probably similar issues with diffing/patching.
When rebasing, if a conflict occurs and is resolved in a way the rebased
revision becomes empty, it is not skipped, unlike revisions being emptied
without conflicts.
The reason is:
- File 'x' is merged and resolved, merge.update() marks it as 'm' in the
dirstate.
- rebase.concludenode() calls localrepo.commit(), which calls
localrepo.status() which calls dirstate.status(). 'x' shows up as 'm' and is
unconditionnally added to the modified files list, instead of being checked
again.
- localrepo.commit() detects 'x' as changed an create a new revision where only
the manifest parents and linkrev differ.
Marking 'x' as modified without checking it makes sense for regular merges. But
in rebase case, the merge looks normal but the second parent is usually
discarded. When this happens, 'm' files in dirstate are a bit irrelevant and
should be considered 'n' possibly dirty instead. That is what the current patch
does.
Another approach, maybe more efficient, would be to pass another flag to
merge.update() saying the 'branchmerge' is a bit of a lie and recordupdate()
should call dirstate.normallookup() instead of merge().
It is also tempting to add this logic to dirstate.setparents(), moving from two
to one parent is what invalidates the 'm' markers. But this is a far bigger
change to make.
v2: succumb to the temptation and move the logic in dirstate.setparents(). mpm
suggested trying _filecommit() first but it is called by commitctx() which
knows nothing about the dirstate and comes too late into the game. A second
approach was to rewrite the 'm' state into 'n' on the fly in dirstate.status()
which failed for graft in the following case:
$ hg init repo
$ cd repo
$ echo a > a
$ hg ci -qAm0
$ echo a >> a
$ hg ci -m1
$ hg up 0
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg mv a b
$ echo c > b
$ hg ci -m2
created new head
$ hg graft 1 --tool internal:local
grafting revision 1
$ hg --config extensions.graphlog= glog --template '{rev} {desc|firstline}\n'
@ 3 1
|
o 2 2
|
| o 1 1
|/
o 0 0
$ hg log -r 3 --debug --patch --git --copies
changeset: 3:19cd7d1417952af13161b94c32e901769104560c
tag: tip
phase: draft
parent: 2:b5c505595c9e9a12d5dd457919c143e05fc16fb8
parent: -1:0000000000000000000000000000000000000000
manifest: 3:3d27ce8d02241aa59b60804805edf103c5c0cda4
user: test
date: Thu Jan 01 00:00:00 1970 +0000
extra: branch=default
extra: source=a03df74c41413a75c0a42997fc36c2de97b26658
description:
1
Here, revision 3 is created because there is a copy record for 'b' in the
dirstate and thus 'b' is considered modified. But this information is discarded
at commit time since 'b' content is unchanged. I do not know if discarding this
information is correct or not, but at this time we cannot represent it anyway.
This patch therefore implements the last solution of moving the logic into
dirstate.setparents(). It does not sound crazy as 'm' files makes no sense with
only one parent. It also makes dirstate.merge() calls .lookupnormal() if there
is one parent, to preserve the invariant.
I am a bit concerned about introducing this kind of stateful behaviour to
existing code which historically treated setparents() as a basic setter without
side-effects. And doing that during the code freeze.
file in nested directory causes unexpected abort.
problems below should be fixed for recursive normalization route in
dirstate._normalize():
1. rsplit() may cause unpacking into more than 2 elements.
it should be called with 'maxsplit' argument to unpack
into 'd, f'
2. 'd' is replaced by normalized value prefixed with
'self._root', but this makes 'folded' as absolute path,
and it is unexpected one for caller of recursive
normalization
on icasefs, "hg qnew" fails to import changing letter case of filename
already occurred in working directory, for example:
$ hg rename a tmp
$ hg rename tmp A
$ hg qnew casechange
$ hg status
R a
$
"hg qnew" invokes 'dirstate.walk()' via 'localrepository.commit()'
with 'exact match' matching object having exact filenames of targets
in ones 'files()'.
current implementation of 'dirstate.walk()' always normalizes letter
case of filenames from 'match.files()' on icasefs, even though exact
matching is required.
then, files only different in letter case are treated as one file.
this patch prevents 'dirstate.walk()' from normalizing, if exact
matching is required, even on icasefs.
filenames for 'exact matching' are given not from user command line,
but from dirstate walk result, manifest of changecontext, patch files
or fixed list for specific system files (e.g.: '.hgtags').
in such case, case normalization should not be done, so this patch
works well.
path to repo root may contains case sensitive part, even though repo
is located in case insensitive filesystem: e.g. repo in FAT32 device
mounted on Unix.
so, case normalized root causes failure of stat(2).
this patch uses case preserved root for 'util.fspath()' invocation to
avoid this problem.
case preserved root for 'util.fspath()' may decrease efficiency of
fspath cache, but 'util.fspath()' is currently called only from
dirstate, so this fix has less impact.
this patch adds 'dirs()' to changectx/workingctx, which returns map of
all directories deduced from manifest, to examine whether specified
pattern is related to the context as directory or not quickly.
'workingctx.dirs()' uses 'dirstate.dirs()' rather than building
another copy of it.
'dirstate._normalize()', the only caller of 'util.fspath()', has
already normcase()-ed path before invocation of it.
normcase()-ed root can be cached on dirstate side, too.
so, this patch changes 'util.fspath()' API specification to avoid
normcase()-ing in it.
at first of dirstate.walk() on case insensitive filesystem,
normalization of '.' causes util.fspath() invocation, but '.' is not
cached in it.
this invocation is not only useless, but also harmful: initial "hg
tag" causes creation of ".hgtags" file after dirstate.walk(), and
looking up ".hgtags" in cache will fail, because directory contents of
root is already cached at util.fspath() invocation for '.'.
Complex merges with divergent renames can cause a file to be 'moved'
twice, causing dirstate.drop() to be called twice. Rather than try to
ensure there are no unexpected corner cases where this can happen, we
simply ignore drops of files that aren't tracked.
Before this patch, Windows always did the wrong thing with exec bits
when committing a merge: consult the flags in first parent.
Now we manually recompute the result of merging flags at commit time,
which almost always does the right thing (except when there are
conflicts between symlink and exec flags).
To do this, we:
- pull flag synthesis out into its own function
- delay building this function unless it's needed
- add a merge case that compares flags in local and other against the ancestor
This has been tested in multiple ways on Linux:
- running the whole test suite with both old and new code in place,
checking for differences in each flags() result
- running the whole test suite while comparing real on-disk flags
against synthetic ones for merges
- test-issue1802 (from Martin Geisler) which disables exec bit
checking on Unix
The usual contract is that close() makes your writes permanent, so
atomictempfile's use of close() to *discard* writes (and rename() to
keep them) is rather unexpected. Thus, change it so close() makes
things permanent and add a new discard() method to throw them away.
discard() is only used internally, in __del__(), to ensure that writes
are discarded when an atomictempfile object goes out of scope.
I audited mercurial.*, hgext.*, and ~80 third-party extensions, and
found no one using the existing semantics of close() to discard
writes, so this should be safe.
It has substantially different semantics from forget at the command
layer, so change it to avoid confusion.
We can't simply combine it with remove because we need to explicitly
drop non-added files in some cases like commit.
These leaks may occur in environments that don't employ a reference
counting GC, i.e. PyPy.
This implies:
- changing opener(...).read() calls to opener.read(...)
- changing opener(...).write() calls to opener.write(...)
- changing open(...).read(...) to util.readfile(...)
- changing open(...).write(...) to util.writefile(...)
We can get rid of the _lastnormal set by using the filesystem mtimes to
identify the problematic "lastnormal" files on status(), forcing a file
content-comparison if the file's mtime timeslot is equal to _lastnormaltime.
- consistently use mtime as mapped to dirstate granularity (needed for
filesystems like NTFS, which have sub-second resolution)
- no need to add files with mtime < _lastnormaltime
- improve comments
(issue2264, issue2516)
The race happens when two commits in a row change the same file
without changing its size, *if* those two commits happen in the same
second in the same process while holding the same repo lock. For
example:
commit 1:
M a
M b
commit 2: # same process, same second, same repo lock
M b # modify b without changing its size
M c
This first manifested in transplant, which is the most common way to
do multiple commits in the same process. But it can manifest in any
script or extension that does multiple commits under the same repo
lock. (Thus, the test script tests both transplant and a custom script.)
The problem was that dirstate.status() failed to notice the change to
b when localrepo is about to do the second commit, meaning that change
gets left in the working directory. In the context of transplant, that
means either a crash ("RuntimeError: nothing committed after
transplant") or a silently inaccurate transplant, depending on whether
any other files were modified by the second transplanted changeset.
The fix is to make status() work a little harder when we have
previously marked files as clean (state 'normal') in the same process.
Specifically, dirstate.normal() adds files to self._lastnormal, and
other state-changing methods remove them. Then dirstate.status() puts
any files in self._lastnormal into state 'lookup', which will make
localrepository.status() read file contents to see if it has really
changed. So we pay a small performance penalty for the second (and
subsequent) commits in the same process, without affecting the common
case. Anything that does lots of status updates and checks in the
same process could suffer a performance hit.
Incidentally, there is a simpler fix: call dirstate.normallookup() on
every file updated by commit() at the end of the commit. The trouble
with that solution is that it imposes a performance penalty on the
common case: it means the next status-dependent hg command after every
"hg commit" will be a little bit slower. The patch here is more
complex, but only affects performance for the uncommon case.
Add missing calls to close() to many places where files are
opened. Relying on reference counting to catch them soon-ish is not
portable and fails in environments with a proper GC, such as PyPy.
split can be more readable for longer lists like the list in
dirstate.invalidate. As dirstate.invalidate is used in wlock() and therefoe
used heavily, I think it's worth avoiding a split there too.
Previously, branch names were ideally manipulated as UTF-8 strings,
because they were stored as UTF-8 in the dirstate and the changelog
and could not be safely converted to the local encoding and back.
However, only about 80% of branch name code was actually using the
right encoding conventions. This patch uses the localstr addition to
allow working on branch names as local strings, which simplifies
handling so that the previously incorrect code becomes correct.