Before this patch, largefiles gotten from revisions other than the
parent of the working directory at "hg revert" become "clean"
unexpectedly in steps below:
1. "repo.status()" is invoked (for status check before reverting)
1-1 "dirstate" entry for standinfile SF is "normal"-ed
(1-2 "lfdirstate" entry of largefile LF (for SF) is "normal"-ed)
2. "cmdutil.revert()" is invoked
2-1 standinfile SF is updated in the working directory
2-2 "dirstate" entry for SF is NOT updated
3. "lfcommands.updatelfiles()" is invoked (by "overrides.overriderevert()")
3-1 largefile LF (for SF) is updated in the working directory
3-2 "dirstate" returns "n" and valid timestamp for SF (by 1-1 and 2-2)
3-3 "lfdirstate" entry for LF is "normal"-ed
3-4 "lfdirstate" is written into ".hg/largefiles/dirstate", and
timestamp of LF is stored into "lfdirstate" file (by 3-3)
(ASSUMPTION: timestamp of LF differs from one of "lfdirstate" file)
Then, "hs status" treats LF as "clean", even though LF is updated by
"other" revision (by 3-1), because "lfilesrepo.status()" always treats
"normal"-ed files (by 3-3 and 3-4) as "clean".
When largefiles are reverted, they should be "normallookup"-ed
forcibly.
This patch uses "normallookup" on "lfdirstate" while reverting, by
passing "True" to newly added argument "normallookup".
Forcible "normallookup"-ing is not so expensive, because list of
target largefiles is explicitly specified in this case.
This patch uses "[debug] dirstate.delaywrite" feature in the test, to
ensure that timestamp of the largefile gotten from "other" revision is
stored into ".hg/largefiles/dirstate" (for ASSUMPTION at 3-4)
Before this patch, largefiles gotten from "other" revision (with
conflict) at "hg merge" become "clean" unexpectedly in steps below:
1. "repo.status()" is invoked (for status check before merging)
1-1 "dirstate" entry for standinfile SF is "normal"-ed
1-2 "lfdirstate" entry of largefile LF (for SF) is "normal"-ed
2. "merge.update()" is invoked
2-1 SF is updated in the working directory
(ASSUMPTION: user choice "other" at conflict)
2-2 "dirstate" entry for SF is "merge"-ed
3. "lfcommands.updatelfiles()" is invoked (by "overrides.hgmerge()")
3-1 largefile LF (for SF) is updated in the working directory
3-2 "dirstate" returns "m" for SF (by 2-2)
3-3 "lfdirstate" entry for LF is left as it is
3-4 "lfdirstate" is written into ".hg/largefiles/dirstate", and
timestamp of LF is stored into "lfdirstate" file (by 1-2)
(ASSUMPTION: timestamp of LF differs from one of "lfdirstate" file)
Then, "hs status" treats LF as "clean", even though LF is updated by
"other" revision (by 3-1), because "lfilesrepo.status()" always treats
"normal"-ed files (by 1-2 and 3-4) as "clean".
When state of standinfile in "dirstate" is "m", largefile should be
"normallookup"-ed.
This patch invokes "normallookup" on "lfdirstate" for merged files.
This patch uses "[debug] dirstate.delaywrite" feature in the test, to
ensure that timestamp of the largefile gotten from "other" revision is
stored into ".hg/largefiles/dirstate". (for ASSUMPTION at 3-4)
Before this patch, largefiles gotten from "other" revision (without
conflict) at "hg merge" become "clean" unexpectedly in steps below:
1. "merge.update()" is invoked
1-1 standinfile SF is updated in the working directory
1-2 "dirstate" entry for SF is "normallookup"-ed
2. "lfcommands.updatelfiles()" is invoked (by "overrides.hgmerge()")
2-1 largefile LF (for SF) is updated in the working directory
2-2 "dirstate" returns "n" for SF (by 1-2)
2-3 "lfdirstate" entry for LF is "normal"-ed
2-4 "lfdirstate" is written into ".hg/largefiles/dirstate", and
timestamp of LF is stored into "lfdirstate" file
(ASSUMPTION: timestamp of LF differs from one of "lfdirstate" file)
Then, "hs status" treats LF as "clean", even though LF is updated by
"other" revision (by 2-1), because "lfilesrepo.status()" always treats
"normal"-ed files (by 2-3 and 2-4) as "clean".
When timestamp is not set (= negative value) for standinfile in
"dirstate", largefile should be "normallookup"-ed regardless of
rebasing or not, because "n" state in "dirstate" doesn't ensure
"clean"-ness of a standinfile at that time.
This patch uses "normallookup" instead of "normal", if "mtime" of
standin is unset
This is a temporary way to fix with less changes. For fundamental
resolution of this kind of problems in the future, "lfdirstate" should
be updated with "dirstate" simultaneously while "merge.update"
execution: maybe by hooking "recordupdates"
It is also why this patch (temporarily) uses internal field "_map" of
"dirstate" directly.
This patch uses "[debug] dirstate.delaywrite" feature in the test, to
ensure that timestamp of the largefile gotten from "other" revision is
stored into ".hg/largefiles/dirstate". (for ASSUMPTION at 2-4)
This patch newly adds "test-largefiles-update.t", to avoid increasing
cost to run other tests for largefiles by subsequent patches
(especially, "[debug] dirstate.delaywrite" causes so).
Even though "dirstate.write()" is invoked explicitly after "normal"
invocations, timestamp field of entries may be still "unset" in the
"dirstate" file itself , because "pack_dirstate" drops it when it is
equal to the timestamp of "dirstate" file itself.
This can avoid overlooking modification of files, which are updated at
same time in the second. But on the other hand, this may hide timing
critical problems.
For example, incorrect "normal"-ing (or lack of "normallookup"-ing on
the already "normal"-ed entry) is visible only when:
- the target file is modified in the working directory at T1, and
- "dirstate" file is written out at T2 (!= T1)
Otherwise, T1 is dropped by "pack_dirstate" in "dirstate.write()"
invocation, and "unset" is stored into "dirstate" file.
It often fails to reproduce problems from incorrect "normal"-ing by
Mercurial testset, because automated actions in the small repository
almost always causes that T1 and T2 are same.
This patch adds the debug feature to delay writing out to ensure
timestamp of each entries explicitly.
This feature is used to make timing critical "dirstate" problems
reproducable in subsequent patches.
This previously died in _revdescendants() taking the min() of the first set to
only(), when it was empty. An empty second set already worked. Likewise,
descendants() already handled an empty set.
This patch makes commit message shown in text editor customizable by
template. For example, this can advertise:
- sample commit messages for routine works,
- points to call attention before commit,
- message of the day, and so on
To show commit message correctly even in problematic encoding, this
patch chooses the latter below:
- replace "buildcommittext" with "buildcommittemplate" completely
- invoke "buildcommittemplate" only if '[committemplate] changeset'
is configured explicitly
For example, if multibyte character ending with backslash (0x5c) is
followed by ASCII character 'n' in the customized template, sequence
of backslash and 'n' is treated as line-feed unexpectedly (and
multibyte character is broken, too).
This corruption occurs in 'decode("string-escape")' while parsing
template string.
Add a new internal:tagmerge merge tool which implements an automatic merge
algorithm for mercurial's tag files
The tagmerge algorithm is able to resolve most merge conflicts that
currently would trigger a .hgtags merge conflict. The only case that
it does not (and cannot) handle is that in which two tags point to
different revisions on each merge parent _and_ their corresponding tag
histories have the same rank (i.e. the same length). In all other
cases the merge algorithm will choose the revision belonging to the
parent with the highest ranked tag history. The merged tag history is
the combination of both tag histories (special care is taken to try to
combine common tag histories where possible).
The algorithm also handles cases in which tags have been manually
removed from the .hgtags file and other similar corner cases.
In addition to actually merging the tags from two parents, taking into
account the base, the algorithm also tries to minimize the difference
between the merged tag file and the first parent's tag file (i.e. it
tries to make the merged tag order as as similar as possible to the
first parent's tag file order).
The algorithm works as follows:
1. read the tags from p1, p2 and the base
- when reading the p1 tags, also get the line numbers associated to each
tag node (these will be used to sort the merged tags in a way that
minimizes the diff to p1). Ignore the file numbers when reading p2 and
the base
2. recover the "lost tags" (i.e. those that are found in the base but not on p1
or p2) and add them back to p1 and/or p2
- at this point the only tags that are on p1 but not on p2 are those new
tags that were introduced in p1. Same thing for the tags that are on p2
but not on p2
3. take all tags that are only on p1 or only on p2 (but not on the base)
- Note that these are the tags that were introduced between base and p1 and
between base and p2, possibly on separate clones
4. for each tag found both on p1 and p2 perform the following merge algorithm:
- the tags conflict if their tag "histories" have the same "rank" (i.e.
length) _AND_ the last (current) tag is _NOT_ the same
- for non conflicting tags:
- choose which are the high and the low ranking nodes
- the high ranking list of nodes is the one that is longer.
In case of draw favor p1
- the merged node list is made of 3 parts:
- first the nodes that are common to the beginning of both the
low and the high ranking nodes
- second the non common low ranking nodes
- finally the non common high ranking nodes (with the last one
being the merged tag node)
- note that this is equivalent to putting the whole low ranking node
list first, followed by the non common high ranking nodes
- note that during the merge we keep the "node line numbers", which will
be used when writing the merged tags to the tag file
5. write the merged tags taking into account to their positions in the first
parent (i.e. try to keep the relative ordering of the nodes that come
from p1). This minimizes the diff between the merged and the p1 tag files
This is done by using the following algorithm
- group the nodes for a given tag that must be written next to each other
- A: nodes that come from consecutive lines on p1
- B: nodes that come from p2 (i.e. whose associated line number is None)
and are next to one of the a nodes in A
- each group is associated with a line number coming from p1
- generate a "tag block" for each of the groups
- a tag block is a set of consecutive "node tag" lines belonging to the
same tag and which will be written next to each other on the merged
tags file
- sort the "tag blocks" according to their associated number line
- put blocks whose nodes come all from p2 first
- write the tag blocks in the sorted order
Notes:
- A few tests have been added to test-tag.t. These tests are very specific to
the new internal:tagmerge tool, so perhaps they should be moved to their own
test file.
- The merge algorithm was discussed in a thread on the mercurial mailing list.
In http://markmail.org/message/anqaxldup4tmgyrx a slightly different algorithm
was suggested. In it the p1 and p2 tags would have been interleaved instead of
put one before the other. It would be possible to implement that but my tests
suggest that the merge result would be more confusing and harder to understand.
As extensively detailed by Pierre-Yves[1], simplemerge's minimal
markers can result in hopeless confusion for many common merges. As it
happens, we accidentally inherited this behavior when we borrowed
simplemerge from bzr; it is not the behavior used by RCS's merge(1),
Since merge(1) (and not bzr) is what we aim to emulate when emulating
RCS's merge markers, we simply turn this feature off. This brings us
in line with the behavior of CVS, SVN, and Git as a bonus.
(NB: using conflict markers with Mercurial is discouraged.)
[1] http://markmail.org/message/wj5mh3lc46czlvld
convert glob tessa
We are going to introduce a setting to control the "minimisation" feature of
``internal:merge``. So we need more interesting conflicting content. We change
the content in an independent changeset to make sure the coming code change
leave the output unchanged.
Before this patch, 'detailed' is used as the default of '[ui]
mergemarkers'. This embeds non-ASCII characters in tags, branches,
bookmarks, author and/or commit descriptions into merged files in the
encoding specified by '--encoding' global option, 'HGENCODING' or
other locale setting environment variables.
But, if files to be merged use another encoding, this behavior breaks
consistency of encoding in merged files.
For example, ISO-2022-JP or EUC-JP are sometimes used as the file
encoding for Japanese characters, because of historical and/or
environmental reasons, even though UTF-8 or Shift-JIS are ordinarily
used as the terminal encoding.
This can't be resolved automatically, because Mercurial doesn't aware
encoding of managed files.
This patch uses 'basic' as the default of '[ui] mergemarkers' to avoid
embedding encoding sensitive characters for safety.
This patch puts '[ui] mergemarkers = detailed' into default hgrc file
for tests, to reduce changes for tests in this patch.
Previously, the directory '.hg/largefiles' would always be created if it didn't
exist when the lfdirstate was opened. If there were no standin files, no
dirstate file would be created in the directory. The end result was that
enabling the largefiles extension globally, but not explicitly adding a
largefile would result in the repository eventually sprouting this directory.
Creation of this directory effectively changes readonly operations like summary
and status into operations that require write access. Without write access,
commands that would succeed without the extension loaded would abort with a
surprising error when the extension is loaded, but not actively used:
$ hg sum -R /tmp/thg --config extensions.largefiles=
parent: 16541:00dc703d5aed
repowidget: specify incoming bundle by plain file path to avoid url parsing
branch: default
abort: Permission denied: '/tmp/thg/.hg/largefiles'
This change is simpler than changing the callers of openlfdirstate() to use the
'create' parameter that was introduced in 74522122b97d, and probably how that
should have been implemented in the first place.
Python lookups are slow, so do all lookup outside of the for loop.
This provide a small but still significant speedup:
revset #0: 0::
0) wall 0.063258 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
1) wall 0.057776 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
Attribute lookup is slow in python. So this version is going to be a bit
faster. This does not have a visible impact since the rest of the stack is much
slower but this shaves the yak a few extra nanometers.
Moreover the new version is more readable so it worth doing this change for code
quality purpose.
This optimisation was approved by a core python dev.
Instead of iterating over all files in the context and ignoring those
that are not standins, pass a standin-matcher to the context and
iterate over only the files matching.
Apart from making the intent clearer, this implementation will also
benefit from any future optimizations done to the manifest walking
code.
The variable 'lfiles' is first used for a set of the names of all the
large files. It is then overwritten with a tuple like the ones
returned from status(). To reduce confusion, let's create a separate
variable for the second use.
At the end of lfilesrepo.status(), we clear the lists of unknown,
ignored and clean files, depending on the values of 'listunknown'
etc. The lists originate from other calls to status(), and it is only
'clean' that may get updated after the calls. Let's remove the need to
clear any of the lists by explicitly only adding to 'clean' when
'listclean' is true.
Don't try to append empty lines to HG patch headers - instead, add them in str
method.
This minor change removes some apparently redundant code and makes the code
more robust.
There would in some cases be an empty line between headers and the description -
that does not seem right.
There should also be an empty line between description and diff - but that was
missing.
These two mistakes would sometimes make it up for each other so we fix both at
once to just show the improvement.
Instead of writing an extra newline when writing a header line, write an extra
line when it not is written as a part of the description but is necessary
anyway.
In match.__init__(), we create the matchfn predicate by and-ing
together the individual predicates for includes, excludes (negated)
and patterns. Instead of the current set of nested if/else blocks, we
can simplify by adding the predicates to a list and defining the
overall predicate in a generic way based on the components. We can
still optimize it for the 0-length and 1-length cases. This way, there
is no combinatorial explosion to deal with if new component predicates
are added, and there is less risk of getting the overall predicate
wrong.
This default mirrors the default for 'git diff'. Other commands have slightly
different defaults -- for example, the move/copy detection for 'git blame'
assumes that a hunk is moved if more than 40 alphanumeric characters are the
same, or copied if more than 20 alphanumeric characters are the same. 50% seems
to be the most common default, though.
"And" operation with something that contains the whole repo should be super
cheap. Check method docstring for details.
This provide massive boost to simple revset that use `subset & xxx`
revset #0: p1(20000)
0) wall 0.002447 comb 0.010000 user 0.010000 sys 0.000000 (best of 767)
1) wall 0.000529 comb 0.000000 user 0.000000 sys 0.000000 (best of 3947)
revset #1: p2(10000)
0) wall 0.002464 comb 0.000000 user 0.000000 sys 0.000000 (best of 913)
1) wall 0.000530 comb 0.000000 user 0.000000 sys 0.000000 (best of 4226)
No other regression spotted.
More performance improvements are expected in the future as more
revset predicate are converted to use `subset & xxx`
The relaxed way `fullreposet` handles "&" operation may cause some trouble for
people comparing smartset from different filter levels. I'm not sure such people
exist and we can improve that aspect in later patches.
We rename the `spanset` class to `_spanset`. `spanset` is now a function that
builds either a `fullreposet` or a `_spanset` according to the argument passed.
At some point, we may force people to explicitly use the `fullreposet`
constructor, but the current approach makes it easier to ensure we use the new
class whenever possible and focus on the benefits of this class.
Every revset evaluation starts from `subset = spanset(repo)` and a lot of
revset predicates build a `spansetrepo` for their internal needs.
`spanset` is a generic class that can handle any situation. As a result a lot
of operation between spanset result in an `orderedlazyset`, a safe object but
suboptimal in may situation.
So we introduce a `fullreposet` class where some of the operation will be
overwritten to produce more interesting results.
When a set of revisions was empty, we were using an empty tuple. We now return an
empty frozenset to ensure the object could be used in an operation that requires a
set.
The old code relied on the subset contents to get rid of invalid values. We would
like to be able to rely more on the computation in parents() so we filter out
the invalid value.
Both paths are doing similar thing in the end. We refactor the function so that
the `ps` set is commonly used at the end.
This will end excluding `nullrev` from this set in a future patch
The old code relied on the subset contents to get rid of invalid values. We would
like to be able to rely more on the computation in p1() and p2() so we filter out
the invalid value
