When commiting to a repo with lots of history (>400000 changesets)
checking the results of revset.py:descendants against the subset takes
some time. Since the subset equals the entire changelog, the check
isn't necessary. Avoiding it in that case saves 0.1 seconds off of
a 1.78 second commit. A 6% gain.
We use the length of the subset to determine if it is the entire repo.
There is precedence for this in revset.py:stringset.
bundle() revset expression returns all changes that are present
in the bundle file (no matter whether they are in the repo or not).
Bundle file should be specified via -R option.
The old name was not very good for two reasons:
- caller does not care about "cache",
- set of revision returned may not be obsolete at all.
The new name was suggested by Kevin Bullock.
For changelog level filtering to take effect it need to be used for any
iteration. Some remaining use of `xrange` in revset code is replace by proper
use of `changelog.revs` or direct iteration over changelog.
We don't need to compute the set of all branchpoints. We can just check the
number of children that element of subset have. The extra work did not seems to
had particular performance impact but the code is simpler this way.
The branchpoint() function returns changesets with more than one child.
Eventually I would like to be able to see only branch points and merge
points in a graphical log to see the topology of the repository.
This changeset introduces caches on the `obsstore` that keeps track of sets of
revisions meaningful for obsolescence related logics. For now they are:
- obsolete: changesets used as precursors (and not public),
- extinct: obsolete changesets with osbolete descendants only,
- unstable: non obsolete changesets with obsolete ancestors.
The cache is accessed using the `getobscache(repo, '<set-name>')` function which
builds the cache on demand. The `clearobscaches(repo)` function takes care of
clearing the caches if any.
Caches are cleared when one of these events happens:
- a new marker is added,
- a new changeset is added,
- some changesets are made public,
- some public changesets are demoted to draft or secret.
Declaration of more sets is made easy because we will have to handle at least
two other "troubles" (latecomer and conflicting).
Caches are now used by revset and changectx. It is usually not much more
expensive to compute the whole set than to check the property of a few elements.
The performance boost is welcome in case we apply obsolescence logic on a lot of
revisions. This makes the feature usable!
"extinct" and "unstable" predicates use "obsolete" implementation
internally, but own predicate name should be used in error messages of
them instead of "obsolete".
This predicate is used to find csets that were created because of a graft,
transplant or rebase --keep. An optional revset can be supplied, in which case
the result will be limited to those copies which specified one of the revs as
the source for the command.
hg log -r destination() # csets copied from anywhere
hg log -r destination(branch(default)) # all csets copied from default
hg log -r origin(x) or destination(origin(x)) # all instances of x
This predicate will follow a cset through different types of copies. Given a
repo with a cset 'S' that is grafted to create G(S), which itself is
transplanted to become T(G(S)):
o-S
/
o-o-G(S)
\
o-T(G(S))
hg log -r destination( S ) # { G(S), T(G(S)) }
hg log -r destination( G(S) ) # { T(G(S)) }
The implementation differences between the three different copy commands (see
the origin() predicate) are not intentionally exposed, however if the
transplant was a graft instead:
hg log -r destination( G(S) ) # {}
because the 'extra' field in G(G(S)) is S, not G(S). The implementation cannot
correct this by following sources before G(S) and then select the csets that
reference those sources because the cset provided to the predicate would also
end up selected. If there were more than two copies, sources of the argument
would also get selected.
Note that the convert extension does not currently update the 'extra' map in its
destination csets, and therefore copies made prior to the convert will be
missing from the resulting set.
Instead of the loop over 'subset', the following almost works, but does not
select a transplant of a transplant. That is, 'destination(S)' will only
select T(S).
dests = set([r for r in subset if _getrevsource(repo, r) in args])
This predicate is used to find the original source of csets created by a graft,
transplant or rebase --keep. If a copied cset is itself copied, only the
source of the original copy is selected.
hg log -r origin() # all src csets, anywhere
hg log -r origin(branch(default)) # all srcs of copies on default
By following through different types of copy commands and only selecting the
original cset, the implementation differences between the copy commands are
hidden. (A graft of a graft preserves the original source in its 'extra' map,
while transplant and rebase use the immediate source specified for the
command).
Given a repo with a cset S that is grafted to create G(S), which itself is
grafted to become G(G(S))
o-S
/
o-o-G(S)
\
o-G(G(S))
hg log -r origin( G(S) ) # { S }
hg log -r origin( G(G(S)) ) # { S }, NOT { G(S) }
Even if the last graft were a transplant
hg log -r origin( T(G(S)) ) # { S }
A rebase without --keep essentially strips the source, so providing the cset
that results to this predicate will yield an empty set.
Note that the convert extension does not currently update the 'extra' map in
its destination csets, and therefore copies made prior to the convert will be
unable to find their source.
`extinct` changesets are obsolete changesets with obsolete descendants only. They
are of no interest anymore and can be:
- exclude from exchange
- hidden to the user in most situation
- safely garbage collected
This changeset just allows mercurial to detect them.
The implementation is a bit naive, as for unstable changesets. We better use a
simple revset query and a cache, but simple version comes first.
An unstable changeset is a changeset *not* obsolete but with some obsolete
ancestors.
The current logic to decide if a changeset is unstable is naive and very
inefficient. A better solution is to compute the set of unstable changeset with
a simple revset and to cache the result. But this require cache invalidation
logic. Simpler version goes first.
The new "diff" field lets you use the matching revset keyword to find revisions
that apply the same change as the selected revisions.
The match must be exact (i.e. same additions, same deletions, same modified
lines and same change context, same file renames and copies).
Two revisions matching their diff must also match their files. Thus, to match
the diff much faster we will always check that the 'files' match first, and only
then check that the 'diff' matches as well.
graft, transplant and rebase all embed a different type of source marker in
extra, and each with a different name. The current implementation of each is
such that there will never be more than one of these markers on a node.
Note that the rebase marker can only be resolved if the source is
still present, which excludes the typical rebase usage (without
--keep) from consideration (unless the resulting bundle in
strip-backup is overlayed). There probably isn't any reason to use
rebase --keep as a substitute for transplant or graft at this point,
but maybe there was at one point and there are even a few rebases in
the hg repo, so it may be of historical interest.
This selects changesets added because of repo conversions. For example
hg log -r "converted()" # all csets created by a convertion
hg log -r "converted(rev)" # the cset converted from rev in the src repo
The converted(rev) form is analogous to remote(id), where the remote repo is
the source of the conversion. This can be useful for cross referencing an old
repository into the current one.
The source revision may be the short changeset hash or the full hash from the
source repository. The local identifier isn't useful. An interesting
ramification of this is if a short revision is specified, it may cause more
than one changeset to be selected. (e.g. converted(6) matches changesets with
a convert_revision field of 6e..e and 67..0)
The convert.hg.saverev option must have been specified when converting the hg
source repository for this to work. The other sources automatically embed the
converted marker.
Caching has no performance effect on the revset aliases which triggered
the recent recursive evaluation bug. I wrote it not to feel bad about
expanding several times the same complicated expression.
If the string provided to the 'tag' predicate starts with 're:', the rest
of the string will be treated as a regular expression and matched against
all tags in the repository.
There is a slight backwards-compatibility problem for people who actually
have tags that start with 're:'. As a workaround, these tags can be matched
using a 'literal:' prefix.
If no tags match the pattern, an error is raised. This matches the behaviour
of the previous exact-match code.
There have been quite a few places where we pop elements off the
front of a list. This can turn O(n) algorithms into something more
like O(n**2). Python has provided a deque type that can do this
efficiently since at least 2.4.
As an example of the difference a deque can make, it improves
perfancestors performance on a Linux repo from 0.50 seconds to 0.36.
The alias expansion code it changed from:
1- Get replacement tree
2- Substitute arguments in the replacement tree
3- Expand the replacement tree again
into:
1- Get the replacement tree
2- Expand the replacement tree
3- Expand the arguments
4- Substitute the expanded arguments in the replacement tree
and fixes cases like:
[revsetalias]
level1($1, $2) = $1 or $2
level2($1, $2) = level1($2, $1)
$ hg log -r "level2(level1(1, 2), 3)"
where the original version incorrectly aborted on infinite expansion
error, because it was confusing the expanded aliases with their
arguments.
The current revset alias expansion code works like:
1- Get the replacement tree
2- Substitute the variables in the replacement tree
3- Expand the replacement tree
It makes it easy to substitute alias arguments because the placeholders
are always replaced before the updated replacement tree is expanded
again. Unfortunately, to fix other alias expansion issues, we need to
reorder the sequence and delay the argument substitution. To solve this,
a new "virtual" construct called _aliasarg() is introduced and injected
when parsing the aliases definitions. Only _aliasarg() will be
substituted in the argument expansion phase instead of all regular
matching string. We also check user inputs do not contain unexpected
_aliasarg() instances to avoid argument injections.
The original motivation was changectx.phase() had special logic to
correctly lookup in repo._phaserev, including invalidating it when
necessary. And at other places, repo._phaserev was accessed directly.
This led to the discovery that phases state including _phaseroots,
_phaserev and _dirtyphase was manipulated in localrepository.py,
phases.py, repair.py, etc. phasecache helps encapsulating that.
This patch replaces all phase state in localrepo with phasecache and
adjust related code except for advance/retractboundary() in phases.
These still access to phasecache internals directly. This will be
addressed in a followup.
current description of 'matching()' revset predicate can't be
formatted well on "hg help revset" output.
each descriptions for revset predicates (or something like them) are
split-ed into lines, and spaces on left side of them are stripped
before minirst processing. so, bullet list can't be nested.
this patch just flattens description of 'matching()' predicate to be
formatted well.
The fast path was triggered if the argument was not like "type:value", with
type a known pattern type. This is wrong for several reasons:
- path:value is valid for the fast path
- '*' is interpreted as a glob by default and is not valid for fast path
Fast path detection is now done after the pattern is parsed, and the normalized
path is extracted for direct comparison. All this seems a bit complicated, it
is tempting to drop the fast path completely. Also, the hasfile() revset does
something similar (only check .files()), without a fast path. If the fast path
is really that efficient maybe it should be used there too.
Note that:
$ log 'modifies("set:modified()")'
is different from:
$ log 'modifies("*")'
because of the usual merge ctx.files()/status(ctx.p1(), ctx) differences.
Reported by Steffen Eichenberg <steffen.eichenberg@msg-gillardon.de>
match
This patch sorts the fields that are passed to the matching function so that it
always starts by matching those fields that take less time to match.
Not all fields take the same amount of time to match. I've done several
measurements running the following command:
hg --time log -r "matching(1, field)"
on the mercurial repository, and where 'field' was each one of the fields
accepted by match. In order to avoid the print overhead (which could be
different for different fields, given the different number of matches) I used a
modified version of the matching() function which always returns no matches.
These tests showed that different fields take wildly different amounts of time
to match. Particulary the substate field takes up to 25 seconds to match on my
machine, compared to the 0.3 seconds that takes to match the phase field or the
2 seconds (approx) that takes to match most fields. With this patch, matching
both the phase and the substate of a revision takes the same amount of time as
matching the phase.
The field match order introduced by this patch is as follows:
phase, parents, user, date, branch, summary, files, description, substate
An extra nice thing about this patch is that it makes the match time stable.
Rather than getting all the fields that are being matches from every revision
and then comparing them to those of the target revision, compare each field one
by one and stop the match as soon as there is a match failure.
This can greatly reduce the match time when matching multiple fields.
The impact on match time when matching a single field seems negligible
(according to my measurements).
Apparently the "import x as xy" doesn't manage to update xy in the
current scope's dictionary after load, which causes nodemod.nullrev to do a huge amount of demandload magic in the inner loop.
This solves a similar problem than the previous --follow/--rev patch. This time
we need changelog.ancestors()/descendants() filtering on first parent.
Duplicating the code looked better than introducing keyword arguments. Besides,
the ancestors() version was already implemented in follow() revset.
This keyword can be used to find revisions that "match" one or more fields of a
given set of revisions.
A revision matches another if all the selected fields (description, author,
branch, date, files, phase, parents, substate, user, summary and/or metadata)
match the corresponding values of those fields on the source revision.
By default this keyword looks for revisions that whose metadata match
(description, author and date) making it ideal to look for duplicate revisions.
matching takes 2 arguments (the second being optional):
1.- rev: a revset represeting a _single_ revision (e.g. tip, ., p1(.), etc)
2.- [field(s) to match]: an optional string containing the field or fields
(separated by spaces) to match.
Valid fields are most regular context fields and some special fields:
* regular fields:
- description, author, branch, date, files, phase, parents,
substate, user.
Note that author and user are synonyms.
* special fields: summary, metadata.
- summary: matches the first line of the description.
- metatadata: It is equivalent to matching 'description user date'
(i.e. it matches the main metadata fields).
Examples:
1.- Look for revisions with the same metadata (author, description and date)
as the 11th revision:
hg log -r "matching(11)"
2.- Look for revisions with the same description as the 11th revision:
hg log -r "matching(11, description)"
3.- Look for revisions with the same 'summary' (i.e. same first line on their
description) as the 11th revision:
hg log -r "matching(11, summary)"
4.- Look for revisions with the same author as the current revision:
hg log -r "matching(., author)"
You could use 'user' rather than 'author' to get the same result.
5.- Look for revisions with the same description _AND_ author as the tip of the
repository:
hg log -r "matching(tip, 'author description')"
6.- Look for revisions touching the same files as the parent of the tip of the
repository
hg log -r "matching(p1(tip), files)"
7.- Look for revisions whose subrepos are on the same state as the tip of the
repository or its parent
hg log -r "matching(p1(tip):tip, substate)"
8.- Look for revisions whose author and subrepo states both match those of any
of the revisions on the stable branch:
hg log -r "matching(branch(stable), 'author substate')"
When _followfirst() revset was introduced it seemed to be the sole user of such
an argument, so filectx.ancestors() was duplicated and modified instead. It now
appears this argument could be used when computing the set of files to be
considered when --patch or --stat are passed along with --follow FILE.
This subtlety is not documented yet but:
- pats/--include/--exclude filesets are evaluated against the working directory
- --rev filesets are reevaluated against every revisions
log --graph --follow-first FILE cannot be compared with the regular version
because it never worked: --follow-first is not taken in account in
walkchangerevs() fast path and is explicitely bypassed in FILE case in
walkchangerevs() nested iterate() function.
The filtering logic of match objects cannot be reproduced with the existing
revsets as it operates at changeset files level. A changeset touching "a" and
"b" is matched by "-I a -X b" but not by "file(a) and not file(b)".
To solve this, a new internal "_matchfiles(...)" revset is introduced. It works
like "file(x)" but accepts more than one argument and its arguments are
prefixed with "p:", "i:" and "x:" to be used as patterns, include patterns or
exclude patterns respectively.
The _matchfiles revset is kept private for now:
- There are probably smarter ways to pass the arguments in a user-friendly way
- A "rev:" argument is likely appear at some point to emulate log command
behaviour with regard to filesets: they are evaluated for the parent revision
and applied everywhere instead of being reevaluated for each revision.
current documentation for 'remote()' predicate is wrong about
specification of parameters.
there are 3 patterns:
# of
param: id: remote:
- 0 current branch "defult" remote
- 1 specified "defult" remote
- 2 specified specified
The revset aliases expansion worked like:
expr = "some revset"
for alias in aliases:
expr = alias.process(expr)
where "process" was replacing the alias with its *unexpanded* substitution,
recursively. So it only worked when aliases were applied in proper dependency
order.
This patch rewrites the expansion process so all aliases are expanded
recursively at every tree level, after parent alias rewriting and variable
expansion.
Previously we always included '.', which may not touch a file.
Instead, find the file revision present in '.' and add its linkrev.
This matches the results of 'hg log --follow file'.
The large or-expressions we used to build required a substantial
amount of subset filtering in orset() which was inefficient. Instead we build a
single string which we process in one go with a special internal predicate.
Simplifies client logic in multiple places since it encapsulates the
computation of the common and, more importantly, the missing node lists.
This also allows an upcomping patch to communicate precomputed versions of
these lists to clients.
some problematic encoding (e.g.: cp932) uses ASCII alphabet characters
in byte sequence of multi byte characters.
"str.lower()" on such byte sequence may treat distinct characters as
same one, and cause unexpected log matching.
this patch uses "encoding.lower()" instead of "str.lower()" to
normalize strings for compare.
This allows folding external revsets or lists of revsets into a revset
expression. Revsets are pre-parsed for validity so that syntax errors
don't escape.
This patch adds two new revset descriptions:
- 'goods': the list of topologicaly-good csets:
- if good csets are topologically before bad csets, yields '::good'
- else, yields 'good::'
- and conversely for 'bads'
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
The 'ignored' changesets are outside the bisection range, but are
changesets that may have an impact on the outcome of the bisection.
For example, in case there's a merge between the good and bad csets,
but the branch-point is out of the bisection range, and the issue
originates from this branch, the branch will not be visited by bisect
and bisect will find that the culprit cset is the merge.
So, the 'ignored' set is equivalent to:
( ( ::bisect(bad) - ::bisect(good) )
| ( ::bisect(good) - ::bisect(bad) ) )
- bisect(range)
- all ancestors of bad csets that are not ancestors of good csets, or
- all ancestors of good csets that are not ancestors of bad csets
- but that are not in the bisection range.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
Use repo.set() wherever possible, instead of locally trying to
reproduce complex graph computations.
'pruned' now means 'all csets that will no longer be visited by the
bisection'. The change is done is this very patch instead of its own
dedicated one becasue the code changes all over the place, and the
previous 'pruned' code was totally rewritten by the cleanup, so it
was easier to just change the behavior at the same time.
The previous series went in too fast for this cleanup pass to be
included, so here it is. ;-)
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
The 'untested' set is made of changesets that are in the bisection range
but for which the status is still unknown, and that can later be used to
further decide on the bisection outcome.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
The 'pruned' set is made of changesets that did participate to
the bisection. They are made of
- all good changesets
- all bad changsets
- all skipped changesets, provided they are in the bisection range
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
The 'range' set is made of all changesets that make the bisection
range, that is
- csets that are ancestors of bad csets and descendants of good csets
or
- csets that are ancestors of good csets and descendants of bad csets
That is, roughly equivalent of:
bisect(good)::bisect(bad) | bisect(bad)::bisect(good)
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
Computing the ranges of csets in the bisection belongs to the hbisect
code. This allows for reusing the status computation from many places,
not only the revset code, but also to later display the bisection status
of a cset...
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
Rename the 'bisected' keyword to simply 'bisect'.
Still accept the old name, but no longer advertise it.
As discussed with Matt on IRC.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
Given an operator ^ that's either postfix or infix and an operator :
that's either prefix or infix, the parser can't figure out the right
thing to do. So we rewrite the expression to be sensible in the optimizer.
The existing code seemed to have incorrect assumptions about how parameter
lists are represented by the parser.
Now the match and replace functions have been merged and simplified by using
getlist().
Like keyword(), but does not search in filenames and users.
No grepdesc() or descgrep() added, because it might be bad to introduce
grepfoo() versions of too many string searches.
- Handle 'subset' argument
- Stop returning the null rev from p1 and parents, as in the non-dirstate case
- Order parents as in the non-dirstate case (ascending revs)
This patch makes the 'set' argument to revset function parents() optional.
Like p1() and p2(), if no argument is given, returns the parent(s) of the
working directory.
Morally equivalent to 'p1()+p2()', as expected.
This patch makes the 'set' argument to revset functions p1() and p2()
optional. If no argument is given, p1() and p2() return the first or second
parent of the working directory.
If the working directory is not an in-progress merge (no 2nd parent), p2()
returns the empty set. For a checkout of the null changeset, both p1() and
p2() return the empty set.
For the boolean operators, the subset optimization works by calculating
the cheaper argument first, and passing the subset to the second
argument to restrict the revision domain. This works well for filtering
predicates.
But parents() don't work like a filter: it may return revisions outside the
specified set. So, combining it with boolean operators may easily yield
incorrect results. For instance, for the following revision graph:
0 -- 1
the expression '0 and parents(1)' should evaluate as follows:
0 and parents(1) ->
0 and 0 ->
0
But since [0] is passed to parents() as a subset, we get instead:
0 and parents(1 and 0) ->
0 and parents([]) ->
0 and [] ->
[]
This also affects children(), p1() and p2(), for the same reasons.
Predicates that call these (like heads()) are also affected.
We work around this issue by ignoring the subset when propagating
the call inside those predicates.
hg log -r 'outgoing(..)' ignored #branch in some cases.
This patch fixes it.
The cases where it misbehaved are now covered by the added
test-revset-outgoing.t
This adds support for r'...' and r"..." as string literals. Strings
with the "r" prefix will not have their escape characters interpreted.
This is especially useful for grep(), where, with regular string
literals, \number is interpreted as an octal escape code, and \b is
interpreted as the backspace character (\x08).
A query like
head() and (descendants("bad") and not descendants("fix"))
(testing if repo heads are affected by a bug) will abort with a
RepoLookupError if either badrev or fixrev aren't found inside
the repository, which is not very informative.
The new predicate returns an empty set for lookup errors, so
head() and (descendants(present("bad")) and not descendants(present("fix")))
will behave as wanted even if those revisions are not found.
Rather than dynamically optimize in methods, we pre-optimize the parse tree
directly. This also lets us do some substitution on some of the
symbols like - and ::.
discovery.findoutgoing used to be a useful hook for extensions like
hgsubversion. This patch reintroduces this version of findcommonincoming
which is meant to be used when computing outgoing changesets.
When limit, last, min and max were evaluated they worked on a reduced set in the
wrong way. Now they work on an unrestricted set (the whole repo) and get
limited later on.
This is a long desired cleanup and paves the way for new discovery.
To specify subsets for bundling changes, all code should use the heads
of the desired subset ("heads") and the heads of the common subset
("common") to be excluded from the bundled set. These can be used
revlog.findmissing instead of revlog.nodesbetween.
This fixes an actual bug exposed by the change in test-bundle-r.t
where we try to bundle a changeset while specifying that said changeset
is to be assumed already present in the target. This used to still
bundle the changeset. It no longer does. This is similar to the bugs
fixed by the recent switch to heads/common for incoming/pull.
^ (Nth parent) and ~ (Nth first ancestor) are infix operators that match
certain ancestors of the set:
set^0
the set
set^1 (also available as set^)
the first parent of every changeset in set
set^2
the second parent of every changeset in set
set~0
the set
set~1
the first ancestor (i.e. the first parent) of every changeset in set
set~2
the second ancestor (i.e. first parent of first parent) of every changeset
in set
set~N
the Nth ancestor (following first parents only) of every changeset in set;
set~N is equivalent to set^1^1..., with ^1 repeated N times.
if range(len(repo)) is passed to stringset and x is a valid rev
(checked before) then x is guaranteed to be in subset, we can check
for that by comparing the lengths of the sets
This is valuable because now revsets like 'bookmarks() or tip' will
always show tip after bookmarks unless tip was itself a bookmark. This
is a somewhat contrived example, but this behavior is useful for
"where am I" type aliases that use log and revsets.
# Date 1289564504 -3600
# Node ID b75264c15cc888cf38c3c7b8f619801e3c2589c7
# Parent 89b2e5d940f669e590096c6be70eee61c9172fff
revsets: overload the branch() revset to also take a branch name.
This should only change semantics in the specific case of a tag/branch
conflict where the tag wasn't done on the branch with the same
name. Previously, branch(whatever) would resolve to the branch of the
tag in that case, whereas now it will resolve to the branch of the
name. The previous behaviour, while documented, seemed very
counter-intuitive to me.
An alternate approach would be to introduce a new revset such as
branchname() or namedbranch(). While this would retain backwards
compatibility, the distinction between it and branch() would not be
readily apparent to users. The most intuitive behaviour would be to
have branch(x) require 'x' to be a branch name, and something like
branchof(x) or samebranch(x) do what branch(x) currently
does. Unfortunately, our backwards compatibility guarantees prevent us
from doing that.
Please note that while 'hg tag' guards against shadowing a branch, 'hg
branch' does not. Besides, even if it did, that wouldn't solve the
issue of conversions with such tags and branches...
