Windows was previously getting this test failure:
--- e:/Projects/hg/tests/test-hgignore.t
+++ e:/Projects/hg/tests/test-hgignore.t.err
@@ -230,6 +230,7 @@
$ hg status
? dir1/file2
+ ? dir1/subdir/subfile3
? dir1/subdir/subfile4
? dir2/file1
@@ -241,4 +242,4 @@
$ echo "glob:file*2" > dir1/.hgignoretwo
$ hg status | grep file2
- [1]
+ ? dir1/file2
The problem was 'source' would be in the form "F:\test-hgignore.t\.hgignore", so
when pathutil.dirname() split on '/', 'sourceroot' was empty. Therefore, 'path'
ended up being relative instead of absolute.
This adds a new rule syntax that allows the user to include a pattern file, but
only have those patterns match against files underneath the subdirectory of the
pattern file.
This is useful when you have nested projects in a repository and the inner
projects wants to set up ignore rules that won't affect other projects in the
repository. It is also useful in high commit rate repositories for removing the
root .hgignore as a point of contention.
There was a bug in my recent change to visitdir (cb39606d374e) due to
the stored generator being iterated over twice. Making the generator into a
list at the start fixes this.
A future patch will be allowing nested matchers. To support that, let's refactor
_buildmatch to build a list of matchers then return True if any match.
We were already doing that for filesets + regex patterns, but this way will be
more generic.
A future patch will make _buildmatch able to expand relative include patterns.
Doing so will require knowing the root of the repo, so let's go ahead and pass
it in.
tl;dr: This is another step towards a (previously unstated) goal of
eliminating match.files() in conditions.
There are four types of matchers:
* always: Matches everything, checked with always(), files() is empty
* exact: Matches exact set of files, checked with isexact(), files()
contains the files to match
* patterns: Matches more complex patterns, checked with anypats(),
files() contains roots of the matched patterns
* prefix: Matches simple 'path:' patterns as prefixes ('foo' matches
both 'foo' and 'foo/bar'), no single method to check, files()
contains the prefixes to match
For completeness, it would be nice to have a method for checking for
the "prefix" type of matcher as well, so let's add that, making it
return True simply when none of the others do.
The larger goal here is to eliminate uses of match.files() in
conditions (i.e. bool(match.files())). The reason for this is that
there are scenarios when you would like to create a "prefix" matcher
that happens to match no files. One example is for 'hg files -I foo
bar'. The narrowmatcher also restricts the set of files given and it
would not surprise me if have bugs caused by that already. Note that
'if m.files() and not m.anypats()' and similar is sometimes used to
catch the "exact" and "prefix" cases above.
match.visitdir() used to only look at the match's primary pattern roots to
decide if a treemanifest traverser should descend into a particular directory.
This change logically makes visitdir also consider the match's include and
exclude pattern roots (if applicable) to make this decision.
This is especially important for situations like using narrowhg with multiple
treemanifest revlogs.
Now that the matcher supports 'include:' rules, let's change the dirstate.ignore
creation to just create a matcher with a bunch of includes. This allows us to
completely delete ignore.py.
I moved some of the syntax documentation over to readpatternfile in match.py so
we don't lose it.
This allows the matcher to understand 'include:path/to/file' style rules. The
files support the standard hgignore syntax and any rules read from the file are
included in the matcher without regard to the files location in the repository
(i.e. if the included file is in somedir/otherdir, all of it's rules will still
apply to the entire repository).
Occasionally the matcher will want to print warning messages instead of throwing
exceptions (like if it encounters a bad syntax parameter when parsing files).
Let's add an optional warn argument that can provide this. The next patch will
actually use this argument.
Future patches will be adding the ability to recursively include pattern files
in a match rule expression. Part of that behavior will require tracking which
file each pattern came from so we can report errors correctly.
Let's add a 'source' arg to the kindpats list to track this. Initially it will
only be populated by listfile rules.
Match's visitdir() was prematurely optimized to return 'all' in some cases, so
that the caller would not have to call it for directories within the current
directory. This change makes the visitdir system less flexible for future
changes, such as making visitdir consider the match's include and exclude
patterns.
As a demonstration of this optimization not actually improving performance,
I ran 'hg files -r . media' on the Mozilla repository, stored as treemanifest
revlogs.
With best of ten tries, the command took 1.07s both with and without the
optimization, even though the optimization reduced the calls from visitdir()
from 987 to 51.
This will work for any command that creates its matcher via scmutil.match(), but
only the files command is tested here (both workingctx and basectx based tests).
The previous behavior was to completely ignore the files in the subrepo, even
though -S was given.
My first attempt was to teach context.walk() to optionally recurse, but once
that was in place and the complete file list was built up, the predicate test
would fail with 'path in nested repo' when a file in a subrepo was accessed
through the parent context.
There are two slightly surprising behaviors with this functionality. First, any
path provided inside the fileset isn't narrowed when it is passed to the
subrepo. I dont see any clean way to do that in the matcher. Fortunately, the
'subrepo()' fileset is the only one to take a path.
The second surprise is that status predicates are resolved against the subrepo,
not the parent like 'hg status -S' is. I don't see any way to fix that either,
given the path auditor error mentioned above.
match.ispartial() will return the opposite of match.always() in core, but this
function will be extensible by extensions to produce another result even
if match.always() will be untouched.
This will be useful for narrowhg, where ispartial() will return False even if
the match won't always match. This would happen in the case where the only
time the match function is False is when the path is outside of the narrow
spec.
Several methods print files relative to the repo root, unless files are named on
the command line, in which case they are printed relative to cwd. Since the
check relies on the 'pats' parameter, which needs to be replaced by a matcher
when adding subrepo support, this logic gets folded into the matcher to tidy up
the callers.
Prior to 7d5fcea60c78, this style decision was based off of whether or not the
'pats' list was empty. That change altered the check to test match.anypats()
instead, in order to make paths printed consistent when -I/-X is specified.
That however, changed the style when a file is given to the command. So now we
test the pattern list to get the old behavior for files, as well as test -I/-X
to get the consistency for patterns.
The 'always' class calls its parent constructor with an empty list of
patterns, which will result in a matcher that always matches. The
parent constructor will set self._always to True in such cases, so
there is no need to set it again.
In match.__init__(), we create the matchfn predicate by and-ing
together the individual predicates for includes, excludes (negated)
and patterns. Instead of the current set of nested if/else blocks, we
can simplify by adding the predicates to a list and defining the
overall predicate in a generic way based on the components. We can
still optimize it for the 0-length and 1-length cases. This way, there
is no combinatorial explosion to deal with if new component predicates
are added, and there is less risk of getting the overall predicate
wrong.
For a pathological .hgignore with over 2500 glob lines and over 200000 calls to
re.escape, and with re2 available, this speeds up parsing the .hgignore from
0.75 seconds to 0.20 seconds. This causes e.g. 'hg status' with hgwatchman
enabled to go from 1.02 seconds to 0.47 seconds.
Previously, a glob pattern of the form 'foo/**/bar' would match 'foo/a/bar' but
not 'foo/bar'. That was because the '**' in 'foo/**/bar' would be translated to
'.*', making the final regex pattern 'foo/.*/bar'. That pattern doesn't match
the string 'foo/bar'.
This is a bug because the '**/' glob matches the empty string in standard Unix
shells like bash and zsh.
Fix that by making the ending '/' optional if an empty string can be matched.
No real changes.
pattern: 'kind:pat' as specified on the command line
patterns, pats: list of patterns
kind: 'path', 'glob' or 're' or ...
pat: string in the corresponding 'kind' format
kindpats: list of (kind, pat) tuples
match.dir is currently called in two different places:
(1) noting when a directory specified explicitly is visited.
(2) noting when a directory is visited during a recursive walk.
purge cares about both, but commit only cares about the first.
Upcoming patches will split the two cases into two different callbacks. Why
bother? Consider a hypothetical extension that can provide more efficient walk
results, via e.g. watching the filesystem. That extension will need to
fall back to a full recursive walk if a callback is set for (2), but not if a
callback is only set for (1).
The fall-back root for walking is the repo root, not no root.
The "roots" do however also end up in m.files() which is used in various ways,
for instance to indicate whether matches are exact. The change could thus have
other impacts.
This improves the performance of log --patch and --stat by about
20% for moderately large manifests (e.g. mozilla-central) for the
common case of no -I/-X patterns.
There are two sets of Python re2 bindings available on the internet;
this code works with both.
Using re2 can greatly improve "hg status" performance when a .hgignore
file becomes even modestly complex.
Example: "hg status" on a clean tree with 134K files, where "hg
debugignore" reports a regexp 4256 bytes in size.
no .hgignore: 1.76 sec
Python re: 2.79
re2: 1.82
The overhead of regexp matching drops from 1.03 seconds with stock
re to 0.06 with re2.
(For comparison, a git repo with the same contents and .gitignore
file runs "git status -s" in 1.71 seconds, i.e. only slightly faster
than hg with re2.)
'exact' match objects are sometimes created with a non-list 'pattern'
argument:
- using 'set' in queue.refresh():hgext/mq.py
match = scmutil.matchfiles(repo, set(c[0] + c[1] + c[2] + inclsubs))
- using 'dict' in revert():mercurial/cmdutil.py (names = {})
m = scmutil.matchfiles(repo, names)
'exact' match objects return specified 'pattern' to callers of
'match.files()' as it is, so it is a non-list object.
but almost all implementations expect 'match.files()' to return a list
object, so this may causes problems: e.g. exception for "+" with
another list object.
this patch ensures that '_files' of 'exact' match objects is a list
object.
for non 'exact' match objects, parsing specified 'pattern' already
ensures that it it a list one.
Introduce match.always() to check if a match object always says yes, i.e.
None was passed in. If so, mfmatches should not bother iterating every file in
the repository.
Matt suggested this on IRC, I do not think the choice is obvious, but this one
makes things simpler because while filesets are turned into a list of files
into the match objects, it would more be difficult to tell invalid files passed
in pats from those expanded from filesets.
We want util.readfile() to operate in binary mode, so EOLs have to be handled
correctly depending on the platform. It seems both easier and more convenient
to treat LF and CRLF the same way on all platforms.
These leaks may occur in environments that don't employ a reference
counting GC, i.e. PyPy.
This implies:
- changing opener(...).read() calls to opener.read(...)
- changing opener(...).write() calls to opener.write(...)
- changing open(...).read(...) to util.readfile(...)
- changing open(...).write(...) to util.writefile(...)
For GUI clients its sometimes important to know which files will be ignored and
which files will be important. This allows the GUI client to skipping redoing a
'hg status' when the files are ignored but have changed. (For instance, a
typical case is that the "build" directory inside some project is ignored but
files in it frequently change.)
