Summary:
gitignore could have performance issues stating .gitignore files everywhere.
That happens if watchman returns O(working copy) files. Add a config to
disable it as we're finding solutions.
Reviewed By: DurhamG
Differential Revision: D7482499
fbshipit-source-id: 4c9247b0318bf034c8e9af4b74c21110cc598714
Summary:
Use the new gitignore matcher powered by Rust.
The hgignore matcher has some laziness, but is not tree-aware - with N
"hgignore" files loaded, it needs O(N) time to match. The gitignore matcher
is tree-aware and backed by native code with decent time complexity.
We have been maintaining a translation script that collects all gitignores,
generate hgignore files with very long regexp for them. That script has
issues with sparse recently. This diff allows us to remove those generated
hgignore files from the repo.
Note: fsmonitor state does not contain ignored files. And ignore
invalidation is generally broken in fsmonitor (it only checks top-level
.hgignore). That means, once a file is ignored, it cannot be "unignored" by
just removing the matched pattern from ".gitignore". The file has to be
"touched" or so.
Reviewed By: markbt
Differential Revision: D7319608
fbshipit-source-id: 1763544aedb44676413efb6d14ffd3917ed3b1cd
# skip-blame because this was mechanically rewritten the following script. I
ran it on both *.t and *.py, but none of the *.py changes were proper. All *.t
ones appear to be, and they run without addition failures on both Windows and
Linux.
import argparse
import os
import re
ap = argparse.ArgumentParser()
ap.add_argument('path', nargs='+')
opts = ap.parse_args()
globre = re.compile(r'^(.*) \(glob\)(.*)$')
for p in opts.path:
tmp = p + '.tmp'
with open(p, 'rb') as src, open(tmp, 'wb') as dst:
for line in src:
m = globre.match(line)
if not m or '$LOCALIP' in line or '*' in line:
dst.write(line)
continue
if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
dst.write(line)
continue
dst.write(m.group(1) + m.group(2) + '\n')
os.unlink(p)
os.rename(tmp, p)
Automatic replacement seems better than trying to figure out a check-code rule.
I didn't bother looking to see why the error message and file name is reversed
in the annotate and histedit tests, based on Windows or not.
I originally had this as a list of tuples, conditional on the platform. But
there are a couple of 'No such file or directory' messages emitted by Mercurial
itself, so unconditional is required for stability. There are also several
variants of what I assume is 'connection refused' and 'unknown host' in
test-clone.t and test-clonebundles.t for Docker, FreeBSD jails, etc. Yes, these
are handled by (re) tags, but maybe it would be better to capture those strings
in order to avoid whack-a-mole in future tests. All of this points to using a
dictionary containing one or more strings-to-be-replaced values.
Using a matcher for this command allows processing the named file(s) as
relative to cwd. It also leverages the icasefs normalization logic the same
way the status command does. (However, a false indicator is given for a
nonexistent file in some cases, e.g. passing 'foo.REJ' when that file doesn't
exist, and the rule is '*.rej'. Maybe the regex itself needs to be case
insensitive on these platforms, at least for the debug command.) Finally, the
file printed is relative to cwd and uses platform specific slashes, so a few
(glob)s were needed in seemingly unrelated tests.
c01965ab5195 introduced a deterministic `__repr__` for ignores. However, it
didn't account for when ignore was `util.never`. This broke fsmonitor's ignore
change detection -- with an empty hgignore, it would kick in all the time.
Introduce `nevermatcher` and switch to it. This neatly parallels
`alwaysmatcher`.
The includematcher will always get at least one include pattern and
will never get any non-include patterns, so we can remove most of the
code in it. This patch does mostly straight-forward deletions of
code. We will clean up further later.
At this point the includematcher is an exact copy of the main matcher
class. We will specialize and simplify both classes in the following
patches. This initial unmodified copy is just to make the differences
clearer. We also rename the main matcher to "patternmatcher" for
consistency.
I may eventually merge this new includematcher back into the main
matcher, but I think doing it this way makes the intermediate steps
clearer regardless.
Exact matching is now handled by the exactmatcher class.
We can safely remove _files from the __repr__() implementation,
because even though the field is set, the patternspat field is enough
for the representation to be unambiguous (which was not the case when
the matcher could handle exact matches).
fsmonitor and debugignore currently access matcher fields that I would
consider implementation details, namely patternspat, includepat, and
excludepat. Let' instead implement __repr__() and have the few users
use that instead.
Marked (API) because the fields can now be None.
With an ignore pattern containing a '/' and a Windows style path containing '\',
status was properly ignoring the file, but debugignore was stating that it
wasn't ignored.
This patch adds a capability to hg debugignore: to explain why a given file is
being ignores by mercurial. We display the filename, line and linenumber of the
rule that lead us to ignore the file.
Before this patch debugignore was just displaying the list of ignore patterns.
This patch makes it support a list of filename as argument and tells the user
if those given files are ignored or not.
When reading pattern files, we just call open(path), which is relative to the
current directory. Let's fix this by resolving the paths before attempting to
read the file.
Previously the hgignore test just called hg init in the test directory. A future
patch needs to test hgignore stuff from outside of the repo, so let's move the
entire test repo into a subdirectory.
This adds a new rule syntax that allows the user to include a pattern file, but
only have those patterns match against files underneath the subdirectory of the
pattern file.
This is useful when you have nested projects in a repository and the inner
projects wants to set up ignore rules that won't affect other projects in the
repository. It is also useful in high commit rate repositories for removing the
root .hgignore as a point of contention.
Now that the matcher supports 'include:' rules, let's change the dirstate.ignore
creation to just create a matcher with a bunch of includes. This allows us to
completely delete ignore.py.
I moved some of the syntax documentation over to readpatternfile in match.py so
we don't lose it.
This allows the matcher to understand 'include:path/to/file' style rules. The
files support the standard hgignore syntax and any rules read from the file are
included in the matcher without regard to the files location in the repository
(i.e. if the included file is in somedir/otherdir, all of it's rules will still
apply to the entire repository).
Previously these would be considered to be relative to the current working
directory. That behavior is both undocumented and doesn't really make sense.
There are two reasonable options for how to resolve relative paths:
- relative to the repo root
- relative to the config file
Resolving these files relative to the repo root matches existing behavior with
hooks. An earlier discussion about this is available at
http://mercurial.markmail.org/thread/tvu7yhzsiywgkjzl.
Thanks to Isaac Jurado <diptongo@gmail.com> for the initial patchset that
spurred the discussion.
Previously, a glob pattern of the form 'foo/**/bar' would match 'foo/a/bar' but
not 'foo/bar'. That was because the '**' in 'foo/**/bar' would be translated to
'.*', making the final regex pattern 'foo/.*/bar'. That pattern doesn't match
the string 'foo/bar'.
This is a bug because the '**/' glob matches the empty string in standard Unix
shells like bash and zsh.
Fix that by making the ending '/' optional if an empty string can be matched.
If a directory matched a regex in hgignore but the files inside the directory
did not match the regex, they would appear as deleted in hg status. This
change fixes them to appear normally in hg status.
Removing the ignore(nf) conditional here is ok because it just means we might
stat more files than we had before. My testing on a large repo shows this
causes no performance regression since the only additional files being stat'd
are the ones that are missing (i.e. status=!), which are generally rare.
Many tests didn't change back from subdirectories at the end of the tests ...
and they don't have to. The missing 'cd ..' could always be added when another
test case is added to the test file.
This change do that tests (99.5%) consistently end up in $TESTDIR where they
started, thus making it simpler to extend them or move them around.
Globbing is usually used for filenames, so on windows it is reasonable and very
convenient that glob patterns accepts '\' or '/' when the pattern specifies
'/'.
For GUI clients its sometimes important to know which files will be ignored and
which files will be important. This allows the GUI client to skipping redoing a
'hg status' when the files are ignored but have changed. (For instance, a
typical case is that the "build" directory inside some project is ignored but
files in it frequently change.)
This adds a " (glob)" marker that works like a simpler version of
(re): "*" is converted to ".*", and "?" is converted to ".".
Both special characters can be escaped using "\", and the backslash
itself can be escaped as well.
Other glob-style syntax, like "**", "[chars]", or "[!chars]", isn't
supported.
Consider this test:
$ hg glog --template '{rev}:{node|short} "{desc}"\n'
@ 2:20c4f79fd7ac "3"
|
| o 1:38f24201dcab "2"
|/
o 0:2a18120dc1c9 "1"
Because each line beginning with "|" can be compiled as a regular
expression (equivalent to ".*|"), they will match any output.
Similarly:
$ echo foo
The blank output line can be compiled as a regular expression and will
also match any output.
With this patch, none of the above output lines will be matched as
regular expressions. A line must end in " (re)" in order to be matched
as one.
Lines are still matched literally first, so the following will pass:
$ echo 'foo (re)'
foo (re)
