Summary: Also change the internal API so it no longer accepts the "heads" argument.
Reviewed By: ryanmce
Differential Revision: D6745865
fbshipit-source-id: 368742be49b192f7630421003552d0a10eb0b76d
# skip-blame because this was mechanically rewritten the following script. I
ran it on both *.t and *.py, but none of the *.py changes were proper. All *.t
ones appear to be, and they run without addition failures on both Windows and
Linux.
import argparse
import os
import re
ap = argparse.ArgumentParser()
ap.add_argument('path', nargs='+')
opts = ap.parse_args()
globre = re.compile(r'^(.*) \(glob\)(.*)$')
for p in opts.path:
tmp = p + '.tmp'
with open(p, 'rb') as src, open(tmp, 'wb') as dst:
for line in src:
m = globre.match(line)
if not m or '$LOCALIP' in line or '*' in line:
dst.write(line)
continue
if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
dst.write(line)
continue
dst.write(m.group(1) + m.group(2) + '\n')
os.unlink(p)
os.rename(tmp, p)
Automatic replacement seems better than trying to figure out a check-code rule.
I didn't bother looking to see why the error message and file name is reversed
in the annotate and histedit tests, based on Windows or not.
I originally had this as a list of tuples, conditional on the platform. But
there are a couple of 'No such file or directory' messages emitted by Mercurial
itself, so unconditional is required for stability. There are also several
variants of what I assume is 'connection refused' and 'unknown host' in
test-clone.t and test-clonebundles.t for Docker, FreeBSD jails, etc. Yes, these
are handled by (re) tags, but maybe it would be better to capture those strings
in order to avoid whack-a-mole in future tests. All of this points to using a
dictionary containing one or more strings-to-be-replaced values.
Due to a bug in the test runner (fixed by the next commit), the regex
used for matching lines like " foobar | 2 +-" stoppped at the "|" and
the test passed even though the rest of the line did not match. The
test seems to have been supposed to match "|" and "+" literally on
those lines, so this changes the regex to escape those characters. It
also changes a "\s*" to "\s+" since I think we'll always include a
space after the "|" in the diffstat output.
Differential Revision: https://phab.mercurial-scm.org/D1545
We change subrepos.allowed from a list of allowed subrepo types to
a combination of a master switch and per-type boolean flag.
If the master switch is set, subrepos can be disabled wholesale.
If subrepos are globally enabled, then per-type options are
consulted. Mercurial repos are enabled by default. Everything else
is disabled by default.
We have a security issue with git subrepos. I'm not sure if svn subrepo is
vulnerable, but it seems not 100% safe to allow writing arbitrary data into
a metadata directory. So for now, only hg subrepo is enabled by default.
Maybe we should improve the help to describe why git/svn subrepos are
disabled.
This allows us to minimize the behavior change introduced by the next patch.
I have no idea which config style is preferred in UX POV, but I decided to
get things done.
a) list: 'allowed = hg, git, svn'
b) sub option: 'allowed.hg = True' or 'allowed:hg = True'
c) per-type action: 'hg = allow', 'git = abort'
This is an alternative workaround for the issue5730.
Perhaps this is the simplest way of disabling subrepo operations. It does
nothing clever, but just aborts if Mercurial starts accessing to a subrepo.
I think Greg's patch is more useful since it allows us to at least check
out the parent repository. However, that would be confusing if the default
is flipped to checkout=False and subrepos are silently ignored.
I don't like the config name 'allowed', but I couldn't get any better name.
It wasn't easy to extend the pathauditor to check symlink traversal across
subrepos because pathauditor._checkfs() rejects a directory having ".hg"
directory. That's why I added the explicit islink() check.
No idea if this patch is necessary after we've fixed the issue5730 by
splitting submerge() into planning and execution phases.
Upon pull or unbundle, we display a message with the range of new revisions
fetched. This revision range could readily be used after a pull to look out
what's new with 'hg log'. The algorithm takes care of filtering "obsolete"
revisions that might be present in transaction's "changes" but should not be
displayed to the end user.
When ui.origbackuppath is set, .orig files are stored outside of the working
copy, however they still have a .orig suffix appended to them. This can cause
unexpected conflicts, particularly when tracked files or directories have .orig
at the end.
This change removes the .orig suffix from files stored in an out-of-tree
origbackuppath.
Test Plan:
Update and run unit tests.
Differential Revision: https://phab.mercurial-scm.org/D679
This vulnerability was fixed by the previous patch and there were more ways
to exploit than using '|shellcmd'. So it doesn't make sense to reject only
pipe character.
Test cases are updated to actually try to exploit the bug. As the SSH bridge
of git/svn subrepos are not managed by our code, the tests for non-hg subrepos
are just removed.
This may be folded into the original patches.
'ssh://' has an exploit that will pass the url blindly to the ssh
command, allowing a malicious person to have a subrepo with
'-oProxyCommand' which could run arbitrary code on a user's machine. In
addition, at least on Windows, a pipe '|' is able to execute arbitrary
commands.
When this happens, let's throw a big abort into the user's face so that
they can inspect what's going on.
Well, mostly. The annotation on subrepo functions tacks on a parenthetical to
the abort message, which seems reasonable for a generic mechanism. But now all
messages consistently spell out 'subrepository', and double quote the name of
the repo. I noticed the inconsistency in the change for the last commit.
In the next patch, I'm gonna removing the global command aliases that force
the epoch date but some tests either fail or their output change after that.
Instead I'm copying the needed aliases in the test files that will otherwise
change.
Update test-rebase-obsolete.t because a revision hash is based on the epoch
date after a 'commit --amend' and the output will change after removing date
aliases.
Update test-subrepo-git.t as the git subrepo doesn't use traditional date
mechanisms. I'm not sure that updating the git subrepo to support default-date
make sense. Add the commit alias to the test in order for making it pass after
removing the date aliases globally.
Rather than sometimes using a complicated shell construct to dump pwned.txt
(if it wasn't expected to exist, but might, if something were broken) or
just cat (if it was expected to exist), just use the "f" utility, which
will be consistent in its behavior across different platforms.
Also make sure that *something* gets put into pwned.txt, even if we ended
up typoing the message variable.
Having a single "pwned" message which may or may not be emitted during the
tests for CVE-2016-3068 leads to extra confusion. Allow each test to emit
a more detailed message based on what the expectations are.
In both cases, we expect a version of git which has had the vulnerability
plugged, as well as a version of mercurial which also knows about
GIT_ALLOW_PROTOCOL. For the first test, we make sure GIT_ALLOW_PROTOCOL is
unset, meaning that the ext-protocol subrepo should be ignored; if it
isn't, there's either a problem with mercurial or the installed copy of
git.
For the second test, we explicitly allow ext-protocol subrepos, which means
that the subrepo will be accessed and a message emitted confirming that
this was, in fact, our intention.
The "pwned" message from this test gets gets sent to stderr, and so may get
emitted in different places from run to run in the rest of mercurial's
output. This patch forces the message to go to a specific file instead,
whose existence and contents we can examine at a stable point in the test's
execution.
CVE-2016-3068 (1/1)
Git's git-remote-ext remote helper provides an ext:: URL scheme that
allows running arbitrary shell commands. This feature allows
implementing simple git smart transports with a single shell shell
command. However, git submodules could clone arbitrary URLs specified
in the .gitmodules file. This was reported as CVE-2015-7545 and fixed
in git v2.6.1.
However, if a user directly clones a malicious ext URL, the git client
will still run arbitrary shell commands.
Mercurial is similarly effected. Mercurial allows specifying git
repositories as subrepositories. Git ext:: URLs can be specified as
Mercurial subrepositories allowing arbitrary shell commands to be run
on `hg clone ...`.
The Mercurial community would like to thank Blake Burkhart for
reporting this issue. The description of the issue is copied from
Blake's report.
This commit changes submodules to pass the GIT_ALLOW_PROTOCOL env
variable to git commands with the same list of allowed protocols that
git submodule is using.
When the GIT_ALLOW_PROTOCOL env variable is already set, we just pass it
to git without modifications.
Before this patch, some tests using external "diff" command via
extdiff extension fail on Solaris, because system standard "diff" (=
/usr/bin/diff) on Solaris always formats chunk header in the style
below:
@@ -X.x +Y.y @@
even though "diff" on Linux sometimes omits ".x" and/or ".y" in it.
This patch makes chunk header of external diff glob-ed for portability
of tests, and adds check-code.py rules to detect such diff output in
tests.
This patch also changes "hg diff" output in test-subrepo-git to
simplify detection rules, even though it is certainly portable because
these lines are generated by "git" command.
This patch is a part of making tests using external "diff" portable,
and tests below aren't yet portable even after this patch.
test-largefiles-update.t
test-subrepo-deep-nested-change.t
d6f3cca7f3a5 is adding a new tar call but forgot to filter out the
pax_global_header file that some tar versions write. This is similar to what
happened in 686775e575f3.
Unix utilities like tar will happily prefix the files it packs with './', but
annoyingly, Windows Explorer will not show these packed files when it opens the
archive. Since there doesn't seem to be a point in including './' in the path
names, just drop it. The default 'hg archive' prefix is the basename of the
archive, so specifying '.' allows for that default to be disabled completely.
'git diff-index' by default does not recurse into subdirectories
when changes are found. As a result, the directory is shown as changed,
rather than the files in this directory.
Adding '-r' results in recursing until the blobs themselves are checked.
The previous behavior when archiving a subrepo 's' on Windows was to internally
name the file under it 's\file', due to the use of vfs.reljoin(). When printing
the file list from the archive on Windows or Linux, the file was named
's\\file'. The archive extracted OK on Windows, but if the archive was brought
to a Linux system, it created a file named 's\file' instead of a directory 's'
containing 'file'.
*.zip format achives seemed not to have the problem, but this was definitely an
issue with *.tgz archives.
Largefiles actually got this right, but a test is added to keep this from
regressing. The subrepo-deep-nested-change.t test was repurposed to archive to
a file, since there are several subsequent tests that archive to a directory.
The output change is losing the filesystem prefix '../archive_lf' and not
listing the directories 'sub1' and 'sub1/sub2'.
One case where that would happen is while trying to resolve a subrepo, if the
path to the subrepo was actually a broken symlink. This bug was exposed by an
hg-git test.
In git 2.2.0 and higher, removing files and directories is changed:
removing an object that does not exist returns success rather than failure.
As a result, even though .hg/hgrc does not exist, success is returned
and the .hg/ directory is removed.
To handle this correctly, use 'rm -rf' to allow successful removing
for all git versions.
The exact changeset where this was introduced in git:
1054af7d04aef64378d69a0496b45cdbf6a0bef2
wrapper.c: remove/unlink_or_warn: simplify, treat ENOENT as success
This follows normal Mercurial rules, and the message is lifted from
workingctx.add(). The file is printed with abs() to be consistent with how it
is printed in workingctx, even though that is inconsistent with how added files
are printed in verbose mode. Further, the 'already tracked' notifications come
after all of the files that are added are printed, like in Mercurial.
As a side effect, we now have the reject list to return to the caller, so that
'hg add' exits with the proper code. It looks like an abort occurs if git fails
to add the file. Prior to touching 'snake.python' in the test, this was the
result of attempting to add the file after a 'git rm':
fatal: pathspec 'snake.python' did not match any files
abort: git add error 128 in s (in subrepo s)
I'm not sure what happens when git is a deep subrepo, but the 'in s' and
'in subrepo s' from @annotatesubrepoerror are redundant here. Maybe we should
stat the files before invoking git to catch this case and print out the prettier
hg message? The other thing missing from workingctx.add() is the call to
scmutil.checkportable(), but that would need to borrow the parent's ui object.
The previous test gave a false success because only an hg-ignored pattern was
specified. Therefore match.files() was empty, and it fell back to the files
unknown to git. The simplest fix is to always consider what is unknown to git,
as well as anything specified explicitly. Files that are ignored by git can
only be introduced by an explicit mention in match.files().
V2: use 'self._ctx.node()' instead of 'rev' in makefileobj.
As Matt Harbison mentioned, using 'rev' does not make sense,
since we'd be passing a git revision to the top-level
Mercurial repository.
Previously, git subrepo diffs did not have a newline at the end.
This caused multiple subrepo diffs to be joined on the same line.
Additionally, the command prompt after the diff still contained
a part of the diff.
Previously, revert was only possible if the '--no-backup'
switch was specified.
Now, to support backups, we explicitly go over all modified
files in the subrepo.
'git diff --stat' output changed with regard to the amount
of changes/insertions/deletions shown.
In older git versions (1.7.7.6), output was shown as:
2 files changed, 2 insertions(+), 1 deletions(-)
In newer versions, output is shown as:
2 files changed, 2 insertions(+), 1 deletion(-)
This patch uses a regex to handle both cases.
Previously, git subrepos did not support reverting.
This change adds basic support for reverting
when '--no-backup' is specified.
A warning is given (and the current state is kept)
when a revert is done without the '--no-backup' flag.
The output of 'git diff --stat' changed in git 1.7.10 and 1.7.11.
To ensure the tests work with earlier versions of git as well,
the output is now wrapped with a whitespace regex.
So far, git subrepositories were silently ignored for diffs.
This patch adds support for git subrepositories,
with the remark that --include and --exclude are not supported.
If --include or --exclude are used, the subrepo is ignored.
