If we use the normal vfs, store encoding will be applied when we
.join() the path to be copied. This results in attempting to copy
a file that (likely) doesn't exist. Using the rawvfs operates on
the raw file path, which is returned by vfs.readdir().
Users at Mozilla are encountering this, as I've instructed them to
run `hg debugupgraderepo` to upgrade to generaldelta. While Mercurial
shouldn't deposit any files under .hg/store that require encoding, it
is possible for e.g. .DS_Store files to be created by the operating
system.
`hg debugupgraderepo` is currently buggy with regards to path
handling when copying files in .hg/store/. Specifically, it applies
the store filename encoding to paths instead of operating on raw
files.
This commit adds a test demonstrating the buggy behavior.
Previously, the "oldheads" variable was a list. On a repository at
Mozilla with 46,492 heads, profiling revealed that list membership
testing was dominating execution time of applying small changegroups.
This patch converts the list of old heads to a set. This makes
membership testing significantly faster. On the aforementioned
repository with 46,492 heads:
$ hg unbundle <file with 1 changeset>
before: 18.535s wall
after: 1.303s
Consumers of this variable only check for truthiness (`if oldheads`),
length (`len(oldheads)`), and (most importantly) item membership
(`h not in oldheads` - which occurs twice). So, the change to a set
should be safe and suitable for stable.
The practical effect of this change is that changegroup application
and related operations (like `hg push`) no longer exhibit an O(n^2)
CPU explosion as the number of heads grows.
Previously, `hg bundle zstd` on a non-generaldelta repo would
attempt to use a v1 bundle. This would fail because zstd is not
supported on v1 bundles.
This patch changes the behavior to automatically use a v2 bundle
when the user explicitly requests a bundlespec that is a compression
engine not supported on v1. If the bundlespec is <engine>-v1, it is
still explicitly rejected because that request cannot be fulfilled.
Version 1 bundles only support a fixed set of compression engines.
Before this change, we would accept any compression engine for v1
bundles, even those that may not work on v1. This could lead to
an error.
We define a fixed set of compression engines known to work with v1
bundles and we add checking to ensure a newer engine (like zstd)
won't work with v1 bundles.
I also took the liberty of adding test coverage for unknown compression
names because I noticed we didn't have coverage of it before.
This should catch the bug fixed by "worker: ignore meaningless exit status
indication returned by os.waitpid()."
Before, worker.py was untested since test repositories are relatively small.
Default-push has been deprecated in favour of default:pushurl. But "hg clone" still
inserts this in every hgrc file it creates. This patch updates the message by replacing
default-push with default:pushurl and also makes the necessary changes to test files.
Before this patch, worker implementation assumes that os.waitpid()
with os.WNOHANG returns '(0, 0)' for still running child process. This
is explicitly specified as below in Python API document.
os.WNOHANG
The option for waitpid() to return immediately if no child
process status is available immediately. The function returns
(0, 0) in this case.
On the other hand, POSIX specification doesn't define the "stat_loc"
value returned by waitpid() with WNOHANG for such child process.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/waitpid.html
CPython implementation for os.waitpid() on POSIX doesn't take any care
of this gap, and this may cause unexpected "exit status indication"
even on POSIX conformance platform.
For example, os.waitpid() with os.WNOHANG returns non-zero "exit
status indication" on FreeBSD. This implies os.kill() with own pid or
sys.exit() with non-zero exit code, even if no child process fails.
To ignore meaningless exit status indication returned by os.waitpid(),
this patch skips subsequent steps forcibly, if os.waitpid() returns 0
as pid.
This patch also arranges examination of 'p' value for readability.
FYI, there are some issues below about this behavior reported for
CPython.
https://bugs.python.org/issue21791https://bugs.python.org/issue27808
Previously Abort raised during 'getbundle' call poorly reported (HTTP-500 for
http, some scary messages for ssh). Abort error have been properly reported for
"push" for a long time, there is not reason to be different for 'getbundle'. We
properly catch such error and report them back the best way available. For
bundle, we issue a valid bundle2 reply (as expected by the client) with an
'error:abort' part. With bundle1 we do as best as we can depending of http or
ssh.
bundle2 allow the server to report error explicitly. This was initially
implemented for push but there is not reason to not use it for pull too. This
changeset add logic similar to the one in 'unbundle' to the
client side of 'getbundle'. That logic make sure the error is properly reported
as "remote". This will allow the server side of getbundle to send clean "Abort"
message in the next changeset.
Changeset a0966f529e1b introduced a config option to have the server deny pull
using bundle1. The original protocol has not really been design to allow that
kind of error reporting so some hack was used. It turned the hack only works on
HTTP and that ssh server hangs forever when this is used. After further
digging, there is no way to report the error in a unified way. Using `ooberror`
freeze ssh and raising 'Abort' makes HTTP return a HTTP-500 without further
details. So with sadness we implement a version that dispatch according to the
protocol used.
Now the error is properly reported, but we still have ungraceful abort after
that. The protocol do not allow anything better to happen using bundle1.
We are about to add a test for ssh pull/cloning being denied because of bundle1
usage. For this, it is cleaner to not operate from the clone using http. So we
update the test beforehand for clarity. This is more churns that what I'm happy
to see on stable, but the rests of the series is worth it in my opinion.
Changeset a0966f529e1b introduced a config option to have the server deny push
using bundle1. The original protocol has not really be design to allow such kind
of error reporting so some hack was used. It turned the hack only works on HTTP
and that ssh wire peer hangs forever when the same hack is used. After further
digging, there is no way to report the error in a unified way. Using 'ooberror'
freeze ssh and raising 'Abort' makes HTTP return a HTTP500 without further
details. So with sadness we implement a version that dispatch according to the
protocol used.
We also add a test for pushing over ssh to make sure we won't regress in the
future. That test show that the hint is missing, this is another bug fixed in
the next changeset.
The remote hint message was ignored when reporting the remote error and
passed to the local generic abort error. I think I might initially have
tried to avoid reimplementing logic controlling the hint display depending of
the verbosity level. However, first, there does not seems to have such verbosity
related logic and second the resulting was wrong as the primary error and the
hint were split apart. We now properly print the hint as remote output.
This patch also makes some expected output lines in tests glob-ed for
persistence of them.
BTW, files below aren't yet changed in 2017, but this patch also
updates copyright of them, because:
- mercurial/help/hg.1.txt
almost all of "man hg" output comes from online help of hg
command, and is already changed in 2017
- mercurial/help/hgignore.5.txt
- mercurial/help/hgrc.5
"copyright 2005-201X Matt Mackall" in them mentions about
copyright of Mercurial itself
This patch replaces domain of mercurial-devel ML address by
mercurial-scm.org for "Report-Msgid-Bugs-To" property of each *.po
files.
This avoids releasing 4.1.1 with invalid "Report-Msgid-Bugs-To"
in *.mo file, if corresponded *.po file isn't msgmerge-ed with recent
hg.pot by translator.
These *.po files aren't covered by check-code.py pattern newly added
in subsequent patch, because it ignores them.
This patch also adds new check-code.py pattern to detect invalid usage
of "mercurial@selenic.com".
Change for test-convert-tla.t is tested, but similar change for almost
same test-convert-baz.t isn't yet tested actually, because I couldn't
find out the way to get "GNU Arch baz client".
AFAIK, buildbot skips test-convert-baz.t, too. Does anybody have
appropriate environment for testing?
0b5f1f2efc77 introduced handling of a crash in this case. A review comment
suggested that it was not entirely obvious that a 'dm' always would have a 'r'
for the source file.
To mitigate that risk, make the code more conservative and make less
assumptions.
When this test was introduced, it used the short-form of all the flags
on this update invocation. I suspect, based on the "start with clean
dirstates" comment and the fact that the no-exec branch of the #if
guard leaves dirstate clean, that this should have been 'update -qCr'
instead of 'update -qcr', but that a bug in largefiles --check
handling left this problem unnoticed.
I'll leave a breadcrumb further up about the current failure mode in
the hopes that we can fix this some day.
This was previously discussed in [0] but the trail in that thread goes
cold after a few replies. Given that this is still a flaky test, that
appears to only be passing by bad fortune, I think it's worth
correcting the code of the test to make a correct assertion, and to
keep track of the suspected bug with some other mechanism than an
invalid test (if we had support for "expected failure" blocks this
might be a worthwhile use of them?).
0: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089501.html
I spent some time confused by this test. I'm pretty sure that this
line intends to be cleaning the dirstate, not checking that it's clean
before updating: the preceding #if block leaves the dirstate clean in
the noexec case, and dirty in the exec case, so we can't expect
consistent behavior across that platform variation. A subsequent patch
will modify this command to use --clean instead of --check.
I'll elaborate in that patch about the hypothetical bug here.
Work around that 'dm' in the data model only can have one operation for the
target file, but still can have multiple and conflicting operations on the
source file where the other operation is a 'rm'. The move would thus fail with
'abort: No such file or directory'.
In this case it is "obvious" that the file should be removed, either before or
after moving it. We thus keep the 'rm' of the source file but drop the 'dm'.
This is not a pretty fix but quite "obviously" safe (famous last words...) as
it only touches a rare code path that used to crash. It is possible that it
would be better to swap the files for 'dm' as suggested on
https://bz.mercurial-scm.org/show_bug.cgi?id=5020#c13 but it is not entirely
obvious that it not just would create conflicts on the other file. That can be
revisited later.
dict.keys() is documented to return a copy, so it's surprising that
sortdict.keys() did not. I noticed this because we have an extension
that calls readlocaltags(). That method tries to remove any tags that
point to non-existent revisions (most likely stripped). However, since
it's unintentionally working on the instance it's modifying, it
sometimes fails to remove tags when there are multiple bad tags in a
row. This was not caught because localrepo.tags() does an additional
layer of filtering.
sortdict is also used in other places, but I have not checked whether
its keys() and/or __delitem__() methods are used there.
GNU grep, when emitting a matching line that doesn't have a terminating
newline, will add an extra newline. Solaris grep passes the original line
through without the newline. This causes differences in test output when
looking at the last line of the output of get-with-headers.py, which
doesn't usually emit (and certainly doesn't guarantee) a terminating
newline.
Both grep implementations succeed in matching the requested pattern,
though, so rely on specifying the full pattern on grep's commandline
instead of expecting it in the output, and send the output to /dev/null.
I've seen this in a (misconfigured) FreeBSD jail which has ::1 as an
entry for localhost, but IPv6 support is disabled in the jail. It took
me months to figure out what was going on (and I only figured it out
when tinyproxy.py got confused by similar IPv4-level misconfiguration
of the localhost domain in /etc/hosts.)
I don't feel strongly about this patch: on the one hand, it's papering
over a host-level misconfiguration, but on the other it avoids some
weird and hard to diagnose problems that can occur in weirdly
restricted environments.
outgoing() and remote() may stall for long due to network I/O, which seems
unsafe per definition, "whether a predicate is safe for DoS attack." But I'm
not 100% sure about this. If our concern isn't elapsed time but CPU resource,
these predicates are considered safe. Perhaps that would be up to the
web/application server configuration?
Anyway, outgoing() and remote() wouldn't be useful in hgweb, so I think
it's okay to ban them.
Only FreeBSD seems to be this picky. Note that this explicit
absolute-path `cd` exposes a defect in the test, in that we end up
still inside the cwd-vanish repository, but that's not a regression in
this change. Since we're in a code freeze, I'm doing the smallest
thing possible to try and fix bugs on FreeBSD, rather than cleaning up
the entire problem. I'll follow up with a more complete fix after the
freeze.
We've made chg utilize the common code path implemented in pager.py (by
e8fb65f5e551 and e97133c7a9dc), but the chg server does not always start
with a tty. Because of this, uisetup() of the pager extension could be
skipped on the chg server.
Kudos given to Sean Farley for dogfooding new chg and spotting this problem.
This patch makes it possible to unshelve while having missing files
in your repo as long as shelved changes don't touch those missing files.
It also makes error message better otherwise.
statprof has a __main__ handler that allows viewing of previously
written data files. As Yuya pointed out during review, 82ee01726a77
broke this. This patch fixes that.
