--insecure is our psuedo-supported footgun for disabling connection
security.
The flag already disables CA verification. I think allowing the use of
TLS 1.0 when specified is appropriate.
TIL that ui instances for remote/peer repos don't automagically inherit
config options from .hg/hgrc files.
This patch makes remote ui instances inherit options from the
[hostsecurity] section. We were already inheriting options
from [hostfingerprints] and [auth]. So adding [hostsecurity] to the
list seems appropriate.
When the inhibit extension from mutable-history is enabled, it attempts to
iterate over the rebaseset to prevent the nodes being rebased from being
marked obsolete. This happens at the same time as rebase's
_filterobsoleterevs function trying to iterate over the rebaseset to figure
out which ones are obsolete. The two of these iterating over the same
revset generatorset cause a 'generator already executing' exception. This is
probably a flaw in the revset implementation, since iterating over the same
set twice should be supported.
This regression was introduced in 5d16ebe7b14, since it changed
_filterobsoleterevs to be called before the rebaseset was turned into a
set(). For now let’s just make the rebaseset an actual set again before
calling that function. This was caught by the inhibit tests.
The relevant call stack from test-inhibit.t:
File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 285, in _preparenewrebase
obsrevs = _filterobsoleterevs(self.repo, rebaseset)
File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/inhibit.py", line 197, in _filterobsoleterevswrap
r = orig(repo, rebasesetrevs, *args, **kwargs)
File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 1380, in _filterobsoleterevs
return set(r for r in revs if repo[r].obsolete())
File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 1380, in <genexpr>
return set(r for r in revs if repo[r].obsolete())
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3079, in _iterordered
val2 = next(iter2)
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3417, in gen
yield nextrev()
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3424, in _consumegen
for item in self._gen:
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 71, in iterate
cl = repo.changelog
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 319, in changelog
revs = filterrevs(unfi, self.filtername)
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 261, in filterrevs
repo.filteredrevcache[filtername] = func(repo.unfiltered())
File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/directaccess.py", line 65, in _computehidden
hidden = repoview.filterrevs(repo, 'visible')
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 261, in filterrevs
repo.filteredrevcache[filtername] = func(repo.unfiltered())
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 175, in computehidden
hideable = hideablerevs(repo)
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 33, in hideablerevs
return obsolete.getrevs(repo, 'obsolete')
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/obsolete.py", line 1097, in getrevs
repo.obsstore.caches[name] = cachefuncs[name](repo)
File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/inhibit.py", line 255, in _computeobsoleteset
if getrev(n) not in blacklist:
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3264, in __contains__
return x in self._r1 or x in self._r2
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3348, in __contains__
for l in self._consumegen():
File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3424, in _consumegen
for item in self._gen:
ValueError: generator already executing
The code used self._rbcnamescount as if it was the length of self._names ...
but actually it is just the number of good entries on disk. This caused the
cache to be populated inefficiently. In some cases very inefficiently.
Instead of checking the length before lookup, just try a lookup in self._names
- that is also in most cases faster.
Comments and debug messages are tweaked to help understanding the issue
and the fix.
versiontuple() was previously only splitting on '+' and strings
like "3.9-rc" were causing it to misreport the version as
(3, None). By splitting on either '+' or '-' we can handle
our version strings with "-rc" in them.
The inventory property was deprecated in favor of root_inventory in bzr
2.5.0. Current version is 2.7.0.
I noticed this when testing locally on Python 2.6.9, which has warnings
turned on by default. The failure that occurs without this patch can be
seen on Python 2.7 by running with warnings enabled:
$ PYTHONWARNINGS=::DeprecationWarning make 'test-convert-bzr*'
Now setpgid has 2 main purposes: better handling for terminal-generated
SIGTSTP, SIGINT, and process-exit-generated SIGHUP. Update the comment to
explain things more clearly.
These signals are meant to send to a process group, instead of a single
process: SIGINT is usually emitted by the terminal and sent to the process
group. SIGHUP usually happens to a process group if termination of a process
causes that process group to become orphaned.
Before this patch, chg will only forward these signals to the single server
process. This patch changes it to the server process group.
This will allow us to properly kill processes started by the forked server
process, like a ssh process. The behavior difference can be observed by
setting SSH_ASKPASS to a dummy script doing "sleep 100" and then run
"chg push ssh://dest-need-password-auth". Before this patch, the first Ctrl+C
will kill the hg process while ssh-askpass and ssh will remain alive. This
patch will make sure they are killed properly.
It was in some cases possible to end up writing to the cache file without
growing it first. The range assignment in _setcachedata would append instead of
writing at the requested position and thus write the new record in the wrong
place.
To fix this, we avoid looking up in too small caches, and when growing the
cache, do it right before writing the new record to it so we know it has been
done correctly.
We recently discovered a case in production that chg uses 100% CPU and is
trying to read data forever:
recvfrom(4, "", 1814012019, 0, NULL, NULL) = 0
Using gdb, apparently readchannel() got wrong data. It was reading in an
infinite loop because rsize == 0 does not exit the loop, while the server
process had ended.
(gdb) bt
#0 ... in recv () at /lib64/libc.so.6
#1 ... in readchannel (...) at /usr/include/bits/socket2.h:45
#2 ... in readchannel (hgc=...) at hgclient.c:129
#3 ... in handleresponse (hgc=...) at hgclient.c:255
#4 ... in hgc_runcommand (hgc=..., args=<optimized>, argsize=<optimized>)
#5 ... in main (argc=...486922636, argv=..., envp=...) at chg.c:661
(gdb) frame 2
(gdb) p *hgc
$1 = {sockfd = 4, pid = 381152, ctx = {ch = 108 'l',
data = 0x7fb05164f010 "st):\nTraceback (most recent call last):\n"
"Traceback (most recent call last):\ne", maxdatasize = 1814065152,"
" datasize = 1814064225}, capflags = 16131}
This patch addresses the infinite loop issue by detecting continuously empty
responses and abort in that case.
Note that datasize can be translated to ['l', ' ', 'l', 'a']. Concatenate
datasize and data, it forms part of "Traceback (most recent call last):".
This may indicate a server-side channeledoutput issue. If it is a race
condition, we may want to use flock to protect the channels.
The Python ssl module conditionally sets the TLS 1.1 and TLS 1.2
constants depending on whether HAVE_TLSv1_2 is defined. Yes, these
are both tied to the same constant (I would think there would be
separate constants for each version). Perhaps support for TLS 1.1
and 1.2 were added at the same time and the assumption is that
OpenSSL either has neither or both. I don't know.
As part of developing this patch, it was discovered that Apple's
/usr/bin/python2.7 does not support TLS 1.1 and 1.2 (only TLS 1.0)!
On OS X 10.11, Apple Python has the modern ssl module including
SSLContext, but it doesn't appear to negotiate TLS 1.1+ nor does
it expose the constants related to TLS 1.1+. Since this code is
doing more robust feature detection (and not assuming modern ssl
implies TLS 1.1+ support), we now get TLS 1.0 warnings when running
on Apple Python. Hence the test changes.
I'm not super thrilled about shipping a Mercurial that always
whines about TLS 1.0 on OS X. We may want a follow-up patch to
suppress this warning.
It's been broken since 799db3fe9866, which made ui argument mandatory. I've
tried several combinations of HTTP/HTTPS proxying on old/new Python versions,
but I couldn't figure out how to reach this code path. Also, wrapping HTTP
connection by SSLSocket seems wrong. My understanding is that self.realhostport
is set by _generic_start_transaction() if HTTPS connection is tunneled.
This patch removes proxy tunneling from httpconnection.connect() assuming
that it was dead code from the beginning. Note that HTTPS over tunneling
should be handled by httpsconnection class.
The bundle2 changegroup part has an advisory param saying how many
changesets are in the part. Before this patch, we were setting
this part when generating bundle2 parts via the wire protocol but
not when generating local bundle2 files.
A side effect of not setting the changeset count part is that progress
bars don't work when applying changesets. As the tests show, this
impacted clone bundles, shelve, backup bundles, `hg unbundle`, and
anything touching bundle2 files.
This patch adds a backdoor to allow us to pass state from
changegroup generation into the unbundler. We store the number
of changesets in the changegroup in this state and use it to
populate the aforementioned advisory part parameter when generating
the bundle2 bundle.
I concede that I'm not thrilled by how state is being passed in
changegroup.py (it feels a bit hacky). I would love to overhaul the
rather confusing set of functions in changegroup.py with something that
passes rich objects around instead of e.g. low-level generators.
However, given the code freeze for 3.9 is imminent, I'd rather not
undertake this endeavor right now. This feels like the easiest way
to get the parameter added to the changegroup part.
`hg debugbundle` is calling repr() on bundle2 part params, which are
now util.sortdict instances. Unfortunately, repr() doesn't appear
to be deterministic for util.sortdict. So, we implement one.
We include the type name because that's the common convention for
__repr__ implementations. Having the type name in `hg debugbundle`
is a bit ugly. But it's a debug command and I don't care enough to
fix it.
An upcoming change that introduces a 2nd part parameter to a part
reveals that `hg debugbundle` isn't deterministic because parameters
are stored on n plain, unsorted dict.
While we could change that command to sort before output, I think
the more important underlying issue is that bundle2 reading is taking
an ordered data structure and converting it to an unordered one.
Plugging in util.sortdict() fixes that problem while preserving API
compatibility.
This patch also appears to shine light on the fact that we don't
have tests verifying parts with multiple parameters roundtrip
correctly. That would be a good thing to test (and fuzz)... someday.
As part of teaching Mozilla's replication extension to better handle
repositories with obsolescence data, I encountered a few scenarios
where I wanted built-in wire protocol commands from replication clients
to operate on unfiltered repositories so they could have access to
obsolete changesets.
While the undocumented "web.view" config option provides a mechanism
to choose what filter/view hgweb operates on, this doesn't apply
to wire protocol commands because wireproto.dispatch() is always
operating on the "served" repo.
This patch extracts the line for obtaining the repo that
wireproto commands operate on to its own function so extensions
can monkeypatch it to e.g. return an unfiltered repo.
I stopped short of exposing a config option because I view the use
case for changing this as a niche feature, best left to the domain
of extensions.
As a server operator, I've always wanted to know what Mercurial
version clients are running so I can track version adoption and
make informed decisions about which versions of Mercurial to
support in extensions. Unfortunately, there is no easy way to discern
this today: the best you can do is look for high-level feature usage
(e.g. bundle2) or sniff capabilities from bundle2 commands. And these
things aren't changed frequently enough to tell you anything that
interesting.
Nearly every piece of software talking HTTP sends its version in
the user agent. This includes web browsers, curl, and even Git.
This patch adds the distribution name and version to the user-agent
HTTP request header. We choose "Mercurial" for the distribution
name because that seems appropriate. The version string comes
from __version__.
The value is inside parenthesis for a few reasons:
* The version *may* contain spaces
* Alternate forms like "Mercurial/<version>" imply structure and
since the user agent should not be used by servers for protocol
or feature negotiation/detection, we don't want to even give the
illusion that the value should be parsed. A free form field is
the most hostile to parsing.
Flagging the patch as BC so it shows up in release notes. This
change should be backwards compatible. But I wouldn't be surprised if
a server somewhere is filtering on the exact old user agent string. So
I want to make noise about this change.
The old value 5 was arbitrary chosen. Since there's no practical reason to
limit the backlog, this patch simply uses SOMAXCONN as a value large enough.
pycompat.py includes hack to import modules whose names are changed in Python 3.
We use try-except to load module according to the version of python. But this
method forces us to import the modules to raise an ImportError and hence making
it demandimport unfriendly.
This patch changes the try-except blocks to a single if-else block. To avoid
test-check-pyflakes.t complain about unused imports, pycompat.py is excluded
from the test.
Python 2.7 supports specifying a custom cipher list to TLS sockets.
Advanced users may wish to specify a custom cipher list to increase
security. Or in some cases they may wish to prefer weaker ciphers
in order to increase performance (e.g. when doing stream clones
of very large repositories).
This patch introduces a [hostsecurity] config option for defining
the cipher list. The help documentation states that it is for
advanced users only.
Honestly, I'm a bit on the fence about providing this because
it is a footgun and can be used to decrease security. However,
there are legitimate use cases for it, so I think support should
be provided.
To check importing modules in perf.py for historical portability, this
patch lists up files by "hg files" both for "1.2" and tip, and builds
up "module whitelist" check from those files.
This patch uses "1.2" as earlier side version of "module whitelist",
because "mercurial.error" module is a blocker for loading perf.py with
Mercurial earlier than 1.2, and just importing "mercurial.error"
separately isn't enough.
This patch introduces tests/check-perf-code.py as a preparation for
adding extra checks on contrib/perf.py in subsequent patches (mainly,
for historical portability).
At this change, check-perf-code.py doesn't add any extra check, and is
equal to check-code.py. This makes subsequent patch focus only on
adding an extra check on perf.py check-perf-code.py.
check-perf-code.py adds extra checks on perf.py by wrapping
contrib/check-code.py, because "filtering" by check-code.py (e.g.
normalize characters in string literal or comment line) is useful to
simplify regexp for check, and avoid false positive matching.
This patch makes an extra check pattern to be prepared by
"_preparepats()" as similarly as existing patterns, if it is added to
"checks" array before invocation of "main()" in check-code.py.
This is a part of preparation for adding check-code.py extra checks by
another python script in subsequent patch.
This is also useful for SkeletonExtensionPlan.
https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan
demandimport of early Mercurial loads an imported module immediately,
if a module is imported absolutely by "from a import b" style. Recent
perf.py satisfies this condition, because it does:
- have "from __future__ import absolute_import" line
- use "from a import b" style for modules in "mercurial" package
Before this patch, importing modules below prevents perf.py from being
loaded by earlier Mercurial, because these aren't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9.
- branchmap 2.5 (or e3354ba12eef)
- repoview 2.5 (or 7d207cb7e38a)
- obsolete 2.3 (or b50a017cd9ac)
- scmutil 1.9 (or 065064cdde5f)
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with Mercurial earlier than 1.8 (or
1299f0c14572).
After this patch, "mercurial.error" is the only blocker in "from
mercurial import" statement for loading perf.py with Mercurial earlier
than 1.2. This patch ignores it, because just importing it separately
isn't enough.
