In order to make this work, we have to wrap the io streams in a
TextIOWrapper so that __builtins__.input() can do unicode IO on Python
3. We can't just restore the original (unicode) sys.std* because we
might be running a cmdserver, and if we blindly restore sys.* to the
original values then we end up breaking the cmdserver. Sadly,
TextIOWrapper tries to close the underlying stream during its __del__,
so we have to make a sublcass to prevent that.
If you see errors like:
TypeError: a bytes-like object is required, not 'str'
On an input() or print() call on Python 3, the substitution of
sys.std* is probably the root cause.
A previous version of this change tried to put the bytesinput() method
in pycompat - it turns out we need to do some encoding handling, so we
have to be in a higher layer that's allowed to use
mercurial.encoding.encoding. As a result, this is in util for now,
with the TextIOWrapper subclass hiding in encoding.py. I'm not sure of
a better place for the time being.
Differential Revision: https://phab.mercurial-scm.org/D299
To list up available compression types instead of
".. bundlecompressionmarker" in "hg help bundlespec" output, proxy
object "docobject" is used, because:
- current online help system requires that __doc__ of registered
object (maybe, function) is already well formatted in reST syntax
- bundletype() method of compressionengine classes is used to list up
available compression types, but
- __doc__ of bundletype() object (= "instancemethod") is read-only
On the other hand, hggettext requires original function object, in
order to get document location in source code.
Therefore, description of each compression types isn't yet
translatable. Even if translatable, translators should make much
effort to determine location of original texts in source code.
To get actual function information, this patch makes hggettext use
function object saved as "_origfunc", if it is available. This patch
also changes bundlecompressiontopics() side, in order to explain how
these changes work easily.
This patch is a part of preparations for making description of each
compression types translatable.
a3022f57803b made `util.nogc` a no-op. It was to optimize PyPy. But it slows
down CPython if many objects (like 300k+) are created.
For example, running `hg log -r .` without extensions in `hg-committed` with
14k+ obsmarkers have the following times:
before | after
hg | chg | hg | chg
-----------------------------
1.262 | 0.860 | 1.077 | 0.619 (seconds, best of 20 runs)
Therefore let's re-enable nogc for CPython.
Differential Revision: https://phab.mercurial-scm.org/D402
We have at least three types with a close() and a release() method
where the close() method is supposed to be called on success and the
release() method is supposed to be called last, whether successful or
not. Two of them (transaction and dirstateguard) already have
identical implementations of __enter__ and __exit__. Let's extract a
base class for this, so we reuse the code and so the third type
(transactionmanager) can also be used as a context manager.
Differential Revision: https://phab.mercurial-scm.org/D392
This vulnerability was fixed by the previous patch and there were more ways
to exploit than using '|shellcmd'. So it doesn't make sense to reject only
pipe character.
Test cases are updated to actually try to exploit the bug. As the SSH bridge
of git/svn subrepos are not managed by our code, the tests for non-hg subrepos
are just removed.
This may be folded into the original patches.
'ssh://' has an exploit that will pass the url blindly to the ssh
command, allowing a malicious person to have a subrepo with
'-oProxyCommand' which could run arbitrary code on a user's machine. In
addition, at least on Windows, a pipe '|' is able to execute arbitrary
commands.
When this happens, let's throw a big abort into the user's face so that
they can inspect what's going on.
Our use of SSH has an exploit that will parse the first part of an url
blindly as a hostname. Prior to this set of security patches, a url
with '-oProxyCommand' could run arbitrary code on a user's machine. In
addition, at least on Windows, a pipe '|' can be abused to execute
arbitrary commands in a similar fashion.
We defend against this by checking ssh:// URLs and looking for a
hostname that starts with a - or contains a |.
When this happens, let's throw a big abort into the user's face so
that they can inspect what's going on.
This was previously landed as 4bc0c14fb501 but backed out in b63351f6a2 because
it broke hooks mid-rebase and caused conflict resolution data loss in the event
of unexpected exceptions. This new version adds the behavior back but behind a
config flag, since the performance improvement is notable in large repositories.
The old commit message was:
Recently we switched rebases to run the entire rebase inside a single
transaction, which dramatically improved the speed of rebases in repos with
large working copies. Let's also move the dirstate into a single dirstateguard
to get the same benefits. This let's us avoid serializing the dirstate after
each commit.
In a large repo, rebasing 27 commits is sped up by about 20%.
I believe the test changes are because us touching the dirstate gave the
transaction something to actually rollback.
(grafted from 9e3dc3a1638b9754b58a0cb26aaa75d868058109)
(grafted from 7d38b41d2266d9a02a15c64229fae0da5738dcec)
Differential Revision: https://phab.mercurial-scm.org/D135
rebase will have similar logic, so let's extract it. Besides, it makes
the histedit code more readable.
We may want to parametrize acceptintervention() by the exception(s)
that should result in transaction close.
Differential Revision: https://phab.mercurial-scm.org/D66
This was meant as a substitute for Python's "with" with multiple
context managers before we moved to Python 2.7. We're now on 2.7, so
we should have no reason to keep ctxmanager. "hg grep --all
ctxmanager" says that it was never used anyway.
Differential Revision: https://phab.mercurial-scm.org/D73
Inspired by the dirstate fix in 39954a8760cd, this should fix any race
conditions with the fsmonitor state changing from underneath.
Since we now grab the wlock for any non-invalidate writes, the only situation
this appears to happen in is with a concurrent invalidation. Test that.
Since 1d07d9da84a0, pycompat.bytestr() wrapped by win32mbcs returns
unicode object, if an argument is not byte-str object. And this causes
unexpected failure at colorization.
pycompat.bytestr() is used to convert from color effect "int" value to
byte-str object in color module. Wrapped pycompat.bytestr() returns
unicode object for such "int" value, because it isn't byte-str.
If this returned unicode object is used to colorize non-ASCII byte-str
in cases below, UnicodeDecodeError is raised at an operation between
them.
- colorization uses "ansi" color mode, or
Even though this isn't default on Windows, user might use this
color mode for third party pager.
- ui.write() is buffered with labeled=True
Buffering causes "ansi" color mode internally, regardless of
actual color mode. With "win32" color mode, extra escape sequences
are omitted at writing data out.
For example, with "win32" color mode, "hg status" doesn't fail for
non-ASCII filenames, but "hg log" does for non-ASCII text, because
the latter implies buffered formatter.
There are many "color effect" value lines in color.py, and making them
byte-str objects isn't suitable for fixing on stable. In addition to
it, pycompat.bytestr will be used to get byte-str object from any
types other than int, too.
To resolve this issue, this patch does:
- replace pycompat.bytestr in checkwinfilename() with newly added
hook point util._filenamebytestr, and
- make win32mbcs reverse-wrap util._filenamebytestr
(this is a replacement of 1d07d9da84a0)
This patch does two things above at same time, because separately
applying the former change adds broken revision (from point of view of
win32mbcs) to stable branch.
"_" prefix is added to "filenamebytestr", because it is win32mbcs
specific hook point.
Our current deprecation warning mechanism relies on ui object. They are case
where we cannot have access to the UI object. On a general basis we avoid using
the python mechanism for deprecation warning because up to Python 2.6 it is
exposing warning to unsuspecting user who cannot do anything to deal with them.
So we build a "safe" strategy to hide this warnings behind a flag in an
environment variable. The test runner set this flag so that tests show these
warning. This will help us marker API as deprecated for extensions to update
their code.
We have to do the % formatting over the sysstr, since the things we're
going to splat into it are themselves sysstrs. This is probably
technically wrong-ish, since bt is probably actually a bytestr here,
but this fixes the immediate issue, which was that hg was broken on
Python 3.
An upcoming patch will add support for documenting bundle
specifications in more detail. As part of this, we'd like to
enumerate available bundle compression formats. In order to do
this, we need to provide the help mechanism a dict of names
and objects with docstrings.
This patch adds docstrings to compengine.bundletype and adds
a function for retrieving a dict of them. The code is not yet
used.
Previously, when copying a file, copyfiles will compare src's st_dev with
dirname(dst)'s st_dev, to decide whether to enable hardlink or not.
That could have issues on Linux's overlayfs, where stating directories could
result in different st_dev from st_dev of stating files, even if both the
directories and the files exist in the overlay's upperdir.
This patch fixes it by checking dirname(src) instead. It's more consistent
because we are checking directories for both src and dest.
That fixes test-hardlinks.t running on common Docker setups.
This patch removes the global variable "allowhardlinks" that disables
hardlink in all cases, so hardlink gets enabled if the filesystem type is
whitelisted.
Third party extensions wanting to enable hardlink support unconditionally
can replace "_hardlinkfswhitelist.__contains__".
Since osutil.getfstype is available, use it to detect filesystem types. The
whitelist currently includes common local filesystems on Linux where they
should have good hardlink support. We may add new filesystems for other
platforms later.
pycompat.urlreq.unquote and pycompat.urlunquote effectively alias the
same thing. pycompat.urlunquote is only used once in the code base.
So let's switch to urlreq.unquote.
"Effectively" in the above paragraph is because pycompat.urlreq.unquote
aliases urllib.unquote and pycompat.urlunquote aliases urlparse.unquote
on Python 2. You might think one of urllib.unquote and urlparse.unquote
is an alias to the other, but you would be incorrect. In fact, these
functions are copies of each other. There is even a comment in the
CPython source code saying to keep them in sync. You can't make this
up.
