Summary:
btrfs has a bug involving truncate not returning the correct error
code. We already handle it in other cases, so let's extend it to these uses of
truncate. Hopefully the need for this disappears soon.
Reviewed By: quark-zju
Differential Revision: D8152297
fbshipit-source-id: f55602ff5e0ec36346c547bfd4b6d0f6e4127500
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
Summary: D7840688 broke building RPM on OSX. This commit fixes the same.
Reviewed By: quark-zju
Differential Revision: D7854180
fbshipit-source-id: 4d3e4c87da777930780ad53888c13a41aab6c6e4
Summary:
Update the buck build rules to depend on the pyre2 third-party library, and to
try importing it using the module name `re2`.
Reviewed By: ryanmce
Differential Revision: D7840688
fbshipit-source-id: 21156958f42cdcf61f4dfdb2c6eccf95e657fcd1
Summary:
This logs timing data for:
- update
- merge.update
- mergedriver
- applyupdates
- workers
- fetches
- pulls
- remotefilelog prefetches
- treemanifest prefetches
- status
- dirstate walks
- fsmonitor walks
- watchman queries
Hopefully this will let us narrow down where time is going in a number of cases
where the profile data is ambigiuous or hard to come by.
Reviewed By: quark-zju
Differential Revision: D7681026
fbshipit-source-id: e6fe65c9a4d2f4e128f62ccb767a7cbe73b2649a
Summary:
This avoids potential SEGSEGVs when the system re2 is ABI incompatible.
See D7622774 for more details.
Reviewed By: markbt
Differential Revision: D7622771
fbshipit-source-id: bae1500e7468881a7a19d3cb9074db583b2444a6
Summary:
Update hg.copystore and util.copyfiles to use the new-style progress bar
context manager.
Reviewed By: ryanmce
Differential Revision: D7329482
fbshipit-source-id: 4ac1def57e0ae4e7819c7c0c4d6f1715b8cbb625
Summary:
This check is useful and detects real errors (ex. fbconduit). Unfortunately
`arc lint` will run it with both py2 and py3 so a lot of py2 builtins will
still be warned.
I didn't find a clean way to disable py3 check. So this diff tries to fix them.
For `xrange`, the change was done by a script:
```
import sys
import redbaron
headertypes = {'comment', 'endl', 'from_import', 'import', 'string',
'assignment', 'atomtrailers'}
xrangefix = '''try:
xrange(0)
except NameError:
xrange = range
'''
def isxrange(x):
try:
return x[0].value == 'xrange'
except Exception:
return False
def main(argv):
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
content = open(path).read()
try:
red = redbaron.RedBaron(content)
except Exception:
print(' warning: failed to parse')
continue
hasxrange = red.find('atomtrailersnode', value=isxrange)
hasxrangefix = 'xrange = range' in content
if hasxrangefix or not hasxrange:
print(' no need to change')
continue
# find a place to insert the compatibility statement
changed = False
for node in red:
if node.type in headertypes:
continue
# node.insert_before is an easier API, but it has bugs changing
# other "finally" and "except" positions. So do the insert
# manually.
# # node.insert_before(xrangefix)
line = node.absolute_bounding_box.top_left.line - 1
lines = content.splitlines(1)
content = ''.join(lines[:line]) + xrangefix + ''.join(lines[line:])
changed = True
break
if changed:
# "content" is faster than "red.dumps()"
open(path, 'w').write(content)
print(' updated')
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
For other py2 builtins that do not have a py3 equivalent, some `# noqa`
were added as a workaround for now.
Reviewed By: DurhamG
Differential Revision: D6934535
fbshipit-source-id: 546b62830af144bc8b46788d2e0fd00496838939
Summary:
Solves issues below:
```
hgext/backups.py:18:1: F811 redefinition of unused 'registrar' from line 17
hgext/catnotate.py:1:1: F811 redefinition of unused 'util' from line 1
hgext/remotenames.py:57:5: F811 redefinition of unused 'registrar' from line 34
hgsubversion/setup.py:103:5: F401 'mercurial' imported but unused
hgsubversion/setup.py:109:5: F401 'hgsubversion.svnwrap.svn_swig_wrapper' imported but unused
i18n/polib.py:1281:29: F841 local variable 'exc' is assigned to but never used (Python 2)
i18n/polib.py:1427:13: F841 local variable 'typ' is assigned to but never used
i18n/polib.py:28:1: F401 'sys' imported but unused
mercurial/manifest.py:411:5: F811 redefinition of unused '_lazymanifest' from line 168
mercurial/posix.py:419:5: F811 redefinition of unused 'normcasefallback' from line 362
mercurial/posix.py:425:5: F811 redefinition of unused 'checkexec' from line 167
mercurial/posix.py:431:5: F811 redefinition of unused 'checklink' from line 234
mercurial/pycompat.py:29:5: F401 'http.cookiejar as cookielib' imported but unused
mercurial/pycompat.py:30:5: F401 'http.client as httplib' imported but unused
mercurial/pycompat.py:31:5: F401 'pickle' imported but unused
mercurial/pycompat.py:33:5: F401 'socketserver' imported but unused
mercurial/pycompat.py:34:5: F401 'xmlrpc.client as xmlrpclib' imported but unused
mercurial/statprof.py:573:36: F812 list comprehension redefines 'parent' from line 562 (Python 2)
mercurial/util.py:1076:5: F811 redefinition of unused 'nogc' from line 1051
mercurial/util.py:3221:5: F811 redefinition of unused 'dirs' from line 3184
tests/silenttestrunner.py:24:5: F811 redefinition of unused 'main' from line 6
tests/test-context.py:90:1: F811 redefinition of unused 'scmutil' from line 4
tests/test-fb-hgext-cstore-treemanifest.py:146:5: F811 redefinition of unused 'testDeeplyNested' from line 134
tests/test-fb-hgext-extutil.py:46:5: F811 redefinition of unused 'testbgcommandfailure' from line 37
tests/test_hgsubversion_util.py:47:1: F811 redefinition of unused 'svnwrap' from line 31 (Python 2)
tests/test_hgsubversion_util.py:49:1: F811 redefinition of unused 'svnwrap' from line 47 (Python 2)
```
Reviewed By: ryanmce
Differential Revision: D6934533
fbshipit-source-id: 8b51851a76fec88bb59107ed05a901d42c7326f8
Summary:
This allows `HGDATAPATH` to override where to look for data files
like builtin configs, templates, help texts.
It'll be useful for running tests with a packed hg binary.
Reviewed By: DurhamG
Differential Revision: D6879865
fbshipit-source-id: 4638a9eff62a5fab088215cbeb72740bb67cac04
Summary:
When pushing a treeonly commit to a tree+flat hybrid server, the server
needs to execute the hooks in treeonly mode so we never try to access the flat
manifests. Previously we did this for shell hooks by setting some environment
variables, but we didn't do it for python hooks that ran in process.
This diff makes the python hooks run against a bundle repo that is instantiated
in treeonly mode. No changes to any hooks are required, as the repo object they
are given is already in the correct mode.
Reviewed By: quark-zju
Differential Revision: D6840971
fbshipit-source-id: 9fcb97d972076911b35bccf3f79b60972bcafc33
Signals could be blocked by something like:
#include <unistd.h>
#include <signal.h>
int main(int argc, char * const argv[]) {
sigset_t set;
sigfillset(&set);
sigprocmask(SIG_BLOCK, &set, NULL);
execv("/bin/hg", argv);
return 0;
}
One of the problems is if SIGCHLD is blocked, chgserver would not reap
zombie workers since it depends on SIGCHLD handler entirely.
While it's the parent process to blame but it seems a good idea to just
unblock the signal from hg. FWIW git does that for SIGPIPE already [1].
Unfortunately Python 2 does not reset or provide APIs to change signal
masks. Therefore let's add one in osutil. Note: Python 3.3 introduced
`signal.pthread_sigmask` which solves the problem.
`sigprocmask` is part of POSIX [2] so there is no feature testing in
`setup.py`.
[1]: 7559a1be8a
[2]: http://pubs.opengroup.org/onlinepubs/7908799/xsh/sigprocmask.html
Differential Revision: https://phab.mercurial-scm.org/D1736
This utility function allows clearing of the cached value set up
@propertycache, if there is one.
Differential Revision: https://phab.mercurial-scm.org/D1337
This is done by:
sed -i "s/pycompat\.sysplatform == 'darwin'/pycompat.isdarwin/" **/*.py
Plus a manual change to `sslutil.py` which involves indentation change that
cannot be done by `sed`.
Differential Revision: https://phab.mercurial-scm.org/D1035
This function finds a name which does not clash with any other name in the
manifest, and so can be used to safely rename a file.
Differential Revision: https://phab.mercurial-scm.org/D783
These are all the httpserver and urllib.* aliases. They seem to make
more sense in the slightly-more-specific urllibcompat package than the
general-purpose pycompat.
Differential Revision: https://phab.mercurial-scm.org/D935
Previously, chg's `verify` call could take 30+ms loading and checking new
config files. With one socket redirection, that adds up to around 70ms,
which is a lot for fast commands (ex. `bookmark --hidden`).
When investigating closer, A lot of time was spent on actually spent on ui
copying, which is mainly about `config.config` and `dict` copying.
This patch makes that 20x faster by adopting copy-on-write. The
copy-on-write is performed at config section level.
Before:
In [1]: %timeit ui.copy()
100 loops, best of 3: 2.32 ms per loop
After:
In [1]: %timeit ui.copy()
10000 loops, best of 3: 128 us per loop
2ms may look not that bad, but it adds up pretty quickly with multiple
calls. A typical chg run may call it 4 times, which is about 10ms.
Differential Revision: https://phab.mercurial-scm.org/D808
This is useful for large files that are only partly touched.
Test Plan:
Will be used and tested in a later patch.
Differential Revision: https://phab.mercurial-scm.org/D476
Make it clear that `fp` (`file` object) is different from `fd` (low-level
file descriptor number).
Differential Revision: https://phab.mercurial-scm.org/D642
Previously, if `.hg/store/00manifest.d.hgtmp1` exists, hg will copy the
entire `00manifest.d` every time when appending new manifest revisions.
That could happen if Mercurial or the machine crashed when `.hgtmp1` was
just created but not deleted yet.
This patch changes the fixed name to a random generated name. To be
consistent with D468, `~` suffix was used.
Differential Revision: https://phab.mercurial-scm.org/D611
Previously, we were performing membership testing against a
list. Change it to a set for a minor perf win. While we're at it,
explode the assignment in place so less work is needed at module
import time.
Differential Revision: https://phab.mercurial-scm.org/D600
Tools like Buck have patterns to ignore the creation of files (in the working
copy) that match certain patterns:
39278a4f07/src/com/facebook/buck/cli/Main.java (L259-L299)
When Buck sees a new source file (as reported by Watchman), it has to invalidate
a number of caches associated with the directory that contains the file.
Using a standard suffix, such as `~`, would make it easier for Buck and others
to filter out these types of file creation events.
The other uses of `tempfile.mkstemp()` in Hg do not appear to be problematic
because they (generally speaking) do not specify the `dir` parameter, so the
new file is created in the system-appropriate temp directory, which is outside
the working copy.
Test Plan:
`make tests`
Differential Revision: https://phab.mercurial-scm.org/D468
In order to make this work, we have to wrap the io streams in a
TextIOWrapper so that __builtins__.input() can do unicode IO on Python
3. We can't just restore the original (unicode) sys.std* because we
might be running a cmdserver, and if we blindly restore sys.* to the
original values then we end up breaking the cmdserver. Sadly,
TextIOWrapper tries to close the underlying stream during its __del__,
so we have to make a sublcass to prevent that.
If you see errors like:
TypeError: a bytes-like object is required, not 'str'
On an input() or print() call on Python 3, the substitution of
sys.std* is probably the root cause.
A previous version of this change tried to put the bytesinput() method
in pycompat - it turns out we need to do some encoding handling, so we
have to be in a higher layer that's allowed to use
mercurial.encoding.encoding. As a result, this is in util for now,
with the TextIOWrapper subclass hiding in encoding.py. I'm not sure of
a better place for the time being.
Differential Revision: https://phab.mercurial-scm.org/D299
To list up available compression types instead of
".. bundlecompressionmarker" in "hg help bundlespec" output, proxy
object "docobject" is used, because:
- current online help system requires that __doc__ of registered
object (maybe, function) is already well formatted in reST syntax
- bundletype() method of compressionengine classes is used to list up
available compression types, but
- __doc__ of bundletype() object (= "instancemethod") is read-only
On the other hand, hggettext requires original function object, in
order to get document location in source code.
Therefore, description of each compression types isn't yet
translatable. Even if translatable, translators should make much
effort to determine location of original texts in source code.
To get actual function information, this patch makes hggettext use
function object saved as "_origfunc", if it is available. This patch
also changes bundlecompressiontopics() side, in order to explain how
these changes work easily.
This patch is a part of preparations for making description of each
compression types translatable.
a3022f57803b made `util.nogc` a no-op. It was to optimize PyPy. But it slows
down CPython if many objects (like 300k+) are created.
For example, running `hg log -r .` without extensions in `hg-committed` with
14k+ obsmarkers have the following times:
before | after
hg | chg | hg | chg
-----------------------------
1.262 | 0.860 | 1.077 | 0.619 (seconds, best of 20 runs)
Therefore let's re-enable nogc for CPython.
Differential Revision: https://phab.mercurial-scm.org/D402
We have at least three types with a close() and a release() method
where the close() method is supposed to be called on success and the
release() method is supposed to be called last, whether successful or
not. Two of them (transaction and dirstateguard) already have
identical implementations of __enter__ and __exit__. Let's extract a
base class for this, so we reuse the code and so the third type
(transactionmanager) can also be used as a context manager.
Differential Revision: https://phab.mercurial-scm.org/D392
This vulnerability was fixed by the previous patch and there were more ways
to exploit than using '|shellcmd'. So it doesn't make sense to reject only
pipe character.
Test cases are updated to actually try to exploit the bug. As the SSH bridge
of git/svn subrepos are not managed by our code, the tests for non-hg subrepos
are just removed.
This may be folded into the original patches.
'ssh://' has an exploit that will pass the url blindly to the ssh
command, allowing a malicious person to have a subrepo with
'-oProxyCommand' which could run arbitrary code on a user's machine. In
addition, at least on Windows, a pipe '|' is able to execute arbitrary
commands.
When this happens, let's throw a big abort into the user's face so that
they can inspect what's going on.
