Summary:
Generally `$()` is preferred over `\`\`` -- so I was surprised it was here. And `(( ))` is Bash's useful arithmetic operator.
Per IRL discussion with quark-zju, we don't really care about non-Bash shells, which this is meant to support.
Reviewed By: quark-zju
Differential Revision: D8721467
fbshipit-source-id: 2a05ca486036fcbdb45d127dbe18e5539f0c956a
Summary:
Previously only function names are checked. Let's extend the check to
variable names.
Existing names breaking the rule are whitelisted.
Reviewed By: DurhamG
Differential Revision: D8744130
fbshipit-source-id: ca52d938305ccc3b2e1ea59e4e8f9c0dadff5970
Summary:
Previous code format attempt (D8173629) didn't cover all files due to `**/*.py`
was not expanded recursively by bash. That makes certain changes larger than
they should be (ex. D8675439). Now use zsh's `**/*.py` to format them.
Also fix Python syntax so black can run on more files, and all lint issues.
Reviewed By: phillco
Differential Revision: D8696912
fbshipit-source-id: 95f07aa0c5eb1b63947b0f77f534957f4ab65364
Summary:
Print the output from check-code.py in a way amenable to sorting (mostly, print each offense on one line instead of several)
The next diff will explain why.
Reviewed By: quark-zju
Differential Revision: D8721178
fbshipit-source-id: d814f94c151c6251f827ed6fe099e170e0b5353a
Summary:
The new entry point was added by D7840237.
With it, it is now pointless to pass `--config` arguments to the chg server.
So let's just remove the related logic, to avoid accidental profiling
related configs to the server (although the code path should ignore the
config flags).
Since we no longer use the old `hg serve` command code path, raise a
ProgrammingError explicitly to avoid surprises.
Reviewed By: singhsrb
Differential Revision: D8370965
fbshipit-source-id: 6a54cd54b41dc66c10f87e821ceb8e79adef09c7
Summary:
Re-apply D8302882. This was causing problems because I didn't realize
commitcloud was using zstd bundles. Now we do have Rust-backed zstd
compression support so we can remove the python bindings.
Reviewed By: DurhamG
Differential Revision: D8361250
fbshipit-source-id: 981289734793a4c3401577426180649fdc7eb1b9
Summary: This causing problems with the new alpha build w/ pulling std bundles and we need to back kit out until quark-zju can ship a Rust version.
Reviewed By: singhsrb
Differential Revision: D8339855
fbshipit-source-id: 2ff8b4d023e3d248ecc7e70924302bbf394ca268
Summary:
We use lz4 compression in production. Regarding on zstd, our approach would
be using the Rust `lib/zstdelta` library. So there is no need to keep the
Python binding.
This makes `make local` faster and also makes internal code search about
zstd cleaner.
Reviewed By: DurhamG, phillco
Differential Revision: D8302882
fbshipit-source-id: a6c34d6fea59140caeac158274388ba75a28fb29
Summary:
Usually the handshake process is pretty quick (<0.01 seconds):
chg: debug: 0.000148 try connect to ...
chg: debug: 0.000338 connected to server
chg: debug: 0.000359 initialize context buffer with size 4096
chg: debug: 0.008225 hello received: ...
chg: debug: 0.008269 capflags=0x7b03, pid=31941
chg: debug: 0.008282 request setprocname, block size 17
chg: debug: 0.008316 request attachio
chg: debug: 0.008978 response read from channel r, size 4
chg: debug: 0.009045 request chdir, block size 45
chg: debug: 0.009092 version matched (6119653365548183087)
However, we have seen some OSX cases where the handshake and basically
everything takes much longer:
chg: debug: 0.000139 try connect to ...
chg: debug: 0.000297 connected to server
chg: debug: 0.000321 initialize context buffer with size 4096
chg: debug: 0.192316 hello received: ...
chg: debug: 0.192362 capflags=0x7b03, pid=55634
chg: debug: 0.192373 request setprocname, block size 17
chg: debug: 0.192420 request attachio
chg: debug: 0.229009 response read from channel r, size 4
chg: debug: 0.229072 request chdir, block size 34
chg: debug: 0.229111 version matched (6119653365548183087)
(See P59677258 for the full paste)
If restart the chg server, the problem goes away and commands will be fast
again.
Unfortunately I'm not sure about the root cause of the problem. Maybe it's
Python's GC doing something very expensive? Maybe it's OSX thinking the server
process is "inactive" and put it to some state that's very slow to recover? Or maybe
it's some weird 3rdparty service?
For now, what we do know are:
- The slowness *sometimes* reproduces with chg.
- The slowness goes away if chg server is restarted.
As a last resort, detect the slowness by measuring the handshake time, then
restart the server accordingly. To avoid an infinite restart loop on slow machines,
the restart can only happen once.
The threshold is set to 0.05 seconds, which is roughly 5x the normal value, and
can be disabled by `CHGSTARTTIMECHECK=0`.
Reviewed By: phillco
Differential Revision: D8294468
fbshipit-source-id: 75246ea4d872045664e7feadb0acc47dfa1d8eae
Summary:
They're actively fighting against the clang-format config
and don't have an auto-fix.
Reviewed By: quark-zju
Differential Revision: D8283622
fbshipit-source-id: 2de45f50e6370a5ed14915c6ff23dc843ff14e8a
Summary:
I've been troubleshooting eden integration test failures on my
devserver and traced it to some slightly off behavior in the telemetry
wrapper.
The wrapper was setting `CHGHG` to `hg.real` rather than the computed
path to the `hg.real` executable. In the eden integration tests this
path is the buck generated `hg.par`. The problem this caused was running
the installed hg.real rather than the one from the test environment
and this caused resolution of the eden extension to fail.
Once I fixed that up I found that chg had detected a problem with the
paths to the hg executable that were being used; we were picking up `chg`
from the system path and had a similar issue to above.
I introduced an environmental variable `CHG_BIN` to hold the desired path
and set it to the buck built `chg` binary.
In the process of this I found that `chg` was triggering a UBSAN issue
by passing a nullptr as the second argument to `memcpy`. I've included
the trivial fix for that in this diff also.
Reviewed By: quark-zju
Differential Revision: D8274636
fbshipit-source-id: 7ee0740cbfb447ab41b9e08308767d42790ba296
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
We're seeing "Permission denied" errors on some of our automation, but
it's difficult to track down where it's coming from. Let's make each message
more descriptive.
Also updates, hgsql to handle the hg-rsh hook, which I noticed while
investigating.
Reviewed By: phillco, farnz
Differential Revision: D8188414
fbshipit-source-id: 5f8c99e8ba896c2636b1a04716125bc6a9df0591
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
Summary:
The clang-format file in scm/hg basically only applies to things in the
scm/hg/mercurial directory.
There are 180 C and C++ files under scm/hg, and the vast majority of them do
not follow the style specified in this clang-format file. All but 11 of these
files were present in scm/hg/contrib/clang-format-blacklist.
Of the 11 files that do follow this style, 10 are in the scm/hg/mercurial/
directory. (10 of the 21 files in this directory use this style.) The 1
other file is in scm/hg/contrib/xdiff.
The majority of the C/C++ files in scm/hg/hgext/extlib and scm/hg/lib/ follow
a style closer to Facebook and Google's C++ style guidelines.
Therefore this moves the .clang-format file to scm/hg/mercurial, and lets the
main fbsource clang-format file apply to the other files under scm/hg
Reviewed By: quark-zju
Differential Revision: D8131512
fbshipit-source-id: 622a33abc39eb240eff4ca27f69a675a7ed54a89
Summary:
Generate a `u64` integer about the "version" at build time, and make chg
client check the version before connecting to the server.
This would ensure a chg client would only connect to a matched version of
the server.
- In setup.py, compute the "versionhash", write it as
`mercurial.__version__.versionhash`.
- In dispatch.py, `mercurial.__version__` needs to be explicitly loaded
before forking.
- In commandserver.py, send the versionhash to the client with the "hello"
message.
- In chg.c, verify the versionhash. If it does not match, unlink the socket
path and reconnect.
Reviewed By: farnz
Differential Revision: D7978131
fbshipit-source-id: 50acc923e72e40a4f66a96f01a194cf1a57fe832
Summary:
This is a precursor to splitting the fsmonitor extension, as both
it and the new extension will use pywatchman.
Reviewed By: quark-zju
Differential Revision: D8002713
fbshipit-source-id: 37983fe2898d23223d1178eb3f15685f17ff8868
Summary:
The mercurial Eden extension writes a `.hg/dirstate` file now, so scm-prompt.sh
no longer needs logic to look for Eden's snapshot file when the `dirstate` file
does not exist.
Reviewed By: wez
Differential Revision: D7874269
fbshipit-source-id: d36445e99de42f135088f38f3ce4ce372be9245e
Summary:
When the share extension is in use the remotenames file lives in the shared
repository, not the current working direcotry's .hg directory.
Reviewed By: wez
Differential Revision: D7872628
fbshipit-source-id: f4faae3411e6cef14cef5d52151092ce3ecebd47
Summary:
I did some extra xdiff changes in upstream, namely:
- Remove unused features
- Replace "long" (32-bit in MSVC) with int64_t to support large files
- Add comment on some key variables
This backports them. It also includes Matt's fixes about Windows compatibility.
Reviewed By: ryanmce
Differential Revision: D7223939
fbshipit-source-id: 9287d5be22dae4ab41b05b3a4c160d836b5714a6
Summary:
Use the new gitignore matcher powered by Rust.
The hgignore matcher has some laziness, but is not tree-aware - with N
"hgignore" files loaded, it needs O(N) time to match. The gitignore matcher
is tree-aware and backed by native code with decent time complexity.
We have been maintaining a translation script that collects all gitignores,
generate hgignore files with very long regexp for them. That script has
issues with sparse recently. This diff allows us to remove those generated
hgignore files from the repo.
Note: fsmonitor state does not contain ignored files. And ignore
invalidation is generally broken in fsmonitor (it only checks top-level
.hgignore). That means, once a file is ignored, it cannot be "unignored" by
just removing the matched pattern from ".gitignore". The file has to be
"touched" or so.
Reviewed By: markbt
Differential Revision: D7319608
fbshipit-source-id: 1763544aedb44676413efb6d14ffd3917ed3b1cd
Summary: sed -i without arguments doesn't work on OSX.
Reviewed By: farnz
Differential Revision: D7195193
fbshipit-source-id: a8eead927c94404a37ce5df956de82d29bc1b6a8
Summary:
Let's switch to xdiff for its better diff quality and performance!
The test changes demonstrate xdiff's better diff quality.
Reviewed By: ryanmce
Differential Revision: D7135206
fbshipit-source-id: 1775df6fc0f763df074b4f52779835d6ef0f3a4e
Summary:
Implement a `mercurial.cext.xdiff` module that exposes the xdiff algorithm.
`xdiff.blocks` should be a drop-in replacement for `bdiff.blocks`.
In theory we can change the pure C version of `bdiff.c` directly. However
that means we lose bdiff entirely. It seems more flexible to have both at
the same time so they can be easily switched via Python code. Hence the
Python module approach.
Reviewed By: ryanmce
Differential Revision: D7135205
fbshipit-source-id: 48cd3b5be7fd5ef41b64eab6c76a5c8a6ce99e05
Summary:
Add a simple binary that runs xdiff in a minimal way. This is mainly for
exposing xdiff logic so it can be used in command line for testing purpose.
It also serves as an example of how to use xdiff.
Reviewed By: ryanmce
Differential Revision: D7133531
fbshipit-source-id: ceb608f5754b61eaa95804730b3c89643ff1837b
Summary:
Vendor git's xdiff library from git commit
d7c6c2369d7c6c2369ac21141b7c6cceaebc6414ec3da14ad using GPL2+ license.
There is another recent user report that hg diff generates suboptimal
result. It seems the fix to issue4074 isn't good enough. I crafted some
other interesting cases, and hg diff barely has any advantage compared with
gnu diffutils or git diff.
| testcase | gnu diffutils | hg diff | git diff |
| | lines time | lines time | lines time |
| patience | 6 0.00 | 602 0.08 | 6 0.00 |
| random | 91772 0.90 | 109462 0.70 | 91772 0.24 |
| json | 2 0.03 | 1264814 1.81 | 2 0.29 |
"lines" means the size of the output, i.e. the count of "+/-" lines. "time"
means seconds needed to do the calculation. Both are the smaller the better.
"hg diff" counts Python startup overhead.
Git and GNU diffutils generate optimal results. For the "json" case, git can
have an optimization that does a scan for common prefix and suffix first,
and match them if the length is greater than half of the text. See
https://neil.fraser.name/news/2006/03/12/. That would make git the fastest
for all above cases.
About testcases:
patience:
Aiming for the weakness of the greedy "patience diff" algorithm. Using
git's patience diff option would also get suboptimal result. Generated using
the Python script:
```
open('a', 'w').write('\n'.join(list('a' + 'x' * 300 + 'u' + 'x' * 700 + 'a\n')))
open('b', 'w').write('\n'.join(list('b' + 'x' * 700 + 'u' + 'x' * 300 + 'b\n')))
```
random:
Generated using the script in `test-issue4074.t`. It practically makes the
algorithm suffer. Impressively, git wins in both performance and diff
quality.
json:
The recent user reported case. It's a single line movement near the end of a
very large (800K lines) JSON file.
Reviewed By: ryanmce
Differential Revision: D7124455
fbshipit-source-id: 832651115da770f9d2ed5fdff2e200453c0013f8
Summary:
Don't think this is required or used anymore and reveals information
about the structure of our project if we open source our mercurial.
Reviewed By: quark-zju
Differential Revision: D7128203
fbshipit-source-id: 4cdfa008631d08321a4d5a1c8f18cef429c35077
Summary:
Delete hgext/extlib/cstore/bitmanipulation.h, and just use the existing
mercurial/bitmanipulation.h header file instead.
It looks like there are quite a few other copy-and-pasted files between these
directories too, which should probably still be cleaned up.
Reviewed By: quark-zju
Differential Revision: D7008478
fbshipit-source-id: 1fdc779b470242d3083a3948dace3951922e836c
Summary:
In some tests using chg, `--traceback` got accidentally enabled for
commands without `--traceback`. That is a side effect starting
the server using `--traceback`. Let's avoid passing `--traceback`
when starting the server.
Differential Revision: D6942974
fbshipit-source-id: d40697c8a9487ae53ffb5ae43da4f4582ca86545
Summary:
This check is useful and detects real errors (ex. fbconduit). Unfortunately
`arc lint` will run it with both py2 and py3 so a lot of py2 builtins will
still be warned.
I didn't find a clean way to disable py3 check. So this diff tries to fix them.
For `xrange`, the change was done by a script:
```
import sys
import redbaron
headertypes = {'comment', 'endl', 'from_import', 'import', 'string',
'assignment', 'atomtrailers'}
xrangefix = '''try:
xrange(0)
except NameError:
xrange = range
'''
def isxrange(x):
try:
return x[0].value == 'xrange'
except Exception:
return False
def main(argv):
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
content = open(path).read()
try:
red = redbaron.RedBaron(content)
except Exception:
print(' warning: failed to parse')
continue
hasxrange = red.find('atomtrailersnode', value=isxrange)
hasxrangefix = 'xrange = range' in content
if hasxrangefix or not hasxrange:
print(' no need to change')
continue
# find a place to insert the compatibility statement
changed = False
for node in red:
if node.type in headertypes:
continue
# node.insert_before is an easier API, but it has bugs changing
# other "finally" and "except" positions. So do the insert
# manually.
# # node.insert_before(xrangefix)
line = node.absolute_bounding_box.top_left.line - 1
lines = content.splitlines(1)
content = ''.join(lines[:line]) + xrangefix + ''.join(lines[line:])
changed = True
break
if changed:
# "content" is faster than "red.dumps()"
open(path, 'w').write(content)
print(' updated')
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
For other py2 builtins that do not have a py3 equivalent, some `# noqa`
were added as a workaround for now.
Reviewed By: DurhamG
Differential Revision: D6934535
fbshipit-source-id: 546b62830af144bc8b46788d2e0fd00496838939
Summary:
This patch fixes the regex. There are too many methods with underscores.
So a whitelist was added to avoid unnecessary churn of line numbers in
test-check-code.t.
Reviewed By: ryanmce
Differential Revision: D6937917
fbshipit-source-id: 9b7816278fc6f414c21f921b67d1cbb6a735a30f
Summary:
`test-check-commit` gives false positives for some commits. Remove it.
Facebook
The commits that give false positives include commits that are outside of
`fbcode/scm/hg`.
Reviewed By: quark-zju
Differential Revision: D6819543
fbshipit-source-id: ddfaae7350d4ad6503b7a7ec22e899bb7ae743df
Summary:
Port the double blank line and "foobar" naming rules to check-code.
check-commit checks "changes" while check-code checks "snapshot".
Checking changes would have a couple of disadvantages:
- No check after code is committed
- Could have difficulity dealing with commits involving other changes
in a mono repo.
Facebook
Context: https://fb.facebook.com/groups/scm/permalink/1558174000898881
Reviewed By: DurhamG
Differential Revision: D6928820
fbshipit-source-id: 9ea998731778150b60112840bce702d9584ba15b
Summary:
Previously `hg server` uses `HGPORT` that might be in use. This patch uses
`-p 0 --port-file ...` so `hg server` always gets assigned a free port.
The change was first made by the following Ruby script:
```
re = /^ \$ hg serve(.*) -p \$(HGPORT[12]?) (.*[^\\])$\n \$/
Dir['*.t'].each do |path|
old = File.read(path)
new = old.lines.map do |l|
next l if l[/\(glob\)/] or not l['$HGPORT'] or l[/^ [$>]/]
"#{l.chomp} (glob)\n"
end.join.gsub re, <<-'EOS'.chomp
$ hg serve\1 -p 0 --port-file $TESTTMP/.port \3
$ \2=`cat $TESTTMP/.port`
$
EOS
File.write(path, new) if old != new
end
```
Then there are some manual changes:
run-tests.py: It now treats `$HGPORT` in output as glob pattern `*`, since
it does not know the assigned value in tests.
test-bookmarks-pushpull.t, test-https.t: Some `hg pull`s were changed to use
explicit paths instead of relying on `.hgrc` since the test restarts the
server and `.hg/hgrc` having an outdated URL.
test-schemes.t: The test writes `$HGPORT` to `.hgrc` before assigning it.
Changed the order so the correct `$HGPORT` is written.
test-patchbomb-tls.t: Changed `(?) (glob)` to `(glob) (?)`.
Reviewed By: DurhamG
Differential Revision: D6925398
fbshipit-source-id: d5c10476f43ce23f9e99618807580cf8ba92595c
Summary:
This solves issues when the binary is linked with another C library that define
those functions.
Reviewed By: DurhamG
Differential Revision: D6888242
fbshipit-source-id: d714c7eb18bc4c281912df50567e7f176d64a669
Summary:
This fixes `test-check-module-imports.t` running under `buck test`
because `lz4` is not treated as a system module.
Reviewed By: DurhamG
Differential Revision: D6881307
fbshipit-source-id: 26a7e2d36c1ae6815064de15c6bdcb6e762c8ddf
Summary:
This is a resend of https://phab.mercurial-scm.org/D1430, without breaking
Windows.
I encountered "too many opened files" problem due to treemanifest packs on my
laptop. This patch seems to be the easiest solution without side effects. Other
choices are deleting files (seem like an non-ideal workaround), forcing a
repack (could be slow), and rewriting using Rust (could take too long).
The root cause is Python's `mmap` implementation has to keep a fd internally
to support `mmapobj.resize` API. We only need read-only operation on the
mmap object so the fd is unnecessary. Re-implement a minimal mmap interface
for this purpose.
Reviewed By: DurhamG
Differential Revision: D6835890
fbshipit-source-id: 74c429e957cb8677682604eb02fc38b5b8d13ef7
Summary:
I got tired of my prompt taking a long time to return to me sometimes after a command ended, and started to investigate, and found this:
```
aregner 4069767 0.0 0.0 115052 5076 pts/17 Ss+ 12:32 0:00 \_ -bash
aregner 298922 0.0 0.0 115052 3540 pts/17 S+ 14:40 0:00 | \_ -bash
aregner 298923 0.0 0.0 115052 3668 pts/17 S+ 14:40 0:00 | \_ -bash
aregner 273995 0.0 0.0 115052 4824 pts/9 Ss+ 14:32 0:00 \_ -bash
aregner 274061 0.0 0.0 115052 3820 pts/9 S+ 14:32 0:00 \_ -bash
[devbig308 (77bac96|remote/fbcode/warm):~/2nd-fbcode] strace -p 298923
strace: Process 298923 attached
stat("/home/.hg",
```
Sometimes it will hang ong this for 5+ minutes. I'm assuming this has something to do with autofs and NFS home dirs or something or another, but it doesn't matter because when it gets to this point it should just stop looking for a source control directory.
Reviewed By: ryanmce
Differential Revision: D6790747
fbshipit-source-id: d32c74476046aa6b62bc26725eea6b60a4eaa9f7
Summary:
It's pretty handy. The implementation is simple and clean. So move it from
contrib and enable it by default. "import"s are adjusted to make the module
checker happy.
Security and UX wise, since we have `--debugger` already, adding another
REPL seems fine.
Test Plan: Ran all tests
Reviewers: phillco, #mercurial
Reviewed By: phillco
Differential Revision: https://phabricator.intern.facebook.com/D6741283
Signature: 6741283:1516225662:ddc19a663e7ecef2a1fdaa5041f308dc838a8471
Summary:
Reorder the imports to make the module checker happy.
The module checker should be aware of Cython module paths. Do it manually.
Test Plan: Run `test-check-module-imports.t`
Reviewers: durham, #mercurial
Reviewed By: durham
Differential Revision: https://phabricator.intern.facebook.com/D6688384
Signature: 6688384:1515540420:bec02696b24c12cb4fa2595fd235fc8f3246b2a2
Summary:
Get rid of `gettimeofday` and switch to C++11 `std::chrono` for Windows/rare
platform compatibility.
Also format the code using clang-format.
Test Plan:
Make sure it build on both x64 and Power8 platform.
`make local` and try it in IPython:
```
In [1]: from hgext import traceprof
In [2]: def f():
...: g(1)
...:
In [3]: def g(x):
...: print(x+1)
...:
In [4]: from mercurial import ui
In [7]: with traceprof.profile(ui.ui(), sys.stderr):
...: f()
...:
2
| <module> ipython2:3
| start_ipython IPython/__init__.py:93
| launch_instance application.py:650
| start ipapp.py:342
| mainloop interactiveshell.py:479
| interact interactiveshell.py:459
| run_cell interactiveshell.py:2591
| run_ast_nodes interactiveshell.py:2770
| run_code interactiveshell.py:2851
| <module> <ipython-input-7-2e5a012739d1>:1
| __exit__ contextlib.py:21
Total time: 0 ms
```
Note: it crashes on Windows, which will be workarounded in a later patch.
Reviewers: rmcelroy, #mercurial
Reviewed By: rmcelroy
Subscribers: fried
Differential Revision: https://phabricator.intern.facebook.com/D6681062
Tags: aarch64
Signature: 6681062:1515488414:6b7a51eda9e9764560d415350630590e4817fae2