Summary:
When `chg` decides, that the command is unsupported, it executes the real hg
instead. Now that the real `hg` and the `chg` can be the same binary, we need
to make sure that `chg` is definitely not re-executed. Passing `CHGDISABLE`
achieves that.
Reviewed By: markbt
Differential Revision: D13118766
fbshipit-source-id: e41a2c230d17c4c26e590d648e4d0b722cf41ee5
Summary:
Add "--dry-run" for fix-code.py and use it in test-check.
This avoids license header and version = "*" issues.
Reviewed By: ikostia
Differential Revision: D10213070
fbshipit-source-id: 9fdd49ead3dfcecf292d5f42c028f20e5dde65d3
Summary:
`version = "*"` is not a good pattern. This script reads Cargo.lock and writes
down the version explicitly. This helps the OSS build.
Reviewed By: ikostia
Differential Revision: D10213074
fbshipit-source-id: 39720d895a5788b68d3e9d183d5e8941abc8c048
Summary:
In order to be able to run `chg` from the main `hg.rust` binary, we decided
that we will turn it into a static lib. This diff teaches our current build
scripts to do this.
Reviewed By: quark-zju
Differential Revision: D10853906
fbshipit-source-id: 8e0f37aa7e52d4a0610f36d7903eb0a318c193ad
Summary:
This reduces the chance that SIGINT can cause deadlock.
A config option is added in case things go wrong.
Reviewed By: markbt
Differential Revision: D10843397
fbshipit-source-id: 49e61fd404e5bb8ba32f9e66ea7609c427d1f590
Summary:
Currently it's just fixing copyright headers. I plan to make it also normalize
Cargo.toml and add a test-check for it.
Reviewed By: phillco
Differential Revision: D10213072
fbshipit-source-id: 38b7ba6d308bba3e13fe8db2eeb185c822838b78
Summary:
The script uses fanotify to answer "who changes this file" questions on Linux.
Sometimes people complain about files showing up unexpectedly on "hg status".
This could be helpful to debug those kind of issues.
This is to answer "what process" questions for a few selected long-living files
without guarnateed correctness. It does not handle cases like replacing files
(changing inode), renaming, etc.
It is Linux only right now. It seems an OSX version could be built using
fsevents.
Reviewed By: DurhamG
Differential Revision: D10418991
fbshipit-source-id: 55b75d0a324a1e561d85431c92b59a13da87fba0
Summary:
dirstate.drop() is used in two different situations. 1. To make a
tracked file become untracked, and 2. To remove a file from the dirstate
entirely. In the treestate case, this difference matters, so let's split drop
into two functions, one to mark something as untracked, and the other to remove
it entirely.
In this first patch we just rename drop to untrack. The next patch will
introduce the new delete function.
Reviewed By: ikostia
Differential Revision: D10317738
fbshipit-source-id: 85950ab1b1a10cd481edcfbba6da445b3dbf6397
Summary: Reformat all opted-in python code with version `18.9b0` of Black.
Reviewed By: ambv
Differential Revision: D10126605
fbshipit-source-id: 82af0d645dd411ce8ae6b8d239e151b3730cd789
Summary:
The Python hook got "kwargs" without "HG_" prefix. Add an explicit test for it.
Also remove unnecessary configs and change the shell to bash, since we don't
run tests using vanilla "sh".
check-code is updated so it no longer complains about the use of `bash`.
Reviewed By: markbt
Differential Revision: D9561962
fbshipit-source-id: 76a190dde1b0aeb0032a65c173ce6368a28e8cf6
Summary:
`sys.argv[0]` is wrong when hg is called from a compiled binary and thus
`sys.argv[0]` is not necessarily excutable.
Reviewed By: quark-zju
Differential Revision: D9368561
fbshipit-source-id: 80df0c2ae4a892c88a8d46382b88d18314efee70
Summary:
This script isn't used and just adds confusion. We use the one at
`fb/staticfiles/opt/facebook/share/bash_completion`.
Reviewed By: singhsrb
Differential Revision: D9029193
fbshipit-source-id: b718553c4b68c004d284a40f5aca645ebda38853
Summary:
There is an issue on OSX where the chg worker runs much slower. Its root cause
is still yet to be figured out (does OSX compresses background process memory
to make). It seems we hit it again according to https://fburl.com/co1eagws.
The previous workaround was to check "hello" time, and has a threshold to
restart the server. "hello" turned out to be a cheap operation and could be
inaccurate. From P59864987 it seems much more accurate to check the "validate"
operation (0.56s vs 0.026s). So this patch does that instead.
Note: we might need to revisit this if the Rust config parser lands, as that
will speed up the "validate" command.
Reviewed By: singhsrb
Differential Revision: D9019429
fbshipit-source-id: b2c02d08d20cc075d55e2d745d9394c90e543a1b
Summary:
Generally `$()` is preferred over `\`\`` -- so I was surprised it was here. And `(( ))` is Bash's useful arithmetic operator.
Per IRL discussion with quark-zju, we don't really care about non-Bash shells, which this is meant to support.
Reviewed By: quark-zju
Differential Revision: D8721467
fbshipit-source-id: 2a05ca486036fcbdb45d127dbe18e5539f0c956a
Summary:
Previously only function names are checked. Let's extend the check to
variable names.
Existing names breaking the rule are whitelisted.
Reviewed By: DurhamG
Differential Revision: D8744130
fbshipit-source-id: ca52d938305ccc3b2e1ea59e4e8f9c0dadff5970
Summary:
Previous code format attempt (D8173629) didn't cover all files due to `**/*.py`
was not expanded recursively by bash. That makes certain changes larger than
they should be (ex. D8675439). Now use zsh's `**/*.py` to format them.
Also fix Python syntax so black can run on more files, and all lint issues.
Reviewed By: phillco
Differential Revision: D8696912
fbshipit-source-id: 95f07aa0c5eb1b63947b0f77f534957f4ab65364
Summary:
Print the output from check-code.py in a way amenable to sorting (mostly, print each offense on one line instead of several)
The next diff will explain why.
Reviewed By: quark-zju
Differential Revision: D8721178
fbshipit-source-id: d814f94c151c6251f827ed6fe099e170e0b5353a
Summary:
The new entry point was added by D7840237.
With it, it is now pointless to pass `--config` arguments to the chg server.
So let's just remove the related logic, to avoid accidental profiling
related configs to the server (although the code path should ignore the
config flags).
Since we no longer use the old `hg serve` command code path, raise a
ProgrammingError explicitly to avoid surprises.
Reviewed By: singhsrb
Differential Revision: D8370965
fbshipit-source-id: 6a54cd54b41dc66c10f87e821ceb8e79adef09c7
Summary:
Re-apply D8302882. This was causing problems because I didn't realize
commitcloud was using zstd bundles. Now we do have Rust-backed zstd
compression support so we can remove the python bindings.
Reviewed By: DurhamG
Differential Revision: D8361250
fbshipit-source-id: 981289734793a4c3401577426180649fdc7eb1b9
Summary: This causing problems with the new alpha build w/ pulling std bundles and we need to back kit out until quark-zju can ship a Rust version.
Reviewed By: singhsrb
Differential Revision: D8339855
fbshipit-source-id: 2ff8b4d023e3d248ecc7e70924302bbf394ca268
Summary:
We use lz4 compression in production. Regarding on zstd, our approach would
be using the Rust `lib/zstdelta` library. So there is no need to keep the
Python binding.
This makes `make local` faster and also makes internal code search about
zstd cleaner.
Reviewed By: DurhamG, phillco
Differential Revision: D8302882
fbshipit-source-id: a6c34d6fea59140caeac158274388ba75a28fb29
Summary:
Usually the handshake process is pretty quick (<0.01 seconds):
chg: debug: 0.000148 try connect to ...
chg: debug: 0.000338 connected to server
chg: debug: 0.000359 initialize context buffer with size 4096
chg: debug: 0.008225 hello received: ...
chg: debug: 0.008269 capflags=0x7b03, pid=31941
chg: debug: 0.008282 request setprocname, block size 17
chg: debug: 0.008316 request attachio
chg: debug: 0.008978 response read from channel r, size 4
chg: debug: 0.009045 request chdir, block size 45
chg: debug: 0.009092 version matched (6119653365548183087)
However, we have seen some OSX cases where the handshake and basically
everything takes much longer:
chg: debug: 0.000139 try connect to ...
chg: debug: 0.000297 connected to server
chg: debug: 0.000321 initialize context buffer with size 4096
chg: debug: 0.192316 hello received: ...
chg: debug: 0.192362 capflags=0x7b03, pid=55634
chg: debug: 0.192373 request setprocname, block size 17
chg: debug: 0.192420 request attachio
chg: debug: 0.229009 response read from channel r, size 4
chg: debug: 0.229072 request chdir, block size 34
chg: debug: 0.229111 version matched (6119653365548183087)
(See P59677258 for the full paste)
If restart the chg server, the problem goes away and commands will be fast
again.
Unfortunately I'm not sure about the root cause of the problem. Maybe it's
Python's GC doing something very expensive? Maybe it's OSX thinking the server
process is "inactive" and put it to some state that's very slow to recover? Or maybe
it's some weird 3rdparty service?
For now, what we do know are:
- The slowness *sometimes* reproduces with chg.
- The slowness goes away if chg server is restarted.
As a last resort, detect the slowness by measuring the handshake time, then
restart the server accordingly. To avoid an infinite restart loop on slow machines,
the restart can only happen once.
The threshold is set to 0.05 seconds, which is roughly 5x the normal value, and
can be disabled by `CHGSTARTTIMECHECK=0`.
Reviewed By: phillco
Differential Revision: D8294468
fbshipit-source-id: 75246ea4d872045664e7feadb0acc47dfa1d8eae
Summary:
They're actively fighting against the clang-format config
and don't have an auto-fix.
Reviewed By: quark-zju
Differential Revision: D8283622
fbshipit-source-id: 2de45f50e6370a5ed14915c6ff23dc843ff14e8a
Summary:
I've been troubleshooting eden integration test failures on my
devserver and traced it to some slightly off behavior in the telemetry
wrapper.
The wrapper was setting `CHGHG` to `hg.real` rather than the computed
path to the `hg.real` executable. In the eden integration tests this
path is the buck generated `hg.par`. The problem this caused was running
the installed hg.real rather than the one from the test environment
and this caused resolution of the eden extension to fail.
Once I fixed that up I found that chg had detected a problem with the
paths to the hg executable that were being used; we were picking up `chg`
from the system path and had a similar issue to above.
I introduced an environmental variable `CHG_BIN` to hold the desired path
and set it to the buck built `chg` binary.
In the process of this I found that `chg` was triggering a UBSAN issue
by passing a nullptr as the second argument to `memcpy`. I've included
the trivial fix for that in this diff also.
Reviewed By: quark-zju
Differential Revision: D8274636
fbshipit-source-id: 7ee0740cbfb447ab41b9e08308767d42790ba296
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
We're seeing "Permission denied" errors on some of our automation, but
it's difficult to track down where it's coming from. Let's make each message
more descriptive.
Also updates, hgsql to handle the hg-rsh hook, which I noticed while
investigating.
Reviewed By: phillco, farnz
Differential Revision: D8188414
fbshipit-source-id: 5f8c99e8ba896c2636b1a04716125bc6a9df0591
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
Summary:
The clang-format file in scm/hg basically only applies to things in the
scm/hg/mercurial directory.
There are 180 C and C++ files under scm/hg, and the vast majority of them do
not follow the style specified in this clang-format file. All but 11 of these
files were present in scm/hg/contrib/clang-format-blacklist.
Of the 11 files that do follow this style, 10 are in the scm/hg/mercurial/
directory. (10 of the 21 files in this directory use this style.) The 1
other file is in scm/hg/contrib/xdiff.
The majority of the C/C++ files in scm/hg/hgext/extlib and scm/hg/lib/ follow
a style closer to Facebook and Google's C++ style guidelines.
Therefore this moves the .clang-format file to scm/hg/mercurial, and lets the
main fbsource clang-format file apply to the other files under scm/hg
Reviewed By: quark-zju
Differential Revision: D8131512
fbshipit-source-id: 622a33abc39eb240eff4ca27f69a675a7ed54a89
Summary:
Generate a `u64` integer about the "version" at build time, and make chg
client check the version before connecting to the server.
This would ensure a chg client would only connect to a matched version of
the server.
- In setup.py, compute the "versionhash", write it as
`mercurial.__version__.versionhash`.
- In dispatch.py, `mercurial.__version__` needs to be explicitly loaded
before forking.
- In commandserver.py, send the versionhash to the client with the "hello"
message.
- In chg.c, verify the versionhash. If it does not match, unlink the socket
path and reconnect.
Reviewed By: farnz
Differential Revision: D7978131
fbshipit-source-id: 50acc923e72e40a4f66a96f01a194cf1a57fe832
Summary:
This is a precursor to splitting the fsmonitor extension, as both
it and the new extension will use pywatchman.
Reviewed By: quark-zju
Differential Revision: D8002713
fbshipit-source-id: 37983fe2898d23223d1178eb3f15685f17ff8868
Summary:
The mercurial Eden extension writes a `.hg/dirstate` file now, so scm-prompt.sh
no longer needs logic to look for Eden's snapshot file when the `dirstate` file
does not exist.
Reviewed By: wez
Differential Revision: D7874269
fbshipit-source-id: d36445e99de42f135088f38f3ce4ce372be9245e
Summary:
When the share extension is in use the remotenames file lives in the shared
repository, not the current working direcotry's .hg directory.
Reviewed By: wez
Differential Revision: D7872628
fbshipit-source-id: f4faae3411e6cef14cef5d52151092ce3ecebd47
Summary:
I did some extra xdiff changes in upstream, namely:
- Remove unused features
- Replace "long" (32-bit in MSVC) with int64_t to support large files
- Add comment on some key variables
This backports them. It also includes Matt's fixes about Windows compatibility.
Reviewed By: ryanmce
Differential Revision: D7223939
fbshipit-source-id: 9287d5be22dae4ab41b05b3a4c160d836b5714a6
Summary:
Use the new gitignore matcher powered by Rust.
The hgignore matcher has some laziness, but is not tree-aware - with N
"hgignore" files loaded, it needs O(N) time to match. The gitignore matcher
is tree-aware and backed by native code with decent time complexity.
We have been maintaining a translation script that collects all gitignores,
generate hgignore files with very long regexp for them. That script has
issues with sparse recently. This diff allows us to remove those generated
hgignore files from the repo.
Note: fsmonitor state does not contain ignored files. And ignore
invalidation is generally broken in fsmonitor (it only checks top-level
.hgignore). That means, once a file is ignored, it cannot be "unignored" by
just removing the matched pattern from ".gitignore". The file has to be
"touched" or so.
Reviewed By: markbt
Differential Revision: D7319608
fbshipit-source-id: 1763544aedb44676413efb6d14ffd3917ed3b1cd
Summary: sed -i without arguments doesn't work on OSX.
Reviewed By: farnz
Differential Revision: D7195193
fbshipit-source-id: a8eead927c94404a37ce5df956de82d29bc1b6a8
Summary:
Let's switch to xdiff for its better diff quality and performance!
The test changes demonstrate xdiff's better diff quality.
Reviewed By: ryanmce
Differential Revision: D7135206
fbshipit-source-id: 1775df6fc0f763df074b4f52779835d6ef0f3a4e
Summary:
Implement a `mercurial.cext.xdiff` module that exposes the xdiff algorithm.
`xdiff.blocks` should be a drop-in replacement for `bdiff.blocks`.
In theory we can change the pure C version of `bdiff.c` directly. However
that means we lose bdiff entirely. It seems more flexible to have both at
the same time so they can be easily switched via Python code. Hence the
Python module approach.
Reviewed By: ryanmce
Differential Revision: D7135205
fbshipit-source-id: 48cd3b5be7fd5ef41b64eab6c76a5c8a6ce99e05
Summary:
Add a simple binary that runs xdiff in a minimal way. This is mainly for
exposing xdiff logic so it can be used in command line for testing purpose.
It also serves as an example of how to use xdiff.
Reviewed By: ryanmce
Differential Revision: D7133531
fbshipit-source-id: ceb608f5754b61eaa95804730b3c89643ff1837b
Summary:
Vendor git's xdiff library from git commit
d7c6c2369d7c6c2369ac21141b7c6cceaebc6414ec3da14ad using GPL2+ license.
There is another recent user report that hg diff generates suboptimal
result. It seems the fix to issue4074 isn't good enough. I crafted some
other interesting cases, and hg diff barely has any advantage compared with
gnu diffutils or git diff.
| testcase | gnu diffutils | hg diff | git diff |
| | lines time | lines time | lines time |
| patience | 6 0.00 | 602 0.08 | 6 0.00 |
| random | 91772 0.90 | 109462 0.70 | 91772 0.24 |
| json | 2 0.03 | 1264814 1.81 | 2 0.29 |
"lines" means the size of the output, i.e. the count of "+/-" lines. "time"
means seconds needed to do the calculation. Both are the smaller the better.
"hg diff" counts Python startup overhead.
Git and GNU diffutils generate optimal results. For the "json" case, git can
have an optimization that does a scan for common prefix and suffix first,
and match them if the length is greater than half of the text. See
https://neil.fraser.name/news/2006/03/12/. That would make git the fastest
for all above cases.
About testcases:
patience:
Aiming for the weakness of the greedy "patience diff" algorithm. Using
git's patience diff option would also get suboptimal result. Generated using
the Python script:
```
open('a', 'w').write('\n'.join(list('a' + 'x' * 300 + 'u' + 'x' * 700 + 'a\n')))
open('b', 'w').write('\n'.join(list('b' + 'x' * 700 + 'u' + 'x' * 300 + 'b\n')))
```
random:
Generated using the script in `test-issue4074.t`. It practically makes the
algorithm suffer. Impressively, git wins in both performance and diff
quality.
json:
The recent user reported case. It's a single line movement near the end of a
very large (800K lines) JSON file.
Reviewed By: ryanmce
Differential Revision: D7124455
fbshipit-source-id: 832651115da770f9d2ed5fdff2e200453c0013f8
Summary:
Don't think this is required or used anymore and reveals information
about the structure of our project if we open source our mercurial.
Reviewed By: quark-zju
Differential Revision: D7128203
fbshipit-source-id: 4cdfa008631d08321a4d5a1c8f18cef429c35077
Summary:
Delete hgext/extlib/cstore/bitmanipulation.h, and just use the existing
mercurial/bitmanipulation.h header file instead.
It looks like there are quite a few other copy-and-pasted files between these
directories too, which should probably still be cleaned up.
Reviewed By: quark-zju
Differential Revision: D7008478
fbshipit-source-id: 1fdc779b470242d3083a3948dace3951922e836c
Summary:
In some tests using chg, `--traceback` got accidentally enabled for
commands without `--traceback`. That is a side effect starting
the server using `--traceback`. Let's avoid passing `--traceback`
when starting the server.
Differential Revision: D6942974
fbshipit-source-id: d40697c8a9487ae53ffb5ae43da4f4582ca86545
Summary:
This check is useful and detects real errors (ex. fbconduit). Unfortunately
`arc lint` will run it with both py2 and py3 so a lot of py2 builtins will
still be warned.
I didn't find a clean way to disable py3 check. So this diff tries to fix them.
For `xrange`, the change was done by a script:
```
import sys
import redbaron
headertypes = {'comment', 'endl', 'from_import', 'import', 'string',
'assignment', 'atomtrailers'}
xrangefix = '''try:
xrange(0)
except NameError:
xrange = range
'''
def isxrange(x):
try:
return x[0].value == 'xrange'
except Exception:
return False
def main(argv):
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
content = open(path).read()
try:
red = redbaron.RedBaron(content)
except Exception:
print(' warning: failed to parse')
continue
hasxrange = red.find('atomtrailersnode', value=isxrange)
hasxrangefix = 'xrange = range' in content
if hasxrangefix or not hasxrange:
print(' no need to change')
continue
# find a place to insert the compatibility statement
changed = False
for node in red:
if node.type in headertypes:
continue
# node.insert_before is an easier API, but it has bugs changing
# other "finally" and "except" positions. So do the insert
# manually.
# # node.insert_before(xrangefix)
line = node.absolute_bounding_box.top_left.line - 1
lines = content.splitlines(1)
content = ''.join(lines[:line]) + xrangefix + ''.join(lines[line:])
changed = True
break
if changed:
# "content" is faster than "red.dumps()"
open(path, 'w').write(content)
print(' updated')
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
For other py2 builtins that do not have a py3 equivalent, some `# noqa`
were added as a workaround for now.
Reviewed By: DurhamG
Differential Revision: D6934535
fbshipit-source-id: 546b62830af144bc8b46788d2e0fd00496838939
