Summary:
It was used to reduce process count on laptops. Now chg server can be
reused across different `--config`, `HGPLAIN` settings, it's no longer
necessary to keep the hack.
Reviewed By: singhsrb
Differential Revision: D7847150
fbshipit-source-id: d4f98debb5e9eb4f2c3f8575532cc833c61e4b1d
Summary:
This makes it further harder to have multiple chg servers.
- MODULEPOLICY: We effectively only support 'c' now.
- LD_: It might be intentionally to trace chg server but not the client.
Assuming the user knows what they are doing.
- PATH: It's rare that `PATH` change can cause issues.
- PYTHON: Assume the current Python works, it's reasonable to not start a
new Python process.
- PYTHONPATH: Since we now bundle most extensions, side effects on
extensions should be considered rare.
Reviewed By: singhsrb
Differential Revision: D7845333
fbshipit-source-id: 264d67b687173fb4f2bdef1ef45937d8f098ed3d
Summary:
Previously, i18n are initialized at module import time, and cannot be
reinitialized if HGPLAIN is changed. That makes chg server has to hash
HGPLAIN, and use different servers if HGPLAIN is set.
Change i18n to be only initialized at `dispatch._dispatch` time, after chg
server updating environment variables. So a same chg server could serve
different languages or encodings.
Reviewed By: singhsrb
Differential Revision: D7845334
fbshipit-source-id: 6fba19cc07efdfa60a0e328cf2cc981cfed4bcc8
Summary:
Pre-import modules so when they are imported by `extensions.py`, they will
get a cache hit.
Note: use a dict explicitly since it's much faster than re-`import`:
In [2]: %timeit import hgext.rebase
The slowest run took 13.43 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 444 ns per loop
In [3]: d=sys.modules
In [4]: %timeit d['hgext.rebase']
10000000 loops, best of 3: 38 ns per loop
In [7]: %timeit sys.modules['hgext.rebase']
The slowest run took 31.89 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 67.3 ns per loop
Even importing two modules and constructing a tuple is 2.5x faster than directly
"import" one module:
In [8]: %timeit (sys.modules['hgext.rebase'], sys.modules['hgext.absorb'])
The slowest run took 10.78 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 177 ns per loop
Reviewed By: singhsrb
Differential Revision: D7840236
fbshipit-source-id: ca351c61ec63ffdaf401e561944e97963b434b3c
Summary:
Motivated by recent D7784903 which kills chg because it holds blackbox.log
file descriptor, and that patch is causing race conditions running with chg
(chg's sock atomic rename might fail if the directory is deleted).
There are other ways to solve the direct issue. This diff takes a more
aggressive but much cleaner approach. Basically, the `hg serve` framework is
too late for chg's usecase - the repo was already loaded, extension
side-effects have been already done at that time - chg has to use
workarounds to be compatible with that. Even with a best effort, it is still
possible to have weird interactions with shared repo because how hg loads
extensions.
The new approach is to pre-import a list of bundled extensions but do not
run their `uisetup`s. This solves a couple of hard problems:
- Compatibility - `uisetup` runs per request. That behaves exactly as what
an extension author expects.
- Less memory usage - there is no `repo` object is loaded in memory.
- Reduced process count - since extension config change does not require a
new chg server, the count of server processes would decrease (ex.
`--config extensions.blackbox=!` won't require a new chg server)
- Not holding fd to edenfs, since neither blackbox nor repo is loaded. This
makes it possible to remove the hacky killing chg logic in D7784903.
The downside is performance, since extension loading, and `uisetup` will run
every time. Benchmark shows that's could be 50ms-ish. But we could move
forward by moving extension logic to core incrementally to get rid of that
cost too.
This is basically a simplified version of my previous stack starting
with [1]. The original commit message was:
This is the beginning of a series that does a re-architect to chg mentioned
in [1], to achieve better compatibility.
The compatibility issues are mainly around "uisetup"s and "reposetup"s:
- Developers are usually unaware that uisetup runs only once per chg
process. We cannot reliably devel-warn them. The result is, potential
broken code are written. For example, it's really hard for chg to deal
with "experimental.evolution" changed from unset to manually set in
config files because setconfig is used if that config option is not set.
- An unnecessary "reposetup" caused by "hg serve" may have unwanted
side effects. This can become troublesome if the repo requires things
like remotefilelog or lz4revlog, and the user sets HGRCPATH to run
tests.
The current chg implementation assumes that "loading" an extension is not
side effect free - if extension related config has changed, a restart is
needed. The new idea is, "loading" = "importing" + "run ui/extsetup", the
"importing" part can be side-effect free for some extensions. And benchmark
shows "import" takes most of the time consumed, while "uisetup" is usually
very fast. We can afford running "uisetup"s per request.
To be able to (pre-)"import" extensions without running any "uisetup"s, a
different entry point is needed. Otherwise as long as we go through the
normal dispatch / runcommand ("hg serve") flow, "uisetup"s cannot be
avoided.
Aside from better compatibility, we can also remove some hacks:
- chg client: no longer needs to extract sensitive argv
- chg server: confighash can be changed to only hash environment variables
(reduce the number of server processes)
- chg server: srcui.walkconfig hack is no longer necessary
This patch adds a new script "chgserve" as the new entry point. Currently,
it is just a minimal implementation that makes "CHGHG=chgserve chg ..."
work, without doing any pre-importing. The change could also be done in the
"hg" script. But since chg is still experimental, let's keep "hg" untouched
for now.
[1]: www.mercurial-scm.org/pipermail/mercurial-devel/2016-July/085965.html
[1]: 6f91a1a69f
Reviewed By: singhsrb
Differential Revision: D7840237
fbshipit-source-id: e3d613b41fe4b721238b86c5bf84434d32cf0609
Summary:
This is an attempt to fix the wrong status issue. A formal fix would be
D7909172, which merges fsmonitor.state into treestate. See the comment in
`__init__.py` change for why status can be wrong.
Although it fixes the "status" path, I'm not sure whether other code paths
writing dirstates could be problematic or not. Anyway, D7909172 should be the
preferred final fix.
Reviewed By: r4-in
Differential Revision: D7916344
fbshipit-source-id: beab1c825b970bb47ffe03617a14eb6f203feafa
Summary:
This helps to avoid the following problem:
1. hg creates a temporary lock file, writes some stuff there
2. os writes this stuff into its buffer
3. hg closes the file, the metadata is written out (or journaled)
4. hg renames the file, which is again a metadata-only operation
5. the buffer is still not flushed
6. the OS crashes
7. upon reload, the os has a file with a correct name and a correct length,
but unexpected contents
Reviewed By: quark-zju
Differential Revision: D7889111
fbshipit-source-id: a0a152c9e7efef34847fa2d2ab9b94191bde43f4
Summary:
While establishing an ssh connection, ssh may need to interact with the user
(e.g. to collect passwords). Suspend the progress bar for the duration of
this, so it doesn't interfere.
Reviewed By: quark-zju
Differential Revision: D7876414
fbshipit-source-id: 5fa82f0f40fcffa6b94fa0210d102c76d3618a1d
Summary:
The mercurial Eden extension writes a `.hg/dirstate` file now, so scm-prompt.sh
no longer needs logic to look for Eden's snapshot file when the `dirstate` file
does not exist.
Reviewed By: wez
Differential Revision: D7874269
fbshipit-source-id: d36445e99de42f135088f38f3ce4ce372be9245e
Summary:
If `hg pushbackup` or `hg cloudsync` timed out waiting on the backup lock it
would still exit successfully. This updates the code to exit unsuccessfully so
that callers can tell that their commits were not actually backed up.
Reviewed By: quark-zju
Differential Revision: D7874624
fbshipit-source-id: 501b76d7c98d16f0128a77a20c881bf55d11a1d4
Summary:
This diff prepares the importer to cleanly handle moving files in perforce while
keeping the structure in hg untouched.
It introduces a magic string that has to be present in the changelist description:
`IMPORTER_IGNORE_REORG@`
When that is present, move/add move/delete actions where hg path is the same are
ignored.
Differential Revision: D7832814
fbshipit-source-id: e8323d0c3cc79ee81cb819bee63435f345069861
Summary:
Rework the commit cloud commands:
* Arrange them under a top-level `cloud` command.
* Make `join` call `register` if necessary and `sync`. This gives one-step
on-boarding.
* Add help text to all commands; clean up existing help text.
* Simplify output a little.
Differential Revision: D7876295
fbshipit-source-id: 1f7c0cabfa3d426ced34b90bdca64bef78d78211
Summary: This lets you find profiles that'll include a given filename.
Reviewed By: markbt
Differential Revision: D7876496
fbshipit-source-id: 3b375e5d8257a80853f854a6be27c74f805687e3
Summary:
The prompt was recently fixed to work with remotenames and we need to
update the test.
Reviewed By: ryanmce
Differential Revision: D7876171
fbshipit-source-id: 95ebc8d9d598cf943e67e1d38e9d7ee061882cca
Summary:
When the share extension is in use the remotenames file lives in the shared
repository, not the current working direcotry's .hg directory.
Reviewed By: wez
Differential Revision: D7872628
fbshipit-source-id: f4faae3411e6cef14cef5d52151092ce3ecebd47
Summary:
On systems with long TESTTMP paths, the test-subcommands test wraps the
filename in the help output. This means the TESTTMP matching no longer works,
and the test fails.
Move the alias definitions into the extension, where we can set the origin
manually.
Reviewed By: mjpieters
Differential Revision: D7859229
fbshipit-source-id: a3b62079ec302f7d6de1e4444a3d5bd740b39703
Summary:
Replace the monkey-patched subcommands in fbsparse with the new subcommand
infra.
Reviewed By: mjpieters
Differential Revision: D7849195
fbshipit-source-id: 7f1fc21f555dc2bea49d3a80efdbfd42a6e70cb5
Summary:
Add generalised support for subcommands. This is similar to the monkey-patched
version in `fbsparse`, but fully supported by the command infrastructure.
Subcommands are the same structure as normal commands, but are attached to a
table in the `subcommands` attribute of the main command. Normally, if no
subcommand is provided, the normal command function is called. This can be
made into an error by setting `subonly` on the top-level command.
In order to make `fbsparse` continue to work, I've temporarily hacked how it
handles help text. This will be fixed in a later diff that switches fbsparse
to use this infrastructure.
Reviewed By: mjpieters
Differential Revision: D7849476
fbshipit-source-id: b988edabb17da77bf91a278e0faa2feecd0c1db9
Summary:
WindowsError is only defined on Windows platforms. On non-Windows platforms
this could cause rage failures like this one: P59496908
This updates the code to just catch `OSError` since `WindowsError` derives from
`OSError` on Windows.
Reviewed By: phillco
Differential Revision: D7856903
fbshipit-source-id: cf56c24eb52bb1da5dcd94ef56a3f8eb5fd849e1
Summary: D7840688 broke building RPM on OSX. This commit fixes the same.
Reviewed By: quark-zju
Differential Revision: D7854180
fbshipit-source-id: 4d3e4c87da777930780ad53888c13a41aab6c6e4
Summary:
The logic was added by D2594568 and is intentional for "repairing" dirstate.
It's O(working copy). That makes absorb super slow. Therefore let's use the
new "exact" flag to skip the slow path.
Reviewed By: mitrandir77
Differential Revision: D7818728
fbshipit-source-id: a3a7d6074bd0240b9e1919e18d3e0b95daf74a64
Summary:
Previously, absorb patches fsmonitor to skip invalidating fsmonitor state
during dirstate.rebuild. Now let's use the added "exact" parameter to avoid
the monkey patching.
Reviewed By: mitrandir77
Differential Revision: D7818729
fbshipit-source-id: 0db41c7609277acd4823101e5569cbbe80f7580e
Summary: This is a hint for performing certain fast paths.
Reviewed By: mitrandir77
Differential Revision: D7818730
fbshipit-source-id: 4adcf8724b462d8d652e8e580d6a36eebc46a0f8
Summary:
Previously it is not actually used.
`test-hgext-repogenerator.t` changed because treedirstate uses random
number to generate file names.
`fakedirstatewritetime.py` was updated to be treedirstate-aware. This
makes test-revert.t test-merge-tools.t test-merge1.t pass.
Reviewed By: singhsrb
Differential Revision: D7844960
fbshipit-source-id: 33a1d0d4a8e22ea5e6bb6454956884571fcf6bab
Summary:
Either fix or ignore issues in `fb/` so the test is consistent if run
externally.
Reviewed By: DurhamG
Differential Revision: D7850647
fbshipit-source-id: 0f7faa3be2dff1dcf61a3b765c1827583fafc14f
Summary:
`GetFileInformationByHandle` returns a `BY_HANDLE_FILE_INFORMATION` structure,
which is similar to what a `stat` call returns. In particular, this structure
contains:
- the `VolumeSerialNumber` field
- the `CreationTime` fields
- the `LastWriteTime` fields
- the `FileSize` field
- the `FileIndex` fields
All of these are self-explanatory, except for the `FileIndex`. Here's what MSDN says:
```
The identifier that is stored in the nFileIndexHigh and nFileIndexLow members is called the file ID.
...
In the NTFS file system, a file keeps the same file ID until it is deleted.
You can replace one file with another file without changing the file ID
by using the ReplaceFile function. However, the file ID of the replacement file,
not the replaced file, is retained as the file ID of the resulting file.
```
Basically, every change to a file, except replacing it with some other file,
results in a changed file Id. Calling `ReplaceFile` however results in
`CreationTime` preserved from the replaced file and `LastWriteTime` preserved
from the replacement file:
```
C:\Code\tries\windowstries
λ python fileinfo.py
1.txt: Attr;32;Create;4064609256;30663014;Write;3046340864;30663166;Volume;1792064959;Size;0;5;Idx;655360;547898
2.txt: Attr;32;Create;3030045984;30663166;Write;3030172944;30663166;Volume;1792064959;Size;0;5;Idx;786432;565725
Replacing 1.txt with 2.txt; result is: 1
1.txt: Attr;32;Create;4064609256;30663014;Write;3030172944;30663166;Volume;1792064959;Size;0;5;Idx;786432;565725
```
Thus comparing all of these fields seems to be enough to replicate the `cachestat` beharior from `posix.py` (We
cache the `stat` of a file, which we almost always expect to change by renaming into it. We only use this `cachestat`
while our process is alive. One notable exception is the `.hgignore` file, which the user can change as they please,
but which we still `cachestat`.)
This change has performance implications for `status` if we use `.hgignore`: it's nearly 0.1s faster.
If we use `.gitignore`, there are no performance implications (at least I did not find any), but I'd still like
to land it for the sake of feature parity between Posix and Windows.
Reviewed By: quark-zju
Differential Revision: D7843746
fbshipit-source-id: f6f69ee12bdce054d7ea77917e83a95bcec17f83
Summary: Next diff in the stack accesses it from `windows.py`.
Reviewed By: quark-zju
Differential Revision: D7843747
fbshipit-source-id: fa9458a3ac4e66013f61c92d8ebc41b0859d4e37
Summary:
Update the buck build rules to depend on the pyre2 third-party library, and to
try importing it using the module name `re2`.
Reviewed By: ryanmce
Differential Revision: D7840688
fbshipit-source-id: 21156958f42cdcf61f4dfdb2c6eccf95e657fcd1
Summary:
There is a proper way [1] to skip the dict check-code check. Let's use it.
[1]: a61ed1c2d7
Reviewed By: DurhamG
Differential Revision: D7831336
fbshipit-source-id: a5e654e9e94cbfb1c5a07b047eb6e5451904c48e
Summary: Our CI didn't catch when I landed the previous diff.
Reviewed By: singhsrb
Differential Revision: D7834066
fbshipit-source-id: a51c2a294ea550917836f8b1eede2570838b60b7
Summary:
This would make tests run on treedirstate.
To avoid issues with Eden pulling from a non-eden treedirstate repo,
treedirstate is changed to be "always on" and disables itself on an eden repo.
The extension list is changed to a set for efficient `__contains__` test.
Reviewed By: phillco
Differential Revision: D7769804
fbshipit-source-id: d328fe51ef67c4730cfc53f43bdfc48c2765c541
Summary:
We mark some hg commands as appearing in short help, let's add this
to our parser
Reviewed By: quark-zju
Differential Revision: D7779270
fbshipit-source-id: 0c2b790f1994205ae4dbf7cd12ac3ba7f5ef39ad
Summary:
Let's check-in the definitions for now. In the future those should be generated
every build with all the extensions enabled - like in prod.
Reviewed By: quark-zju
Differential Revision: D7779273
fbshipit-source-id: f0d5c5260be74c5f64c0945004bf60399a6e8c4c
Summary:
This allows writing `TreeState` state in two ways - save as a new file, or
incrementally update an existing file.
Reviewed By: markbt
Differential Revision: D7748822
fbshipit-source-id: 472b78af6cf7ea79968460a51ec824eaa96e4973
Summary: They are used by the next diff.
Reviewed By: markbt
Differential Revision: D7748834
fbshipit-source-id: 9562204975d83a8dce6eb80d2677387e24f8f0a0
Summary:
The method names are inspired by std HashMap. The types are slightly
different due to `Tree` implementation details.
Reviewed By: markbt
Differential Revision: D7748828
fbshipit-source-id: fc24481cdf0054c8e879d760082e192e52afc7f5
Summary:
The `Tree` object can return an `&mut` entry easily. Let's expose the
interface. This could be useful when the caller only wants to modify part of
the file state. For example, changing `copied` without touching anything
else.
Reviewed By: markbt
Differential Revision: D7748820
fbshipit-source-id: 430fa8ee310297c61866695a692134daf519e78d
Summary:
Unlike TreeDirstate, this struct does not have two trees, and uses
FileStateV2.
Reviewed By: markbt
Differential Revision: D7748826
fbshipit-source-id: e637fad64e6b3e9b2a122e26a29fd04014181d6b
Summary: This lets you select on substrings in fields and paths.
Reviewed By: quark-zju
Differential Revision: D7788826
fbshipit-source-id: f92b8cc646fd36f4cb3b8a4dc6116576db80eb42
Summary: Calculation on how many profiles were hidden is only done when the hint is actually shown.
Reviewed By: quark-zju
Differential Revision: D7774192
fbshipit-source-id: a7196b2cc5640d0a7cb9c4d572e1a31ebfa41598
Summary:
This lets us produce a list of profiles with the title set, or find all profiles that are missing a description.
Convert the 'hidden' filter to use this feature, rather than special-case it in `_discover()`.
Reviewed By: quark-zju
Differential Revision: D7774029
fbshipit-source-id: 3bcba75e6da97bf0e560e11ce1ae7cbcee49ee45
Summary:
Infinitepush stores bundles uncompressed, which can make them quite large.
Enable zstd compression of bundles, both when sending them to the server during
backups, and when storing the bundle in the bundle store on the server.
Reviewed By: quark-zju
Differential Revision: D7777371
fbshipit-source-id: ef9360bf05bf2a6d2b76e26cd40ad1e4ee8ae076
Summary:
check-seclevel.py complains about using section markers inside a container.
The help text about sparse file format and config options can be put at
extension-level.
Reviewed By: mjpieters
Differential Revision: D7776355
fbshipit-source-id: 0b7d813c5eee352a054b21897682f6064f384829
Summary:
On OSX developers kept having to set their $PATH or change the make
file to actually build against the homebrew python. Let's build against it by
default if it exists.
Reviewed By: ryanmce
Differential Revision: D7791395
fbshipit-source-id: c69e41a469c5f94825814b4b30bc8ea144112167
