Summary:
commitcloud: do not use filtered set of heads when update infinitepush
state
hg cloud sync command can be interrupted when pulling
some of commits will be pulled, but not written to the commit cloud and
infinitepush states
the next run of cloud sync is smart, it will not pull what has been already pulled in but will fix the
commit cloud state
but because we used the filtered list of heads for infinitepush state, it will not fix the infinitepush
state
So, we shouldn't use the filtered set
Reviewed By: markbt
Differential Revision: D7972200
fbshipit-source-id: 94c01694d4ac77beeed647f77cbdb30fe3f7a404
Summary:
I noticed some tests were running slower without chg. Upon investigation,
`alias hg=...` seems stop working. Similar to D7563731, let's define
functions explicitly instead.
Reviewed By: singhsrb
Differential Revision: D7964291
fbshipit-source-id: 08a69c865ffef6be7e84dd66a7cece9284b94e60
Summary:
`pkill -f chgunix` is a convenient, but not a safe way to invalidate chg
servers. We're going to do the server invalidation in a better way. So just
revert the change.
Reviewed By: singhsrb
Differential Revision: D7978130
fbshipit-source-id: c13b75204ae1097ffe992b2e26d80d028022ff0d
Summary:
`DETACHED_PROCESS` simply means that child process won't inherit
current process' console (it does get one allocated by the OS though). `CREATE_NO_WINDOW` means that child
process does not get an allocated console altogether.
For details: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684863(v=vs.85).aspx
Reviewed By: quark-zju
Differential Revision: D7969621
fbshipit-source-id: 58d5e69808fb1064bcc1101c971f52088a7fd2de
Summary:
The infinitepushbackupstate file must be written using the vfs of the source
repo when the share extension is in use.
Reviewed By: mitrandir77
Differential Revision: D7936097
fbshipit-source-id: a57e241ca969632ced65029cb5ccf61373bd8aeb
Summary:
In order to prevent newly added obsmarkers from being lost while we are syncing
them, we separate out the obsmarkers that we are in the process of syncing into
a separate `syncing` file. New obsmarkers that are created during sync go into
the `pending` file and are synced on the next sync, which means it's safe to
delete the `syncing` file when sync completes.
Differential Revision: D7932299
fbshipit-source-id: bd86e836ded10e75790b87ca6734a29f068f3571
Summary:
A command that creates obsmarkers while they are being synced can cause the
obsmarkers to be dropped.
In this test, the obsmarker for the rebase restack is lost.
Differential Revision: D7932300
fbshipit-source-id: 11988f2a2c77eed9f9fab258a6623abd8c50e1cd
Summary:
They got randomly printed out and broke tests. Let's limit it to
"debugtreedirstate" command.
Reviewed By: singhsrb
Differential Revision: D7904376
fbshipit-source-id: 4f0b3708fdc55ed207c3c3baa2926f4b2374730f
Summary:
It's flaky and fastmanifest is no longer used in production. So let's just
remove the test.
Reviewed By: singhsrb
Differential Revision: D7945558
fbshipit-source-id: a6a4b34d40f0994223f9eebc41a34ffdcb260032
Summary:
D7845334 seems to "fix" the tests - the warnings are gone like other
`test-gendoc-*.t`. So let's update the test.
Reviewed By: DurhamG
Differential Revision: D7937819
fbshipit-source-id: 00ec52f5860c9ad7a5ab3b42a3d6db87124b7fbe
Summary:
Previously we only enabled on demand tree generation when treeonly was
set, but we also need it when sendtrees is set since we'll need to generate
trees for sending.
Reviewed By: quark-zju
Differential Revision: D7927856
fbshipit-source-id: a69d6c7920a92e4f90bdcd1d04aad9ef59e9c778
Summary:
hgsql depends on repositories being byte for byte identical, but the
current pull after a streaming clone can cause the repository to be different
(like if different delta decisions were made, or the commits were ordered
slightly differently). Let's disable that pull when the repository is an hgsql
repo.
Reviewed By: ryanmce
Differential Revision: D7925300
fbshipit-source-id: 6eba7ad4ccdd37f6d7c5090522867d1a54f722b7
Summary:
The destination revset could contain `SRC` that will break tweakdefaults.
Change the ordering so it could work.
Reviewed By: singhsrb
Differential Revision: D7924127
fbshipit-source-id: cb3eb1379425303607e4cb6f57f534133d457dea
Summary:
The "sensitive config sections" was used because "hg serve" loading
different extensions are incompatible with each other. Now we neither load
extensions nor run their uisetups, and just use one chg server. So
sensitive config sections can be removed.
Reviewed By: singhsrb
Differential Revision: D7847149
fbshipit-source-id: 758c1df21d280bf0f88d91432e1201c8417df532
Summary:
`pkill -f chgunix` was used to kill chg server processes. That could kill
worker processes if the worker hasn't changed its process name. Let's change
the process name early after forking to reduce the window a chg worker can
be killed by `pkill -f chgunix`.
Reviewed By: singhsrb
Differential Revision: D7845335
fbshipit-source-id: 3b3fc64c4328a058e0c6aad3cb11b0bc4efd9d13
Summary:
It was used to reduce process count on laptops. Now chg server can be
reused across different `--config`, `HGPLAIN` settings, it's no longer
necessary to keep the hack.
Reviewed By: singhsrb
Differential Revision: D7847150
fbshipit-source-id: d4f98debb5e9eb4f2c3f8575532cc833c61e4b1d
Summary:
This makes it further harder to have multiple chg servers.
- MODULEPOLICY: We effectively only support 'c' now.
- LD_: It might be intentionally to trace chg server but not the client.
Assuming the user knows what they are doing.
- PATH: It's rare that `PATH` change can cause issues.
- PYTHON: Assume the current Python works, it's reasonable to not start a
new Python process.
- PYTHONPATH: Since we now bundle most extensions, side effects on
extensions should be considered rare.
Reviewed By: singhsrb
Differential Revision: D7845333
fbshipit-source-id: 264d67b687173fb4f2bdef1ef45937d8f098ed3d
Summary:
Previously, i18n are initialized at module import time, and cannot be
reinitialized if HGPLAIN is changed. That makes chg server has to hash
HGPLAIN, and use different servers if HGPLAIN is set.
Change i18n to be only initialized at `dispatch._dispatch` time, after chg
server updating environment variables. So a same chg server could serve
different languages or encodings.
Reviewed By: singhsrb
Differential Revision: D7845334
fbshipit-source-id: 6fba19cc07efdfa60a0e328cf2cc981cfed4bcc8
Summary:
Pre-import modules so when they are imported by `extensions.py`, they will
get a cache hit.
Note: use a dict explicitly since it's much faster than re-`import`:
In [2]: %timeit import hgext.rebase
The slowest run took 13.43 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 444 ns per loop
In [3]: d=sys.modules
In [4]: %timeit d['hgext.rebase']
10000000 loops, best of 3: 38 ns per loop
In [7]: %timeit sys.modules['hgext.rebase']
The slowest run took 31.89 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 67.3 ns per loop
Even importing two modules and constructing a tuple is 2.5x faster than directly
"import" one module:
In [8]: %timeit (sys.modules['hgext.rebase'], sys.modules['hgext.absorb'])
The slowest run took 10.78 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 177 ns per loop
Reviewed By: singhsrb
Differential Revision: D7840236
fbshipit-source-id: ca351c61ec63ffdaf401e561944e97963b434b3c
Summary:
Motivated by recent D7784903 which kills chg because it holds blackbox.log
file descriptor, and that patch is causing race conditions running with chg
(chg's sock atomic rename might fail if the directory is deleted).
There are other ways to solve the direct issue. This diff takes a more
aggressive but much cleaner approach. Basically, the `hg serve` framework is
too late for chg's usecase - the repo was already loaded, extension
side-effects have been already done at that time - chg has to use
workarounds to be compatible with that. Even with a best effort, it is still
possible to have weird interactions with shared repo because how hg loads
extensions.
The new approach is to pre-import a list of bundled extensions but do not
run their `uisetup`s. This solves a couple of hard problems:
- Compatibility - `uisetup` runs per request. That behaves exactly as what
an extension author expects.
- Less memory usage - there is no `repo` object is loaded in memory.
- Reduced process count - since extension config change does not require a
new chg server, the count of server processes would decrease (ex.
`--config extensions.blackbox=!` won't require a new chg server)
- Not holding fd to edenfs, since neither blackbox nor repo is loaded. This
makes it possible to remove the hacky killing chg logic in D7784903.
The downside is performance, since extension loading, and `uisetup` will run
every time. Benchmark shows that's could be 50ms-ish. But we could move
forward by moving extension logic to core incrementally to get rid of that
cost too.
This is basically a simplified version of my previous stack starting
with [1]. The original commit message was:
This is the beginning of a series that does a re-architect to chg mentioned
in [1], to achieve better compatibility.
The compatibility issues are mainly around "uisetup"s and "reposetup"s:
- Developers are usually unaware that uisetup runs only once per chg
process. We cannot reliably devel-warn them. The result is, potential
broken code are written. For example, it's really hard for chg to deal
with "experimental.evolution" changed from unset to manually set in
config files because setconfig is used if that config option is not set.
- An unnecessary "reposetup" caused by "hg serve" may have unwanted
side effects. This can become troublesome if the repo requires things
like remotefilelog or lz4revlog, and the user sets HGRCPATH to run
tests.
The current chg implementation assumes that "loading" an extension is not
side effect free - if extension related config has changed, a restart is
needed. The new idea is, "loading" = "importing" + "run ui/extsetup", the
"importing" part can be side-effect free for some extensions. And benchmark
shows "import" takes most of the time consumed, while "uisetup" is usually
very fast. We can afford running "uisetup"s per request.
To be able to (pre-)"import" extensions without running any "uisetup"s, a
different entry point is needed. Otherwise as long as we go through the
normal dispatch / runcommand ("hg serve") flow, "uisetup"s cannot be
avoided.
Aside from better compatibility, we can also remove some hacks:
- chg client: no longer needs to extract sensitive argv
- chg server: confighash can be changed to only hash environment variables
(reduce the number of server processes)
- chg server: srcui.walkconfig hack is no longer necessary
This patch adds a new script "chgserve" as the new entry point. Currently,
it is just a minimal implementation that makes "CHGHG=chgserve chg ..."
work, without doing any pre-importing. The change could also be done in the
"hg" script. But since chg is still experimental, let's keep "hg" untouched
for now.
[1]: www.mercurial-scm.org/pipermail/mercurial-devel/2016-July/085965.html
[1]: 6f91a1a69f
Reviewed By: singhsrb
Differential Revision: D7840237
fbshipit-source-id: e3d613b41fe4b721238b86c5bf84434d32cf0609
Summary:
This is an attempt to fix the wrong status issue. A formal fix would be
D7909172, which merges fsmonitor.state into treestate. See the comment in
`__init__.py` change for why status can be wrong.
Although it fixes the "status" path, I'm not sure whether other code paths
writing dirstates could be problematic or not. Anyway, D7909172 should be the
preferred final fix.
Reviewed By: r4-in
Differential Revision: D7916344
fbshipit-source-id: beab1c825b970bb47ffe03617a14eb6f203feafa
Summary:
This helps to avoid the following problem:
1. hg creates a temporary lock file, writes some stuff there
2. os writes this stuff into its buffer
3. hg closes the file, the metadata is written out (or journaled)
4. hg renames the file, which is again a metadata-only operation
5. the buffer is still not flushed
6. the OS crashes
7. upon reload, the os has a file with a correct name and a correct length,
but unexpected contents
Reviewed By: quark-zju
Differential Revision: D7889111
fbshipit-source-id: a0a152c9e7efef34847fa2d2ab9b94191bde43f4
Summary:
While establishing an ssh connection, ssh may need to interact with the user
(e.g. to collect passwords). Suspend the progress bar for the duration of
this, so it doesn't interfere.
Reviewed By: quark-zju
Differential Revision: D7876414
fbshipit-source-id: 5fa82f0f40fcffa6b94fa0210d102c76d3618a1d
Summary:
The mercurial Eden extension writes a `.hg/dirstate` file now, so scm-prompt.sh
no longer needs logic to look for Eden's snapshot file when the `dirstate` file
does not exist.
Reviewed By: wez
Differential Revision: D7874269
fbshipit-source-id: d36445e99de42f135088f38f3ce4ce372be9245e
Summary:
If `hg pushbackup` or `hg cloudsync` timed out waiting on the backup lock it
would still exit successfully. This updates the code to exit unsuccessfully so
that callers can tell that their commits were not actually backed up.
Reviewed By: quark-zju
Differential Revision: D7874624
fbshipit-source-id: 501b76d7c98d16f0128a77a20c881bf55d11a1d4
Summary:
This diff prepares the importer to cleanly handle moving files in perforce while
keeping the structure in hg untouched.
It introduces a magic string that has to be present in the changelist description:
`IMPORTER_IGNORE_REORG@`
When that is present, move/add move/delete actions where hg path is the same are
ignored.
Differential Revision: D7832814
fbshipit-source-id: e8323d0c3cc79ee81cb819bee63435f345069861
Summary:
Rework the commit cloud commands:
* Arrange them under a top-level `cloud` command.
* Make `join` call `register` if necessary and `sync`. This gives one-step
on-boarding.
* Add help text to all commands; clean up existing help text.
* Simplify output a little.
Differential Revision: D7876295
fbshipit-source-id: 1f7c0cabfa3d426ced34b90bdca64bef78d78211
Summary: This lets you find profiles that'll include a given filename.
Reviewed By: markbt
Differential Revision: D7876496
fbshipit-source-id: 3b375e5d8257a80853f854a6be27c74f805687e3
Summary:
The prompt was recently fixed to work with remotenames and we need to
update the test.
Reviewed By: ryanmce
Differential Revision: D7876171
fbshipit-source-id: 95ebc8d9d598cf943e67e1d38e9d7ee061882cca
Summary:
When the share extension is in use the remotenames file lives in the shared
repository, not the current working direcotry's .hg directory.
Reviewed By: wez
Differential Revision: D7872628
fbshipit-source-id: f4faae3411e6cef14cef5d52151092ce3ecebd47
Summary:
On systems with long TESTTMP paths, the test-subcommands test wraps the
filename in the help output. This means the TESTTMP matching no longer works,
and the test fails.
Move the alias definitions into the extension, where we can set the origin
manually.
Reviewed By: mjpieters
Differential Revision: D7859229
fbshipit-source-id: a3b62079ec302f7d6de1e4444a3d5bd740b39703
Summary:
Replace the monkey-patched subcommands in fbsparse with the new subcommand
infra.
Reviewed By: mjpieters
Differential Revision: D7849195
fbshipit-source-id: 7f1fc21f555dc2bea49d3a80efdbfd42a6e70cb5
Summary:
Add generalised support for subcommands. This is similar to the monkey-patched
version in `fbsparse`, but fully supported by the command infrastructure.
Subcommands are the same structure as normal commands, but are attached to a
table in the `subcommands` attribute of the main command. Normally, if no
subcommand is provided, the normal command function is called. This can be
made into an error by setting `subonly` on the top-level command.
In order to make `fbsparse` continue to work, I've temporarily hacked how it
handles help text. This will be fixed in a later diff that switches fbsparse
to use this infrastructure.
Reviewed By: mjpieters
Differential Revision: D7849476
fbshipit-source-id: b988edabb17da77bf91a278e0faa2feecd0c1db9
Summary:
WindowsError is only defined on Windows platforms. On non-Windows platforms
this could cause rage failures like this one: P59496908
This updates the code to just catch `OSError` since `WindowsError` derives from
`OSError` on Windows.
Reviewed By: phillco
Differential Revision: D7856903
fbshipit-source-id: cf56c24eb52bb1da5dcd94ef56a3f8eb5fd849e1
Summary: D7840688 broke building RPM on OSX. This commit fixes the same.
Reviewed By: quark-zju
Differential Revision: D7854180
fbshipit-source-id: 4d3e4c87da777930780ad53888c13a41aab6c6e4
Summary:
The logic was added by D2594568 and is intentional for "repairing" dirstate.
It's O(working copy). That makes absorb super slow. Therefore let's use the
new "exact" flag to skip the slow path.
Reviewed By: mitrandir77
Differential Revision: D7818728
fbshipit-source-id: a3a7d6074bd0240b9e1919e18d3e0b95daf74a64
Summary:
Previously, absorb patches fsmonitor to skip invalidating fsmonitor state
during dirstate.rebuild. Now let's use the added "exact" parameter to avoid
the monkey patching.
Reviewed By: mitrandir77
Differential Revision: D7818729
fbshipit-source-id: 0db41c7609277acd4823101e5569cbbe80f7580e
Summary: This is a hint for performing certain fast paths.
Reviewed By: mitrandir77
Differential Revision: D7818730
fbshipit-source-id: 4adcf8724b462d8d652e8e580d6a36eebc46a0f8
Summary:
Previously it is not actually used.
`test-hgext-repogenerator.t` changed because treedirstate uses random
number to generate file names.
`fakedirstatewritetime.py` was updated to be treedirstate-aware. This
makes test-revert.t test-merge-tools.t test-merge1.t pass.
Reviewed By: singhsrb
Differential Revision: D7844960
fbshipit-source-id: 33a1d0d4a8e22ea5e6bb6454956884571fcf6bab
Summary:
Either fix or ignore issues in `fb/` so the test is consistent if run
externally.
Reviewed By: DurhamG
Differential Revision: D7850647
fbshipit-source-id: 0f7faa3be2dff1dcf61a3b765c1827583fafc14f
Summary:
`GetFileInformationByHandle` returns a `BY_HANDLE_FILE_INFORMATION` structure,
which is similar to what a `stat` call returns. In particular, this structure
contains:
- the `VolumeSerialNumber` field
- the `CreationTime` fields
- the `LastWriteTime` fields
- the `FileSize` field
- the `FileIndex` fields
All of these are self-explanatory, except for the `FileIndex`. Here's what MSDN says:
```
The identifier that is stored in the nFileIndexHigh and nFileIndexLow members is called the file ID.
...
In the NTFS file system, a file keeps the same file ID until it is deleted.
You can replace one file with another file without changing the file ID
by using the ReplaceFile function. However, the file ID of the replacement file,
not the replaced file, is retained as the file ID of the resulting file.
```
Basically, every change to a file, except replacing it with some other file,
results in a changed file Id. Calling `ReplaceFile` however results in
`CreationTime` preserved from the replaced file and `LastWriteTime` preserved
from the replacement file:
```
C:\Code\tries\windowstries
λ python fileinfo.py
1.txt: Attr;32;Create;4064609256;30663014;Write;3046340864;30663166;Volume;1792064959;Size;0;5;Idx;655360;547898
2.txt: Attr;32;Create;3030045984;30663166;Write;3030172944;30663166;Volume;1792064959;Size;0;5;Idx;786432;565725
Replacing 1.txt with 2.txt; result is: 1
1.txt: Attr;32;Create;4064609256;30663014;Write;3030172944;30663166;Volume;1792064959;Size;0;5;Idx;786432;565725
```
Thus comparing all of these fields seems to be enough to replicate the `cachestat` beharior from `posix.py` (We
cache the `stat` of a file, which we almost always expect to change by renaming into it. We only use this `cachestat`
while our process is alive. One notable exception is the `.hgignore` file, which the user can change as they please,
but which we still `cachestat`.)
This change has performance implications for `status` if we use `.hgignore`: it's nearly 0.1s faster.
If we use `.gitignore`, there are no performance implications (at least I did not find any), but I'd still like
to land it for the sake of feature parity between Posix and Windows.
Reviewed By: quark-zju
Differential Revision: D7843746
fbshipit-source-id: f6f69ee12bdce054d7ea77917e83a95bcec17f83
Summary: Next diff in the stack accesses it from `windows.py`.
Reviewed By: quark-zju
Differential Revision: D7843747
fbshipit-source-id: fa9458a3ac4e66013f61c92d8ebc41b0859d4e37
Summary:
Update the buck build rules to depend on the pyre2 third-party library, and to
try importing it using the module name `re2`.
Reviewed By: ryanmce
Differential Revision: D7840688
fbshipit-source-id: 21156958f42cdcf61f4dfdb2c6eccf95e657fcd1
Summary:
There is a proper way [1] to skip the dict check-code check. Let's use it.
[1]: a61ed1c2d7
Reviewed By: DurhamG
Differential Revision: D7831336
fbshipit-source-id: a5e654e9e94cbfb1c5a07b047eb6e5451904c48e
Summary: Our CI didn't catch when I landed the previous diff.
Reviewed By: singhsrb
Differential Revision: D7834066
fbshipit-source-id: a51c2a294ea550917836f8b1eede2570838b60b7
Summary:
This would make tests run on treedirstate.
To avoid issues with Eden pulling from a non-eden treedirstate repo,
treedirstate is changed to be "always on" and disables itself on an eden repo.
The extension list is changed to a set for efficient `__contains__` test.
Reviewed By: phillco
Differential Revision: D7769804
fbshipit-source-id: d328fe51ef67c4730cfc53f43bdfc48c2765c541
