Summary:
In a previous diff we added logic to delete corrupt packs in the cache
directories, but the logic only caught issues after the pack was loaded. We also
need to catch issues while loading the pack. This happens when a pack file is 0
length.
Reviewed By: quark-zju
Differential Revision: D8324248
fbshipit-source-id: c2fcb4f69c4627cce316d626e91f752ef562a0d9
Summary:
We use lz4 compression in production. Regarding on zstd, our approach would
be using the Rust `lib/zstdelta` library. So there is no need to keep the
Python binding.
This makes `make local` faster and also makes internal code search about
zstd cleaner.
Reviewed By: DurhamG, phillco
Differential Revision: D8302882
fbshipit-source-id: a6c34d6fea59140caeac158274388ba75a28fb29
Summary:
The reason caching is disabled seems to be that symlinks can change where
paths points to, while some properties of that path are cached.
Sometimes we know that symlinks are not used in the repo, so we'd like hg
to let us shoot ourselves in the foot.
The motivation here is that I have a specially crafted rebase operation,
which makes on the order of 400000 stat-like syscalls without caching and
on the order of 6000 with caching. This can save ~20s on Windows.
Note that this setting is disabled by default so we can't break anything.
FWIW, we already have an unsafe config option, which allows us to opt into the "we don't do crazy stuff with symlinks" behavior, it's called `unsafe.filtersuspectsymlink`, setting it to false speeds up Windows `status` and we do so for `ovrsource` on Windows.
Reviewed By: quark-zju
Differential Revision: D8297129
fbshipit-source-id: ec751d18b50b042f28f68c9e1a3634ab94a4e7c3
Summary:
An updated version of D8174246 with a test. Max memory usage is very useful to us to understand which commands are using too much memory, and the existing wrapper metrics don't work when chg is used (they measure chg's usage).
Windows support will come later as it is a bit more involved.
Reviewed By: DurhamG
Differential Revision: D8318584
fbshipit-source-id: 323450bc7ab376014d70106beb5d4fdcc7fba0c8
Summary:
`hgsubversion` and `lz4revlog` were moved to `hgext`. So the top-level
directories can be removed.
Reviewed By: DurhamG, phillco
Differential Revision: D8302880
fbshipit-source-id: 3b783b97f5d77585d406c9e405b4dffe17dd284e
Summary:
In the new corruption fix strategy, we're only going to strip the
entries from the root treemanifest revlog. This will leave all the
still-referenced sub-trees available for fetching if clients still have old
references to them, while preventing new references from being taken from the
root nodes.
Reviewed By: phillco
Differential Revision: D8284287
fbshipit-source-id: 4cc20bfdbf7ae5d32074dae22da4d37717ee5728
Summary:
Needed for windows to allow us write to a tempfile created in Rust. Should do it in a different way?
Context: we create a NameTempFile from Rust that we pass as argument to hg as profiling.output=named_temp_file. When open with 'w', the mercurial command fails with "abort: Permission denied". As far as I read, it might be because of how the Win32 API works. Under the hood, Python's open function is calling the CreateFile function, and if that fails, it translates the Windows error code into a Python IOError. From CreateFile documentation:
If CREATE_ALWAYS and FILE_ATTRIBUTE_NORMAL are specified, CreateFile fails and sets the last error to ERROR_ACCESS_DENIED if the file exists and has the FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM attribute. To avoid the error, specify the same attributes as the existing file.
I might be wrong though... any other suggestions?
Reviewed By: ikostia
Differential Revision: D8099420
fbshipit-source-id: c3077cc5d7bc03c8f2eeafdac3467db62c20a669
Summary:
See t30174052 -- new version of FB git seems to have trouble pushing to other dirs. Let's turn these tests off for now to resume the contbuild.
Disable test-hggit-branch-bookmark-suffix.t too.
Reviewed By: DurhamG
Differential Revision: D8306205
fbshipit-source-id: b8f10f0158e38608ed5b6a96bd8da52aa211f8d7
Summary:
Template function requires input arguments. They do not always exist.
If any argument is missing, just return an empty value, instead of crash.
Reviewed By: mitrandir77
Differential Revision: D8221246
fbshipit-source-id: ff3839398bdc620bff61d1b19b1512010c06388e
Summary:
Use the command-level template and the `nodechanges` template keyword.
Drop the use of formatter.
`-Tjson` is not treated specially so it will just print `json`. If we'd like
to make it output `{"nodehcnages": ...}` (probably not a list with one
element), it's doable by having some mapping from `json` to a default
template.
Reviewed By: mitrandir77
Differential Revision: D8221076
fbshipit-source-id: 75575a98324d0d069789e0e915d5aa282ca0d4bb
Summary: Will be used by the next change.
Reviewed By: mitrandir77
Differential Revision: D8221077
fbshipit-source-id: c9713175dfaab0b090f81d2c9d4f4d9d63a0771c
Summary:
There mercurial template language is a powerful language. But it has some
limitations right now:
- It's focused on "commit". If there is no commit, things become harder.
- The related formatter concept is coupled with "a list of things". If there
is no list, things become impossible.
- Some commands use formatter/template partially, and that does not play
very well with `ui.write`, and is difficult to document.
This patch starts to solve all the issues, by introducing command-level
template state, and with the vision:
- Anything could be used as an "environment" passed to the template
language, a commit, a file, a line, a "command" with customized
properties, etc.
- If a template keyword function is incompatible with the environment, it
should fail gracefully.
- The template language is a super set of other things. `-Tjson` should not
be magical, but an alias of `{some complex expression | json}`.
- Discourage the use of formatter.
- Template keywords can be registered and documented.
This would remove the limitations of the template language and unleash its
full potential.
Note there are still some cases where the formatter is desirable. For example,
`hg log` with a large revset should be able to stream its output. The command
template feature added right now only renders things at the end of a command,
and is not (and requires significant effort to be) streaming-friendly.
Reviewed By: mitrandir77
Differential Revision: D8221078
fbshipit-source-id: 26fed50025c4676af277686382ce2d36f1faed97
Summary:
The added method takes an environment (`props`), and a program written in
the template language, execute the program, and pass output to `ui.write`.
The API is similar to the upstream [1]. But it avoids `ctx` intentionally.
Since the "environment" should be more flexible than having to have a `ctx`.
[1]: aa32940279
Reviewed By: mitrandir77
Differential Revision: D8221079
fbshipit-source-id: 94ee5c563f943330ef91966a4883c5733cc8dac9
Summary:
According to scuba chart https://fburl.com/scuba/mn5nfdna, these are
not used by any critical client, don't seem useful and can be removed.
Reviewed By: quark-zju
Differential Revision: D8232616
fbshipit-source-id: ae07cfb00bfe45cd6f8c31f3fa0871c9ba1fdadf
Summary:
The pull request [1] is still open, which means `quickcheck::rand` is still
private when building with `cargo`. It only works with a patched quickcheck.
We cannot revert D8234503 since that will break buck build. So there is no
choice but upgrade to rand 0.5.
[1]: https://github.com/BurntSushi/quickcheck/pull/204
Reviewed By: DurhamG
Differential Revision: D8297404
fbshipit-source-id: 19937c49ae96a39e326b1b54eb00e6e2944193c2
Summary: Implement most methods, similar to treedirstatemap.
Reviewed By: markbt
Differential Revision: D7909176
fbshipit-source-id: d20e4642ee381cc1cfe7c774071c8c3c1e493f45
Summary:
Add a class with methods defined in `dirstate.dirstatemap` so it could be
used as a drop-in replacement. The actual logic is not implemented yet.
Reviewed By: markbt
Differential Revision: D7909170
fbshipit-source-id: 5c1ae1bf4341d6368f5c9824487e7702f93043a7
Summary: The Ubuntu and Windows builders have an older rustc that doesn't support this syntax.
Reviewed By: DurhamG
Differential Revision: D8301570
fbshipit-source-id: 56990a804053a4dc78e41789c7b577bcf82868d7
Summary:
Usually the handshake process is pretty quick (<0.01 seconds):
chg: debug: 0.000148 try connect to ...
chg: debug: 0.000338 connected to server
chg: debug: 0.000359 initialize context buffer with size 4096
chg: debug: 0.008225 hello received: ...
chg: debug: 0.008269 capflags=0x7b03, pid=31941
chg: debug: 0.008282 request setprocname, block size 17
chg: debug: 0.008316 request attachio
chg: debug: 0.008978 response read from channel r, size 4
chg: debug: 0.009045 request chdir, block size 45
chg: debug: 0.009092 version matched (6119653365548183087)
However, we have seen some OSX cases where the handshake and basically
everything takes much longer:
chg: debug: 0.000139 try connect to ...
chg: debug: 0.000297 connected to server
chg: debug: 0.000321 initialize context buffer with size 4096
chg: debug: 0.192316 hello received: ...
chg: debug: 0.192362 capflags=0x7b03, pid=55634
chg: debug: 0.192373 request setprocname, block size 17
chg: debug: 0.192420 request attachio
chg: debug: 0.229009 response read from channel r, size 4
chg: debug: 0.229072 request chdir, block size 34
chg: debug: 0.229111 version matched (6119653365548183087)
(See P59677258 for the full paste)
If restart the chg server, the problem goes away and commands will be fast
again.
Unfortunately I'm not sure about the root cause of the problem. Maybe it's
Python's GC doing something very expensive? Maybe it's OSX thinking the server
process is "inactive" and put it to some state that's very slow to recover? Or maybe
it's some weird 3rdparty service?
For now, what we do know are:
- The slowness *sometimes* reproduces with chg.
- The slowness goes away if chg server is restarted.
As a last resort, detect the slowness by measuring the handshake time, then
restart the server accordingly. To avoid an infinite restart loop on slow machines,
the restart can only happen once.
The threshold is set to 0.05 seconds, which is roughly 5x the normal value, and
can be disabled by `CHGSTARTTIMECHECK=0`.
Reviewed By: phillco
Differential Revision: D8294468
fbshipit-source-id: 75246ea4d872045664e7feadb0acc47dfa1d8eae
Summary:
It could affect pushrebase behavior - pushrebase server might have a large
set of "missing" nodes, and take that as the commits to rebase. Then fail
with "cannot rebase public commits".
Maybe there is a better way to fix it so it only affects pull but not push.
But for now it seems safer to disable it for automation.
Reviewed By: DurhamG
Differential Revision: D8049229
fbshipit-source-id: 3612e2fb2c1c825e8974156c27b03966ee745c98
Summary:
They're actively fighting against the clang-format config
and don't have an auto-fix.
Reviewed By: quark-zju
Differential Revision: D8283622
fbshipit-source-id: 2de45f50e6370a5ed14915c6ff23dc843ff14e8a
Summary:
Previously, when adding temporary pack files to the local data store,
we added them at the end. This causes problems with the recently added ondemand
generating datastores, since when we go down the line of stores we will hit the
generating store (which will create a tree from flat manifest, which is slow)
before we hit the bundle store that actually contains the data.
Let's reorder the stores so the bundle stores go at the front, since they are
likely to be used.
Reviewed By: phillco
Differential Revision: D8291415
fbshipit-source-id: 4f9799cf755b2b1f7fe8d8d1c041a3b3f2c7e463
Summary:
The windows and ubuntu builds don't have a version of rust that
supports these features, so this breaks the build.
Reviewed By: phillco, quark-zju, singhsrb
Differential Revision: D8289651
fbshipit-source-id: d08b141b4d9996e3b899ac0604225ad34f863990
Summary:
Optimized builds were giving a warning about shadowed variables, which
broke some of our continuous builds. Let's fix those.
Reviewed By: phillco
Differential Revision: D8287865
fbshipit-source-id: 8a2d023be93499ed6458a35ba8e8cb4d83d59e4a
Summary:
I've been troubleshooting eden integration test failures on my
devserver and traced it to some slightly off behavior in the telemetry
wrapper.
The wrapper was setting `CHGHG` to `hg.real` rather than the computed
path to the `hg.real` executable. In the eden integration tests this
path is the buck generated `hg.par`. The problem this caused was running
the installed hg.real rather than the one from the test environment
and this caused resolution of the eden extension to fail.
Once I fixed that up I found that chg had detected a problem with the
paths to the hg executable that were being used; we were picking up `chg`
from the system path and had a similar issue to above.
I introduced an environmental variable `CHG_BIN` to hold the desired path
and set it to the buck built `chg` binary.
In the process of this I found that `chg` was triggering a UBSAN issue
by passing a nullptr as the second argument to `memcpy`. I've included
the trivial fix for that in this diff also.
Reviewed By: quark-zju
Differential Revision: D8274636
fbshipit-source-id: 7ee0740cbfb447ab41b9e08308767d42790ba296
Summary:
We want to better understand why people are getting "scanning working copy", so this is a first step towards that -- log every time we call `invalidate()` with the reason.
If one reason ends up dominating the others (e.g., "exception") we can drill down later.
Reviewed By: quark-zju
Differential Revision: D8272450
fbshipit-source-id: 46cb963eefa5ca60df81bdcd8f1e232855c106b7
Summary:
Previously, if a client sent a gettreepack request and specified a
non-root directory, the server would go back to the root of the tree and send
the whole tree. Now that treemanifest has a walksubdirtrees function for
iterating over a specific subset of a tree, we can make it return just the
subset.
This diff does delete the treememoizer class. This was previously used to keep
trees in memory if they would be needed again (for instance, as a base node for
a future tree). We aren't able to provide this functionality when using
walksubdirtrees, because python doesn't have access to the underlying sub
directory tree objects. This class was probably unnecessary anyway, since we
usually only request a small number of trees at once.
Reviewed By: quark-zju
Differential Revision: D8217360
fbshipit-source-id: 1db970659eabe2829acc6d49a15eeb9345ce4a5b
Summary:
In a future diff we'll be adding the ability to request just a
subsection of a tree. In order to do that, we need to expose an api that can
iterate of just a subsection of the tree.
Since the treemanifest object itself acts mostly like a dictionary, and doesn't
expose sub-tree functionality, we expose sub-tree walks as a limited use class
level function.
Reviewed By: quark-zju
Differential Revision: D8216726
fbshipit-source-id: 0d72ef041fd6a973387f975b21e55c82f7ec789c
Summary:
A recent change renamed the methods assuming they were rust methods
but these methods are used in the python code and therefore, already had the
correct naming. This broke the one of the python tests calling the methods
which will pass now.
Reviewed By: DurhamG, quark-zju
Differential Revision: D8279742
fbshipit-source-id: 01761ad033c6669a0ae54dc1beb853a0d56df27d
Summary:
just refactoring to improve the code quality
the main improvement is that I separated TcpReceiver to a different service,
any other services can register callbacks with TcpReceiver service.
For WorkspaceSubscriberService callbacks are implemented using mpsc channel to notify the main WorkspaceSubscriberService thread and single atomic flag that allows running subscriptions to join.
Another improvement is that I added logic to run cloud sync on the first keep alive after connection errors
Reviewed By: markbt
Differential Revision: D8226109
fbshipit-source-id: 3fe513da9273b28b2262948ecdf620821e7ab313
Summary:
Use the new stack-by-stack commit backup for infinitepushbackup as well as
commitcloud.
Reviewed By: DurhamG
Differential Revision: D8241380
fbshipit-source-id: 0c1481ad1ab5759ab642d0b02c7fe08523056292
Summary:
Rather than pushing a bundle containing all unpushed draft commits, instead
push stack by stack. This reduces the individual bundle size, and means
failure to push some commits means we can still make progress.
Reviewed By: DurhamG
Differential Revision: D8241383
fbshipit-source-id: 6f1528613fa228627fe156a8e4720dd46929ebe2
Summary:
Use the repo connectionpool to connect to the server. This allows sensible
re-use of connections whilst gracefully handling errors.
Reviewed By: DurhamG
Differential Revision: D8241382
fbshipit-source-id: f43669a32e4b1ff57193ea0959c57ff14526a998
Summary:
Split the pushing of commitcloud commits and infinitepush bookmarks into
separate functions, in preparation for pushing commits stack-by-stack.
Reviewed By: DurhamG
Differential Revision: D8241385
fbshipit-source-id: dc82c3470259b0ff738ebcdf6901ee8052405379
Summary:
Refactor pushing of draft heads into infinitepush; we want to call this from
infinitepushbackup, too.
Reviewed By: DurhamG
Differential Revision: D8241381
fbshipit-source-id: d5eb06efd6cd9382e016b0738d9b3e89e524de53
Summary:
Remove the cleanup from pushbackupbundle. As we now wait for the bundle
response, this is no longer necessary, and it prevents re-use of the
connection.
Reviewed By: DurhamG
Differential Revision: D8241379
fbshipit-source-id: a885198c49dce1beb1385a61a209d299ed3d2d02
Summary:
Split the infinitepush backups code into a separate module.
The infinitepush extension is responsible for providing scratch pushes, as well
as the server-side implementation for storing scratch bundles.
The infinitepushbackup extension uses this feature to provide automatic backup
support. It is an alternative to the commitcloud extension, and the two will
be mutually exclusive.
Reviewed By: DurhamG
Differential Revision: D8204418
fbshipit-source-id: 9279c426da35d451a5e7b39a72f2c91d364e3fd4
Summary:
Re-using long-running connections can cause problems, as the `hg serve` process
on the server may be using large amounts of resources. Add a new configuration
option `connectionpool.lifetime`. Connections that are older than that time
will be closed and not re-used.
Reviewed By: DurhamG
Differential Revision: D8204415
fbshipit-source-id: ac937778e3dc79e77e367017db2141066bd17263
Summary:
build rust binaries as part of mercurial build
by default they appear together with hg and chg in the folder called scripts but it can be configured.
Reviewed By: quark-zju
Differential Revision: D8270436
fbshipit-source-id: df928b89e6530c285c3eab438769c53f545d4098
Summary:
Without this change, code doing `index.get(...).values().collect()` might
end up with an infinite loop.
Reviewed By: DurhamG
Differential Revision: D8156510
fbshipit-source-id: 5497aa354de7d49cfc4308a025856608ce981a1e
Summary:
Previously, the index API optionally takes a root offset. This is
inconvenient for the caller since they probably need to record both
valid file length and root offsets. Since root nodes are always at
the end of the index. Let's just simplify the API to take a logical
file length instead of a root offset.
Reviewed By: DurhamG
Differential Revision: D8156512
fbshipit-source-id: 7029272a61c9990e6484bca7ebbff64e2233c6cd
Summary:
Previously, `mmap_readonly` always reads file length, and uses that for mmap
length. In many cases we do know the desired file length and it's cleaner to
not `mmap` unused bytes. So let's add a parameter to do that.
Note: The `stat` call is still needed. Since `mmap` wouldn't return an error
of the requested length is greater than the file length.
Reviewed By: DurhamG
Differential Revision: D8156523
fbshipit-source-id: 991aa28f3542eaff24387dcc6a7302122fb6962f
Summary: The function will be reused in another module.
Reviewed By: DurhamG
Differential Revision: D8156522
fbshipit-source-id: 2aff6f2e4b8fc9b5d2c000e12ac2d940f7fab407
