Summary:
Similar to D29404057 (cedddd1c8d), add a way to disable resolving IDs by setting
a limit using `EDENSCM_REMOTE_ID_THRESHOLD`.
Reviewed By: andll
Differential Revision: D29440143
fbshipit-source-id: 30409089493ae2cd5c189e37b0d4f88df9a6d8e8
Summary: The Rust NameSet does not have nullid. Do not bother resolving remotely.
Reviewed By: kulshrax
Differential Revision: D29437983
fbshipit-source-id: f1eb73e738281b3b5096fba22ba92833f5a4f3ee
Summary:
The remotenames information might be out of sync with the DAG.
Use the DAG's master group's head for the pull fast path "old" master so that
even if the remotenames is stale for some reason, the fast path still works.
Reviewed By: andll
Differential Revision: D29434719
fbshipit-source-id: 7bbd0757dc6a9428f86663ac4989f7c30b365b46
Summary: Will be used by the next change.
Reviewed By: andll
Differential Revision: D29434722
fbshipit-source-id: 74dbec506fb0985379480815380118cd41058aec
Summary:
If a user Ctrl+C-ed (or something else interrupts) between flushing changelog
and flushing metalog (remotenames). The pulled commits will exist in the local
graph and break the next time using the fast path.
Reviewed By: andll
Differential Revision: D29434720
fbshipit-source-id: e7ca7542691279644679effb06707a1d305541af
Summary:
This is a better practice. It avoids debug messages from other places (ex.
discovery).
Reviewed By: andll
Differential Revision: D29434721
fbshipit-source-id: 5990f0174e9b61aed7d9d56252df63b571364070
Summary:
The remotenames has logic to work with legacy clones to write the remotenames
state. However, modern clone paths take care of remotenames directly so there
is no need to do an extra pull by remotenames.
Reviewed By: DurhamG
Differential Revision: D29428883
fbshipit-source-id: a73c0ee716b09f4e34d6fa30997f961284678d13
Summary:
This makes it pick up some `%include` configs and can avoid some surprises.
Also remove the manual `paths.default` setconfig. That seems to cause
`destrepo.edenapi` being `None` and break lazy changelog clone paths.
Reviewed By: DurhamG
Differential Revision: D29428882
fbshipit-source-id: bafeb195a560e35be17355b793613b60d97fbecf
Summary: Make dynamicconfig pick up the repo name because hgrc gets written.
Reviewed By: DurhamG
Differential Revision: D29428878
fbshipit-source-id: fbf578cd7c770a4541fff3b85ff40c40cd5a6cc5
Summary: This will be used by the next change.
Reviewed By: DurhamG
Differential Revision: D29428879
fbshipit-source-id: 69e0ffac12fb9c442488d59ea8faa0ea4b47a2c1
Summary: This makes it easier to figure out a traceback using an empty repo name.
Reviewed By: DurhamG
Differential Revision: D29428881
fbshipit-source-id: 95a09c691e3d921ad4f960a39002f71ec879d927
Summary:
This makes it easier to debug wrong reponame issues.
In theory those need to be checked and the config needs to be regenerated if
they are changed for correctness. Given that username and reponame are rarely
changed, I saved it for later.
Reviewed By: DurhamG
Differential Revision: D29428880
fbshipit-source-id: f996af6a7a1e329faaa8b0a53dac8621fa94dac8
Summary: Left over of D29404901 (410769c529).
Reviewed By: DurhamG
Differential Revision: D29429146
fbshipit-source-id: b37c89745d924efc28110d8b96e9b51162b6570b
Summary: This avoids issues in assertions used by metadataonlyctx.
Reviewed By: DurhamG
Differential Revision: D29430672
fbshipit-source-id: 4de305d62f65a9d6c4fb2e3f375ef20d9d94d41c
Summary:
We do not respect `$PAGER`, and the default is the internal streampager.
Also document streampager config options.
Reviewed By: markbt
Differential Revision: D29412598
fbshipit-source-id: 706f09f8805e9067b46eec7d83c6742b61084b09
Summary:
See also D29400532 (909411bb1c). It turns out that it might be more desirable to just mix
stdout and stderr streams in streampager. For example, having them mixed then
the graph log output can show what network fetches or calculations are done
before outputting the graph lines. This is also more consistent with the
vanilla terminal (no pager) behavior.
Reviewed By: markbt
Differential Revision: D29412531
fbshipit-source-id: c07f68b12498a7cee6152bbecbb58d5a7e64097a
Summary: Move the scmstore implementation from the `specialized` module to the root of the `scmstore` module.
Reviewed By: kulshrax
Differential Revision: D29405779
fbshipit-source-id: ae2ef9cc05337a0ff81f5ba5b7051792207fee82
Summary: `scmstore` is dead, long live `scmstore`.
Reviewed By: kulshrax
Differential Revision: D29405613
fbshipit-source-id: 3252a545f5b944d14c15b2a777b84a99a2d4c293
Summary: Update unit tests in `revisionstore::indexedlogdatastore` to use new scmstore instead of old scmstore.
Reviewed By: kulshrax
Differential Revision: D29405258
fbshipit-source-id: 3d2e8cd313dbe66a257433702402804f490bdf47
Summary:
Update unit tests in `revisionstore::edenapi::data` to use new scmstore. There's not really a wrapper to exercise anymore for edenapi specifically, so it's probably better to just make these `scmstore` unit tests instead of edenapi (or indexedlogdatastore as in the next change)-specific.
For ease of unit testing, make fetch_logger optional and introduce `empty` constructor.
Reviewed By: kulshrax
Differential Revision: D29397495
fbshipit-source-id: d7ef0df16cf83a2506606c55c78fcbfa684904d7
Summary:
The server is expected to provide head (of all segs), parents (of each seg),
roots (of all segs). We checked roots and parents but only check head in debug
build. Let's check head in release build too.
Reviewed By: andll
Differential Revision: D29405816
fbshipit-source-id: 1a97eb52a9a0d1d444ae5dabd1a01f0786be9fa9
Summary:
The external grep command (originally from fb-hgext's tweaks of grep command)
was run without redirecting its stdout. That could be problematic for
streampager, because writing directly to stdout will mess up with streampager
interface.
For example, the following command will show "nothing", while it should print
something:
fbcode/antlir/fbpkg/build % lhg grep -l scuba --config pager.pager=internal:streampager --config grep.usebiggrep=False --config pager.interface=full
Fix it by using a PIPE and forward the external command's output to streampager
buffer properly.
Reviewed By: andll
Differential Revision: D29408821
fbshipit-source-id: 4b2a0a6bbd64aa00d09f921d830b173cc56ae630
Summary:
Found by xavierd. Recent `os_info` bump now detects CentOS as OracleLinux.
Workaround it to keep our repo functional.
Reviewed By: xavierd
Differential Revision: D29410415
fbshipit-source-id: 1bd8183f46e3c2265aef119e9f96d9d05a5dbae6
Summary: Previously, hg would always try to fetch data via EdenAPI, even when doing so wouldn't make sense (for example, a local test repo). The only time it makes sense to try to use EdenAPI at all is when the repo in question is supported by Mononoke. As a quick heuristic, let's disable EdenAPI if `paths.default` is not set to a `mononoke://` URL.
Differential Revision: D29372739
fbshipit-source-id: b3601e529ed44c355cb192aedf4332317f4e3132
Summary: Add an option to allow manually forcing EdenAPI to be enabled or disabled. This is useful in a variety of cases, such as bypassing the normal EdenAPI activation logic in tests, or to forcibly disable EdenAPI in cases where it isn't working correctly.
Differential Revision: D29377923
fbshipit-source-id: f408efe2a46ef3f1bd2914669310c3445c7d4121
Summary:
Users get used to hints at the end of the output, not in a separate panel in
streampager. Make it so.
Reviewed By: andll
Differential Revision: D29400532
fbshipit-source-id: 41dc5d20f44a315bc0235ca3cb7857ae8955d3ad
Summary: This will be used by the next change.
Reviewed By: andll
Differential Revision: D29400533
fbshipit-source-id: e6b90bedd8d8a6cf9452dfb5c5f14f9980e12f62
Summary:
More straightforward way of D29404055 (e6ea02372c). Return the non-lazy set directly from
Rust. This avoids some overheads.
Note: ignoring whitespace will make reviewing easier.
Reviewed By: andll
Differential Revision: D29404901
fbshipit-source-id: 02e4766256863fe3fe258bcb318473355cd1efe4
Summary: This was used to narrow down issues.
Reviewed By: andll
Differential Revision: D29404054
fbshipit-source-id: 3bfdac332d63bdb13f40d5cf23dacec242b46d52
Summary:
Running `hasnode` in a loop is inefficient with lazy changelog.
Update it so it does batch filtering and does not ask remote service.
With the change, BackupState.heads will only contain locally known heads.
So the extra filtering is removed.
Reviewed By: andll
Differential Revision: D29404056
fbshipit-source-id: 327bd4c292ad75fd14a47587482440b6ece1d2d5
Summary:
Previously, the obsolete() set (from repo.dageval) is a "meta" (lazy) set that
might trigger frequent remote lookups. Calculate the obsolete() set into a
static set form so it does not trigger remote lookups.
This avoids expensive remote lookups in:
File "/data/users/quark/fbsource/fbcode/een/scm/edenscm/hgext/smartlog.py", line 432, in <lambda>
hidenodes = repo.dageval(lambda: obsnodes - heads(obsnodes) - roots(obsnodes))d
This issue was found out by setting env vars:
EDENSCM_LOG=dag::protocol=debug
to find the commit hashes, and:
EDENSCM_DISABLE_REMOTE_RESOLVE=a8209cbe13bcdc42d4999ffaa15dc9881660d5d,76d91d23d583f92b6c909b6747db1b75229d5be5
to find the traceback.
Reviewed By: andll
Differential Revision: D29404055
fbshipit-source-id: cc8bdd5347b38c1681565455874ed22f1a87321e
Summary:
With lazy changelog, for every commit hash that is unknown to the repo, it
needs to be resolved remotely. For commit hash that is also unknown to the
remote server, it can be bad because we don't have cross-process negative
caching and will trigger the remote resolution over and over. Practically,
the solution is to avoid remote lookup if the "minor" correctness issue
is acceptable (ex. D29111710 (ac6c6cf3fa), D29114049 (880b5c3cd8)).
That has been tricky to debug - the remote fetching happens in Rust, and
cannot be easily inserted something like `import ipdb; ipdb.set_trace()`
like Python. I have been inserting sleeps and use gdb to understand the
call stack when writing the above 2 diffs.
This diff makes debugging easier by supporting "break point" setting by
environ variables. So one can do something like:
$ EDENSCM_LOG=dag::protocol=debug hg ...
DEBUG dag::protocol: resolve names [435c235d65ccc4f95595d74478a617450c96c2e] remotely
$ EDENSCM_DISABLE_REMOTE_RESOLVE=435c235d65cc4f95595d74478a617450c96c2ec hg ... --traceback --debugger
Reviewed By: andll
Differential Revision: D29404057
fbshipit-source-id: d8a631f279f32e2ee88f097796cdc85d8ca27b77
Summary:
The logic was wrong. If local is True, vertexes is a filtered list, while nodes
isn't. They cannot be mix-used.
Reviewed By: andll
Differential Revision: D29403466
fbshipit-source-id: 06f0853282b0f284d5b2970bfa37589ef59f6169
Summary:
We don't need the full precision of execution time in the telemetry, thus
reduce the precision before logging it.
Reviewed By: kulshrax
Differential Revision: D29384354
fbshipit-source-id: cab02ab4db6b260525e56e0395ca1071fdb4b442
Summary:
Straightforward conversion of native checkout implementation from old scmstore API to new (non-async, batched) scmstore API.
We'll meet again someday soon, async.
Reviewed By: andll
Differential Revision: D29321512
fbshipit-source-id: 1e3e0d92c95730a5c2df610061f6faf5b1eb9068
Summary: The returned value now includes roots. Rename the function to clarify.
Reviewed By: kulshrax
Differential Revision: D29383072
fbshipit-source-id: 02a255ce20d9797f482f6fe1c716f2d79a12d4e0
Summary: There is a regression in 1.7.0 (which we're on at the moment) so we might as well update.
Reviewed By: zertosh, farnz
Differential Revision: D29358047
fbshipit-source-id: 226393d79c165455d27f7a09b14b40c6a30d96d3
Summary:
Use segmented changelog fast path during master branch pull
This change is gated by a config
Reviewed By: quark-zju
Differential Revision: D29347804
fbshipit-source-id: f0cc7633e67fb04266ce15506375873d58ea77fc
Summary:
Make it possible for Python to detect the error and fallback to slow path
accordingly.
Reviewed By: andll
Differential Revision: D29369067
fbshipit-source-id: 1de6162532b145469c09070bdf843ec96d38e431
Summary:
An alternative to D29363808 (e396cab669). The benefit is that parents_and_head is used by
both the client and the server. So we don't need to duplicate D29363808 (e396cab669) in
Mononoke code.
Reviewed By: andll
Differential Revision: D29365079
fbshipit-source-id: bca60ba2b3df477929d8e72b2363e5a0f744b35d
Summary:
Pull fast path uses `reload` which drops pending changes.
To avoid misuse, raise an error if pending changes are present.
Reviewed By: andll
Differential Revision: D29363799
fbshipit-source-id: 8f520d2c5553432abc452bc7b2b59d7af80e0a99
Summary:
The import pull data logic used low-level locking, persisting APIs, it does not
write cached idmap to disk. So we need to manually insert the idmap remote
lookup result to the actual local idmap explicitly.
This addressed an issue that verify_missing fails in the pull fast path.
Reviewed By: andll
Differential Revision: D29363813
fbshipit-source-id: 2749855a6c8c924bd1b772173de066d400f73764
Summary:
For a NameDag, `IdConvert` on `self.map` cannot resolve names remotely, but
`IdConvert` on `self` can. Use the latter. This is similar to D27547584 (af3c3b3fd0) where
some `self.map` are updated to `self`.
This addressed an issue found in the pull fast path test. Note there is another
issues to solve.
Reviewed By: andll
Differential Revision: D29363810
fbshipit-source-id: 28ba583ed14bbc5d52af81d4128d965f24eef011