Summary:
If the repo changelog was migrated to lazy, existing processes including the
import helpers might start failing to resolve commits. Detect that, and request
a restart of the import helper.
An example traceback:
edenscm.mercurial.error.RepoLookupError: unknown revision 'ce48ccc4197b2066a8c7f2a930cee89e8aea6379'
2021-06-29 12:33:12,738 error processing command 4
Traceback (most recent call last):
File "edenscm/mercurial/commands/eden.py", line 363, in process_request
cmd_function(req)
File "edenscm/mercurial/commands/eden.py", line 604, in cmd_manifest_node_for_commit
node = self.get_manifest_node(rev_name)
File "edenscm/mercurial/commands/eden.py", line 821, in get_manifest_node
return self._get_manifest_node_impl(rev)
File "edenscm/mercurial/commands/eden.py", line 790, in _get_manifest_node_impl
node_hash = ctx.manifestnode()
File "edenscm/mercurial/context.py", line 569, in manifestnode
return self._changeset.manifest
File "edenscm/mercurial/util.py", line 987, in __get__
result = self.func(obj)
File "edenscm/mercurial/context.py", line 540, in _changeset
return self._repo.changelog.changelogrevision(self._node)
File "edenscm/mercurial/changelog2.py", line 374, in changelogrevision
return changelogrevision(self.revision(nodeorrev))
File "edenscm/mercurial/changelog2.py", line 463, in revision
p1, p2 = self.parents(node)[:2]
File "edenscm/mercurial/changelog2.py", line 610, in parents
parents = list(self.dag.parentnames(node))
error.CommitLookupError: '<int> cannot be found'
Reviewed By: andll
Differential Revision: D29470181
fbshipit-source-id: 40ad6eebe1f656c63d3f96b2b70ad60b350eed21
Summary:
Previously the None edenapi error isn't great for understanding what's going on:
File "edenscm/mercurial/changelog2.py", line 123, in _openhybrid
inner = bindings.dag.commits.openhybrid(
TypeError: Expected type that converts to client but received NoneType
Let's make edenapi non-nullable by default, with nicer error messages, and make
places using the nullable edenapi opt-in explicitly.
Example errors:
% lhg sl --config edenapi.url=
abort: edenapi is required but disabled by missing edenapi.url config
% lhg sl --config edenapi.enable=
abort: edenapi is required but disabled by edenapi.enable being empty
% lhg sl --config paths.default=/tmp
abort: edenapi is required but disabled by paths.default being 'file:'
% lhg sl --config paths.default=ssh://x --config ui.ssh=dummyssh
abort: edenapi is required but disabled by paths.default being 'ssh:' and dummyssh in use
Reviewed By: kulshrax
Differential Revision: D29462258
fbshipit-source-id: 84fab5a7cce837c52ccb8108d1f4fe44ef2b6d1e
Summary:
D29372739 (ef79c2d811) caused issues that repo.edenapi becomes None due to scheme being ssh
(hit by markbt) or fb (hit by StanislavGlebik). Given that it's just legacy tests
that do not want EdenAPI, let's use a blacklist for legacy tests, instead of a
(fragile) whitelist for where EdenAPI should be enabled.
Reviewed By: kulshrax
Differential Revision: D29461305
fbshipit-source-id: 4a0317747d9ee3412ffdfdb25dc0d8383305a25c
Summary:
To construct a lazy changelog, the repo object needs to have `name` and `edenapi`. Implement them.
Besides, print why changelog fails to open with `--debug` or `--traceback`.
Reviewed By: andll
Differential Revision: D29468005
fbshipit-source-id: 5815e0d9313dac897f20fa32668cae05ba90c727
Summary: Looks like another test needs updating after D29437983 (5b1b1febc1).
Reviewed By: quark-zju
Differential Revision: D29446063
fbshipit-source-id: c71684609830b6ec047a2efc615b806dd2a0066b
Summary:
With lazy pull fast path, the legacy commit scan will trigger one-by-one id
resolution, which is terrible and defeat the purpose of laziness. Let's just
force graphql path instead.
Reviewed By: andll
Differential Revision: D29440142
fbshipit-source-id: 4b0d10e6f5a3eb0533b2d95b1889a1f1c1f281cb
Summary:
Similar to D29404057 (cedddd1c8d), add a way to disable resolving IDs by setting
a limit using `EDENSCM_REMOTE_ID_THRESHOLD`.
Reviewed By: andll
Differential Revision: D29440143
fbshipit-source-id: 30409089493ae2cd5c189e37b0d4f88df9a6d8e8
Summary: The Rust NameSet does not have nullid. Do not bother resolving remotely.
Reviewed By: kulshrax
Differential Revision: D29437983
fbshipit-source-id: f1eb73e738281b3b5096fba22ba92833f5a4f3ee
Summary:
The remotenames information might be out of sync with the DAG.
Use the DAG's master group's head for the pull fast path "old" master so that
even if the remotenames is stale for some reason, the fast path still works.
Reviewed By: andll
Differential Revision: D29434719
fbshipit-source-id: 7bbd0757dc6a9428f86663ac4989f7c30b365b46
Summary: Will be used by the next change.
Reviewed By: andll
Differential Revision: D29434722
fbshipit-source-id: 74dbec506fb0985379480815380118cd41058aec
Summary:
If a user Ctrl+C-ed (or something else interrupts) between flushing changelog
and flushing metalog (remotenames). The pulled commits will exist in the local
graph and break the next time using the fast path.
Reviewed By: andll
Differential Revision: D29434720
fbshipit-source-id: e7ca7542691279644679effb06707a1d305541af
Summary:
This is a better practice. It avoids debug messages from other places (ex.
discovery).
Reviewed By: andll
Differential Revision: D29434721
fbshipit-source-id: 5990f0174e9b61aed7d9d56252df63b571364070
Summary:
The remotenames has logic to work with legacy clones to write the remotenames
state. However, modern clone paths take care of remotenames directly so there
is no need to do an extra pull by remotenames.
Reviewed By: DurhamG
Differential Revision: D29428883
fbshipit-source-id: a73c0ee716b09f4e34d6fa30997f961284678d13
Summary:
This makes it pick up some `%include` configs and can avoid some surprises.
Also remove the manual `paths.default` setconfig. That seems to cause
`destrepo.edenapi` being `None` and break lazy changelog clone paths.
Reviewed By: DurhamG
Differential Revision: D29428882
fbshipit-source-id: bafeb195a560e35be17355b793613b60d97fbecf
Summary: Make dynamicconfig pick up the repo name because hgrc gets written.
Reviewed By: DurhamG
Differential Revision: D29428878
fbshipit-source-id: fbf578cd7c770a4541fff3b85ff40c40cd5a6cc5
Summary: This will be used by the next change.
Reviewed By: DurhamG
Differential Revision: D29428879
fbshipit-source-id: 69e0ffac12fb9c442488d59ea8faa0ea4b47a2c1
Summary: This makes it easier to figure out a traceback using an empty repo name.
Reviewed By: DurhamG
Differential Revision: D29428881
fbshipit-source-id: 95a09c691e3d921ad4f960a39002f71ec879d927
Summary:
This makes it easier to debug wrong reponame issues.
In theory those need to be checked and the config needs to be regenerated if
they are changed for correctness. Given that username and reponame are rarely
changed, I saved it for later.
Reviewed By: DurhamG
Differential Revision: D29428880
fbshipit-source-id: f996af6a7a1e329faaa8b0a53dac8621fa94dac8
Summary: Left over of D29404901 (410769c529).
Reviewed By: DurhamG
Differential Revision: D29429146
fbshipit-source-id: b37c89745d924efc28110d8b96e9b51162b6570b
Summary: This avoids issues in assertions used by metadataonlyctx.
Reviewed By: DurhamG
Differential Revision: D29430672
fbshipit-source-id: 4de305d62f65a9d6c4fb2e3f375ef20d9d94d41c
Summary:
We do not respect `$PAGER`, and the default is the internal streampager.
Also document streampager config options.
Reviewed By: markbt
Differential Revision: D29412598
fbshipit-source-id: 706f09f8805e9067b46eec7d83c6742b61084b09
Summary:
See also D29400532 (909411bb1c). It turns out that it might be more desirable to just mix
stdout and stderr streams in streampager. For example, having them mixed then
the graph log output can show what network fetches or calculations are done
before outputting the graph lines. This is also more consistent with the
vanilla terminal (no pager) behavior.
Reviewed By: markbt
Differential Revision: D29412531
fbshipit-source-id: c07f68b12498a7cee6152bbecbb58d5a7e64097a
Summary: Move the scmstore implementation from the `specialized` module to the root of the `scmstore` module.
Reviewed By: kulshrax
Differential Revision: D29405779
fbshipit-source-id: ae2ef9cc05337a0ff81f5ba5b7051792207fee82
Summary: `scmstore` is dead, long live `scmstore`.
Reviewed By: kulshrax
Differential Revision: D29405613
fbshipit-source-id: 3252a545f5b944d14c15b2a777b84a99a2d4c293
Summary: Update unit tests in `revisionstore::indexedlogdatastore` to use new scmstore instead of old scmstore.
Reviewed By: kulshrax
Differential Revision: D29405258
fbshipit-source-id: 3d2e8cd313dbe66a257433702402804f490bdf47
Summary:
Update unit tests in `revisionstore::edenapi::data` to use new scmstore. There's not really a wrapper to exercise anymore for edenapi specifically, so it's probably better to just make these `scmstore` unit tests instead of edenapi (or indexedlogdatastore as in the next change)-specific.
For ease of unit testing, make fetch_logger optional and introduce `empty` constructor.
Reviewed By: kulshrax
Differential Revision: D29397495
fbshipit-source-id: d7ef0df16cf83a2506606c55c78fcbfa684904d7
Summary:
The server is expected to provide head (of all segs), parents (of each seg),
roots (of all segs). We checked roots and parents but only check head in debug
build. Let's check head in release build too.
Reviewed By: andll
Differential Revision: D29405816
fbshipit-source-id: 1a97eb52a9a0d1d444ae5dabd1a01f0786be9fa9
Summary:
The external grep command (originally from fb-hgext's tweaks of grep command)
was run without redirecting its stdout. That could be problematic for
streampager, because writing directly to stdout will mess up with streampager
interface.
For example, the following command will show "nothing", while it should print
something:
fbcode/antlir/fbpkg/build % lhg grep -l scuba --config pager.pager=internal:streampager --config grep.usebiggrep=False --config pager.interface=full
Fix it by using a PIPE and forward the external command's output to streampager
buffer properly.
Reviewed By: andll
Differential Revision: D29408821
fbshipit-source-id: 4b2a0a6bbd64aa00d09f921d830b173cc56ae630
Summary:
Found by xavierd. Recent `os_info` bump now detects CentOS as OracleLinux.
Workaround it to keep our repo functional.
Reviewed By: xavierd
Differential Revision: D29410415
fbshipit-source-id: 1bd8183f46e3c2265aef119e9f96d9d05a5dbae6
Summary: Previously, hg would always try to fetch data via EdenAPI, even when doing so wouldn't make sense (for example, a local test repo). The only time it makes sense to try to use EdenAPI at all is when the repo in question is supported by Mononoke. As a quick heuristic, let's disable EdenAPI if `paths.default` is not set to a `mononoke://` URL.
Differential Revision: D29372739
fbshipit-source-id: b3601e529ed44c355cb192aedf4332317f4e3132
Summary: Add an option to allow manually forcing EdenAPI to be enabled or disabled. This is useful in a variety of cases, such as bypassing the normal EdenAPI activation logic in tests, or to forcibly disable EdenAPI in cases where it isn't working correctly.
Differential Revision: D29377923
fbshipit-source-id: f408efe2a46ef3f1bd2914669310c3445c7d4121
Summary:
Users get used to hints at the end of the output, not in a separate panel in
streampager. Make it so.
Reviewed By: andll
Differential Revision: D29400532
fbshipit-source-id: 41dc5d20f44a315bc0235ca3cb7857ae8955d3ad
Summary: This will be used by the next change.
Reviewed By: andll
Differential Revision: D29400533
fbshipit-source-id: e6b90bedd8d8a6cf9452dfb5c5f14f9980e12f62
Summary:
More straightforward way of D29404055 (e6ea02372c). Return the non-lazy set directly from
Rust. This avoids some overheads.
Note: ignoring whitespace will make reviewing easier.
Reviewed By: andll
Differential Revision: D29404901
fbshipit-source-id: 02e4766256863fe3fe258bcb318473355cd1efe4
Summary: This was used to narrow down issues.
Reviewed By: andll
Differential Revision: D29404054
fbshipit-source-id: 3bfdac332d63bdb13f40d5cf23dacec242b46d52
Summary:
Running `hasnode` in a loop is inefficient with lazy changelog.
Update it so it does batch filtering and does not ask remote service.
With the change, BackupState.heads will only contain locally known heads.
So the extra filtering is removed.
Reviewed By: andll
Differential Revision: D29404056
fbshipit-source-id: 327bd4c292ad75fd14a47587482440b6ece1d2d5
Summary:
Previously, the obsolete() set (from repo.dageval) is a "meta" (lazy) set that
might trigger frequent remote lookups. Calculate the obsolete() set into a
static set form so it does not trigger remote lookups.
This avoids expensive remote lookups in:
File "/data/users/quark/fbsource/fbcode/een/scm/edenscm/hgext/smartlog.py", line 432, in <lambda>
hidenodes = repo.dageval(lambda: obsnodes - heads(obsnodes) - roots(obsnodes))d
This issue was found out by setting env vars:
EDENSCM_LOG=dag::protocol=debug
to find the commit hashes, and:
EDENSCM_DISABLE_REMOTE_RESOLVE=a8209cbe13bcdc42d4999ffaa15dc9881660d5d,76d91d23d583f92b6c909b6747db1b75229d5be5
to find the traceback.
Reviewed By: andll
Differential Revision: D29404055
fbshipit-source-id: cc8bdd5347b38c1681565455874ed22f1a87321e
Summary:
With lazy changelog, for every commit hash that is unknown to the repo, it
needs to be resolved remotely. For commit hash that is also unknown to the
remote server, it can be bad because we don't have cross-process negative
caching and will trigger the remote resolution over and over. Practically,
the solution is to avoid remote lookup if the "minor" correctness issue
is acceptable (ex. D29111710 (ac6c6cf3fa), D29114049 (880b5c3cd8)).
That has been tricky to debug - the remote fetching happens in Rust, and
cannot be easily inserted something like `import ipdb; ipdb.set_trace()`
like Python. I have been inserting sleeps and use gdb to understand the
call stack when writing the above 2 diffs.
This diff makes debugging easier by supporting "break point" setting by
environ variables. So one can do something like:
$ EDENSCM_LOG=dag::protocol=debug hg ...
DEBUG dag::protocol: resolve names [435c235d65ccc4f95595d74478a617450c96c2e] remotely
$ EDENSCM_DISABLE_REMOTE_RESOLVE=435c235d65cc4f95595d74478a617450c96c2ec hg ... --traceback --debugger
Reviewed By: andll
Differential Revision: D29404057
fbshipit-source-id: d8a631f279f32e2ee88f097796cdc85d8ca27b77
Summary:
The logic was wrong. If local is True, vertexes is a filtered list, while nodes
isn't. They cannot be mix-used.
Reviewed By: andll
Differential Revision: D29403466
fbshipit-source-id: 06f0853282b0f284d5b2970bfa37589ef59f6169
Summary:
We don't need the full precision of execution time in the telemetry, thus
reduce the precision before logging it.
Reviewed By: kulshrax
Differential Revision: D29384354
fbshipit-source-id: cab02ab4db6b260525e56e0395ca1071fdb4b442
Summary:
Straightforward conversion of native checkout implementation from old scmstore API to new (non-async, batched) scmstore API.
We'll meet again someday soon, async.
Reviewed By: andll
Differential Revision: D29321512
fbshipit-source-id: 1e3e0d92c95730a5c2df610061f6faf5b1eb9068
Summary: The returned value now includes roots. Rename the function to clarify.
Reviewed By: kulshrax
Differential Revision: D29383072
fbshipit-source-id: 02a255ce20d9797f482f6fe1c716f2d79a12d4e0
Summary: There is a regression in 1.7.0 (which we're on at the moment) so we might as well update.
Reviewed By: zertosh, farnz
Differential Revision: D29358047
fbshipit-source-id: 226393d79c165455d27f7a09b14b40c6a30d96d3
