Summary:
If the repo changelog was migrated to lazy, existing processes including the
import helpers might start failing to resolve commits. Detect that, and request
a restart of the import helper.
An example traceback:
edenscm.mercurial.error.RepoLookupError: unknown revision 'ce48ccc4197b2066a8c7f2a930cee89e8aea6379'
2021-06-29 12:33:12,738 error processing command 4
Traceback (most recent call last):
File "edenscm/mercurial/commands/eden.py", line 363, in process_request
cmd_function(req)
File "edenscm/mercurial/commands/eden.py", line 604, in cmd_manifest_node_for_commit
node = self.get_manifest_node(rev_name)
File "edenscm/mercurial/commands/eden.py", line 821, in get_manifest_node
return self._get_manifest_node_impl(rev)
File "edenscm/mercurial/commands/eden.py", line 790, in _get_manifest_node_impl
node_hash = ctx.manifestnode()
File "edenscm/mercurial/context.py", line 569, in manifestnode
return self._changeset.manifest
File "edenscm/mercurial/util.py", line 987, in __get__
result = self.func(obj)
File "edenscm/mercurial/context.py", line 540, in _changeset
return self._repo.changelog.changelogrevision(self._node)
File "edenscm/mercurial/changelog2.py", line 374, in changelogrevision
return changelogrevision(self.revision(nodeorrev))
File "edenscm/mercurial/changelog2.py", line 463, in revision
p1, p2 = self.parents(node)[:2]
File "edenscm/mercurial/changelog2.py", line 610, in parents
parents = list(self.dag.parentnames(node))
error.CommitLookupError: '<int> cannot be found'
Reviewed By: andll
Differential Revision: D29470181
fbshipit-source-id: 40ad6eebe1f656c63d3f96b2b70ad60b350eed21
Summary: Hack to use parexec and the par directly.
Reviewed By: chadaustin
Differential Revision: D29461444
fbshipit-source-id: 9ac6c1aa43728782b8de0a71446109f7fd5dab1d
Summary:
Previously the None edenapi error isn't great for understanding what's going on:
File "edenscm/mercurial/changelog2.py", line 123, in _openhybrid
inner = bindings.dag.commits.openhybrid(
TypeError: Expected type that converts to client but received NoneType
Let's make edenapi non-nullable by default, with nicer error messages, and make
places using the nullable edenapi opt-in explicitly.
Example errors:
% lhg sl --config edenapi.url=
abort: edenapi is required but disabled by missing edenapi.url config
% lhg sl --config edenapi.enable=
abort: edenapi is required but disabled by edenapi.enable being empty
% lhg sl --config paths.default=/tmp
abort: edenapi is required but disabled by paths.default being 'file:'
% lhg sl --config paths.default=ssh://x --config ui.ssh=dummyssh
abort: edenapi is required but disabled by paths.default being 'ssh:' and dummyssh in use
Reviewed By: kulshrax
Differential Revision: D29462258
fbshipit-source-id: 84fab5a7cce837c52ccb8108d1f4fe44ef2b6d1e
Summary:
D29372739 (ef79c2d811) caused issues that repo.edenapi becomes None due to scheme being ssh
(hit by markbt) or fb (hit by StanislavGlebik). Given that it's just legacy tests
that do not want EdenAPI, let's use a blacklist for legacy tests, instead of a
(fragile) whitelist for where EdenAPI should be enabled.
Reviewed By: kulshrax
Differential Revision: D29461305
fbshipit-source-id: 4a0317747d9ee3412ffdfdb25dc0d8383305a25c
Summary:
To construct a lazy changelog, the repo object needs to have `name` and `edenapi`. Implement them.
Besides, print why changelog fails to open with `--debug` or `--traceback`.
Reviewed By: andll
Differential Revision: D29468005
fbshipit-source-id: 5815e0d9313dac897f20fa32668cae05ba90c727
Summary:
This is adds file access logging in the Inode layer of the code. We log the cause (channel, thrift, unknown) and the cause detail (which is the FUSE opcode for fuse, function name for thrift). We do the work on a worker thread since we need to traverse the inode map to translate an inode number to a file path. The workerthread in this code is largely based off of the one in the ProcessNameCache.
At this moment, I am not worrying about logging inode creation as a file access. The nice thing about putting this in FileInode is that it should be able to be used by NFS for free.
I've left a TODO in the code about not logging files that match gitignore patterns since its not a hard blocker for this to land.
Reviewed By: kmancini
Differential Revision: D29128258
fbshipit-source-id: 3e08a3567fed937a381b58847ea83569d70f0ea2
Summary: NfsTaskQueue can be made more generic to be shared across the codebase, so this makes it its own target in `eden/fs/utils` w/ the name EdenTaskQueue.
Reviewed By: xavierd
Differential Revision: D29244762
fbshipit-source-id: 78348f2ff8fa66bc801aefe7d6b3905e0da278e8
Summary: Just to be more specific, lets be explicit about that this is the serverThreadPool that the mount is returning.
Reviewed By: xavierd
Differential Revision: D29244719
fbshipit-source-id: 015a2e8198c418c6fb5a89234274eefb329848fe
Summary: In order to access the kill switch for file access logging, we need to store a EdenConfig pointer in the HiveLogger.
Reviewed By: xavierd
Differential Revision: D29120227
fbshipit-source-id: c828bbbf551c7096877bfa763edc29ef133afb41
Summary: This creates a HiveLogger and threads it through to the ServerState.
Reviewed By: xavierd
Differential Revision: D29119501
fbshipit-source-id: d0b74733d8832d604f8f4cf0af28e767dbddeddf
Summary: This just adds a method to get the repoName out of the EdenMount for logging purposes.
Reviewed By: xavierd
Differential Revision: D29113121
fbshipit-source-id: 9ebb4ac588839a99f25f4e884a7a132f5793e49e
Summary: Looks like another test needs updating after D29437983 (5b1b1febc1).
Reviewed By: quark-zju
Differential Revision: D29446063
fbshipit-source-id: c71684609830b6ec047a2efc615b806dd2a0066b
Summary:
With lazy pull fast path, the legacy commit scan will trigger one-by-one id
resolution, which is terrible and defeat the purpose of laziness. Let's just
force graphql path instead.
Reviewed By: andll
Differential Revision: D29440142
fbshipit-source-id: 4b0d10e6f5a3eb0533b2d95b1889a1f1c1f281cb
Summary:
Similar to D29404057 (cedddd1c8d), add a way to disable resolving IDs by setting
a limit using `EDENSCM_REMOTE_ID_THRESHOLD`.
Reviewed By: andll
Differential Revision: D29440143
fbshipit-source-id: 30409089493ae2cd5c189e37b0d4f88df9a6d8e8
Summary: The Rust NameSet does not have nullid. Do not bother resolving remotely.
Reviewed By: kulshrax
Differential Revision: D29437983
fbshipit-source-id: f1eb73e738281b3b5096fba22ba92833f5a4f3ee
Summary:
See this graph - https://fburl.com/scuba/mononoke_blobstore_trace/77nyw174. It
shows that there was a huge jumps in the number of "hgmanifest" fetches from
scs that didn't exist in the manifold. This is not great for mononoke, because
it uses a significant chunk of our manifold qps capacity (~30K qps on average,
with spikes going up to 100K qps). The main suspect here is
repo_list_hg_manifest method, because few other scs methods (if any) fetch hg
data.
It correlates with the release of new eden version on June, 23, which added
getTreeBatch() method, and looks like it was passing manifest hash and eden id
in the wrong order. See https://fburl.com/code/tsjwss8a
```
proxyHash.revHash(), // this is really the manifest node
```
This is a comment in `getTree()` method which says that `proxyHash.revHash()`
is manifest id, which is exactly what we need to send to scs.
[ScsMetadataImport::getTreeMetadata](https://fburl.com/code/xammvq2k) suggests
that the second parameter should be manifestId (i.e. proxyHash.revHash()), but
we were passing them in the wrong order. Let's fix it.
Reviewed By: xavierd
Differential Revision: D29428581
fbshipit-source-id: d041008f00c7519504c6c67173ca85709e9dc415
Summary:
The remotenames information might be out of sync with the DAG.
Use the DAG's master group's head for the pull fast path "old" master so that
even if the remotenames is stale for some reason, the fast path still works.
Reviewed By: andll
Differential Revision: D29434719
fbshipit-source-id: 7bbd0757dc6a9428f86663ac4989f7c30b365b46
Summary: Will be used by the next change.
Reviewed By: andll
Differential Revision: D29434722
fbshipit-source-id: 74dbec506fb0985379480815380118cd41058aec
Summary:
If a user Ctrl+C-ed (or something else interrupts) between flushing changelog
and flushing metalog (remotenames). The pulled commits will exist in the local
graph and break the next time using the fast path.
Reviewed By: andll
Differential Revision: D29434720
fbshipit-source-id: e7ca7542691279644679effb06707a1d305541af
Summary:
This is a better practice. It avoids debug messages from other places (ex.
discovery).
Reviewed By: andll
Differential Revision: D29434721
fbshipit-source-id: 5990f0174e9b61aed7d9d56252df63b571364070
Summary:
The remotenames has logic to work with legacy clones to write the remotenames
state. However, modern clone paths take care of remotenames directly so there
is no need to do an extra pull by remotenames.
Reviewed By: DurhamG
Differential Revision: D29428883
fbshipit-source-id: a73c0ee716b09f4e34d6fa30997f961284678d13
Summary:
This makes it pick up some `%include` configs and can avoid some surprises.
Also remove the manual `paths.default` setconfig. That seems to cause
`destrepo.edenapi` being `None` and break lazy changelog clone paths.
Reviewed By: DurhamG
Differential Revision: D29428882
fbshipit-source-id: bafeb195a560e35be17355b793613b60d97fbecf
Summary: Make dynamicconfig pick up the repo name because hgrc gets written.
Reviewed By: DurhamG
Differential Revision: D29428878
fbshipit-source-id: fbf578cd7c770a4541fff3b85ff40c40cd5a6cc5
Summary: This will be used by the next change.
Reviewed By: DurhamG
Differential Revision: D29428879
fbshipit-source-id: 69e0ffac12fb9c442488d59ea8faa0ea4b47a2c1
Summary: This makes it easier to figure out a traceback using an empty repo name.
Reviewed By: DurhamG
Differential Revision: D29428881
fbshipit-source-id: 95a09c691e3d921ad4f960a39002f71ec879d927
Summary:
This makes it easier to debug wrong reponame issues.
In theory those need to be checked and the config needs to be regenerated if
they are changed for correctness. Given that username and reponame are rarely
changed, I saved it for later.
Reviewed By: DurhamG
Differential Revision: D29428880
fbshipit-source-id: f996af6a7a1e329faaa8b0a53dac8621fa94dac8
