mononoke: include root count in walker queue stats

Summary:
The main thing this change does is make sure pending roots to visit are represented in the difference between Walked and Children.  Children is the sum of all child nodes discovered, both visited and unvisited.  Walked is a measure of number of nodes visited.  Children-Walked is used as a measure of queue depth remaining to be processed.

When not chunking this is a minor issue as usually just one bookmark root node is not counted in children, but when chunking not counting the roots means mean the chunk of several 100000 roots is not visible as waiting to be processed.

Reviewed By: StanislavGlebik

Differential Revision: D25852526

fbshipit-source-id: df5f21a37be152f0baee40d33fd7dfb7aaa763de
This commit is contained in:
Alex Hornby 2021-01-22 03:10:40 -08:00 committed by Facebook GitHub Bot
parent ca2f1e2824
commit e5307b654d
3 changed files with 24 additions and 19 deletions

View File

@ -46,42 +46,42 @@ count-objects, bonsai core data. total nodes is BONSAICOUNT plus one for the ro
Walking edge types [BookmarkToChangeset, ChangesetToBonsaiParent, ChangesetToFileContent]
Walking node types [Bookmark, Changeset, FileContent]
Seen,Loaded: 7,7
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:3,* FileContent:3,3,0 (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:3,* FileContent:3,3,0 (glob)
count-objects, shallow, bonsai only. No parents, expect just one of each node type. Also exclude FsnodeToFileContent to keep the test intact
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -X hg -x BonsaiHgMapping -X FsnodeToFileContent -i default -i derived_fsnodes 2>&1 | strip_glog
Walking edge types [AliasContentMappingToFileContent, BookmarkToChangeset, ChangesetToFileContent, ChangesetToFsnodeMapping, FileContentMetadataToGitSha1Alias, FileContentMetadataToSha1Alias, FileContentMetadataToSha256Alias, FileContentToFileContentMetadata, FsnodeMappingToRootFsnode, FsnodeToChildFsnode]
Walking node types [AliasContentMapping, Bookmark, Changeset, FileContent, FileContentMetadata, Fsnode, FsnodeMapping]
Seen,Loaded: 9,9
* Type:Walked,Checks,Children AliasContentMapping:3,3,0 Bookmark:1,1,1 Changeset:1,1,3 FileContent:1,1,0 FileContentMetadata:1,0,3 Fsnode:1,1,0 FsnodeMapping:1,1,1 (glob)
* Type:Walked,Checks,Children AliasContentMapping:3,3,0 Bookmark:1,1,2 Changeset:1,1,3 FileContent:1,1,0 FileContentMetadata:1,0,3 Fsnode:1,1,0 FsnodeMapping:1,1,1 (glob)
count-objects, hg only. total nodes is HGCOUNT plus 1 for the root bookmark step, plus 1 for mapping from bookmark to hg. plus 3 for filenode (same blob as envelope)
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I hg 2>&1 | strip_glog
Walking edge types [BonsaiHgMappingToHgChangeset, BookmarkToBonsaiHgMapping, HgChangesetToHgManifest, HgChangesetToHgParent, HgChangesetViaBonsaiToHgChangeset, HgFileEnvelopeToFileContent, HgFileNodeToHgCopyfromFileNode, HgFileNodeToHgParentFileNode, HgFileNodeToLinkedHgChangeset, HgManifestToChildHgManifest, HgManifestToHgFileEnvelope, HgManifestToHgFileNode]
Walking node types [BonsaiHgMapping, Bookmark, FileContent, HgChangeset, HgChangesetViaBonsai, HgFileEnvelope, HgFileNode, HgManifest]
Seen,Loaded: 20,20
* Type:Walked,Checks,Children BonsaiHgMapping:1,1,1 Bookmark:1,1,1 FileContent:3,3,0 HgChangeset:3,*,5 HgChangesetViaBonsai:3,*,2 HgFileEnvelope:3,*,3 HgFileNode:3,*,1 HgManifest:3,3,6 (glob)
* Type:Walked,Checks,Children BonsaiHgMapping:1,1,1 Bookmark:1,1,2 FileContent:3,3,0 HgChangeset:3,*,5 HgChangesetViaBonsai:3,*,2 HgFileEnvelope:3,*,3 HgFileNode:3,*,1 HgManifest:3,3,6 (glob)
count-objects, default shallow walk across bonsai and hg data, but exclude HgFileEnvelope so that we can test that we visit FileContent from fsnodes
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -x HgFileEnvelope -i default -i derived_fsnodes 2>&1 | strip_glog
Walking edge types [AliasContentMappingToFileContent, BonsaiHgMappingToHgChangeset, BookmarkToChangeset, ChangesetToBonsaiHgMapping, ChangesetToFileContent, ChangesetToFsnodeMapping, FileContentMetadataToGitSha1Alias, FileContentMetadataToSha1Alias, FileContentMetadataToSha256Alias, FileContentToFileContentMetadata, FsnodeMappingToRootFsnode, FsnodeToChildFsnode, FsnodeToFileContent, HgChangesetToHgManifest, HgManifestToChildHgManifest, HgManifestToHgFileNode]
Walking node types [AliasContentMapping, BonsaiHgMapping, Bookmark, Changeset, FileContent, FileContentMetadata, Fsnode, FsnodeMapping, HgChangeset, HgFileNode, HgManifest]
Seen,Loaded: 25,25
* Type:Walked,Checks,Children AliasContentMapping:9,9,0 BonsaiHgMapping:1,1,1 Bookmark:1,1,1 Changeset:1,1,4 FileContent:3,*,0 FileContentMetadata:3,0,9 Fsnode:1,1,4 FsnodeMapping:1,1,1 HgChangeset:1,1,1 HgFileNode:3,3,* HgManifest:1,1,3 (glob)
* Type:Walked,Checks,Children AliasContentMapping:9,9,0 BonsaiHgMapping:1,1,1 Bookmark:1,1,2 Changeset:1,1,4 FileContent:3,*,0 FileContentMetadata:3,0,9 Fsnode:1,1,4 FsnodeMapping:1,1,1 HgChangeset:1,1,1 HgFileNode:3,3,* HgManifest:1,1,3 (glob)
count-objects, default shallow walk across bonsai and hg data, including mutable
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -I marker 2>&1 | strip_glog
Walking edge types [AliasContentMappingToFileContent, BonsaiHgMappingToHgChangeset, BookmarkToChangeset, ChangesetToBonsaiHgMapping, ChangesetToFileContent, ChangesetToPhaseMapping, FileContentMetadataToGitSha1Alias, FileContentMetadataToSha1Alias, FileContentMetadataToSha256Alias, FileContentToFileContentMetadata, HgChangesetToHgManifest, HgFileEnvelopeToFileContent, HgManifestToChildHgManifest, HgManifestToHgFileEnvelope, HgManifestToHgFileNode]
Walking node types [AliasContentMapping, BonsaiHgMapping, Bookmark, Changeset, FileContent, FileContentMetadata, HgChangeset, HgFileEnvelope, HgFileNode, HgManifest, PhaseMapping]
Seen,Loaded: 27,27
* Type:Walked,Checks,Children AliasContentMapping:9,9,0 BonsaiHgMapping:1,1,1 Bookmark:1,1,1 Changeset:1,1,4 FileContent:3,*,0 FileContentMetadata:3,0,9 HgChangeset:1,1,1 HgFileEnvelope:3,3,* HgFileNode:3,3,0 HgManifest:1,1,6 PhaseMapping:1,1,0 (glob)
* Type:Walked,Checks,Children AliasContentMapping:9,9,0 BonsaiHgMapping:1,1,1 Bookmark:1,1,2 Changeset:1,1,4 FileContent:3,*,0 FileContentMetadata:3,0,9 HgChangeset:1,1,1 HgFileEnvelope:3,3,* HgFileNode:3,3,0 HgManifest:1,1,6 PhaseMapping:1,1,0 (glob)
count-objects, default shallow walk across bonsai and hg data, including mutable for all public heads
$ mononoke_walker -L sizing scrub -q --walk-root PublishedBookmarks -I shallow -I marker 2>&1 | strip_glog
Walking edge types [AliasContentMappingToFileContent, BonsaiHgMappingToHgChangeset, ChangesetToBonsaiHgMapping, ChangesetToFileContent, ChangesetToPhaseMapping, FileContentMetadataToGitSha1Alias, FileContentMetadataToSha1Alias, FileContentMetadataToSha256Alias, FileContentToFileContentMetadata, HgChangesetToHgManifest, HgFileEnvelopeToFileContent, HgManifestToChildHgManifest, HgManifestToHgFileEnvelope, HgManifestToHgFileNode, PublishedBookmarksToBonsaiHgMapping, PublishedBookmarksToChangeset]
Walking node types [AliasContentMapping, BonsaiHgMapping, Changeset, FileContent, FileContentMetadata, HgChangeset, HgFileEnvelope, HgFileNode, HgManifest, PhaseMapping, PublishedBookmarks]
Seen,Loaded: 28,28
* Type:Walked,Checks,Children AliasContentMapping:9,9,0 BonsaiHgMapping:2,2,1 Changeset:1,1,4 FileContent:3,*,0 FileContentMetadata:3,0,9 HgChangeset:1,*,1 HgFileEnvelope:3,*,* HgFileNode:3,3,0 HgManifest:1,1,6 PhaseMapping:1,1,0 PublishedBookmarks:1,1,2 (glob)
* Type:Walked,Checks,Children AliasContentMapping:9,9,0 BonsaiHgMapping:2,2,1 Changeset:1,1,4 FileContent:3,*,0 FileContentMetadata:3,0,9 HgChangeset:1,*,1 HgFileEnvelope:3,*,* HgFileNode:3,3,0 HgManifest:1,1,6 PhaseMapping:1,1,0 PublishedBookmarks:1,1,3 (glob)
count-objects, shallow walk across bonsai and changeset_info
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -i bonsai -i derived_changeset_info 2>&1 | strip_glog
@ -102,70 +102,70 @@ count-objects, shallow walk across bonsai and unodes
Walking edge types [BookmarkToChangeset, ChangesetToUnodeMapping, UnodeFileToFileContent, UnodeManifestToUnodeFileChild, UnodeManifestToUnodeManifestChild, UnodeMappingToRootUnodeManifest]
Walking node types [Bookmark, Changeset, FileContent, UnodeFile, UnodeManifest, UnodeMapping]
Seen,Loaded: 10,10
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:1,* FileContent:3,* UnodeFile:3,* UnodeManifest:1,* UnodeMapping:1,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:1,* FileContent:3,* UnodeFile:3,* UnodeManifest:1,* UnodeMapping:1,* (glob)
count-objects, deep walk across bonsai and unodes
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I deep -i bonsai -i derived_unodes -X ChangesetToBonsaiParent 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToUnodeMapping, UnodeFileToLinkedChangeset, UnodeFileToUnodeFileParent, UnodeManifestToLinkedChangeset, UnodeManifestToUnodeFileChild, UnodeManifestToUnodeManifestChild, UnodeManifestToUnodeManifestParent, UnodeMappingToRootUnodeManifest]
Walking node types [Bookmark, Changeset, UnodeFile, UnodeManifest, UnodeMapping]
Seen,Loaded: 13,13
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:3,* UnodeFile:3,* UnodeManifest:3,* UnodeMapping:3,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:3,* UnodeFile:3,* UnodeManifest:3,* UnodeMapping:3,* (glob)
count-objects, shallow walk across blame
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -i bonsai -i derived_unodes -i derived_blame -X ChangesetToFileContent -X UnodeFileToFileContent 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToUnodeMapping, UnodeFileToBlame, UnodeManifestToUnodeFileChild, UnodeManifestToUnodeManifestChild, UnodeMappingToRootUnodeManifest]
Walking node types [Blame, Bookmark, Changeset, UnodeFile, UnodeManifest, UnodeMapping]
Seen,Loaded: 10,10
* Type:Walked,Checks,Children Blame:3,* Bookmark:1,1,1 Changeset:1,* UnodeFile:3,* UnodeManifest:1,* UnodeMapping:1,* (glob)
* Type:Walked,Checks,Children Blame:3,* Bookmark:1,1,2 Changeset:1,* UnodeFile:3,* UnodeManifest:1,* UnodeMapping:1,* (glob)
count-objects, deep walk across blame
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I deep -i bonsai -i derived_unodes -i derived_blame -X ChangesetToBonsaiParent -X UnodeFileToLinkedChangeset -X UnodeManifestToLinkedChangeset 2>&1 | strip_glog
Walking edge types [BlameToChangeset, BookmarkToChangeset, ChangesetToUnodeMapping, UnodeFileToBlame, UnodeFileToUnodeFileParent, UnodeManifestToUnodeFileChild, UnodeManifestToUnodeManifestChild, UnodeManifestToUnodeManifestParent, UnodeMappingToRootUnodeManifest]
Walking node types [Blame, Bookmark, Changeset, UnodeFile, UnodeManifest, UnodeMapping]
Seen,Loaded: 16,16
* Type:Walked,Checks,Children Blame:3,* Bookmark:1,1,1 Changeset:3,* UnodeFile:3,* UnodeManifest:3,* UnodeMapping:3,* (glob)
* Type:Walked,Checks,Children Blame:3,* Bookmark:1,1,2 Changeset:3,* UnodeFile:3,* UnodeManifest:3,* UnodeMapping:3,* (glob)
count-objects, shallow walk across deleted files manifest
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -i bonsai -i derived_deleted_manifest -X ChangesetToFileContent 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToDeletedManifestMapping, DeletedManifestMappingToRootDeletedManifest, DeletedManifestToDeletedManifestChild]
Walking node types [Bookmark, Changeset, DeletedManifest, DeletedManifestMapping]
Seen,Loaded: 4,4
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:1,* DeletedManifest:1,* DeletedManifestMapping:1,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:1,* DeletedManifest:1,* DeletedManifestMapping:1,* (glob)
count-objects, deep walk across deleted files manifest
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I deep -i bonsai -i derived_deleted_manifest 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToBonsaiParent, ChangesetToDeletedManifestMapping, DeletedManifestMappingToRootDeletedManifest, DeletedManifestToDeletedManifestChild, DeletedManifestToLinkedChangeset]
Walking node types [Bookmark, Changeset, DeletedManifest, DeletedManifestMapping]
Seen,Loaded: 8,8
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:3,* DeletedManifest:1,* DeletedManifestMapping:3,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:3,* DeletedManifest:1,* DeletedManifestMapping:3,* (glob)
count-objects, shallow walk across skeleton manifest
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -i bonsai -i derived_skeleton_manifests -X ChangesetToFileContent 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToSkeletonManifestMapping, SkeletonManifestMappingToRootSkeletonManifest, SkeletonManifestToSkeletonManifestChild]
Walking node types [Bookmark, Changeset, SkeletonManifest, SkeletonManifestMapping]
Seen,Loaded: 4,4
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:1,* SkeletonManifest:1,* SkeletonManifestMapping:1,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:1,* SkeletonManifest:1,* SkeletonManifestMapping:1,* (glob)
count-objects, deep walk across skeleton manifest
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I deep -i bonsai -i derived_skeleton_manifests 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToBonsaiParent, ChangesetToSkeletonManifestMapping, SkeletonManifestMappingToRootSkeletonManifest, SkeletonManifestToSkeletonManifestChild]
Walking node types [Bookmark, Changeset, SkeletonManifest, SkeletonManifestMapping]
Seen,Loaded: 10,10
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:3,* SkeletonManifest:3,* SkeletonManifestMapping:3,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:3,* SkeletonManifest:3,* SkeletonManifestMapping:3,* (glob)
count-objects, shallow walk across fastlog
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -i bonsai -i derived_unodes -i derived_fastlog -X ChangesetToFileContent -X UnodeFileToFileContent 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToUnodeMapping, FastlogBatchToPreviousBatch, FastlogDirToPreviousBatch, FastlogFileToPreviousBatch, UnodeFileToFastlogFile, UnodeManifestToFastlogDir, UnodeManifestToUnodeFileChild, UnodeManifestToUnodeManifestChild, UnodeMappingToRootUnodeManifest]
Walking node types [Bookmark, Changeset, FastlogBatch, FastlogDir, FastlogFile, UnodeFile, UnodeManifest, UnodeMapping]
Seen,Loaded: 11,11
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:1,* FastlogDir:1,* FastlogFile:3,* UnodeFile:3,* UnodeManifest:1,* UnodeMapping:1,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:1,* FastlogDir:1,* FastlogFile:3,* UnodeFile:3,* UnodeManifest:1,* UnodeMapping:1,* (glob)
count-objects, deep walk across fastlog
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I deep -i bonsai -i derived_unodes -i derived_fastlog -X ChangesetToBonsaiParent -X UnodeFileToLinkedChangeset -X UnodeManifestToLinkedChangeset 2>&1 | strip_glog
Walking edge types [BookmarkToChangeset, ChangesetToUnodeMapping, FastlogBatchToChangeset, FastlogBatchToPreviousBatch, FastlogDirToChangeset, FastlogDirToPreviousBatch, FastlogFileToChangeset, FastlogFileToPreviousBatch, UnodeFileToFastlogFile, UnodeFileToUnodeFileParent, UnodeManifestToFastlogDir, UnodeManifestToUnodeFileChild, UnodeManifestToUnodeManifestChild, UnodeManifestToUnodeManifestParent, UnodeMappingToRootUnodeManifest]
Walking node types [Bookmark, Changeset, FastlogBatch, FastlogDir, FastlogFile, UnodeFile, UnodeManifest, UnodeMapping]
Seen,Loaded: 19,19
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:3,* FastlogDir:3,* FastlogFile:3,* UnodeFile:3,* UnodeManifest:3,* UnodeMapping:3,* (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:3,* FastlogDir:3,* FastlogFile:3,* UnodeFile:3,* UnodeManifest:3,* UnodeMapping:3,* (glob)
count-objects, shallow walk across hg
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -I BookmarkToBonsaiHgMapping -i Bookmark -i hg 2>&1 | strip_glog

View File

@ -58,7 +58,7 @@ Output pretty debug to stdout
),
)
Seen,Loaded: 2,2
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:1,1,0 (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:1,1,0 (glob)
Output non-pretty debug to stdout
$ mononoke_walker -L sizing scrub -q -b master_bookmark -I shallow -i bonsai --include-output-node-type=Changeset --output-format=Debug 2>&1 | strip_glog
@ -66,4 +66,4 @@ Output non-pretty debug to stdout
Walking node types [Bookmark, Changeset]
Node Changeset(ChangesetKey { inner: ChangesetId(Blake2(c3384961b16276f2db77df9d7c874bbe981cf0525bd6f84a502f919044f2dabd)), filenode_known_derived: false }): NodeData: Some(Changeset(BonsaiChangeset { inner: BonsaiChangesetMut { parents: [ChangesetId(Blake2(459f16ae564c501cb408c1e5b60fc98a1e8b8e97b9409c7520658bfa1577fb66))], author: "test", author_date: DateTime(1970-01-01T00:00:00+00:00), committer: None, committer_date: None, message: "C", extra: {}, file_changes: {MPath("C"): Some(FileChange { content_id: ContentId(Blake2(896ad5879a5df0403bfc93fc96507ad9c93b31b11f3d0fa05445da7918241e5d)), file_type: Regular, size: 1, copy_from: None })} }, id: ChangesetId(Blake2(c3384961b16276f2db77df9d7c874bbe981cf0525bd6f84a502f919044f2dabd)) }))
Seen,Loaded: 2,2
* Type:Walked,Checks,Children Bookmark:1,1,1 Changeset:1,1,0 (glob)
* Type:Walked,Checks,Children Bookmark:1,1,2 Changeset:1,1,0 (glob)

View File

@ -526,6 +526,11 @@ impl WalkVisitor<(Node, Option<NodeData>, Option<StepStats>), EmptyRoute> for Wa
EmptyRoute,
Vec<OutgoingEdge>,
) {
let queued_roots = if resolved.label.incoming_type().is_none() {
1
} else {
0
};
if route.is_none() || !self.always_emit_edge_types.is_empty() {
outgoing.retain(|e| {
if e.label.incoming_type().is_none() {
@ -551,7 +556,7 @@ impl WalkVisitor<(Node, Option<NodeData>, Option<StepStats>), EmptyRoute> for Wa
self.record_resolved_visit(&resolved, node_data.as_ref());
// Stats
let num_expanded_new = outgoing.len();
let num_expanded_new = outgoing.len() + queued_roots;
let node = resolved.target;
let (error_count, node_data) = match node_data {