Summary: Switching to specify derived data types other than hg explicitly on the command line
Reviewed By: farnz
Differential Revision: D25367323
fbshipit-source-id: 0e0aea1aab46b43b325486ed6161ea322f7cec4b
Summary: Can just pass on the iterator
Reviewed By: ikostia
Differential Revision: D25216892
fbshipit-source-id: 79c08737477ac7ed1f824c50105d5977ee592126
Summary: Reduces boilerplate for binaries usually run in this mode, notably the walker
Reviewed By: ikostia
Differential Revision: D25216883
fbshipit-source-id: e31d2a6aec7da3baafd8bcf208cf79cc696752c0
Summary: This is useful to prevent accidentally consuming too much. Enabled it for the walker
Reviewed By: ikostia
Differential Revision: D25216880
fbshipit-source-id: e80f490d6ece40d64cc8609e7d6b80d0ecbb1671
Summary: Reduces boiler plate on command line for binaries like walker that want different default
Reviewed By: krallin
Differential Revision: D25216876
fbshipit-source-id: 0df474568d28e0726be223e9dc0a760523063d21
Summary: Remove this now it is the walker default. Makes command lines shorter
Reviewed By: ikostia
Differential Revision: D25219551
fbshipit-source-id: bc5ad4237cad35218a0b4c54aa81eb20edb3f3e1
Summary:
This will reduce boilerplate command line for the walker, as most of the time we want to run it with readonly storage
Because the existing --readonly-storage flag can't take a value this introduces a new --with-readonly-storage=<true|false> option
Reviewed By: krallin
Differential Revision: D25216871
fbshipit-source-id: e1b83b428a9c3787f48c18fd396d23ac95991b77
Summary:
Previously needed to pass in cachelib settings once to MononokeAppBuilder and once to parse_and_init_cachelib.
This change adds a MononokeClapApp and MononokeMatches that preserve the settings, thus preventing the need to pass them in twice (and thus avoiding possible inconsistency)
MononokeMatches uses MaybeOwned to hold the inner ArgMatches, which allows us to hold both the usual reference case from get_matches and an owned case for get_matches_from which is used in test cases.
Reviewed By: krallin
Differential Revision: D24788450
fbshipit-source-id: aad5fff2edda305177dcefa4b3a98ab99bc2d811
Summary:
Move expected_item_size_byte into CachelibSettings, seems like it should be there.
To enable its use also exposes a parse_and_init_cachelib method for callers that have different defaults to default cachelibe settings.
Reviewed By: krallin
Differential Revision: D24761024
fbshipit-source-id: 440082ab77b5b9f879c99b8f764e71bec68d270e
Summary: This makes it easier to enable all derived data for scrubbing
Reviewed By: ikostia
Differential Revision: D25188963
fbshipit-source-id: e9c981e33273d6b2eeadcce0d0a341b33e91e42d
Summary:
Show the options for specifying node and edge types in the --help output
These changes removed the last use of lazy_static in the walker, so updated TARGETS as well.
Reviewed By: krallin
Differential Revision: D25188964
fbshipit-source-id: c5ccb4f5a0f3be1b8cb7d51cd5f99236d60d3029
Summary:
It has a build() method and later in stack it will build a mononoke
specific type rather than the clap::App
Differential Revision: D25216827
fbshipit-source-id: 24a531856405a702e7fecf54d60be1ea3d2aa6e7
Summary:
This diff prepares the Mononoke codebase for composition-based extendability of
`ScubaSampleBuilder`. Specifically, in the near future I will add:
- new methods for verbose scuba logging
- new data field (`ObservabilityContext`) to check if verbose logging should
be enabled or disabled
The higher-level goal here is to be able to enable/disable verbose Scuba
logging (either overall or for certain slices of logs, like for a certain
session id) in real time, without restarting Mononoke. To do so, I plan to
expose the aforementioned verbose logging methods, which will run a check
against the stored `ObservabilityContext` and make a decision of whether the
logging is enabled or not. `ObservabilityContext` will of course hide
implementation details from the renamed `ScubaSampleBuilderExt`, and just provide a yes/no
answer based on the current config and sample fields.
At the moment this should be a completely harmless change.
Reviewed By: krallin
Differential Revision: D25211089
fbshipit-source-id: ea03dda82fadb7fc91a2433e12e220582ede5fb8
Summary: Add support for walking fast entries for directories so we can scrub and inspect them
Differential Revision: D25187551
fbshipit-source-id: 812f9fd82459ef49dcd781c318fbe5c398daad21
Summary: Add FastlogFile to walker so it can be inspected and scrubbed. Directory and batch components of fastlog covered in following diffs.
Differential Revision: D25187549
fbshipit-source-id: c046cbf2561cdbbc9563497119e34d1b09d0ebef
Summary: The path isn't needed to load the UnodeManifest, it's only needed as route information for compression etc
Reviewed By: aslpavel
Differential Revision: D25187547
fbshipit-source-id: b8199a81c5dae4caceed5d455fa6a9bbc3f037ac
Summary: The path is not needed to load the unode, only for route tracking
Reviewed By: aslpavel
Differential Revision: D25187553
fbshipit-source-id: baf6df3b7f56f5ca5a0ba086097acba4ecd82e75
Summary: Change from the walk_blame boolean to a bitflags struct so can also represent fastlog. Using bitflags crate so as not to grow the node size.
Reviewed By: aslpavel
Differential Revision: D25185267
fbshipit-source-id: b45574cd6c4d048aec321357ae54e2ec404f84fc
Summary: Few things were not on lexical order, clean them up. Makes it easier to read next diff without this mixed in
Reviewed By: aslpavel
Differential Revision: D25188965
fbshipit-source-id: 38b7ea6f0e2c0583e6701550d6db16d85ba1a455
Summary: convert `BlobRepo::get_bonsai_bookmark` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25188577
fbshipit-source-id: fb6f2b592b9e9f76736bc1af5fa5a08d12744b5f
Summary: Only the id is used when loading the manifest, not the path. Path is only needed for route information
Reviewed By: mitrandir77
Differential Revision: D25185264
fbshipit-source-id: 1af453fa2716d53ea4fb18a81d59867e98ea07f6
Summary: The path is not needed to load the fsnode, only as route info for things like compression
Reviewed By: mitrandir77
Differential Revision: D25184485
fbshipit-source-id: afd84dcd9dbd82c2c9018e86fb676bc57a5d009b
Summary:
We have `HgBlobChangeset`, `HgFileEnvelope`, `HgManifestEnvelope` ... but we
also have `BlobManifest`. Let's be a little consistent.
Reviewed By: markbt
Differential Revision: D25122288
fbshipit-source-id: 9ae0be49986dbcc31cee9a46bd30093b07795c62
Summary: Add them to the walker so we can scrub and inspect them
Reviewed By: StanislavGlebik
Differential Revision: D25124144
fbshipit-source-id: 5df1ca6e48d18d3d55f68905e93fd75fbae92adb
Summary: So we can scrub and inspect them
Reviewed By: StanislavGlebik
Differential Revision: D25120995
fbshipit-source-id: d150e55f0d72f584c15dbaf2bd017d19130312b2
Summary:
It was mistakenly mapping deleted manifest id to a unode id.
I'm surprised the walk.rs code that constructs the DeletedManifestToDeletedManifestChild edge built with this bug. The DeletedManifestId and ManifestUnodeId types presumably coerce to each other.
Reviewed By: markbt
Differential Revision: D25120994
fbshipit-source-id: 1b53037808779c345c163ef32324961938078fc7
Summary: Remove 'static requirement for async methods of Blobstore, propagate this change and fixup low hanging fruits where the code can become 'static free easily.
Reviewed By: ahornby, farnz
Differential Revision: D24839054
fbshipit-source-id: 5d5daa04c23c4c9ae902b669b0a71fe41ee6dee6
Summary: Add DeletedManifest to walker types with edges to linked changeset and its sub-manifests so that they can be scrubbed.
Differential Revision: D25000319
fbshipit-source-id: f146e6132fde0fb13e630d315484cc2c0a964bdc
Summary: Add the mapping from bonsai to deleted manifest to the walker
Reviewed By: ikostia
Differential Revision: D24989424
fbshipit-source-id: 53d622f661629b9b3de91910f4560b641a95a7bf
Summary: Sometimes pretty debug format is too verbose and one per line regular debug format is preferrable.
Reviewed By: ikostia
Differential Revision: D24996432
fbshipit-source-id: 1acda3985658e4c17b57e36734c77b7579e7e28a
Summary: add an output option to walker scrub so we can dump out debug representation of any type the walker can walk
Reviewed By: farnz
Differential Revision: D24996433
fbshipit-source-id: a332d89d65e4d928159155a34bd39b0e2e1131de
Summary:
Previous code used flat_map on the result which discarded the errors.
Fix it and add a test
Reviewed By: StanislavGlebik
Differential Revision: D24989246
fbshipit-source-id: 075c8e40dceebc480ad722fb467a8a9e9bef5905
Summary:
Add Blame to walker so that we can scrub it.
For public commits, if blame is requested we wait for blame to be derived before traversing unodes. This saves traversing the unode tree twice.
For non-public commits they are not expected to have blame, so for those we don't wait for blame to be derived before traversing unodes.
Reviewed By: farnz
Differential Revision: D24896243
fbshipit-source-id: 66226a8e47f115bcda62269ade63874e0fff4ba0
Summary:
This change allows the walk steps to check the underlying WalkState's phase information to see if a changeset is public. Its exposed to the steps via Checker::is_public().
This saves repeat checks from the changeset phase from the bonsai_to_hg_mapping_step if the walk state already knows its public, and makes sure we are passing the get_phase ephemeral_derive flag from just one place (it should always be the opposite of enable_derive).
Reviewed By: farnz
Differential Revision: D24954837
fbshipit-source-id: b911d69837db8ef34fbe2c27f642d6819ea46908
Summary:
The NodeTypes variants and the EdgeType variants generated from them were getting rather long which makes them hard to work with.
This change shortens them by
- changing BonsaiChangeset* to Changeset (this also has better consistency with changeset its blobstore keys)
- updating Bonsai*Mapping to *Mapping )except for BonsaiHgMapping where it disambiguates with HgBonsaiMapping)
Due to the way walker output is sorted it made more sense to do these in one diff rather than two.
Reviewed By: markbt
Differential Revision: D24864061
fbshipit-source-id: dbd395a89be828ac97cf056f03787097d8f1491d
Summary: Fix the walker build. Had a land race on non-conflicting lines.
Reviewed By: farnz
Differential Revision: D24953152
fbshipit-source-id: b3d43d745242542ba1a6df9ba730460112bee6dc
Summary: Step into history via UnodeFile parents to check they are valid
Reviewed By: markbt
Differential Revision: D24838456
fbshipit-source-id: 3b19973f9f7da6e502595b4ffa33a031e1df03ab
Summary: Check that the UnodeFile's link is good by stepping to it.
Reviewed By: markbt
Differential Revision: D24838458
fbshipit-source-id: c6c8ef5dc580bb591f484df659a76f0ba43a68e6
Summary: add UnodeFile to walker graph to completment UnodeManifest, along with edges from UnodeFile to FileContent
Reviewed By: ikostia
Differential Revision: D24838460
fbshipit-source-id: 4bc860a65c5489ba9b2bff4582967d9db6181f25
Summary: When doing a deep walk step into history via the unode's linked changeset.
Reviewed By: ikostia
Differential Revision: D24838459
fbshipit-source-id: cf2196f95e7b6bf3c3ba2cd1c44fef748a11597f
Summary: add UnodeManifest support to walker so they can be scrubbed
Reviewed By: mitrandir77
Differential Revision: D24604174
fbshipit-source-id: c60a020802fae109005263393ea5618f22f21888
Summary: Add bonsai to unode manifest mapping to the walker so we can scrub from bonsais
Reviewed By: mitrandir77
Differential Revision: D24604188
fbshipit-source-id: d98938c5ec640c479007b59163899c16d0e3fb9d
Summary:
Track ChangesetInfo derivation in the same way as Fsnode. This fixes repeated visits to ChangesetInfo
As part of this the matching in WalkState::needs_visit is made exhaustive, as it the default match had previously been returning true for ChangesetInfo nodes.
Reviewed By: mitrandir77
Differential Revision: D24838455
fbshipit-source-id: 33b8201984b294a1560da104b2cc9c65849b9297
Summary: This is in preparation to prevent marking visited for ChangesetInfo if it is not derived yet.
Reviewed By: mitrandir77
Differential Revision: D24838453
fbshipit-source-id: 0262b7f28754724bbfcc2407b4e649a90175cbcc
Summary:
Adding automatically generated derived_xxxx node groups so that less typing is
needed and we're also checking the nodes are mapped correctly to derived data types.
Reviewed By: mitrandir77
Differential Revision: D24838738
fbshipit-source-id: 2bc8ff03a82c5d18f749affba2e67d214fb7ace7
