Summary: Now that `derive03` is the only version available, rename it to `derive`.
Reviewed By: krallin
Differential Revision: D24900106
fbshipit-source-id: c7fbf9a00baca7d52da64f2b5c17e3fe1ddc179e
Summary:
Generate walker EdgeType::outgoing_type() to reduce boilerplate
When defining edges if the edge variant and destination node at the same no extra parameters needed. If they are different then the node type of destination is passed in parens, e.g. BonsaiParent(BonsaiChangeset)
Reviewed By: StanislavGlebik
Differential Revision: D24687828
fbshipit-source-id: 1616c786d78242c2b3a8c7a1ba58cc1433ea0a26
Summary: Use create_graph to generate EdgeType enum in walker to reduce boiler plate needed when adding new derived node and edge types to the walker
Differential Revision: D24687827
fbshipit-source-id: 63337f4136c649948e0d3039529965c296c6d67e
Summary: Make the naming of bonsai fsnode edge variants consistent with the other edges in preparation for building them programmatically from a macro
Reviewed By: krallin
Differential Revision: D24687833
fbshipit-source-id: 8d46a53c023a4b8f95c0edc42df86e467c054ebb
Summary: Make the naming of linknode edge variants consistent with the other edges in preparation for building them programmatically from a macro
Reviewed By: krallin
Differential Revision: D24687832
fbshipit-source-id: 46525d7bebd17723a130a70f566b24104cc39656
Summary:
Use macro to implement Node::get_type() in walker.
Reduces the boiler plate when adding new types to the graph.
Reviewed By: farnz
Differential Revision: D24687826
fbshipit-source-id: 5f89c6fb69fd9df3fff25a2425a4d2035dbf5ed9
Summary: Generate NodeType::root_edge_type() so less boiler plate when adding new types to the walk.
Reviewed By: farnz
Differential Revision: D24687825
fbshipit-source-id: 083fc57aee8fe01b29ad4a6f9ebe660cc057dfab
Summary:
Define walker graph with macro to reduce repetition which should make adding new derived data types simpler.
Specifically, this removes the duplication between NodeType and Node
Reviewed By: farnz
Differential Revision: D24687831
fbshipit-source-id: 97d67faf02b2a88bb871dc0388d75d3dd3e8528d
Summary: use strum EnumCount instead of own macros, can remove some code. This needed strum upgrade to 0.19
Reviewed By: krallin
Differential Revision: D24680441
fbshipit-source-id: 56e5b66f75c3d8ff949685c26f503571873c0cde
Summary: Update to strum 0.19 as it has improved EnumCount derivation
Reviewed By: mohanz
Differential Revision: D24680442
fbshipit-source-id: 2d3d2a84e994f09bf3b1c7ea748a80a67d100c13
Summary:
This updates the external facing API of the filestore to use 0.3 streams.
Internally, there is still a bit of 0.3 streams, but as of this change, it's
all 0.3 outside.
This required a few changes here and there in places where it was simpler to
just update them to use 0.3 futures instead of `compat()`-ing everything.
Reviewed By: ikostia
Differential Revision: D24731298
fbshipit-source-id: 18a1dc58b27d129970a6aa2d0d23994d5c5de6aa
Summary:
Like it says in the title. This required quite a lot of changes at callsites,
as you'd expect.
Reviewed By: StanislavGlebik
Differential Revision: D24731299
fbshipit-source-id: e58447e88dcc3ba1ab3c951f87f7042e2b03eb2c
Summary:
This updates the metadata APIs in the filestore to futures 0.3 & async / await.
This changes the external API of the filestore, so there's quite a bit of churn
outside of that module.
Reviewed By: markbt
Differential Revision: D24727255
fbshipit-source-id: 59833f185abd6ab9c609c6bcc22ca88ada6f1b42
Summary: We don't need to declare a fake empty version number
Reviewed By: farnz
Differential Revision: D24757981
fbshipit-source-id: 594c97e225704d783bea723efcbb9dfc4d5d800b
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary: Let strum crate do this for us
Reviewed By: krallin
Differential Revision: D24680444
fbshipit-source-id: dbde0077c105d6cc572a0c863bcb4d043714d441
Summary: Update walker to use MappedHgChangesetId derived data now that hg data is modeled like other derived data. This simplifies bonsai_to_hg_mapping_step
Reviewed By: aslpavel
Differential Revision: D24531553
fbshipit-source-id: 62663d8d47ec7145980c4cd567cba3009b1999cb
Summary: Add ChangesetInfo derived data to the walker so that it can be scrubbed and validated
Reviewed By: farnz
Differential Revision: D24312123
fbshipit-source-id: 84b3bba87e5391339f97cd2e5ae0313761726d02
Summary: Add the ability to specify a group of node types in one go on the command line
Reviewed By: farnz
Differential Revision: D24526827
fbshipit-source-id: 59d2f0cd06dbbe2555625023be3725a528256005
Summary:
Prevent the walker from trying to walk derived data node types that are not configured for a repo.
This is done by add a mapping from walker NodeType to derived data ::NAME, connect it up for existing derived data usages and using it for validation in setup.rs
Reviewed By: farnz
Differential Revision: D24391591
fbshipit-source-id: 21ae63f4f210d2e1310b0ee2c509fb492f742db7
Summary: SQLBlob GC (next diff in stack) will need a ConfigStore in SQLBlob. Make one available to blobstore creation
Reviewed By: krallin
Differential Revision: D24460586
fbshipit-source-id: ea2d5149e0c548844f1fd2a0d241ed0647e137ae
Summary: Use the new BonsaiDerived::fetch_derived from the walker when scrubbing fsnode derived data. This saves a round trip to derived data and simplifies bonsai_to_fsnode_mapping_step().
Reviewed By: krallin
Differential Revision: D24311164
fbshipit-source-id: b26c6bdef548ca15c0cdbffde675f1a11d8b510e
Summary: In D24447404, I provided some utility functions that allowed me to avoid constructing and/or passing around a ConfigStore. Remove those functions and fix up the code to run.
Reviewed By: krallin
Differential Revision: D24502692
fbshipit-source-id: 742dbc54fbcf735895d6829745b9317af14dfa0b
Summary: It's designed as a singleton store for normal use - rather than have lots of ways to get different config stores, let's use one global store
Reviewed By: krallin
Differential Revision: D24447404
fbshipit-source-id: 6ecc46351b14794471f654cec98527a11a93d3ef
Summary:
This trait is no longer used all that much outsides of a handful of tests, the
walker, and an admin subcommand, as it has been replaced by the `Manifest`
trait, which works over all kinds of Manifests, and has stronger typing (its
sub-entries always have a path, and they are wrapped in an enum that knows if
they're leaves or trees).
This left a bunch of old legacy code here or there, which is worth removing
to make sure we don't introduce any new callsites to this. Another motivation
is that this legacy code is often not very compatible with new code, and has
historically made it a bit tricky (everything owns a blobstore in this code,
which is pretty awkward and not at all how we do things nowadays).
There is, I think, a bit more potential here since we could also perhaps try to
remove the `HgBlobEntry` struct, but that has a callsites still, so I'm not
doing this here.
Reviewed By: StanislavGlebik
Differential Revision: D24306946
fbshipit-source-id: 8a73dbbf40a904ce19ac65d791b732091c206263
Summary: Reorder scrub walk expansion to reduce queue depth. The bounded_traversal_stream inserts to its unscheduled queue at the front, therefore reversing the order of the children, so we want to add the easiest to complete Nodes to the list of edges last.
Reviewed By: farnz
Differential Revision: D24247625
fbshipit-source-id: 59caa5898e7f38f41cc04a15723370de38f8474f
Summary:
Implement BlobstorePutOps for multiplex blobstores, so restoring the ability to have multiplexes of multiplexes.
Note that this makes BlobstorePutOps::put_behaviour accessor problematic as the inner stores could have different put_behaviour, will remove it in next diff so as to keep this diff reasonable size.
Reviewed By: StanislavGlebik
Differential Revision: D24162235
fbshipit-source-id: 2ace3af5f60607996e449451316c5c0720351f82
Summary: Add error context to file content scrub so that we can tell if an Error has propagated via the scrub stream loading.
Reviewed By: StanislavGlebik
Differential Revision: D23838144
fbshipit-source-id: 40a8a090510959cab1020182c19076b8a3317b1b
Summary: Intern ids to reduce space used in the walk state. This is significant on large repos.
Reviewed By: farnz
Differential Revision: D23691524
fbshipit-source-id: b42f926d88083d06ffc44508db44747f9a14e0a5
Summary: Small change to make it more readable and reduce likelihood of allocation (although the collect might be optimized away anyway)
Reviewed By: farnz
Differential Revision: D23760762
fbshipit-source-id: 5c47352386de128b65052d63b3f3ff1081a462e3
Summary:
Save memory in walker state tracking by not memoing hash values. For large repos this is significant.
I was expecting a small slowdown from this, but so far looks pretty much the same walk rate. Speculation: this may be due to the num cpus lock sharding fix in dashmap 3.11.10 which means there are many more shards than when the memo was tested with 3.11.9, so saving time inside locks is less significant.
Reviewed By: StanislavGlebik
Differential Revision: D23680550
fbshipit-source-id: 351b5ec39885fc30996207c7dccc22c749e30321
Summary: For fsnodes output the filecontent child nodes first as they can be drained without expanding to more nodes.
Reviewed By: farnz
Differential Revision: D23702268
fbshipit-source-id: 26aeca20d47030dbb9145d406db885fe0fce932c
Summary:
In preparation of moving away from SSH as an intermediate entry point for
Mononoke, let Mononoke work with newly introduced Metadata. This removes any
assumptions we now make about how certain data is presented to us, making the
current "ssh preamble" no longer central.
Metadata is primarily based around identities and provides some
backwards-compatible entry points to make sure we can satisfy downstream
consumers of commits like hooks and logs.
Simarly we now do our own reverse DNS resolving instead of relying on what's
been provided by the client. This is done in an async matter and we don't rely
on the result, so Mononoke can keep functioning in case DNS is offline.
Reviewed By: farnz
Differential Revision: D23596262
fbshipit-source-id: 3a4e97a429b13bae76ae1cdf428de0246e684a27
Summary: Noticed there had been an upstream 3.11.10 release with a fix for a performance regression in 3.11.9, PR was https://github.com/xacrimon/dashmap/issues/100
Reviewed By: krallin
Differential Revision: D23690797
fbshipit-source-id: aff3951e5b7dbb7c21d6259e357d06654c6a6923
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: StanislavGlebik
Differential Revision: D23568780
fbshipit-source-id: b4b4a0aa683d236e2fdeb5b96d723ac2d84b9faf
Summary:
In the next diff I'm going to add log-only mode to redaction, and it would be
good to have a way of testing it (i.e. testing that it actually logs accesses
to bad keys).
In this diff let's use a config option that allows logging censored scuba
accesses to file, and let's update redaction integration test to use it
Reviewed By: ikostia
Differential Revision: D23537797
fbshipit-source-id: 69af2f05b86bdc0ff6145979f211ddd4f43142d2
Summary:
Segmented Changelog is a component that has multiple components of each own
that each can be configured in different ways. It seems that it already is
more complicated than other components in how it is set up and it will probably
evolve to have more knobs (caching comes to mind).
Right now we have 3 ways of instantiating SegmentedChangelog:
- Disabled, all requests return errors
- ReadOnly, requests to unprocessed commits return errors
- OnDemandUpdate, requests trigger commit processing when required
Reviewed By: aslpavel
Differential Revision: D23456217
fbshipit-source-id: a6016f05197abbc3722764fa8e9056190a767b36
Summary: Memo the hash values of interned paths in the walker. The interner calls the hash function inside a lock that gets heavily contended, so this reduces the time the lock is held.
Reviewed By: farnz
Differential Revision: D23075260
fbshipit-source-id: 3ee50e3ce56106eadd17dc7d737ba95282640051
Summary: Switch the walker from arc-intern::ArcIntern to internment::ArcIntern as internment does not need to acquire its map's locks on every drop.
Reviewed By: farnz
Differential Revision: D23075265
fbshipit-source-id: 6dd241aed850ec0fd3c8a4e68dda06053ec0b424
Summary: Ctime is an Option<i64>, so rather than as_ctime()/into_ctime() use the fact that it's fairly small and Copy to just .ctime()
Reviewed By: krallin
Differential Revision: D23081739
fbshipit-source-id: be62912eca02e5c29d7473d6f386d98df11000dd
Summary:
Now that the previous diff has pre-computed the hash value using EagerHashMemo, its less expensive to try a read-lock only get() first before committing to a write lock acquiring insert().
The combination of these and the previous diff moved WalkState::visit from dominating the cpu profile to not ( the path interning dominates now ).
Reviewed By: krallin
Differential Revision: D22975881
fbshipit-source-id: 90b2be83282ee2095c517c0d4f13536ddadf6267
Summary:
DashMap takes the hash of its keys multiple times, once outside the lock, and then once or twice inside the lock depending if the key is present in the shard.
Pre-computing the hash value using EagerHashMemo means its done only once and more importantly, outside the lock.
To use EagerHashMemo one needs to supply the BuildHasher, so its added as a struct member and the record method is made a member function.
Reviewed By: farnz
Differential Revision: D22975878
fbshipit-source-id: c2ca362fdfe31e5dca329e6200029207427cd9a1
Summary: This has my into_key() PR https://github.com/xacrimon/dashmap/pull/91 merged so the patch pointing to my fork is also removed.
Reviewed By: farnz
Differential Revision: D22896911
fbshipit-source-id: 188d438ce2aa20cfb3c466a62227d1cd27625f74
