Summary:
Allow users of `tests_utils` to create paths that are not `String`, by supporting any type
that can be converted into `MPath`.
Reviewed By: StanislavGlebik
Differential Revision: D24887002
fbshipit-source-id: 47ad567507185863c1cfa3c6738f30aa9266901a
Summary:
`manifest/test_utils` contains test utilities that are only used by derived data, and only one
of which relates to manifests. Its name (`test_utils`) is also confusing with `tests/utils`.
Move it to `derived_data_test_utils`, and update it to new futures.
Reviewed By: mitrandir77
Differential Revision: D24787536
fbshipit-source-id: 7a4a735132ccf81e3f75683c7f44c9ada11bc9d7
Summary:
This updates the external facing API of the filestore to use 0.3 streams.
Internally, there is still a bit of 0.3 streams, but as of this change, it's
all 0.3 outside.
This required a few changes here and there in places where it was simpler to
just update them to use 0.3 futures instead of `compat()`-ing everything.
Reviewed By: ikostia
Differential Revision: D24731298
fbshipit-source-id: 18a1dc58b27d129970a6aa2d0d23994d5c5de6aa
Summary:
Like it says in the title. This required quite a lot of changes at callsites,
as you'd expect.
Reviewed By: StanislavGlebik
Differential Revision: D24731299
fbshipit-source-id: e58447e88dcc3ba1ab3c951f87f7042e2b03eb2c
Summary:
This updates the metadata APIs in the filestore to futures 0.3 & async / await.
This changes the external API of the filestore, so there's quite a bit of churn
outside of that module.
Reviewed By: markbt
Differential Revision: D24727255
fbshipit-source-id: 59833f185abd6ab9c609c6bcc22ca88ada6f1b42
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary:
Now that fsnodes is async, convert more functions to use references, and tidy
up imports and type names.
Reviewed By: krallin
Differential Revision: D24726145
fbshipit-source-id: 75a619777f19754daf494a3743d26fa2e77aef54
Summary:
Update `fsnodes::derive_fsnode` and its immediate utility functions to use new style
futures and `async`/`.await` syntax.
Reviewed By: krallin
Differential Revision: D24726146
fbshipit-source-id: 0b0d5b1162a73568ef5c47db6e8252267e760e7f
Summary:
This changes the methods from ones that return old `BoxFuture`s to an async method
using `async_trait`.
Reviewed By: krallin
Differential Revision: D24689506
fbshipit-source-id: 7b13010924369f81681e6590898af703c5423385
Summary:
This changes the trait method from one that returns an old `BoxFuture` to an async
method using `async_trait`.
Reviewed By: krallin
Differential Revision: D24686888
fbshipit-source-id: 0ac231cdbb60d256b6d5ad5aafbe8779b96905f3
Summary:
Add BonsaiDerived::derive03 for new callsites that are 0.3/async aware.
Plan is to migrate callsites and then rename BonsaiDerived::derive03 to derive.
Reviewed By: StanislavGlebik
Differential Revision: D24389763
fbshipit-source-id: 4a6f351b9e0c5a2e9a9c47429764f88c47caff08
Summary: No callsites when I search
Reviewed By: StanislavGlebik
Differential Revision: D24389765
fbshipit-source-id: ed1985922a9c1af51afb37fb5e331d32c29fd09e
Summary:
This diff adds a method BonsaiDerived::fetch_derived which is useful for callers currently doing BonsaiDerived::is_derived, and then BonsaiDerived::fetch_derived.
Using the new method cuts out one round trip to derived data, and simplifies calling code. Usage of this in the walker when scrubbing fs nodes is in the next diff.
Reviewed By: StanislavGlebik
Differential Revision: D24311166
fbshipit-source-id: dc52a15076768a6a15ede6dd5872aaaf3f552b66
Summary:
- convert ManifestOps to new style futures
- at this point `//eden/manifest:manifest` crate is completely free from old style futures
Reviewed By: krallin
Differential Revision: D24502214
fbshipit-source-id: f1cdb11bd8234f22af5c905243f71e1e9fca11f1
Summary: `derive_manifest_*` functions are now based on new futures and use new bounded_traversal
Reviewed By: krallin
Differential Revision: D24448824
fbshipit-source-id: 5af06851f9c48a73e53daafe1f4972c186a5ca80
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.
There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.
Reviewed By: jsgf
Differential Revision: D24445298
fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
Summary:
This diff add new mode of tailing based on derived data graph, it uses same functionality as backfill.
- `tail_batch_iteration` uses `bounded_traversal_dag` directly instead of leveraging `DeriveGraph::derive` so we could warm-up dependencies for each node before calling `DerivedUitls::backfill_batch_dangerous`
Reviewed By: StanislavGlebik
Differential Revision: D24306156
fbshipit-source-id: 006cb6d4df9424cd6501cb4a381b95f096e70551
Summary: Convert derived data utils to use new style futures
Reviewed By: StanislavGlebik
Differential Revision: D24331068
fbshipit-source-id: ad658b278802afa1e4ecd44c5a24164135748790
Summary: Rename futures01 types from Foo to Foo01 in the top level lib.rs and derive_impl.rs files in preparation for adding a trait method that returns new futures
Reviewed By: aslpavel
Differential Revision: D24311165
fbshipit-source-id: 4f3b12ba3eaf8023959d6d4bbb4568d191b1fffb
Summary:
This trait is no longer used all that much outsides of a handful of tests, the
walker, and an admin subcommand, as it has been replaced by the `Manifest`
trait, which works over all kinds of Manifests, and has stronger typing (its
sub-entries always have a path, and they are wrapped in an enum that knows if
they're leaves or trees).
This left a bunch of old legacy code here or there, which is worth removing
to make sure we don't introduce any new callsites to this. Another motivation
is that this legacy code is often not very compatible with new code, and has
historically made it a bit tricky (everything owns a blobstore in this code,
which is pretty awkward and not at all how we do things nowadays).
There is, I think, a bit more potential here since we could also perhaps try to
remove the `HgBlobEntry` struct, but that has a callsites still, so I'm not
doing this here.
Reviewed By: StanislavGlebik
Differential Revision: D24306946
fbshipit-source-id: 8a73dbbf40a904ce19ac65d791b732091c206263
Summary:
We don't want to log slow derivations in tests. To do that, let's look at at
unable to decide if we should log.
We only log if the threshold is > 0, so since the default is 0, that means we
won't log unless a config overrides it (which won't be the case in tests, but
will be the case everywhere else: D24329372).
Reviewed By: StanislavGlebik
Differential Revision: D24329377
fbshipit-source-id: e9d70236c614780e383870297d08afaf64450383
Summary: Add support for deriving all types of derived data for a given repository
Reviewed By: StanislavGlebik
Differential Revision: D23842909
fbshipit-source-id: 2fa5c4a9444169b26c5cf70d91a6cc707cca8022
Summary: This change make it possible to find underived changesets for multiple derived data types at once, and then batch derive everything taking into account dependencies between derived data types.
Reviewed By: StanislavGlebik
Differential Revision: D23762152
fbshipit-source-id: 8112fc5deb86f3c55f2fa5079cf917a0506f045c
Summary:
Spotted a TODO in fsnode get_fsnode_id. There was only one user of the function, which didn't really need to call it as it had the blob already.
As well as being a bit tidier this also saves a clone of the fsnode.
Reviewed By: StanislavGlebik
Differential Revision: D23758689
fbshipit-source-id: e0a8c124c929fda3af4c96a76d441a79e5bfbd5b
Summary: Use sorted_vector_map when parsing fsnodes, as inputs are stored sorted, which can result in high cost of BTree insertion when traversing large repos.
Reviewed By: aslpavel
Differential Revision: D23691500
fbshipit-source-id: 1f7a5faf2ef3cb4a72a635d0d8e89037bf4d96b3
Summary:
In preparation of moving away from SSH as an intermediate entry point for
Mononoke, let Mononoke work with newly introduced Metadata. This removes any
assumptions we now make about how certain data is presented to us, making the
current "ssh preamble" no longer central.
Metadata is primarily based around identities and provides some
backwards-compatible entry points to make sure we can satisfy downstream
consumers of commits like hooks and logs.
Simarly we now do our own reverse DNS resolving instead of relying on what's
been provided by the client. This is done in an async matter and we don't rely
on the result, so Mononoke can keep functioning in case DNS is offline.
Reviewed By: farnz
Differential Revision: D23596262
fbshipit-source-id: 3a4e97a429b13bae76ae1cdf428de0246e684a27
Summary: Support for multiple heads in `BonsaiDerived::find_all_underived_ancestors`. This change will be needed to remove manual step of fetching of all changesets in `backfill_derived_data` utilty.
Reviewed By: StanislavGlebik
Differential Revision: D23705295
fbshipit-source-id: 32aa97a77f0a4461cbe4bf1864477e3e121e1879
Summary:
This completely converts mercurial changeset to be an instance of derived data:
- Custom lease logic is removed
- Custom changeset traversal logic is removed
Naming scheme of keys for leases has been changed to conform with other derived data types. This might cause temporary spike of cpu usage during rollout.
Reviewed By: farnz
Differential Revision: D23575777
fbshipit-source-id: 8eb878b2b0a57312c69f865f4c5395d98df7141c
Summary:
This change move logic associated with mercurial changeset derivation to `mercurial_derived_data` crate.
NOTE: it is not converted to derived data infrastructure at this point, it is a preparation step to actually do this
Reviewed By: farnz
Differential Revision: D23573610
fbshipit-source-id: 6e8cbf7d53ab5dbd39d5bf5e06c3f0fc5a8305c8
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: StanislavGlebik
Differential Revision: D23568780
fbshipit-source-id: b4b4a0aa683d236e2fdeb5b96d723ac2d84b9faf
Summary:
Fsnodes have a lot of data about files, but right now we can't access it
through a Fsnode lookup or a manifest walk, because the LeafId for a Fsnode is
just the content id and the file type.
This is a bit sad, because it means we e.g. cannot dump a manifest with file
sizes (D23471561 (179e4eb80e)).
Just changing the LeafId is easy, but that brings a new problem with Fsnode
derivation.
Indeed, deriving manifests normally expects us to have the "derive leaf"
function produce a LeafId (so we'd want to produce a `FsnodeFile`), but in
Fsnodes, this currently happens in deriving trees instead.
Unfortunately, we cannot easily just move the code that produces `FsnodeFile`
from the tree derivation to the leaf derivation, that is, do:
```
fn check_fsnode_leaf(
leaf_info: LeafInfo<FsnodeFile, (ContentId, FileType)>,
) -> impl Future<Item = (Option<FsnodeSummary>, FsnodeFile), Error = Error>
```
Indeed, the performance of Fsnode derivation relies on all the leaves for a
given tree being derived together with the tree and its parents in context.
So, we'd need the ability for deriving a new leaf to return something different
from the actual leaf id. This means we want to return a `(ContentId,
FileType)`, even though our `LeafId` is a `FsnodeFile`.
To do this, this diff introduces a new `IntermediateLeafId` type in the
derivation. This represents the type of the leaf that is passed from deriving a
leaf to deriving a tree. We need to be able to turn a real `LeafId` into it,
because sometimes we don't re-derive leaves.
I think we could also refactor some of the code that passes a context here to
just do this through the `IntermediateLeafId`, but I didn't look into this too
much.
So, this diff does that, and uses it in Mononoke Admin so we can print file
sizes.
Reviewed By: StanislavGlebik
Differential Revision: D23497754
fbshipit-source-id: 2fc480be0b1e4d3d261da1d4d3dcd9c7b8501b9b
Summary:
Most out our APIs throw error when the path doesn't exist. I would like to
argue that's not the right choice for list_file_history.
Errors should be only retuned in abnormal situations and with
`history_across_deletions` param there's no other easy way to check if the file
ever existed other than calling this API - so it's not abnormal to call
it with path that doesn't exist in the repo.
Reviewed By: StanislavGlebik
Differential Revision: D22820263
fbshipit-source-id: 002bda2ef5ee9d6632259a333b7f3652cfb7aa6b
Summary: now the terminator argument is unused - we can get rid of it.
Differential Revision: D22502594
fbshipit-source-id: e8ecec01002421baee38be0c7e048d08068f2d74
Summary:
This new trait is going to replace the `Terminator` argument to fastlog
traversal function. Insted of deciding if we should fetch or/not given fastlog
batch this trait allows us to make decisions based on each visited changeset.
Differential Revision: D22502590
fbshipit-source-id: 19f9218958604b2bcb68203c9646b3c9b433541d
Summary:
The function for finding the commit where the file was deleted
in the fastlog module doesn't depend on fastlog at all.
It also seems generic enough to be a good public API for deleted files
manifests module.
Differential Revision: D22502596
fbshipit-source-id: 2e226bf14339da886668ee8e3402a08e8120266e
Summary:
Let's centralize the logic that adds new nodes to BFS queue during fastlog
traversal, this will allow me to hook into it in the next diffs.
Differential Revision: D22502593
fbshipit-source-id: 63f4e7adb3a7e11b4a2b2dcc65cab3bb4bf6f015
