Summary:
Utility to backfill derived data:
- currently only support `unodes`
Reviewed By: krallin
Differential Revision: D16827897
fbshipit-source-id: aa0bb8ae4fe895233e3f11775b3432184afb37f9
Summary:
This is a part of an effort to move mercurial related stuff into mercurial crate:
- `BlobManifest|HgBlobEntry` -> `mercurial/types/blobs`
- helper `Id` type is no longer required and `BlobManifeset` can be loaded with `HgManifesetId::load`
- `BlobManifest` implements `HgManifest` and typed `Manifest` traits
Reviewed By: StanislavGlebik
Differential Revision: D16984745
fbshipit-source-id: 33d4eb9c33137258956917328efa1f5ec6737ee9
Summary:
This is prerequisite step before moving hg blob `fetch|upload` functionality to mercurial crate.
- Use `Arc<dyn Blobstore>` instead of `RepoBlobstore`
- do not use `BlobRepo` in `UploadHg*` and `fetch_*` structures and functions
Reviewed By: ikostia
Differential Revision: D16963058
fbshipit-source-id: be36599cb430a342348cc077885b2f32eaa4ee47
Summary:
Find entries now support prefixes as well as paths:
- selector can be anything that is convertable to `PathOrPrefix`
- returns `Option<MPath>` instead of `MPath`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D16937401
fbshipit-source-id: 24cce9719d52ce3cbc493575c2a6db5ebf121315
Summary:
The high level goal is to test new unodes functionality. In particular, we want
to test `list_directory()` functionality.
There's a problem though - `list_directory()` returns Nodehash and filetype for
each entry it returns. That's cheap to do with mercurial manifests but not
cheap with unodes:
1) Mercurial manifests store filetype (i.e. whether it's a file,
symlink or executable) in manifests. But we don't store it in unode manifests
(it's stored in a separate file unode entry).
2) Unodes don't have mercurial hashes at all.
However there's currently a single user of `list_directory()` call - this is
scmquery - and it just discards both of the fields - https://fburl.com/8vr5hibo.
So I decided to introduce a separate `list_directory_unodes()` method that
returns a list of directory entries without any additional info. The plan is to
use it in scmquery for repos that already has unodes backfilled. That's
intended to be short-term - ideally we replace current thrift interface with
something more modern or at the very least replace `list_directory` with
`list_directory_unodes()` completely.
Reviewed By: farnz
Differential Revision: D16918232
fbshipit-source-id: f05185be8eecbf0852c7ddb0a482d4a5a63f8b53
Summary:
We needed this hack because in derive_unode_manifest crate we needed to implement Loadable trait (previously defined in
blobstore crate) for ManifestUnodeId which was defined in mononoke_types crate.
Rust doesn't let us do that - see
https://stackoverflow.com/questions/35160995/implement-foreign-trait-for-foreign-type.
So we used this new Id type, which made it awkward to use. This diff fixes it.
Reviewed By: krallin
Differential Revision: D16885044
fbshipit-source-id: 329b09bb3b671a59824d1d2597a3907bebb82642
Summary:
In order to remove struct Id hack in the next diff we need to implement
Manifest trait for ManifestUnode struct. However, there's a problem:
ManifestUnode::TreeId type is ManifestUnodeId which implements
MononokeId trait. And all MononokeId types automatically implement
Loadable<Value=Option<MononokeId>>. But Manifest trait requires TreeId that
implement Loadable<Value=Self> - note that there's no Option<...>, and so
ManifestUnodeId Loadable implementation doesn't match implementation required
by Manifest trait.
There are a few ways to fix it (see one of them in D16885043), and after some
discussion we decided that the best way to proceed is to add a special error
LoadableError that might be `Missing` meaning that id we are trying to fetch
doesn't exist
Reviewed By: aslpavel
Differential Revision: D16915387
fbshipit-source-id: df57ff4a7ab4ddbff5812ce5929a742d949726af
Summary:
mononoke_types doesn't look like a good crate for it. Let's move to blobstore
crate instead
Reviewed By: krallin
Differential Revision: D16914142
fbshipit-source-id: 57602c5a644adc87beea0cbc92beeb0d901d5873
Summary:
This is the first diff in the stack that removes ugly hack `struct Id` that we
used in derived data (see more details about that later in the stack).
This diff reverses the dependency between blobstore crate and mononoke_types
crate. Previously blobstore depended on mononoke_types, and blobstore also had
Loadable/Storable traits. That means that we can't implement Loadable/Storable
traits for types from mononoke_types - we can't even import them without
creating cyclic dependency between crates! And we'll use Loadable in
mononoke_types later in the stack.
This diff reverses the dependency and moves Loadable/Storable in mononoke_types
Reviewed By: krallin
Differential Revision: D16884649
fbshipit-source-id: 20753f3e62980be7bb6a04ab9d19980bc459b40e
Summary:
This updates an earlier diff I had that introduced the typed fetch method on the Blobstore trait. Since we now have Storable and Loadable traits, it makes a little more sense to implement those for MononokeId and use that instead.
I was hoping this would mean we could move the implementation of those traits into the mononoke_types crate, but unfortunately it looks like the `where T: MononokeId` implementation for `Loadable` can't live there (Rust wants it next to the trait definition).
That said, this change does mean we can get rid of some manual BlobstoreBytes creation here and there, so that's still an upside.
In doing so, I updated the Loadable and Storable traits to require only a Blobstore reference, and to take ownership of what they're storing. This saves some clones, especially in the latter.
Reviewed By: aslpavel
Differential Revision: D16733733
fbshipit-source-id: 3ec3a97cef90d6850f2939ba3f5ec6e6e5b459fb
Summary: Use memcache leases to prevent simultaneous generation of derived data for the same changeset id.
Reviewed By: StanislavGlebik
Differential Revision: D16666659
fbshipit-source-id: 47e57449ab854e595a9dc860414c49afa966be01
Summary:
This moves typed fetches from the blobrepo to the blobstore. The upshot is that this allows consumers of a blobstore to do typed fetches, instead of forcing them to get bytes then cast bytes to a blob, then cast the blob to the thing they want.
This required refactoring our crate hierarchy a little bit by moving BlobstoreBytes into Mononoke Types, since we now need the Blobstore crate to depend on Mononoke types, whereas it was the other way around before (since BlobstoreValue was already there, that seems reasonable)
Reviewed By: StanislavGlebik
Differential Revision: D16486774
fbshipit-source-id: 05751986ce3cb7273d68a8b4ebe9957bb892bcd6
Summary:
Implement BonsaiDerived trait for root manifest unode. Note that it mainly just
calls `derive_unode_manifest()`
This diff links together code that derived unode manifest for a single commit and derived data library. After that diff is landed we should be able to run `RootManifestUnodeId::derive(...)` and get unodes for any bonsai commit
Reviewed By: krallin
Differential Revision: D16446044
fbshipit-source-id: 37d5a4ab4332fd6018a0bdd2c1be231dca164afc
Summary:
We have fixed parent order for bonsai changeset i.g. (p1, p2) != (p2, p1). We
need to keep the same in unodes. This diff adds a test for that.
Reviewed By: krallin
Differential Revision: D16446231
fbshipit-source-id: 2e750909496f1b5f84e256e129d61677956b846b
Summary:
Unode parents can be ancestors of each other in some rare cases (see comment). We decided
that preventing this case is not worth it - there's a potential perf hit with
no big benefits.
This diff adds a comment that makes this decision explicit.
Reviewed By: krallin
Differential Revision: D16437748
fbshipit-source-id: 95c74954f8dfde637a2ede35888ea24d54cb7415
Summary:
This diff handles a tricky case of merging two files with the same content but
different histories. See code comment for more details.
Reviewed By: krallin
Differential Revision: D16360117
fbshipit-source-id: 9a21473d49b63b7914e72357da63adcc35f5b612
Summary:
First stab at derived unodes. For now it should work, but only for linear
history e.g. no merges. Support for merges is coming in the next diffs.
The logic of deriving unodes is similar to the logic of deriving hg manifests
which was implemented in D16131665
Reviewed By: krallin
Differential Revision: D16333369
fbshipit-source-id: 52e1841295f1faf0306fb2dbce65674bb60b3d93
Summary: This is generic implementation of tree manifest. It is going to be used to replace current hg manifest generation logic, and to implement any tree manifest bonsai derived structures like UNodes.
Reviewed By: StanislavGlebik
Differential Revision: D16048964
fbshipit-source-id: 6ffbdabca8788f22d87a61daba412cd1f9447d57
