Summary:
The method providing all that's needed for diffing two arbitrary commits.
It doesn't provide diff of the file contents themselves but it provides all
that's necessary to ask for their contents via `file_content` chunk or
diff via `commit_file_diffs`
Reviewed By: markbt
Differential Revision: D18224369
fbshipit-source-id: f814621e0b7904ae4029c64b25fff81f60d8909c
Summary:
This diff updates all license headers to use the new text and style.
Also, a few internal files were missing the header, but now they have it.
`fbcode/common/rust/netstring/` had the internal header, but now it has
GPLV2PLUS - since that goes to Mononoke's Github too.
Differential Revision: D17881539
fbshipit-source-id: b70d2ee41d2019fc7c2fe458627f0f7c01978186
Summary:
This function finds entries that were "introduced" by this manifest i.e.
entries that are present in this manifest but not present in any of
`diff_against` manifests.
This function was used in fastlog, and I'm planning to use it in the new sync
job, so move it to ManifestOps crate (and rename along the way).
One small change in behaviour is that we removed a single traced...() call. I
talked with aslpavel who introduced it and he's fine with removing it -
unfortunately our traces are not in a great shape yet, so this trace call is
not used at all.
Reviewed By: aslpavel
Differential Revision: D17477055
fbshipit-source-id: cba1a55a64299857efe4f4be7a67b6faf31e7019
Summary:
Collect together all derived data in the `derived_data` directory. This moves
the `unodes` derivation crate.
Reviewed By: StanislavGlebik
Differential Revision: D17314208
fbshipit-source-id: f1b575446192cb9799e443efb1cb863b80ef72d6
Summary:
Fsnodes are manifests, and should implement the `Manifest` trait. This will
allow us to derive them using `derive_manifest`.
Fsnodes only exist for directories: there is no fsnode data for the leaves, so the
leaf type and leaf id type are both `(ContentId, FileType)`.
Reviewed By: StanislavGlebik
Differential Revision: D17286037
fbshipit-source-id: 4ffc26f702cd68b938c52471242eac40ac9cec78
Summary: This change extends `derive_manifest` so it would pass additional optional context variable associated with subnodes. This is useful when we want to track some additional information associated with subtree, without either storing it inside entry or re-fetching it from blobstore.
Reviewed By: markbt
Differential Revision: D17397494
fbshipit-source-id: bab87a70cc680fa539c8a26a71032ce776d82127
Summary:
This diff moves initFacebook calls that used to happen just before FFI calls to instead happen at the beginning of main.
The basic assumption of initFacebook is that it happens at the beginning of main before there are additional threads. It must be allowed to modify process-global state like env vars or gflags without the possibility of a data race from other code concurrently reading those things. As such, the previous approach of calling initFacebook through `*fbinit::FACEBOOK` near FFI calls was prone to race conditions.
The new approach is based on attribute macros added in D17245802.
---
The primary remaining situations that still require `*fbinit::FACEBOOK` are when we don't directly control the function arguments surrounding the call to C++, such as in lazy_static:
lazy_static! {
static ref S: Ty = {
let _ = *fbinit::FACEBOOK;
/* call C++ */
};
}
and quickcheck:
quickcheck! {
fn f(/* args that impl Arbitrary */) {
let _ = *fbinit::FACEBOOK;
/* call C++ */
}
}
I will revisit these in a separate diff. They are a small fraction of total uses of fbinit.
Reviewed By: Imxset21
Differential Revision: D17328504
fbshipit-source-id: f80edb763e7f42b3216552dd32f1ea0e6cc8fd12
Summary: I modified statistics_collector tool, so now it calculates both number of files, and total file size in repo.
Reviewed By: krallin
Differential Revision: D17342512
fbshipit-source-id: 94217d8b61c2a7350f1793a2ef33f84d600bbb54
Summary: First version of binary that counts number of files in specific repo and print it.
Reviewed By: StanislavGlebik
Differential Revision: D17285812
fbshipit-source-id: 30bc9e2c11ee75fcfb8d94610bd4e320a56dafc7
Summary:
Extract the test utils from derive_unode_manifest to their own crate so that
they can be re-used in future tests.
Reviewed By: StanislavGlebik
Differential Revision: D17282411
fbshipit-source-id: 50410cffe8a912bd07283bc6ac4e97e28663d854
Summary: This commit tries to improve performance by batching multiple updates of `DerivedDataMapping` into one
Reviewed By: StanislavGlebik
Differential Revision: D17156437
fbshipit-source-id: dd80169b39b43c58cb90b0347cfe95e24e341d0b
Summary: Update bounded_traversal_stream to take IntoIterator of initial values. This allows simultaneous navigation of a graph from multiple roots.
Reviewed By: farnz
Differential Revision: D17163609
fbshipit-source-id: c999e7653cb620c215331ecc46f5a800ced8ef37
Summary:
In order to answer file history requests history we need to avoid doing serial
fetches from a blobstore.
This diff is a first attempt in implementing it. The general idea is to store precalculated history in the blobstore.
Note that at the moment it covers only the very basic case - file history for a commit with no parents, so there's quite a few "TODOs" and "unimplemented!()" in the codebase. This functionality will be extended in the next diffs
Reviewed By: krallin
Differential Revision: D17070901
fbshipit-source-id: 8150afb2509fcd2a428d2369bab58468ab774d72
Summary: If some blob is missing it becomes very hard to debug what is missing. This diff extends `LoadableError::Missing` with string argument.
Reviewed By: krallin
Differential Revision: D17132049
fbshipit-source-id: df1f60b1f739c8a837d1b1798edbaca58b1587f9
Summary: It is often cumbersome to add work with `find_entries` when we only need entry for single path. This function simplifies this case by adding helper function which only works with single path `find_entry`.
Reviewed By: StanislavGlebik
Differential Revision: D17115042
fbshipit-source-id: 0c4ae0d0b61994a2f04a119ec78a0de0394fdf0a
Summary:
Derived data generation is rather slow at the moment - up to 3 mins. I've added
a few printlns that showed that the problem is in the sequential writes.
This diff makes it so that writes to the blobstore happen in parallel.
Reviewed By: aslpavel
Differential Revision: D17093614
fbshipit-source-id: 771ccddfdb742e44a56dbd5be935559447c97298
Summary:
Utility to backfill derived data:
- currently only support `unodes`
Reviewed By: krallin
Differential Revision: D16827897
fbshipit-source-id: aa0bb8ae4fe895233e3f11775b3432184afb37f9
Summary:
This is a part of an effort to move mercurial related stuff into mercurial crate:
- `BlobManifest|HgBlobEntry` -> `mercurial/types/blobs`
- helper `Id` type is no longer required and `BlobManifeset` can be loaded with `HgManifesetId::load`
- `BlobManifest` implements `HgManifest` and typed `Manifest` traits
Reviewed By: StanislavGlebik
Differential Revision: D16984745
fbshipit-source-id: 33d4eb9c33137258956917328efa1f5ec6737ee9
Summary:
This is prerequisite step before moving hg blob `fetch|upload` functionality to mercurial crate.
- Use `Arc<dyn Blobstore>` instead of `RepoBlobstore`
- do not use `BlobRepo` in `UploadHg*` and `fetch_*` structures and functions
Reviewed By: ikostia
Differential Revision: D16963058
fbshipit-source-id: be36599cb430a342348cc077885b2f32eaa4ee47
Summary:
Find entries now support prefixes as well as paths:
- selector can be anything that is convertable to `PathOrPrefix`
- returns `Option<MPath>` instead of `MPath`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D16937401
fbshipit-source-id: 24cce9719d52ce3cbc493575c2a6db5ebf121315
Summary:
The high level goal is to test new unodes functionality. In particular, we want
to test `list_directory()` functionality.
There's a problem though - `list_directory()` returns Nodehash and filetype for
each entry it returns. That's cheap to do with mercurial manifests but not
cheap with unodes:
1) Mercurial manifests store filetype (i.e. whether it's a file,
symlink or executable) in manifests. But we don't store it in unode manifests
(it's stored in a separate file unode entry).
2) Unodes don't have mercurial hashes at all.
However there's currently a single user of `list_directory()` call - this is
scmquery - and it just discards both of the fields - https://fburl.com/8vr5hibo.
So I decided to introduce a separate `list_directory_unodes()` method that
returns a list of directory entries without any additional info. The plan is to
use it in scmquery for repos that already has unodes backfilled. That's
intended to be short-term - ideally we replace current thrift interface with
something more modern or at the very least replace `list_directory` with
`list_directory_unodes()` completely.
Reviewed By: farnz
Differential Revision: D16918232
fbshipit-source-id: f05185be8eecbf0852c7ddb0a482d4a5a63f8b53
Summary:
We needed this hack because in derive_unode_manifest crate we needed to implement Loadable trait (previously defined in
blobstore crate) for ManifestUnodeId which was defined in mononoke_types crate.
Rust doesn't let us do that - see
https://stackoverflow.com/questions/35160995/implement-foreign-trait-for-foreign-type.
So we used this new Id type, which made it awkward to use. This diff fixes it.
Reviewed By: krallin
Differential Revision: D16885044
fbshipit-source-id: 329b09bb3b671a59824d1d2597a3907bebb82642
Summary:
In order to remove struct Id hack in the next diff we need to implement
Manifest trait for ManifestUnode struct. However, there's a problem:
ManifestUnode::TreeId type is ManifestUnodeId which implements
MononokeId trait. And all MononokeId types automatically implement
Loadable<Value=Option<MononokeId>>. But Manifest trait requires TreeId that
implement Loadable<Value=Self> - note that there's no Option<...>, and so
ManifestUnodeId Loadable implementation doesn't match implementation required
by Manifest trait.
There are a few ways to fix it (see one of them in D16885043), and after some
discussion we decided that the best way to proceed is to add a special error
LoadableError that might be `Missing` meaning that id we are trying to fetch
doesn't exist
Reviewed By: aslpavel
Differential Revision: D16915387
fbshipit-source-id: df57ff4a7ab4ddbff5812ce5929a742d949726af
Summary:
mononoke_types doesn't look like a good crate for it. Let's move to blobstore
crate instead
Reviewed By: krallin
Differential Revision: D16914142
fbshipit-source-id: 57602c5a644adc87beea0cbc92beeb0d901d5873
Summary:
This is the first diff in the stack that removes ugly hack `struct Id` that we
used in derived data (see more details about that later in the stack).
This diff reverses the dependency between blobstore crate and mononoke_types
crate. Previously blobstore depended on mononoke_types, and blobstore also had
Loadable/Storable traits. That means that we can't implement Loadable/Storable
traits for types from mononoke_types - we can't even import them without
creating cyclic dependency between crates! And we'll use Loadable in
mononoke_types later in the stack.
This diff reverses the dependency and moves Loadable/Storable in mononoke_types
Reviewed By: krallin
Differential Revision: D16884649
fbshipit-source-id: 20753f3e62980be7bb6a04ab9d19980bc459b40e
Summary:
This updates an earlier diff I had that introduced the typed fetch method on the Blobstore trait. Since we now have Storable and Loadable traits, it makes a little more sense to implement those for MononokeId and use that instead.
I was hoping this would mean we could move the implementation of those traits into the mononoke_types crate, but unfortunately it looks like the `where T: MononokeId` implementation for `Loadable` can't live there (Rust wants it next to the trait definition).
That said, this change does mean we can get rid of some manual BlobstoreBytes creation here and there, so that's still an upside.
In doing so, I updated the Loadable and Storable traits to require only a Blobstore reference, and to take ownership of what they're storing. This saves some clones, especially in the latter.
Reviewed By: aslpavel
Differential Revision: D16733733
fbshipit-source-id: 3ec3a97cef90d6850f2939ba3f5ec6e6e5b459fb
Summary: Use memcache leases to prevent simultaneous generation of derived data for the same changeset id.
Reviewed By: StanislavGlebik
Differential Revision: D16666659
fbshipit-source-id: 47e57449ab854e595a9dc860414c49afa966be01
Summary:
This moves typed fetches from the blobrepo to the blobstore. The upshot is that this allows consumers of a blobstore to do typed fetches, instead of forcing them to get bytes then cast bytes to a blob, then cast the blob to the thing they want.
This required refactoring our crate hierarchy a little bit by moving BlobstoreBytes into Mononoke Types, since we now need the Blobstore crate to depend on Mononoke types, whereas it was the other way around before (since BlobstoreValue was already there, that seems reasonable)
Reviewed By: StanislavGlebik
Differential Revision: D16486774
fbshipit-source-id: 05751986ce3cb7273d68a8b4ebe9957bb892bcd6
Summary:
Implement BonsaiDerived trait for root manifest unode. Note that it mainly just
calls `derive_unode_manifest()`
This diff links together code that derived unode manifest for a single commit and derived data library. After that diff is landed we should be able to run `RootManifestUnodeId::derive(...)` and get unodes for any bonsai commit
Reviewed By: krallin
Differential Revision: D16446044
fbshipit-source-id: 37d5a4ab4332fd6018a0bdd2c1be231dca164afc
Summary:
We have fixed parent order for bonsai changeset i.g. (p1, p2) != (p2, p1). We
need to keep the same in unodes. This diff adds a test for that.
Reviewed By: krallin
Differential Revision: D16446231
fbshipit-source-id: 2e750909496f1b5f84e256e129d61677956b846b
Summary:
Unode parents can be ancestors of each other in some rare cases (see comment). We decided
that preventing this case is not worth it - there's a potential perf hit with
no big benefits.
This diff adds a comment that makes this decision explicit.
Reviewed By: krallin
Differential Revision: D16437748
fbshipit-source-id: 95c74954f8dfde637a2ede35888ea24d54cb7415
Summary:
This diff handles a tricky case of merging two files with the same content but
different histories. See code comment for more details.
Reviewed By: krallin
Differential Revision: D16360117
fbshipit-source-id: 9a21473d49b63b7914e72357da63adcc35f5b612
Summary:
First stab at derived unodes. For now it should work, but only for linear
history e.g. no merges. Support for merges is coming in the next diffs.
The logic of deriving unodes is similar to the logic of deriving hg manifests
which was implemented in D16131665
Reviewed By: krallin
Differential Revision: D16333369
fbshipit-source-id: 52e1841295f1faf0306fb2dbce65674bb60b3d93
Summary: This is generic implementation of tree manifest. It is going to be used to replace current hg manifest generation logic, and to implement any tree manifest bonsai derived structures like UNodes.
Reviewed By: StanislavGlebik
Differential Revision: D16048964
fbshipit-source-id: 6ffbdabca8788f22d87a61daba412cd1f9447d57
