Summary:
I really liked how expressing the problem in terms of data structures
made it fairly straightforward to understand.
Reviewed By: farnz
Differential Revision: D8586383
fbshipit-source-id: d1e3c92a3a5b760a0f13f4912420e2a73b937e8d
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary: Added logic to save `FileChange` as a Mercurial format `HgBlobEntry`
Reviewed By: sunshowers
Differential Revision: D8187792
fbshipit-source-id: 4714c81ab23ebac528cfec15c4a9e66083d4fb6c
Summary:
Implementation of generic `store|fetch` for bonsai types.
- bonsai types have unique typed hashes associated with each bonsai type, I'm leveraging this fact to implement generic `store|fetch` methods on `BlobRepo`
Reviewed By: farnz
Differential Revision: D8254810
fbshipit-source-id: 5f798fade4cb8d1ac851f94c7ad7e64636bbca65
Summary:
- `store|fetch` semantic requires `MononkeId::Value` type but `UnodeId` does not have associated value type.
- `UnodeId` is not used anywhere in codebase
Nuking it for now.
Reviewed By: lukaspiatkowski
Differential Revision: D8351415
fbshipit-source-id: 970ab8d3c0cdca6d77c96ea08643f90a4ce0e624
Summary:
We're going to need to be able to edit memory manifests. Provide
remove and set operations, to match the Bonsai Changeset data structures
Reviewed By: StanislavGlebik
Differential Revision: D7620527
fbshipit-source-id: e85459c5dbfa8855267fd2cf6578c9fc39f223f8
Summary:
Rust 1.26 adds many new language features. In particular `impl Trait` is now
stable, so we no longer need `conservative_impl_trait`.
There also seems to have been changed in the (unstable) TryFrom with respect to
usize, and the behaviour of the never type `!`.
There are still a few deprecation warnings, but they don't cause the build to
fail.
Path remapping is now stable, so the buck config needs to change to use it
rather than the unstable command line option.
TODO:
- get aarch64 rust-crates-io build (can defer to a later update)
Reviewed By: Imxset21
Differential Revision: D7966091
fbshipit-source-id: 2e61e262c21eb01c852a36f49c6a6369cdaddcdb
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary:
We'd like to move away from `RawNodeBlob` and `RawCSBlob` to data structures
serialized by Thrift. This is the first step to doing that.
The most important thing here is that it reuses file content IDs from native
Mononoke storage.
Reviewed By: jsgf
Differential Revision: D7771990
fbshipit-source-id: de4ee0b56aa6610caeff84b2235e19855df086cb
Summary: Want to get this in now before I forget.
Reviewed By: jsgf
Differential Revision: D7781551
fbshipit-source-id: a5e6fa062514d90113c96307f78e20fa05f770ea
Summary: Will be used for Thrift envelopes.
Reviewed By: jsgf
Differential Revision: D7771214
fbshipit-source-id: 3f0dbf77793064f2606ebe34672629c4e49cc7fe
Summary: This is an important consistency check for bonsai changesets.
Reviewed By: jsgf
Differential Revision: D7755664
fbshipit-source-id: 4e64ed532d0730147efec7a2b10b61f625e50dd0
Summary:
This does mean that we need to adjust QuickCheck generation a bit to ensure
validity.
Reviewed By: jsgf
Differential Revision: D7755635
fbshipit-source-id: ebca6f5a52038b64f83ff599079460daf061cb44
Summary: I got annoyed at the lack of this.
Reviewed By: StanislavGlebik
Differential Revision: D7755636
fbshipit-source-id: 0d519dca94ed93e3efe8d12f9004661b92fb8b1a
Summary:
This redefines file changes and deletes slightly -- see the comments
for a full description.
Reviewed By: jsgf, farnz
Differential Revision: D7750040
fbshipit-source-id: f1c52295cd21150fbdc909198104c0571132431d
Summary:
We know that the hashes for non-root-tree-manifests and filenodes
should always be consistent. Verify that.
Reviewed By: farnz
Differential Revision: D7704087
fbshipit-source-id: 7f6207878c5cd372b272aa6970506dd63b5a3c7c
Summary:
This is the first step to getting bonsai changesets into the blob store.
So far the code just waits for content blob uploads to be done. In future diffs Mononoke will synthesize this information into a bonsai changeset as well.
Reviewed By: lukaspiatkowski
Differential Revision: D7627784
fbshipit-source-id: da0c96772ce4d18aed579cecca6135137a8dbe18
Summary:
I was trying to debug something with the new blobimport, and this was
getting in the way.
Reviewed By: StanislavGlebik
Differential Revision: D7664660
fbshipit-source-id: 2ec4ee79fbe13584f35e7dcd9e8df2b8bdf181c0
Summary:
Currently, any sort of `Bytes` can be stored in the blobstore. That
caused me to make several mistakes while writing the code to store bonsai
changesets, because I'd just end up storing the wrong set of `Bytes`.
Introduce stronger typing so that only types that explicitly implement
`BlobStorable` can be stored in the blobstore.
Currently, these sorts of blobs can be stored in the blob store:
* `ChangesetBlob` and `ContentBlob` in `mononoke-types` (these are Thrift-serialized structures)
* The envelope `RawNodeBlob` and `RawCSBlob` types in `blobrepo`, once converted to `EnvelopeBlob` instances
* `HgBlob`, which contains revlog data (manifests or files) exactly as serialized by Mercurial
Reviewed By: StanislavGlebik
Differential Revision: D7627290
fbshipit-source-id: d1bcbde8881e365dec99618556e7b054985bccf7
Summary:
This is not only the newer, more specific type -- it also makes a couple
of upcoming diffs more straightforward.
Reviewed By: StanislavGlebik
Differential Revision: D7622906
fbshipit-source-id: 4e453b827512c538f4f9777ae4d24627f3b124cf
Summary:
Turns out this trait is actually really useful. In this case it'll be
used to generate blobstore types.
Reviewed By: StanislavGlebik
Differential Revision: D7608152
fbshipit-source-id: 792de037c8b028172f868b9b035ede3db9a39788
Summary:
This is going to make life more convenient in upcoming diffs, when these strings are going to be used to generate blobstore keys.
The switch to lowercase is to make those blobstore keys look nicer.
The switch to string literals is because Rust really supports strings a lot better than it does bytestrings. In particular, `concat!` for literals works with strings, but there's no equivalent for bytestring literals. For current purposes the two are equivalent -- there's no runtime cost to using a string literal here.
Reviewed By: StanislavGlebik
Differential Revision: D7598616
fbshipit-source-id: f2fa543429921cca5fd8dc354ea11000fdab3f76
Summary:
If there are multiple parents, it is important to know which parent to
follow for copy information.
Reviewed By: StanislavGlebik
Differential Revision: D7597811
fbshipit-source-id: 74bb0080671b18beb5483b978016bf66a9496967
Summary:
This will help in case we ever need to abandon this format in the
future.
Reviewed By: jsgf
Differential Revision: D7483425
fbshipit-source-id: 0cca098ca861cca0994c06709849ececffb38fc5
Summary:
The current structure makes defining new manifests really verbose and
doesn't easily support nested manifests. Make it really simple: either accept a
path map or a description (a csv).
Also switch to a `BTreeMap` to allow lookups in the next diff.
This is unfortunately hard to separate out because all the bits are
interconnected.
Reviewed By: StanislavGlebik
Differential Revision: D7531115
fbshipit-source-id: ca0fa35d4e0ac6a4e23adb2a7d2b5679ce52b643
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary: Very similar to serialize and deserialize for bonsai changesets.
Reviewed By: farnz
Differential Revision: D7380859
fbshipit-source-id: 42567ae3c2e52597b3708871358165ec54a8c04a
Summary: This name is less misleading. Thanks markbt for suggesting it.
Reviewed By: farnz
Differential Revision: D7380821
fbshipit-source-id: 96abb846fd8979cc322e1f176d15caf24bb00874
Summary: The compact_protocol code is from jsgf.
Reviewed By: farnz
Differential Revision: D7365901
fbshipit-source-id: 4cef925ad60c579188ab016fed3f027b3944a3a1
Summary:
This also means that a direct conversion from a byte slice to a Blake2
is no longer desirable. Forcing people to go through the typed hashes seems
quite desirable.
Reviewed By: farnz
Differential Revision: D7365900
fbshipit-source-id: 36f4afd5a53f857c8952c636b0db0b4b509a6f44
Summary:
Since none of the typed hashes use it, NULL hashes aren't going to be
used anywhere in the system.
Reviewed By: farnz
Differential Revision: D7365068
fbshipit-source-id: d67c021a6c04ca1a120264ec88943a5b35d9b8c4
Summary:
I'm still filling this in, but thought I'd put this up for review
right now.
I haven't done the Thrift conversions yet -- those will happen soon.
Reviewed By: farnz
Differential Revision: D7300834
fbshipit-source-id: 96f5f67dbd3d6bafffa127ad93acd178292ec788
Summary:
A lot of Mononoke expects MPath instances to be non-empty, and some
Mononoke code wants empty MPaths. Have formerly-empty `MPath` instances be
represented as `None` instead.
There's a bit of a wart where we have to check for some instances being empty
even though they should never be. This is not a regression -- this is actually
a deficiency in our current type hierarchy being exposed. `Entry` can sometimes
represent a root manifest, but a root path and a root manifest will never be
joined together in practice. We may want to figure out a better way to
represent that in the future.
Also clean up a bunch of code that checks for empty `MPath` instances. That
just isn't possible any more -- yay.
Reviewed By: StanislavGlebik
Differential Revision: D7359048
fbshipit-source-id: 684f908dabaa06cb25b1dd33f4fd7bb021cb5e1b
Summary:
In this case it's super convenient to just use `RepoPath` instances,
so just do that.
The next diff will remove the ability to create `MPath` instances entirely.
Reviewed By: StanislavGlebik
Differential Revision: D7358368
fbshipit-source-id: 8915e765d7e879ec35abac9dfe3ea9653520395d
Summary:
Lots of places specifically want a non-empty `MPath`, and earlier those cases
weren't type-checked. Now `Option<MPath>` stands for an empty `MPath`. In the
next diff `MPath::empty()` will go away and the only way to represent an
`MPath` that doesn't exist will be with `Option<MPath>`.
Reviewed By: farnz
Differential Revision: D7350970
fbshipit-source-id: 1612aec67134e7a0ebad15dbaa93b5ea972f8ddf
Summary:
The `Option<&MPathElement>` type is more general -- it's easy to
convert from `&Option<MPathElement>` to it, but the other way around can
require a clone.
Reviewed By: farnz
Differential Revision: D7339161
fbshipit-source-id: 0c8ab57a19bc330245c612e3e0e3651e368ab8cb
Summary:
Most places expect MPathElements to be non-empty, and it's good to
have static type checking for that.
The only uses could easily be replaced.
Reviewed By: farnz
Differential Revision: D7337587
fbshipit-source-id: cf700b609478bba1f881a77c82a53f8b9863ad7f
Summary:
A Mononoke tiny changeset is going to have a list of these file
changes.
Reviewed By: farnz
Differential Revision: D7336315
fbshipit-source-id: 25d9908cf825e665ac7605435ce5a4056f221e17
Summary:
For now this just wraps a blob, but in the future this will have
special support for large files.
Reviewed By: farnz
Differential Revision: D7336121
fbshipit-source-id: f4c9e216efdd250fde90307264c1931de97d8592
Summary: Represented as a chrono datetime in Rust and an (i64, i32) pair in Thrift.
Reviewed By: jsgf
Differential Revision: D7325332
fbshipit-source-id: 22c3b17961ffb0b4fdb4e6e8aece3b257b3c718e
Summary:
External code should consider these identifiers opaque -- the fact
that they're hashes is an implementation detail.
Also a "blob" is too generic here -- what we're talking about is file contents.
In the future (e.g. for 10GB+ files) the content hash might point to something
more complicated than file contents.
Reviewed By: farnz
Differential Revision: D7320932
fbshipit-source-id: 11a781345a1973f0af8222b809004daa948c3a47
Summary:
I think it makes a lot of logical sense to have three separate hash
types:
* `ChangesetId` for changesets
* `UnodeHash` for unodes
* `BlobHash` for file blobs
These three are never going to mix. Note that manifests and file unodes are
going to mix in e.g. tree manifest unodes.
Reviewed By: farnz
Differential Revision: D7300742
fbshipit-source-id: 688b410f7a5304fdd5ab3a6800137632887f97b9
Summary:
As part of conversion from Thrift to native Rust data structures,
verify that they're valid.
Reviewed By: farnz
Differential Revision: D7297264
fbshipit-source-id: d0bb46a5a5ed2113d9616a56fc100943802cbc1f
Summary:
Also add conversions to and from the Thrift type to the Rust type.
There are tradeoffs around having parallel sets of data structures vs using
Thrift as the only data structure, but I believe that for these hashes (and
transitively, anything that includes them), having a separate type is best.
That's because:
* these hashes are going to be used all over the place, and using a
heap-allocated vector is just a waste. Even a `SmallVec` that's inline will
consume some extra memory for storing the fact that it's inline.
* something has to verify that the vector is valid, and it's nice for that
to be encoded in the type system.
Reviewed By: farnz
Differential Revision: D7285116
fbshipit-source-id: e887414862821076ef0b11e2d0cb0783be897d4a
Summary:
While writing Thrift deserialization code I realized there was nothing
that actually checked that MPathElement instances don't have embedded nulls or
slashes.
Reviewed By: farnz
Differential Revision: D7296838
fbshipit-source-id: 6a23d559da11e5e935e23d7b9a13f58894efaf62
Summary:
These hashes will be the used throughout Mononoke's data model.
Also add mocks.
Note that a `UnodeHash` cannot be null.
Reviewed By: farnz
Differential Revision: D7260955
fbshipit-source-id: d5be5c67848e7b93218a26282e0a757708df7e62
Summary:
This is ripped off pretty much directly from the `Sha1` type in
`mercurial-types`.
Reviewed By: StanislavGlebik
Differential Revision: D7205225
fbshipit-source-id: facb243735d858c68886ad932920024eaf3de536
