Summary:
Correctly handles case conflicting renames (only change in casing).
- path can now be removed from `CaseConlictingTrie`
- `check_case_conflicts` operates on `BonsaiChangeset` in pushrebase logic
Reviewed By: StanislavGlebik
Differential Revision: D10447522
fbshipit-source-id: d5342e7aa48154debee123b38bf3168e3371baa6
Summary:
Updates rust-crates-io to add several HTTP/2 related crates, namely, `h2`, `httpbis`, and `curl`.
Additionally, fix any breakages introduced by new crate versions. (In particular, the `blake2` crate had breaking changes from 0.7.1 -> 0.8.)
Reviewed By: DurhamG
Differential Revision: D10346164
fbshipit-source-id: 6805261542b5b9c46a34cad6cf6e9fe38f074e87
Summary:
Add a comment about the restriction of cs_id used in CopyInfo. It must match
one of parents mentioned in BonsaiChangeset.
This makes it clear the cs_id won't be the commit introducing the file
revision. It cannot be other things like an ancient commit in the history, or
a future commit in a different branch, either.
Reviewed By: sunshowers
Differential Revision: D10137729
fbshipit-source-id: 9b2afd7689bb93d45514ea9ab66667eb46a3a11f
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
WIP
Mononoke API download for lfs
support get request
curl http://127.0.0.1:8000/{repo_name}/lfs/download/{sha256}
Reviewed By: StanislavGlebik
Differential Revision: D9850413
fbshipit-source-id: 4d756679716893b2b9c8ee877433cd443df52285
Summary:
Let's check that new case conflicts are not added by a commit.
That diff also fixes function check_case_conflict_in_manifest - it needs to
take into account that if one of the conflicting files was removed then there
is no case conflict.
There should be a way to disable this check because we sometimes need to allow
broken commits. For example, during blobimport
Reviewed By: aslpavel
Differential Revision: D9789809
fbshipit-source-id: ca09ee2d3e5340876a8dbf57d13e5135344d1d36
Summary:
Additional 2-step reference for blob:
For each file add an additional blob with:
key = aliases.sha256.sha256(raw_file_contents)
value = blob_key
Pay attention, that sha256 hash is taken `from raw_file_content`, not from a blob content.
Additional blob is sent together with the file content blob.
Reviewed By: lukaspiatkowski, StanislavGlebik
Differential Revision: D9775509
fbshipit-source-id: 4cc997ca5903d0a991fa0310363d6af929f8bbe7
Summary:
In `fetch_file_contents()` `blobstore_bytes.into()` converted the bytes to
`Blob<Id>`. This code calls `MononokeId::from_data()` which calls blake2
hashing. Turns out it causes big problems for large many large files that
getfiles can return.
Since this hash is not used at all, let's avoid generating it.
Reviewed By: jsgf
Differential Revision: D9786549
fbshipit-source-id: 65de6f82c1671ed64bdd74b3a2a3b239f27c9f17
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary:
These are the types that we currently need to be able to serialize if we're to
replace `Asyncmemo`'s caching uses with cachelib. Derive the `Abomonation`
trait for all of them.
Reviewed By: jsgf
Differential Revision: D9082597
fbshipit-source-id: 910e90476a3cc4d18ba758226b8572c3e8d264c6
Summary: See the doc comment for what this state represents and why it is necessary.
Reviewed By: StanislavGlebik
Differential Revision: D9025772
fbshipit-source-id: b0588d037365194bbf2d9889ead60237ef097359
Summary:
Now bonsai changesets are created at the same time as hg changesets, and
the mapping between bonsai and hg changesets is recorded
One important piece is missing. At the moment copy info information is ignored.
I'll add it in the next diffs.
Before diff is landed, I re-run the blobimport to prefill missing bonsai changesets.
Reviewed By: farnz
Differential Revision: D8893504
fbshipit-source-id: 1cc4bbcca2f489a0ef6990d6c04d5b3fd8bef92c
Summary: Previously there was no way to get at the underlying `u64` in a Generation object. I want to use this in order to determine how deep into the commit graph I need to lazily index nodes while answering queries. This will help keep the indexing limited to only what is relevant to incoming queries.
Reviewed By: StanislavGlebik
Differential Revision: D9009058
fbshipit-source-id: 1b4ec44b8245ead75f3097048e85b0a1aafe84f6
Summary:
It's useful to get the hash of the changeset. Looks like we have to clone, but
I'm happy to avoid doing it if that's possible.
Reviewed By: farnz
Differential Revision: D9013977
fbshipit-source-id: bdf06b457fb11c222afddc623c71f3b6b6be30fc
Summary:
Extras value may be non-utf8, for example, hg-git extension generate binary
diff which may be non-utf8.
At the same time it seems that it's better to leave extras key as a utf8
string. However, I'm open to changing it if there is a big pushback
Reviewed By: sunshowers, farnz
Differential Revision: D8991340
fbshipit-source-id: 8f9f33ab3ea77281ae33e0bc735b15201720b761
Summary:
RFC3339 is a standard way to represent dates. See https://tools.ietf.org/html/rfc3339
We're going to use this in code to check for commits older than a certain time.
Reviewed By: farnz
Differential Revision: D9017006
fbshipit-source-id: a4e4325f32d84d4be9b160adf428411d014fb62b
Summary: Those structures will be used in next diffs to store the FilenodeInfo inside memcache for caching purposes
Reviewed By: farnz
Differential Revision: D9014213
fbshipit-source-id: 4952a90415d4b8ab903387fd5cdfaf08d9870c07
Summary: would like there to be an easier way to print these out.
Reviewed By: StanislavGlebik
Differential Revision: D8888556
fbshipit-source-id: 67634ba81ca7ed5789dbc744ef5ab2a4f26be07e
Summary:
I don't like this because particularly the empty string for regular
files looks weird.
Reviewed By: StanislavGlebik
Differential Revision: D8888553
fbshipit-source-id: 20a9048a19b3fdfe681160a637bc2dfc8932c113
Summary:
We need to store relation between Hg changesets and Bonsai changesets.
- `BonsaiHgMapping` is exactly this mapping which establishes injective relation between `{Hg|Bonsai}Changeset`
Reviewed By: StanislavGlebik
Differential Revision: D8801254
fbshipit-source-id: c7df14172e6c2d67c039a24e1bb821e6d92860af
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary: Moved the Generation struct, which is a u64 wrapper, from repogen to mononoke-types, and updated the according usages. This should make it easier to phase out RepoGenCache.
Reviewed By: farnz
Differential Revision: D8725538
fbshipit-source-id: cfd39be03bae56d2288053b7b5e212e6d547c833
Summary:
I really liked how expressing the problem in terms of data structures
made it fairly straightforward to understand.
Reviewed By: farnz
Differential Revision: D8586383
fbshipit-source-id: d1e3c92a3a5b760a0f13f4912420e2a73b937e8d
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary: Added logic to save `FileChange` as a Mercurial format `HgBlobEntry`
Reviewed By: sunshowers
Differential Revision: D8187792
fbshipit-source-id: 4714c81ab23ebac528cfec15c4a9e66083d4fb6c
Summary:
Implementation of generic `store|fetch` for bonsai types.
- bonsai types have unique typed hashes associated with each bonsai type, I'm leveraging this fact to implement generic `store|fetch` methods on `BlobRepo`
Reviewed By: farnz
Differential Revision: D8254810
fbshipit-source-id: 5f798fade4cb8d1ac851f94c7ad7e64636bbca65
Summary:
- `store|fetch` semantic requires `MononkeId::Value` type but `UnodeId` does not have associated value type.
- `UnodeId` is not used anywhere in codebase
Nuking it for now.
Reviewed By: lukaspiatkowski
Differential Revision: D8351415
fbshipit-source-id: 970ab8d3c0cdca6d77c96ea08643f90a4ce0e624
Summary:
We're going to need to be able to edit memory manifests. Provide
remove and set operations, to match the Bonsai Changeset data structures
Reviewed By: StanislavGlebik
Differential Revision: D7620527
fbshipit-source-id: e85459c5dbfa8855267fd2cf6578c9fc39f223f8
Summary:
Rust 1.26 adds many new language features. In particular `impl Trait` is now
stable, so we no longer need `conservative_impl_trait`.
There also seems to have been changed in the (unstable) TryFrom with respect to
usize, and the behaviour of the never type `!`.
There are still a few deprecation warnings, but they don't cause the build to
fail.
Path remapping is now stable, so the buck config needs to change to use it
rather than the unstable command line option.
TODO:
- get aarch64 rust-crates-io build (can defer to a later update)
Reviewed By: Imxset21
Differential Revision: D7966091
fbshipit-source-id: 2e61e262c21eb01c852a36f49c6a6369cdaddcdb
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary:
We'd like to move away from `RawNodeBlob` and `RawCSBlob` to data structures
serialized by Thrift. This is the first step to doing that.
The most important thing here is that it reuses file content IDs from native
Mononoke storage.
Reviewed By: jsgf
Differential Revision: D7771990
fbshipit-source-id: de4ee0b56aa6610caeff84b2235e19855df086cb
Summary: Want to get this in now before I forget.
Reviewed By: jsgf
Differential Revision: D7781551
fbshipit-source-id: a5e6fa062514d90113c96307f78e20fa05f770ea
Summary: Will be used for Thrift envelopes.
Reviewed By: jsgf
Differential Revision: D7771214
fbshipit-source-id: 3f0dbf77793064f2606ebe34672629c4e49cc7fe
Summary: This is an important consistency check for bonsai changesets.
Reviewed By: jsgf
Differential Revision: D7755664
fbshipit-source-id: 4e64ed532d0730147efec7a2b10b61f625e50dd0
Summary:
This does mean that we need to adjust QuickCheck generation a bit to ensure
validity.
Reviewed By: jsgf
Differential Revision: D7755635
fbshipit-source-id: ebca6f5a52038b64f83ff599079460daf061cb44
Summary: I got annoyed at the lack of this.
Reviewed By: StanislavGlebik
Differential Revision: D7755636
fbshipit-source-id: 0d519dca94ed93e3efe8d12f9004661b92fb8b1a
Summary:
This redefines file changes and deletes slightly -- see the comments
for a full description.
Reviewed By: jsgf, farnz
Differential Revision: D7750040
fbshipit-source-id: f1c52295cd21150fbdc909198104c0571132431d
Summary:
We know that the hashes for non-root-tree-manifests and filenodes
should always be consistent. Verify that.
Reviewed By: farnz
Differential Revision: D7704087
fbshipit-source-id: 7f6207878c5cd372b272aa6970506dd63b5a3c7c
Summary:
This is the first step to getting bonsai changesets into the blob store.
So far the code just waits for content blob uploads to be done. In future diffs Mononoke will synthesize this information into a bonsai changeset as well.
Reviewed By: lukaspiatkowski
Differential Revision: D7627784
fbshipit-source-id: da0c96772ce4d18aed579cecca6135137a8dbe18
Summary:
I was trying to debug something with the new blobimport, and this was
getting in the way.
Reviewed By: StanislavGlebik
Differential Revision: D7664660
fbshipit-source-id: 2ec4ee79fbe13584f35e7dcd9e8df2b8bdf181c0
Summary:
Currently, any sort of `Bytes` can be stored in the blobstore. That
caused me to make several mistakes while writing the code to store bonsai
changesets, because I'd just end up storing the wrong set of `Bytes`.
Introduce stronger typing so that only types that explicitly implement
`BlobStorable` can be stored in the blobstore.
Currently, these sorts of blobs can be stored in the blob store:
* `ChangesetBlob` and `ContentBlob` in `mononoke-types` (these are Thrift-serialized structures)
* The envelope `RawNodeBlob` and `RawCSBlob` types in `blobrepo`, once converted to `EnvelopeBlob` instances
* `HgBlob`, which contains revlog data (manifests or files) exactly as serialized by Mercurial
Reviewed By: StanislavGlebik
Differential Revision: D7627290
fbshipit-source-id: d1bcbde8881e365dec99618556e7b054985bccf7
