Summary:
Together with logging bookmark moves, let's also log bundle handle. It will be
used during replay from Mononoke to mercurial.
Reviewed By: ikostia
Differential Revision: D13990929
fbshipit-source-id: 4039322903b13e84fb31c8e65cc2e097ca765213
Summary:
This diff adds some needed boilerplate for saving raw Bundle2 contents into the
blobstore. This is needed for Mercurial replay.
See the attached task for details.
Reviewed By: StanislavGlebik
Differential Revision: D14018131
fbshipit-source-id: 1130f1ae07c53098e2447b0f8b46e5869f49acdf
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
This avoids making blobstore depend on mononoke-types, which itself
has complex dependencies.
Reviewed By: StanislavGlebik
Differential Revision: D13944481
fbshipit-source-id: ade338a03efee2107256a91f6b622df98246a11f
Summary:
File content blobs are thrift encoded in Mononoke. This is done so
that we can change the encoding of content blobs easily. For example, we can
add compression or we can add split the blobs in chunks.
However there is a problem. At the moment file content blob key is a hash of
the actual data that's written to blobstore i.e. of a thrift encoded data. That
means that if we add compression or change thrift encoding in any way, then the
file content blob key changes and it changes the commit hashes.
This is wrong. To fix it let's use hash of the actual file content as the key.
Reviewed By: farnz
Differential Revision: D12884898
fbshipit-source-id: e60a7b326c39dad86e2b26c6f637defcb0acc8e8
Summary: the from_value_opt already knows how to properly parse an integer from sql response, no need to reinvent the wheel and introduce bugs
Reviewed By: aslpavel
Differential Revision: D13651328
fbshipit-source-id: 55af810c99b93bd2f9c67c721a9d0be6034ee466
Summary: Fix Debug formatting for MPath - no need to include the full hex name
Reviewed By: aslpavel
Differential Revision: D13494406
fbshipit-source-id: fc2b5e8f2737f3cb5ac0a804b4e9b69ffca49553
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary: No need to print the hex.
Reviewed By: StanislavGlebik
Differential Revision: D13457386
fbshipit-source-id: f6063b94e4f095d9ffed06b9de6e302b38e29334
Summary:
We'd like to save skiplist data in blobstore so that we won't have to recompute
it. Let's use thrift serialization for it
Reviewed By: lukaspiatkowski
Differential Revision: D13122386
fbshipit-source-id: f44cdcf38fa6a219df9217906a872e60c319e391
Summary:
According to [Git-LFS Plan](https://www.mercurial-scm.org/wiki/LfsPlan), `getfiles` instead of file content should return file in the [following format](https://www.mercurial-scm.org/wiki/LfsPlan#Metadata_format)
```
oid: sha256.SHA256HASH
size: size_int
```
Hg client requests files using sha1 hgfilenode hash. To calculate sha256 of the content, Mononoke is fetching the file from blobstore to memory, and calculate sha256.
It does not give any profit in time and memory consumptions, comparing to non-LFS transfer of Mononoke.
*Solution:*
To put a `key-value` to blobstore, after first request of the file. This means, that after hg client requested sha256 of the file for the first time, after calculation, put it to the blobstore.
Next request of the sha256 of the file content avoid recalcualtion of sha256 in Mononoke. It return sha256 saved in the blob.
Reviewed By: StanislavGlebik
Differential Revision: D13021826
fbshipit-source-id: 692e01e212e7d716bd822fa968e87abed5103aa7
Summary:
Mononoke requires several references to the same blob in the blobstore.
Sha256 aliases are good example. [post](https://fb.facebook.com/groups/scm.mononoke/permalink/739273266435251/)
Short description of alias mechanism:
- we have `key: value` blob in blobstore.
- put a `key1: key` blob in blobstore to have 2-step access from `key1` to `value`.
All the keys in Mononoke are of the form `type_prefix.hash_name.hash`
I expanded MononokeId interface to have an access to the prefix `type_prefix.hash_name` for verification `key` content (see alias mechanism description).
Reviewed By: farnz
Differential Revision: D13084145
fbshipit-source-id: 5b8a4e80869481414a7356ccd7c9aab6e24a5138
Summary:
Correctly handles case conflicting renames (only change in casing).
- path can now be removed from `CaseConlictingTrie`
- `check_case_conflicts` operates on `BonsaiChangeset` in pushrebase logic
Reviewed By: StanislavGlebik
Differential Revision: D10447522
fbshipit-source-id: d5342e7aa48154debee123b38bf3168e3371baa6
Summary:
Updates rust-crates-io to add several HTTP/2 related crates, namely, `h2`, `httpbis`, and `curl`.
Additionally, fix any breakages introduced by new crate versions. (In particular, the `blake2` crate had breaking changes from 0.7.1 -> 0.8.)
Reviewed By: DurhamG
Differential Revision: D10346164
fbshipit-source-id: 6805261542b5b9c46a34cad6cf6e9fe38f074e87
Summary:
Add a comment about the restriction of cs_id used in CopyInfo. It must match
one of parents mentioned in BonsaiChangeset.
This makes it clear the cs_id won't be the commit introducing the file
revision. It cannot be other things like an ancient commit in the history, or
a future commit in a different branch, either.
Reviewed By: sunshowers
Differential Revision: D10137729
fbshipit-source-id: 9b2afd7689bb93d45514ea9ab66667eb46a3a11f
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
WIP
Mononoke API download for lfs
support get request
curl http://127.0.0.1:8000/{repo_name}/lfs/download/{sha256}
Reviewed By: StanislavGlebik
Differential Revision: D9850413
fbshipit-source-id: 4d756679716893b2b9c8ee877433cd443df52285
Summary:
Let's check that new case conflicts are not added by a commit.
That diff also fixes function check_case_conflict_in_manifest - it needs to
take into account that if one of the conflicting files was removed then there
is no case conflict.
There should be a way to disable this check because we sometimes need to allow
broken commits. For example, during blobimport
Reviewed By: aslpavel
Differential Revision: D9789809
fbshipit-source-id: ca09ee2d3e5340876a8dbf57d13e5135344d1d36
Summary:
Additional 2-step reference for blob:
For each file add an additional blob with:
key = aliases.sha256.sha256(raw_file_contents)
value = blob_key
Pay attention, that sha256 hash is taken `from raw_file_content`, not from a blob content.
Additional blob is sent together with the file content blob.
Reviewed By: lukaspiatkowski, StanislavGlebik
Differential Revision: D9775509
fbshipit-source-id: 4cc997ca5903d0a991fa0310363d6af929f8bbe7
Summary:
In `fetch_file_contents()` `blobstore_bytes.into()` converted the bytes to
`Blob<Id>`. This code calls `MononokeId::from_data()` which calls blake2
hashing. Turns out it causes big problems for large many large files that
getfiles can return.
Since this hash is not used at all, let's avoid generating it.
Reviewed By: jsgf
Differential Revision: D9786549
fbshipit-source-id: 65de6f82c1671ed64bdd74b3a2a3b239f27c9f17
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary:
These are the types that we currently need to be able to serialize if we're to
replace `Asyncmemo`'s caching uses with cachelib. Derive the `Abomonation`
trait for all of them.
Reviewed By: jsgf
Differential Revision: D9082597
fbshipit-source-id: 910e90476a3cc4d18ba758226b8572c3e8d264c6
Summary: See the doc comment for what this state represents and why it is necessary.
Reviewed By: StanislavGlebik
Differential Revision: D9025772
fbshipit-source-id: b0588d037365194bbf2d9889ead60237ef097359
Summary:
Now bonsai changesets are created at the same time as hg changesets, and
the mapping between bonsai and hg changesets is recorded
One important piece is missing. At the moment copy info information is ignored.
I'll add it in the next diffs.
Before diff is landed, I re-run the blobimport to prefill missing bonsai changesets.
Reviewed By: farnz
Differential Revision: D8893504
fbshipit-source-id: 1cc4bbcca2f489a0ef6990d6c04d5b3fd8bef92c
Summary: Previously there was no way to get at the underlying `u64` in a Generation object. I want to use this in order to determine how deep into the commit graph I need to lazily index nodes while answering queries. This will help keep the indexing limited to only what is relevant to incoming queries.
Reviewed By: StanislavGlebik
Differential Revision: D9009058
fbshipit-source-id: 1b4ec44b8245ead75f3097048e85b0a1aafe84f6
Summary:
It's useful to get the hash of the changeset. Looks like we have to clone, but
I'm happy to avoid doing it if that's possible.
Reviewed By: farnz
Differential Revision: D9013977
fbshipit-source-id: bdf06b457fb11c222afddc623c71f3b6b6be30fc
Summary:
Extras value may be non-utf8, for example, hg-git extension generate binary
diff which may be non-utf8.
At the same time it seems that it's better to leave extras key as a utf8
string. However, I'm open to changing it if there is a big pushback
Reviewed By: sunshowers, farnz
Differential Revision: D8991340
fbshipit-source-id: 8f9f33ab3ea77281ae33e0bc735b15201720b761
Summary:
RFC3339 is a standard way to represent dates. See https://tools.ietf.org/html/rfc3339
We're going to use this in code to check for commits older than a certain time.
Reviewed By: farnz
Differential Revision: D9017006
fbshipit-source-id: a4e4325f32d84d4be9b160adf428411d014fb62b
Summary: Those structures will be used in next diffs to store the FilenodeInfo inside memcache for caching purposes
Reviewed By: farnz
Differential Revision: D9014213
fbshipit-source-id: 4952a90415d4b8ab903387fd5cdfaf08d9870c07
Summary: would like there to be an easier way to print these out.
Reviewed By: StanislavGlebik
Differential Revision: D8888556
fbshipit-source-id: 67634ba81ca7ed5789dbc744ef5ab2a4f26be07e
Summary:
I don't like this because particularly the empty string for regular
files looks weird.
Reviewed By: StanislavGlebik
Differential Revision: D8888553
fbshipit-source-id: 20a9048a19b3fdfe681160a637bc2dfc8932c113
Summary:
We need to store relation between Hg changesets and Bonsai changesets.
- `BonsaiHgMapping` is exactly this mapping which establishes injective relation between `{Hg|Bonsai}Changeset`
Reviewed By: StanislavGlebik
Differential Revision: D8801254
fbshipit-source-id: c7df14172e6c2d67c039a24e1bb821e6d92860af
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary: Moved the Generation struct, which is a u64 wrapper, from repogen to mononoke-types, and updated the according usages. This should make it easier to phase out RepoGenCache.
Reviewed By: farnz
Differential Revision: D8725538
fbshipit-source-id: cfd39be03bae56d2288053b7b5e212e6d547c833
Summary:
I really liked how expressing the problem in terms of data structures
made it fairly straightforward to understand.
Reviewed By: farnz
Differential Revision: D8586383
fbshipit-source-id: d1e3c92a3a5b760a0f13f4912420e2a73b937e8d
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
