Summary:
Convert to `2018` version and remove `#[macro_use]` and `extern crate` from following folders:
- blobrepo
- blobstore
- mercurial_types
- mononoke_types
Reviewed By: StanislavGlebik
Differential Revision: D14827294
fbshipit-source-id: 4b8a2edb5d9ad1fe75083e172cd0da359fa6dc2e
Summary: Slim down the blobstore trait crate as much as possible.
Reviewed By: aslpavel
Differential Revision: D14542675
fbshipit-source-id: faf09255f7fe2236a491742cd836226474f5967c
Summary:
The problem was in using `file_changes()` of a bonsai object. If a file
replaces a directory, then it just returns an added file, but not a removed
directory.
However `changed_entry_stream` didn't return an entry if just it's mode was changed (i.e. file became executable, or file became a symlink). This diff fixes it as well
Let's use the same computing changing files method instead of `file_changes()`.
Differential Revision: D14279470
fbshipit-source-id: 976b0abd93646f7d68137c83cb07a8564922ce17
Summary:
Make compute_changed_files accept manifest id, it will be used in the next
diffs
Differential Revision: D14279468
fbshipit-source-id: eca92900ed8862bb6db38ff6c5ed5372d8206aa9
Summary: HgFileNodeId is a stronger typed id, so it is prefered to use it instead of HgNodeHash whenever it is identifying a filenode
Reviewed By: aslpavel
Differential Revision: D13986172
fbshipit-source-id: c0334652345acb868e86c38b8c0045e9c023c176
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
`blobrepo_factory` is a crate that knows how to create blobrepo given
a configuration i.e. it creates blobstores, filenodes, changesets etc and
initializes blobrepo with them.
`post_commit` is a small part of blobrepo which can also be extracted from
blobrepo crate.
There are a few upsides with this approach
1) Less dependencies on Blobrepo, meaning we have to rebuild it fewer times
2) BlobRepo compilation is faster
Reviewed By: jsgf
Differential Revision: D13896334
fbshipit-source-id: 1f5701341f01fcefff4e5f9430ddc914b9496064
Summary:
File content blobs are thrift encoded in Mononoke. This is done so
that we can change the encoding of content blobs easily. For example, we can
add compression or we can add split the blobs in chunks.
However there is a problem. At the moment file content blob key is a hash of
the actual data that's written to blobstore i.e. of a thrift encoded data. That
means that if we add compression or change thrift encoding in any way, then the
file content blob key changes and it changes the commit hashes.
This is wrong. To fix it let's use hash of the actual file content as the key.
Reviewed By: farnz
Differential Revision: D12884898
fbshipit-source-id: e60a7b326c39dad86e2b26c6f637defcb0acc8e8
Summary:
When receiving and infinitepush bundle, don't store the filenodes for the
commit in the filenodes table.
When a client pulls these commits, we will reconstruct the filenode info from
the blobstore data. However, they will receive null linknodes, and will need
to use adjustlinknode to compute the correct linknode.
Reviewed By: StanislavGlebik
Differential Revision: D13467110
fbshipit-source-id: 739b06f30a530a159352ffbf612d136c9c831aeb
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary:
Let's check that new case conflicts are not added by a commit.
That diff also fixes function check_case_conflict_in_manifest - it needs to
take into account that if one of the conflicting files was removed then there
is no case conflict.
There should be a way to disable this check because we sometimes need to allow
broken commits. For example, during blobimport
Reviewed By: aslpavel
Differential Revision: D9789809
fbshipit-source-id: ca09ee2d3e5340876a8dbf57d13e5135344d1d36
Summary:
Additional 2-step reference for blob:
For each file add an additional blob with:
key = aliases.sha256.sha256(raw_file_contents)
value = blob_key
Pay attention, that sha256 hash is taken `from raw_file_content`, not from a blob content.
Additional blob is sent together with the file content blob.
Reviewed By: lukaspiatkowski, StanislavGlebik
Differential Revision: D9775509
fbshipit-source-id: 4cc997ca5903d0a991fa0310363d6af929f8bbe7
Summary: This commits change `HgBlob` from an enum into a struct that only contains one Bytes field, completely removes `HgBlobHash` and changes the methods of `HgBlob` from returning `Option`s into directly returning results.
Reviewed By: farnz
Differential Revision: D9317851
fbshipit-source-id: 48030a621874d628602b1c5d3327e635d721facf
Summary: As a bonus this diff also contains unifying the linknode family of methods (they all now accept arguments via reference) and better tracing for get_files request
Reviewed By: StanislavGlebik
Differential Revision: D9031283
fbshipit-source-id: 4526a8446984904bce7d4dcef240088c7f2ffaa3
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary:
This is a split from D8893504. It just enables functionality to create bonsai changesets.
The split was done so that I can land the biggest chunk of work.
Reviewed By: farnz
Differential Revision: D9081430
fbshipit-source-id: 7437c7789998f5691afe83d5b16a8f2c5faac8b4
Summary:
Now bonsai changesets are created at the same time as hg changesets, and
the mapping between bonsai and hg changesets is recorded
One important piece is missing. At the moment copy info information is ignored.
I'll add it in the next diffs.
Before diff is landed, I re-run the blobimport to prefill missing bonsai changesets.
Reviewed By: farnz
Differential Revision: D8893504
fbshipit-source-id: 1cc4bbcca2f489a0ef6990d6c04d5b3fd8bef92c
Summary:
Previously incorrect filenodes were recorded. Instead of filenode "dir/file",
"file" was recorded as a path in filenodes table.
The next diff that adds creating file copies in bonsai changesets relies on
correct filenodes being present in the table.
Reviewed By: farnz
Differential Revision: D8915297
fbshipit-source-id: dcbde286d935bc726847f788779cfdc7c888c023
Summary:
The test was working fine, however, with the next diff in the stack
it started to fail under stress runs. And in that case I assume the problem in
the test itself.
The test creates two commits, that create file with the same content.
However, one filenode wasn't ancestor of another, and that's semantically
incorrect.
Reviewed By: farnz
Differential Revision: D8913129
fbshipit-source-id: 1b3838c6cf5060d768e71ef056cb7765ebbabc99
Summary: Let's pass just one field instead of 4
Reviewed By: farnz
Differential Revision: D8889899
fbshipit-source-id: 8b30496a86950ed534439f5469f8740ee32345b8
Summary:
Correctly resolve file/directory conflict in `MemoryManifest`
- when we have file/directory conflict and have entry in `BonsaiChangeset` that indicates that file should be deleted we need resolve conflict by keeping directory
- more details in [Bonsai changeset actions](https://fb.quip.com/A2kqArd9Nb90#WHBACA4Vitd)
Reviewed By: lukaspiatkowski
Differential Revision: D8658958
fbshipit-source-id: 815a91b847c5c0c9586eb5d317ebb9412196d5f4
Summary: This code handles "trivial merge" (when file blob contains same data but different parents) of `MemoryManifest`
Reviewed By: farnz
Differential Revision: D8420775
fbshipit-source-id: dd0c640b8cc822d2dc42eed8f0e9a8e0a00f2cdc
Summary: more logging more fun
Reviewed By: StanislavGlebik
Differential Revision: D8577655
fbshipit-source-id: 92a160ea8f8c0b8e012a1461fbd3f5d71b4bd171
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.
Reviewed By: farnz
Differential Revision: D8440330
fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary:
Implementation of generic `store|fetch` for bonsai types.
- bonsai types have unique typed hashes associated with each bonsai type, I'm leveraging this fact to implement generic `store|fetch` methods on `BlobRepo`
Reviewed By: farnz
Differential Revision: D8254810
fbshipit-source-id: 5f798fade4cb8d1ac851f94c7ad7e64636bbca65
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: This log is by far the most common one and it makes reading logs much harder. It should probably be changed to ODS counters, but for now lets just make it trace!
Reviewed By: farnz
Differential Revision: D8235663
fbshipit-source-id: 3685b260f1c6c43c1fde8501731583debc8d063b
Summary:
Previously, we assumed that all content hashes came from Mercurial;
this is not going to remain true, as we will want to be able to upload manifests
that have been synthesised from Bonsai Changesets. Turn the previous boolean
into a tri-state, and fix up all callers to get the behaviour they expect.
Reviewed By: StanislavGlebik
Differential Revision: D8014911
fbshipit-source-id: 9156b9fab4542ceb269626ad005e1b28392b5329
Summary:
The commits that are blobimported have out of order or simply incorrect lists of changed files.
Because we have to persists Changesets as is we are passing the untouched list of files here to be used by Changeset.
Reviewed By: farnz
Differential Revision: D7830310
fbshipit-source-id: 56adec2c317896decaa9176b3a6bfb0cab187ed0
Summary: the idea of Mercurial heads in Mononoke will be represented by bookmarks, so there is no need to have them around
Reviewed By: StanislavGlebik
Differential Revision: D7775032
fbshipit-source-id: 1618a1e51862d7c115b2955082f40ee890a045f1
Summary: The eden integration test contains a commit with no content which new_blobimport couldn't import. With this changes the commit API is capable of handling such commits.
Reviewed By: jsgf
Differential Revision: D7709243
fbshipit-source-id: 7d55eb2ec421820d189ab05b0f8cb4411f850a7b
Summary:
We know that the hashes for non-root-tree-manifests and filenodes
should always be consistent. Verify that.
Reviewed By: farnz
Differential Revision: D7704087
fbshipit-source-id: 7f6207878c5cd372b272aa6970506dd63b5a3c7c
Summary:
As the comment explains, sometimes the hashes don't match the
contents. Accept such pushes.
Reviewed By: farnz
Differential Revision: D7699930
fbshipit-source-id: 376f01b6cf03f6cad84c2c878d192d55f8d81812
Summary:
* `Into<Bytes>`, not `Into<String>`
* Centralize in one place because I'm going to be adding more non-trivial code to this in upcoming diffs
Reviewed By: farnz
Differential Revision: D7694764
fbshipit-source-id: e6373cf1562b9a464894466ecf3be59e9892ac78
Summary:
We're going to keep this around for now as part of double-writing.
All the hashes here are definitely Mercurial hashes, so use them that way.
Reviewed By: lukaspiatkowski
Differential Revision: D7683890
fbshipit-source-id: 270091cd11f3cec7ef4cf565de5ef913fcf7adea
