Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary: To make them more explicit
Reviewed By: matthewdippel
Differential Revision: D9132294
fbshipit-source-id: a365d7b58ba095d11fb0570e5ab6994a158873b3
Summary:
This is a split from D8893504. It just enables functionality to create bonsai changesets.
The split was done so that I can land the biggest chunk of work.
Reviewed By: farnz
Differential Revision: D9081430
fbshipit-source-id: 7437c7789998f5691afe83d5b16a8f2c5faac8b4
Summary:
It was using manifest hash as parent hash. One more reason to fix all the types
in Mononoke
Reviewed By: aslpavel
Differential Revision: D9123653
fbshipit-source-id: 0841f7ac64e50e9234d80040b7f286930af53420
Summary:
Now bonsai changesets are created at the same time as hg changesets, and
the mapping between bonsai and hg changesets is recorded
One important piece is missing. At the moment copy info information is ignored.
I'll add it in the next diffs.
Before diff is landed, I re-run the blobimport to prefill missing bonsai changesets.
Reviewed By: farnz
Differential Revision: D8893504
fbshipit-source-id: 1cc4bbcca2f489a0ef6990d6c04d5b3fd8bef92c
Summary:
Make it return manifest hashes. This is not very useful right now, but it will
be useful in the later diffs where we add BonsaiChangeset generation.
Reviewed By: farnz
Differential Revision: D9013976
fbshipit-source-id: 85f0c78bc7b944d300d537dd29cfd66f6f6530c6
Summary: Because get_all_filenodes can go above 1MB (but still less than 3MB) I added a chunking logic to speed up fetching of even the long histories
Reviewed By: farnz
Differential Revision: D9029774
fbshipit-source-id: c2d46a4b49bfebd3c778a0b9c5cf226857f34768
Summary:
I had enough seeing compiler giving me error messages with `blobrepo::Error` whereas it should be `failure::Error`.
`pub type` is exporting a type alias whereas `pub use` is re-exporting. The later apparently will fool the compiler to think that they own the thing they are re-exporting which in this case, failure::Error. If it is for the purpose of declaring the error type the crate is using, type alias is obviously more suitable as it won't take the ownership of the type.
Reviewed By: lukaspiatkowski
Differential Revision: D8926275
fbshipit-source-id: 6b1346adf52bec6b666a3f8b23e7a4ea4e0ab070
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
Previously incorrect filenodes were recorded. Instead of filenode "dir/file",
"file" was recorded as a path in filenodes table.
The next diff that adds creating file copies in bonsai changesets relies on
correct filenodes being present in the table.
Reviewed By: farnz
Differential Revision: D8915297
fbshipit-source-id: dcbde286d935bc726847f788779cfdc7c888c023
Summary:
The test was working fine, however, with the next diff in the stack
it started to fail under stress runs. And in that case I assume the problem in
the test itself.
The test creates two commits, that create file with the same content.
However, one filenode wasn't ancestor of another, and that's semantically
incorrect.
Reviewed By: farnz
Differential Revision: D8913129
fbshipit-source-id: 1b3838c6cf5060d768e71ef056cb7765ebbabc99
Summary:
These types are Hg specific. Since we are going to add bonsai changeset
creation soon, let's make it clear in the types
Reviewed By: farnz
Differential Revision: D8911359
fbshipit-source-id: 8b6cc45122402d7b7e074e66d904d979030de705
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
This proves to be extremely useful for the upcoming bonsai
verification code.
The in-memory stuff is more complicated for the database backends, so punt on
that for now with some warnings.
Reviewed By: farnz
Differential Revision: D8909426
fbshipit-source-id: 1d66d877cfa48ef06bbe614f37c66cf6c2f0446c
Summary:
The thrift client should return false or None for missing objects, but instead
errors out.
Reviewed By: StanislavGlebik
Differential Revision: D8899842
fbshipit-source-id: 81e5433ee63b7c8e6a4e898443a2a24bc784d014
Summary:
I don't like this because particularly the empty string for regular
files looks weird.
Reviewed By: StanislavGlebik
Differential Revision: D8888553
fbshipit-source-id: 20a9048a19b3fdfe681160a637bc2dfc8932c113
Summary: Also make it possible to have mutable clones
Reviewed By: farnz
Differential Revision: D8889900
fbshipit-source-id: 77d304ec5e07c84398e328b361ebf062cd18de45
Summary: Let's pass just one field instead of 4
Reviewed By: farnz
Differential Revision: D8889899
fbshipit-source-id: 8b30496a86950ed534439f5469f8740ee32345b8
Summary: This commit removes the usage of the old manifold client so we can get rid of `tokio_io::Remote`;
Reviewed By: jsgf
Differential Revision: D8751815
fbshipit-source-id: 41ab75076f1b2591496919ea91eb6b1d4d7ae506
Summary:
Bonsai verification tool fails
`buck run mode/opt scm/mononoke:bonsai-verify -- --repo-id 1001 867ca5df5dd87de7a98ffd4495bc97ea57a8af61 --debug --limit 1 --changes`
After debuggin this run with a lot of printfs it turned out we only keeping last entry, `manifest_lookup` was to blame
Reviewed By: farnz
Differential Revision: D8875499
fbshipit-source-id: 0df4c766d55657c8c4cd7005facce715b31e733e
Summary: tokio::runtime can handle multithreading, no need for all this bolierplate
Reviewed By: StanislavGlebik
Differential Revision: D8861170
fbshipit-source-id: 2c489068a55f8cba1854f8a748df1e6efe8b47b7
Summary:
Removed all references to RepoGenCache from publically callable functions in the revset package. This involved:
- Modifying blobrepo so that its get_generation_number method returned a Generation wrapper instead of a raw usize, to allow it to be used in a cleaner manner in the revset code.
- Simultaneously changing the constructors of all the structures in revset. This seems like a big change, but many of them call each other, passing a RepoGenCache object down the line, so eliminating them all at once made for the cleanest update.
- Modifying helper functions in the revset structures which would create streams of nodes by taking ownership of a RepoGenCache object within a closure. Instead they now take ownership of a clone of the repo. This strategy was already done earlier in the same helper functions, so I am assuming the cost of cloning a repo into a closure is small.
- Modifying the only external usage of revset within the mononoke server code.
This is part of a several step process to completely remove RepoGenCache from the code base. The next steps should be:
- Remove all references to RepoGenCache in the testing macros for revset.
- Delete RepoGenCache and clean up any dangling references to it.
Reviewed By: StanislavGlebik
Differential Revision: D8743560
fbshipit-source-id: 125f851075d836d40224d339e1daee912a39f7e4
Summary:
When we add cachelib bindings to Rust, we're going to want to implement a
cachelib blobstore that's more or less the same as the memcache version, but
backed by a cachelib pool instead of a memcache instance.
Split this code up so that we don't duplicate functionality
Reviewed By: StanislavGlebik
Differential Revision: D8523713
fbshipit-source-id: 882298abab8c208103f6d8c74fee60a768c877f6
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
This diff implements hooks which act upon individual files. The results of the hooks are cached using Asyncmemo.
The cache is currently keyed on (changeset_id, hook_name, file path) but this will change to file content hash once we move to Bonsai changesets.
Reviewed By: StanislavGlebik
Differential Revision: D8707726
fbshipit-source-id: ceaf94abd09e1dd7f6b2d8f9c87a9a221439a252
Summary:
Correctly resolve file/directory conflict in `MemoryManifest`
- when we have file/directory conflict and have entry in `BonsaiChangeset` that indicates that file should be deleted we need resolve conflict by keeping directory
- more details in [Bonsai changeset actions](https://fb.quip.com/A2kqArd9Nb90#WHBACA4Vitd)
Reviewed By: lukaspiatkowski
Differential Revision: D8658958
fbshipit-source-id: 815a91b847c5c0c9586eb5d317ebb9412196d5f4
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: This will be pretty useful while debugging.
Reviewed By: jsgf
Differential Revision: D8667770
fbshipit-source-id: 1c776741844d74529415124ab826b013414678c7
Summary:
It's easier to use scuba when you separate the part of log that identifies the log from the part that can have arbitrary data in it.
It's also easier to search for code after a sample was found
Reviewed By: jsgf
Differential Revision: D8625612
fbshipit-source-id: 7d7e382530dd5d7e5d69c6d34caccda4b6d2921b
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary: This code handles "trivial merge" (when file blob contains same data but different parents) of `MemoryManifest`
Reviewed By: farnz
Differential Revision: D8420775
fbshipit-source-id: dd0c640b8cc822d2dc42eed8f0e9a8e0a00f2cdc
Summary: more logging more fun
Reviewed By: StanislavGlebik
Differential Revision: D8577655
fbshipit-source-id: 92a160ea8f8c0b8e012a1461fbd3f5d71b4bd171
Summary:
Manifests are always able to return entries immediately, and never
fail.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D8556499
fbshipit-source-id: e21a2522f1219e47db9b55b24b6ac6c0c463933e
Summary: Going to master adds a latency. Let's go to master only if we haven't found anything in the replica
Reviewed By: lukaspiatkowski
Differential Revision: D8548598
fbshipit-source-id: b0632ccf7f148d20e66763981c8ef4e615fd7f6c
Summary: we want most of the read traffic to be handled by replicas
Reviewed By: lukaspiatkowski
Differential Revision: D8548027
fbshipit-source-id: 297b070b5b33ec114026ecf3f1079fdb26f54b19
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.
Reviewed By: farnz
Differential Revision: D8440330
fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary:
Since file and manifest nodes are going to be stored differently, rename this to
get_hg_raw_content to make it clear that it'll work for both files and
manifests.
I tried allowing a `Chain` to be used, but ran into https://github.com/carllerche/bytes/issues/213. Boo.
Reviewed By: farnz
Differential Revision: D8413001
fbshipit-source-id: 2f06affa265beaed2ac0a48ee4fec8ce922307da
Summary:
Pretty straightforward. Also using this opportunity to add per-repo
prefixes, since all the hashes are going to break anyway.
Note for reviewers: almost all the change is regenerated test fixtures (unfortunately necessary to make atomic). The actual substantive changes are all in the first few files.
Reviewed By: farnz
Differential Revision: D8392234
fbshipit-source-id: c93fc8c6388cb00fe5cff95646ad8c853581cb8c
Summary:
This diff introduces the ChangesetStore trait so the HookManager does not directly depend on the BlobRepo. This means the manager can obtain changesets from a simple in memory implementation of ChangesetStore which can be populated with changesets before they are committed to the server. This means the exact same hook manager logic can be used irrespective of whether the changesets are committed or not.
The diff includes two implementations of ChangesetStore - A simple in memory implementation backed by a HashMap, and one that backs onto BlobStore.
Reviewed By: StanislavGlebik
Differential Revision: D8446430
fbshipit-source-id: 8d14e48cb562fcd10a17370e34f13a662af827df
Summary:
It would be suboptimal if memcache could be abused to examine objects
you should not have access to. Change the key used so that (a) it's a regional
cluster for wider caching, and (b) it's got the blobstore details in it so that
you cannot extract objects not from your blob store.
Reviewed By: StanislavGlebik
Differential Revision: D8424298
fbshipit-source-id: 78f9a1a7302b4a60575f257bda665f719dc1a7b6
Summary: Added logic to save `FileChange` as a Mercurial format `HgBlobEntry`
Reviewed By: sunshowers
Differential Revision: D8187792
fbshipit-source-id: 4714c81ab23ebac528cfec15c4a9e66083d4fb6c
