Summary:
In `fetch_file_contents()` `blobstore_bytes.into()` converted the bytes to
`Blob<Id>`. This code calls `MononokeId::from_data()` which calls blake2
hashing. Turns out it causes big problems for large many large files that
getfiles can return.
Since this hash is not used at all, let's avoid generating it.
Reviewed By: jsgf
Differential Revision: D9786549
fbshipit-source-id: 65de6f82c1671ed64bdd74b3a2a3b239f27c9f17
Summary:
Profiling showed that since we are inserting objects into blobstore
sequentially it takes a lot of time for long stacks of commit. Let's do it in
parallel.
Note that we are still inserting sequentially into changesets table
Reviewed By: farnz
Differential Revision: D9683037
fbshipit-source-id: 8f9496b97eaf265d9991b94243f0f14133f463da
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary:
This diff fills missing parts of push-rebase implementation
- `find_closest_root` - find closest root to specified bookmark
- `find_changed_files` - find file affected by changesets between provided `ancestor` and `descendant`
- `intersect_changed_files` - rejects push rebase if any conflicts have been found
- `create_rebased_changes` - support for merges
- `do_pushrebase` - returns updated bookmark value
Reviewed By: StanislavGlebik
Differential Revision: D9458416
fbshipit-source-id: c0cb53773eba6e966f1a5928c43ebdec761a78d3
Summary: This commits change `HgBlob` from an enum into a struct that only contains one Bytes field, completely removes `HgBlobHash` and changes the methods of `HgBlob` from returning `Option`s into directly returning results.
Reviewed By: farnz
Differential Revision: D9317851
fbshipit-source-id: 48030a621874d628602b1c5d3327e635d721facf
Summary: Since this data is specific to TimedStream and not TimedFuture I split the Stats struct into FutureStats and StreamStats
Reviewed By: StanislavGlebik
Differential Revision: D9355421
fbshipit-source-id: cc2055706574756e2e53f3ccc57abfc50c3a02ba
Summary:
Revsets must use ChangesetId, not HgNodeHash. I'm going to use
`RangeNodeStream` in pushrebase so I thought it was a good time to change it
Reviewed By: farnz
Differential Revision: D9338827
fbshipit-source-id: 50bbe8f73dba3526d70d3f816ddd93507db99be5
Summary: Store where it was copied from
Reviewed By: farnz
Differential Revision: D9132560
fbshipit-source-id: a7a73e1f3de08340f5add5fffa32dd0373eb27fa
Summary:
The issues were fixed, and also thrift manifold works better during bulk
blobimport.
Reviewed By: farnz
Differential Revision: D9132384
fbshipit-source-id: ab4a04eeff86bb4968b80af00c404fad710db183
Summary: - Make `Bookmakrs` work with `ChangsetId` instead of `HgChangesetId`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D9297139
fbshipit-source-id: e18661793d144669354e509271044410caa3502a
Summary: As a bonus this diff also contains unifying the linknode family of methods (they all now accept arguments via reference) and better tracing for get_files request
Reviewed By: StanislavGlebik
Differential Revision: D9031283
fbshipit-source-id: 4526a8446984904bce7d4dcef240088c7f2ffaa3
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary: To make them more explicit
Reviewed By: matthewdippel
Differential Revision: D9132294
fbshipit-source-id: a365d7b58ba095d11fb0570e5ab6994a158873b3
Summary:
This is a split from D8893504. It just enables functionality to create bonsai changesets.
The split was done so that I can land the biggest chunk of work.
Reviewed By: farnz
Differential Revision: D9081430
fbshipit-source-id: 7437c7789998f5691afe83d5b16a8f2c5faac8b4
Summary:
It was using manifest hash as parent hash. One more reason to fix all the types
in Mononoke
Reviewed By: aslpavel
Differential Revision: D9123653
fbshipit-source-id: 0841f7ac64e50e9234d80040b7f286930af53420
Summary:
Now bonsai changesets are created at the same time as hg changesets, and
the mapping between bonsai and hg changesets is recorded
One important piece is missing. At the moment copy info information is ignored.
I'll add it in the next diffs.
Before diff is landed, I re-run the blobimport to prefill missing bonsai changesets.
Reviewed By: farnz
Differential Revision: D8893504
fbshipit-source-id: 1cc4bbcca2f489a0ef6990d6c04d5b3fd8bef92c
Summary:
Make it return manifest hashes. This is not very useful right now, but it will
be useful in the later diffs where we add BonsaiChangeset generation.
Reviewed By: farnz
Differential Revision: D9013976
fbshipit-source-id: 85f0c78bc7b944d300d537dd29cfd66f6f6530c6
Summary: Because get_all_filenodes can go above 1MB (but still less than 3MB) I added a chunking logic to speed up fetching of even the long histories
Reviewed By: farnz
Differential Revision: D9029774
fbshipit-source-id: c2d46a4b49bfebd3c778a0b9c5cf226857f34768
Summary:
I had enough seeing compiler giving me error messages with `blobrepo::Error` whereas it should be `failure::Error`.
`pub type` is exporting a type alias whereas `pub use` is re-exporting. The later apparently will fool the compiler to think that they own the thing they are re-exporting which in this case, failure::Error. If it is for the purpose of declaring the error type the crate is using, type alias is obviously more suitable as it won't take the ownership of the type.
Reviewed By: lukaspiatkowski
Differential Revision: D8926275
fbshipit-source-id: 6b1346adf52bec6b666a3f8b23e7a4ea4e0ab070
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
Previously incorrect filenodes were recorded. Instead of filenode "dir/file",
"file" was recorded as a path in filenodes table.
The next diff that adds creating file copies in bonsai changesets relies on
correct filenodes being present in the table.
Reviewed By: farnz
Differential Revision: D8915297
fbshipit-source-id: dcbde286d935bc726847f788779cfdc7c888c023
Summary:
The test was working fine, however, with the next diff in the stack
it started to fail under stress runs. And in that case I assume the problem in
the test itself.
The test creates two commits, that create file with the same content.
However, one filenode wasn't ancestor of another, and that's semantically
incorrect.
Reviewed By: farnz
Differential Revision: D8913129
fbshipit-source-id: 1b3838c6cf5060d768e71ef056cb7765ebbabc99
Summary:
These types are Hg specific. Since we are going to add bonsai changeset
creation soon, let's make it clear in the types
Reviewed By: farnz
Differential Revision: D8911359
fbshipit-source-id: 8b6cc45122402d7b7e074e66d904d979030de705
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
This proves to be extremely useful for the upcoming bonsai
verification code.
The in-memory stuff is more complicated for the database backends, so punt on
that for now with some warnings.
Reviewed By: farnz
Differential Revision: D8909426
fbshipit-source-id: 1d66d877cfa48ef06bbe614f37c66cf6c2f0446c
Summary:
The thrift client should return false or None for missing objects, but instead
errors out.
Reviewed By: StanislavGlebik
Differential Revision: D8899842
fbshipit-source-id: 81e5433ee63b7c8e6a4e898443a2a24bc784d014
Summary:
I don't like this because particularly the empty string for regular
files looks weird.
Reviewed By: StanislavGlebik
Differential Revision: D8888553
fbshipit-source-id: 20a9048a19b3fdfe681160a637bc2dfc8932c113
Summary: Also make it possible to have mutable clones
Reviewed By: farnz
Differential Revision: D8889900
fbshipit-source-id: 77d304ec5e07c84398e328b361ebf062cd18de45
Summary: Let's pass just one field instead of 4
Reviewed By: farnz
Differential Revision: D8889899
fbshipit-source-id: 8b30496a86950ed534439f5469f8740ee32345b8
Summary: This commit removes the usage of the old manifold client so we can get rid of `tokio_io::Remote`;
Reviewed By: jsgf
Differential Revision: D8751815
fbshipit-source-id: 41ab75076f1b2591496919ea91eb6b1d4d7ae506
Summary:
Bonsai verification tool fails
`buck run mode/opt scm/mononoke:bonsai-verify -- --repo-id 1001 867ca5df5dd87de7a98ffd4495bc97ea57a8af61 --debug --limit 1 --changes`
After debuggin this run with a lot of printfs it turned out we only keeping last entry, `manifest_lookup` was to blame
Reviewed By: farnz
Differential Revision: D8875499
fbshipit-source-id: 0df4c766d55657c8c4cd7005facce715b31e733e
Summary: tokio::runtime can handle multithreading, no need for all this bolierplate
Reviewed By: StanislavGlebik
Differential Revision: D8861170
fbshipit-source-id: 2c489068a55f8cba1854f8a748df1e6efe8b47b7
Summary:
Removed all references to RepoGenCache from publically callable functions in the revset package. This involved:
- Modifying blobrepo so that its get_generation_number method returned a Generation wrapper instead of a raw usize, to allow it to be used in a cleaner manner in the revset code.
- Simultaneously changing the constructors of all the structures in revset. This seems like a big change, but many of them call each other, passing a RepoGenCache object down the line, so eliminating them all at once made for the cleanest update.
- Modifying helper functions in the revset structures which would create streams of nodes by taking ownership of a RepoGenCache object within a closure. Instead they now take ownership of a clone of the repo. This strategy was already done earlier in the same helper functions, so I am assuming the cost of cloning a repo into a closure is small.
- Modifying the only external usage of revset within the mononoke server code.
This is part of a several step process to completely remove RepoGenCache from the code base. The next steps should be:
- Remove all references to RepoGenCache in the testing macros for revset.
- Delete RepoGenCache and clean up any dangling references to it.
Reviewed By: StanislavGlebik
Differential Revision: D8743560
fbshipit-source-id: 125f851075d836d40224d339e1daee912a39f7e4
Summary:
When we add cachelib bindings to Rust, we're going to want to implement a
cachelib blobstore that's more or less the same as the memcache version, but
backed by a cachelib pool instead of a memcache instance.
Split this code up so that we don't duplicate functionality
Reviewed By: StanislavGlebik
Differential Revision: D8523713
fbshipit-source-id: 882298abab8c208103f6d8c74fee60a768c877f6
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
This diff implements hooks which act upon individual files. The results of the hooks are cached using Asyncmemo.
The cache is currently keyed on (changeset_id, hook_name, file path) but this will change to file content hash once we move to Bonsai changesets.
Reviewed By: StanislavGlebik
Differential Revision: D8707726
fbshipit-source-id: ceaf94abd09e1dd7f6b2d8f9c87a9a221439a252
