Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary:
PUT request upload to mononoke API
hg client sends a PUT request to store a file into blobstore during push supporting LFS
Upload file by alias is divied into 2 parts:
- Put alias : blobstore key
- Put blobstore_key: contents
Keep in mind, that file content is thrift encoded
host_address for batch request is from command line flags -H for host, -p for port
Reviewed By: StanislavGlebik
Differential Revision: D10026683
fbshipit-source-id: 6c2726c7fee2fb171582bdcf7ce86b22b0130660
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
We want to be able to do post-commit logging for all changesets. Set
up the data structures I intend to use for now, and arrange to discard all
logging.
A later diff will add logging to a ScribeClient.
Reviewed By: StanislavGlebik
Differential Revision: D9995984
fbshipit-source-id: 796b390f6b83ace576f73a217ac564c4251d7ec5
Summary:
This diff adds a real implementation for CachingChangesetFetcher. Now it
fetches the data for the cache from the blobstore.
The rest is explained in the comments.
Reviewed By: farnz
Differential Revision: D9908320
fbshipit-source-id: 5427f3ed312cb7753434161423cb27b48744347f
Summary:
Initial implementation of ChangesetsFetcher that will use cache smarter.
At the moment it doesn't do anything special, but in the next diffs it will pre
warm cache in case it has a lot of cache misses (that's why it has to have a
reference to the cachelib CachePool).
Reviewed By: farnz
Differential Revision: D9908319
fbshipit-source-id: 6377a947696bae6b060de5a441722c28309b341c
Summary:
High-level goal: we want to make certain big getbundle requests faster. To do
that we'd store blobs of commits that are close to each other in the blobstore
and fetch them only if we had too many cache misses. All this logic will be
hidden in ChangesetFetcher trait implementation. ChangesetFetcher will be
created per request (hence the factory).
Reviewed By: farnz
Differential Revision: D9869659
fbshipit-source-id: 9e3ace3188b3c13f83ef1bd61b668d4f22103f74
Summary:
WIP
Mononoke API download for lfs
support get request
curl http://127.0.0.1:8000/{repo_name}/lfs/download/{sha256}
Reviewed By: StanislavGlebik
Differential Revision: D9850413
fbshipit-source-id: 4d756679716893b2b9c8ee877433cd443df52285
Summary:
Let's check that new case conflicts are not added by a commit.
That diff also fixes function check_case_conflict_in_manifest - it needs to
take into account that if one of the conflicting files was removed then there
is no case conflict.
There should be a way to disable this check because we sometimes need to allow
broken commits. For example, during blobimport
Reviewed By: aslpavel
Differential Revision: D9789809
fbshipit-source-id: ca09ee2d3e5340876a8dbf57d13e5135344d1d36
Summary: Use `ChangesetId` in `DifferenceOfUnionsOfAncestorsNodeStream` instead of `HgNodeHash`. This avoids several bonsai lookups of parent nodes.
Reviewed By: StanislavGlebik
Differential Revision: D9631341
fbshipit-source-id: 1d1be7857bf4e84f9bf5ded70c28ede9fd3a2663
Summary:
Additional 2-step reference for blob:
For each file add an additional blob with:
key = aliases.sha256.sha256(raw_file_contents)
value = blob_key
Pay attention, that sha256 hash is taken `from raw_file_content`, not from a blob content.
Additional blob is sent together with the file content blob.
Reviewed By: lukaspiatkowski, StanislavGlebik
Differential Revision: D9775509
fbshipit-source-id: 4cc997ca5903d0a991fa0310363d6af929f8bbe7
Summary:
In `fetch_file_contents()` `blobstore_bytes.into()` converted the bytes to
`Blob<Id>`. This code calls `MononokeId::from_data()` which calls blake2
hashing. Turns out it causes big problems for large many large files that
getfiles can return.
Since this hash is not used at all, let's avoid generating it.
Reviewed By: jsgf
Differential Revision: D9786549
fbshipit-source-id: 65de6f82c1671ed64bdd74b3a2a3b239f27c9f17
Summary:
Profiling showed that since we are inserting objects into blobstore
sequentially it takes a lot of time for long stacks of commit. Let's do it in
parallel.
Note that we are still inserting sequentially into changesets table
Reviewed By: farnz
Differential Revision: D9683037
fbshipit-source-id: 8f9496b97eaf265d9991b94243f0f14133f463da
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary:
This diff fills missing parts of push-rebase implementation
- `find_closest_root` - find closest root to specified bookmark
- `find_changed_files` - find file affected by changesets between provided `ancestor` and `descendant`
- `intersect_changed_files` - rejects push rebase if any conflicts have been found
- `create_rebased_changes` - support for merges
- `do_pushrebase` - returns updated bookmark value
Reviewed By: StanislavGlebik
Differential Revision: D9458416
fbshipit-source-id: c0cb53773eba6e966f1a5928c43ebdec761a78d3
Summary: This commits change `HgBlob` from an enum into a struct that only contains one Bytes field, completely removes `HgBlobHash` and changes the methods of `HgBlob` from returning `Option`s into directly returning results.
Reviewed By: farnz
Differential Revision: D9317851
fbshipit-source-id: 48030a621874d628602b1c5d3327e635d721facf
Summary: Since this data is specific to TimedStream and not TimedFuture I split the Stats struct into FutureStats and StreamStats
Reviewed By: StanislavGlebik
Differential Revision: D9355421
fbshipit-source-id: cc2055706574756e2e53f3ccc57abfc50c3a02ba
Summary:
Revsets must use ChangesetId, not HgNodeHash. I'm going to use
`RangeNodeStream` in pushrebase so I thought it was a good time to change it
Reviewed By: farnz
Differential Revision: D9338827
fbshipit-source-id: 50bbe8f73dba3526d70d3f816ddd93507db99be5
Summary: Store where it was copied from
Reviewed By: farnz
Differential Revision: D9132560
fbshipit-source-id: a7a73e1f3de08340f5add5fffa32dd0373eb27fa
Summary:
The issues were fixed, and also thrift manifold works better during bulk
blobimport.
Reviewed By: farnz
Differential Revision: D9132384
fbshipit-source-id: ab4a04eeff86bb4968b80af00c404fad710db183
Summary: - Make `Bookmakrs` work with `ChangsetId` instead of `HgChangesetId`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D9297139
fbshipit-source-id: e18661793d144669354e509271044410caa3502a
Summary: As a bonus this diff also contains unifying the linknode family of methods (they all now accept arguments via reference) and better tracing for get_files request
Reviewed By: StanislavGlebik
Differential Revision: D9031283
fbshipit-source-id: 4526a8446984904bce7d4dcef240088c7f2ffaa3
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary: To make them more explicit
Reviewed By: matthewdippel
Differential Revision: D9132294
fbshipit-source-id: a365d7b58ba095d11fb0570e5ab6994a158873b3
Summary:
This is a split from D8893504. It just enables functionality to create bonsai changesets.
The split was done so that I can land the biggest chunk of work.
Reviewed By: farnz
Differential Revision: D9081430
fbshipit-source-id: 7437c7789998f5691afe83d5b16a8f2c5faac8b4
Summary:
It was using manifest hash as parent hash. One more reason to fix all the types
in Mononoke
Reviewed By: aslpavel
Differential Revision: D9123653
fbshipit-source-id: 0841f7ac64e50e9234d80040b7f286930af53420
Summary:
Now bonsai changesets are created at the same time as hg changesets, and
the mapping between bonsai and hg changesets is recorded
One important piece is missing. At the moment copy info information is ignored.
I'll add it in the next diffs.
Before diff is landed, I re-run the blobimport to prefill missing bonsai changesets.
Reviewed By: farnz
Differential Revision: D8893504
fbshipit-source-id: 1cc4bbcca2f489a0ef6990d6c04d5b3fd8bef92c
Summary:
Make it return manifest hashes. This is not very useful right now, but it will
be useful in the later diffs where we add BonsaiChangeset generation.
Reviewed By: farnz
Differential Revision: D9013976
fbshipit-source-id: 85f0c78bc7b944d300d537dd29cfd66f6f6530c6
Summary: Because get_all_filenodes can go above 1MB (but still less than 3MB) I added a chunking logic to speed up fetching of even the long histories
Reviewed By: farnz
Differential Revision: D9029774
fbshipit-source-id: c2d46a4b49bfebd3c778a0b9c5cf226857f34768
Summary:
I had enough seeing compiler giving me error messages with `blobrepo::Error` whereas it should be `failure::Error`.
`pub type` is exporting a type alias whereas `pub use` is re-exporting. The later apparently will fool the compiler to think that they own the thing they are re-exporting which in this case, failure::Error. If it is for the purpose of declaring the error type the crate is using, type alias is obviously more suitable as it won't take the ownership of the type.
Reviewed By: lukaspiatkowski
Differential Revision: D8926275
fbshipit-source-id: 6b1346adf52bec6b666a3f8b23e7a4ea4e0ab070
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
Previously incorrect filenodes were recorded. Instead of filenode "dir/file",
"file" was recorded as a path in filenodes table.
The next diff that adds creating file copies in bonsai changesets relies on
correct filenodes being present in the table.
Reviewed By: farnz
Differential Revision: D8915297
fbshipit-source-id: dcbde286d935bc726847f788779cfdc7c888c023
Summary:
The test was working fine, however, with the next diff in the stack
it started to fail under stress runs. And in that case I assume the problem in
the test itself.
The test creates two commits, that create file with the same content.
However, one filenode wasn't ancestor of another, and that's semantically
incorrect.
Reviewed By: farnz
Differential Revision: D8913129
fbshipit-source-id: 1b3838c6cf5060d768e71ef056cb7765ebbabc99
Summary:
These types are Hg specific. Since we are going to add bonsai changeset
creation soon, let's make it clear in the types
Reviewed By: farnz
Differential Revision: D8911359
fbshipit-source-id: 8b6cc45122402d7b7e074e66d904d979030de705
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
This proves to be extremely useful for the upcoming bonsai
verification code.
The in-memory stuff is more complicated for the database backends, so punt on
that for now with some warnings.
Reviewed By: farnz
Differential Revision: D8909426
fbshipit-source-id: 1d66d877cfa48ef06bbe614f37c66cf6c2f0446c
Summary:
The thrift client should return false or None for missing objects, but instead
errors out.
Reviewed By: StanislavGlebik
Differential Revision: D8899842
fbshipit-source-id: 81e5433ee63b7c8e6a4e898443a2a24bc784d014
Summary:
I don't like this because particularly the empty string for regular
files looks weird.
Reviewed By: StanislavGlebik
Differential Revision: D8888553
fbshipit-source-id: 20a9048a19b3fdfe681160a637bc2dfc8932c113
Summary: Also make it possible to have mutable clones
Reviewed By: farnz
Differential Revision: D8889900
fbshipit-source-id: 77d304ec5e07c84398e328b361ebf062cd18de45
Summary: Let's pass just one field instead of 4
Reviewed By: farnz
Differential Revision: D8889899
fbshipit-source-id: 8b30496a86950ed534439f5469f8740ee32345b8
Summary: This commit removes the usage of the old manifold client so we can get rid of `tokio_io::Remote`;
Reviewed By: jsgf
Differential Revision: D8751815
fbshipit-source-id: 41ab75076f1b2591496919ea91eb6b1d4d7ae506
Summary:
Bonsai verification tool fails
`buck run mode/opt scm/mononoke:bonsai-verify -- --repo-id 1001 867ca5df5dd87de7a98ffd4495bc97ea57a8af61 --debug --limit 1 --changes`
After debuggin this run with a lot of printfs it turned out we only keeping last entry, `manifest_lookup` was to blame
Reviewed By: farnz
Differential Revision: D8875499
fbshipit-source-id: 0df4c766d55657c8c4cd7005facce715b31e733e
Summary: tokio::runtime can handle multithreading, no need for all this bolierplate
Reviewed By: StanislavGlebik
Differential Revision: D8861170
fbshipit-source-id: 2c489068a55f8cba1854f8a748df1e6efe8b47b7
Summary:
Removed all references to RepoGenCache from publically callable functions in the revset package. This involved:
- Modifying blobrepo so that its get_generation_number method returned a Generation wrapper instead of a raw usize, to allow it to be used in a cleaner manner in the revset code.
- Simultaneously changing the constructors of all the structures in revset. This seems like a big change, but many of them call each other, passing a RepoGenCache object down the line, so eliminating them all at once made for the cleanest update.
- Modifying helper functions in the revset structures which would create streams of nodes by taking ownership of a RepoGenCache object within a closure. Instead they now take ownership of a clone of the repo. This strategy was already done earlier in the same helper functions, so I am assuming the cost of cloning a repo into a closure is small.
- Modifying the only external usage of revset within the mononoke server code.
This is part of a several step process to completely remove RepoGenCache from the code base. The next steps should be:
- Remove all references to RepoGenCache in the testing macros for revset.
- Delete RepoGenCache and clean up any dangling references to it.
Reviewed By: StanislavGlebik
Differential Revision: D8743560
fbshipit-source-id: 125f851075d836d40224d339e1daee912a39f7e4
Summary:
When we add cachelib bindings to Rust, we're going to want to implement a
cachelib blobstore that's more or less the same as the memcache version, but
backed by a cachelib pool instead of a memcache instance.
Split this code up so that we don't duplicate functionality
Reviewed By: StanislavGlebik
Differential Revision: D8523713
fbshipit-source-id: 882298abab8c208103f6d8c74fee60a768c877f6
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
This diff implements hooks which act upon individual files. The results of the hooks are cached using Asyncmemo.
The cache is currently keyed on (changeset_id, hook_name, file path) but this will change to file content hash once we move to Bonsai changesets.
Reviewed By: StanislavGlebik
Differential Revision: D8707726
fbshipit-source-id: ceaf94abd09e1dd7f6b2d8f9c87a9a221439a252
Summary:
Correctly resolve file/directory conflict in `MemoryManifest`
- when we have file/directory conflict and have entry in `BonsaiChangeset` that indicates that file should be deleted we need resolve conflict by keeping directory
- more details in [Bonsai changeset actions](https://fb.quip.com/A2kqArd9Nb90#WHBACA4Vitd)
Reviewed By: lukaspiatkowski
Differential Revision: D8658958
fbshipit-source-id: 815a91b847c5c0c9586eb5d317ebb9412196d5f4
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: This will be pretty useful while debugging.
Reviewed By: jsgf
Differential Revision: D8667770
fbshipit-source-id: 1c776741844d74529415124ab826b013414678c7
Summary:
It's easier to use scuba when you separate the part of log that identifies the log from the part that can have arbitrary data in it.
It's also easier to search for code after a sample was found
Reviewed By: jsgf
Differential Revision: D8625612
fbshipit-source-id: 7d7e382530dd5d7e5d69c6d34caccda4b6d2921b
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary: This code handles "trivial merge" (when file blob contains same data but different parents) of `MemoryManifest`
Reviewed By: farnz
Differential Revision: D8420775
fbshipit-source-id: dd0c640b8cc822d2dc42eed8f0e9a8e0a00f2cdc
Summary: more logging more fun
Reviewed By: StanislavGlebik
Differential Revision: D8577655
fbshipit-source-id: 92a160ea8f8c0b8e012a1461fbd3f5d71b4bd171
Summary:
Manifests are always able to return entries immediately, and never
fail.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D8556499
fbshipit-source-id: e21a2522f1219e47db9b55b24b6ac6c0c463933e
Summary: Going to master adds a latency. Let's go to master only if we haven't found anything in the replica
Reviewed By: lukaspiatkowski
Differential Revision: D8548598
fbshipit-source-id: b0632ccf7f148d20e66763981c8ef4e615fd7f6c
Summary: we want most of the read traffic to be handled by replicas
Reviewed By: lukaspiatkowski
Differential Revision: D8548027
fbshipit-source-id: 297b070b5b33ec114026ecf3f1079fdb26f54b19
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.
Reviewed By: farnz
Differential Revision: D8440330
fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary:
Since file and manifest nodes are going to be stored differently, rename this to
get_hg_raw_content to make it clear that it'll work for both files and
manifests.
I tried allowing a `Chain` to be used, but ran into https://github.com/carllerche/bytes/issues/213. Boo.
Reviewed By: farnz
Differential Revision: D8413001
fbshipit-source-id: 2f06affa265beaed2ac0a48ee4fec8ce922307da
Summary:
Pretty straightforward. Also using this opportunity to add per-repo
prefixes, since all the hashes are going to break anyway.
Note for reviewers: almost all the change is regenerated test fixtures (unfortunately necessary to make atomic). The actual substantive changes are all in the first few files.
Reviewed By: farnz
Differential Revision: D8392234
fbshipit-source-id: c93fc8c6388cb00fe5cff95646ad8c853581cb8c
Summary:
This diff introduces the ChangesetStore trait so the HookManager does not directly depend on the BlobRepo. This means the manager can obtain changesets from a simple in memory implementation of ChangesetStore which can be populated with changesets before they are committed to the server. This means the exact same hook manager logic can be used irrespective of whether the changesets are committed or not.
The diff includes two implementations of ChangesetStore - A simple in memory implementation backed by a HashMap, and one that backs onto BlobStore.
Reviewed By: StanislavGlebik
Differential Revision: D8446430
fbshipit-source-id: 8d14e48cb562fcd10a17370e34f13a662af827df
Summary:
It would be suboptimal if memcache could be abused to examine objects
you should not have access to. Change the key used so that (a) it's a regional
cluster for wider caching, and (b) it's got the blobstore details in it so that
you cannot extract objects not from your blob store.
Reviewed By: StanislavGlebik
Differential Revision: D8424298
fbshipit-source-id: 78f9a1a7302b4a60575f257bda665f719dc1a7b6
Summary: Added logic to save `FileChange` as a Mercurial format `HgBlobEntry`
Reviewed By: sunshowers
Differential Revision: D8187792
fbshipit-source-id: 4714c81ab23ebac528cfec15c4a9e66083d4fb6c
Summary:
Update to Rust 1.26.2 toolchain, which has a fix for a soundness
problem in match ergonomics. There was one instance of code affected by this.
Reviewed By: farnz
Differential Revision: D8401773
fbshipit-source-id: 9dfdd933b1e0cf92cdc179a84ea1b67064585ba1
Summary:
Note that no prefix is actually prepended at the moment -- there's an
XXX marking the spots we'll need to update. We'll probably add a prefix once Thrift serialization is turned on.
Reviewed By: farnz
Differential Revision: D8387761
fbshipit-source-id: 0fe2005692183fa91f9787b4c80f600df21d1d93
Summary:
Implementation of generic `store|fetch` for bonsai types.
- bonsai types have unique typed hashes associated with each bonsai type, I'm leveraging this fact to implement generic `store|fetch` methods on `BlobRepo`
Reviewed By: farnz
Differential Revision: D8254810
fbshipit-source-id: 5f798fade4cb8d1ac851f94c7ad7e64636bbca65
Summary:
Unfortunately `HgParents` can't represent all valid parents, because
it can't represent the semantically important case where `p1` is `None` and
`p2` is not. (For incoming changesets we'd like to keep full fidelity with
Mercurial.)
All the Thrift definitions store `p1` and `p2` separately anyway, so just make
that change throughout `RevlogChangeset` and `BlobChangeset`.
Reviewed By: StanislavGlebik
Differential Revision: D8374125
fbshipit-source-id: 63674daaad05d4d4cae3778744dbf1c14b3c2e3b
Summary:
The old blobimport tool will not be able to import commits with the new Thrift serialization they'll be switching to.
`blobrepo::utils::RawNodeBlob` is also used by the admin tool, and it will go away once we start using Thrift serialization.
Reviewed By: farnz
Differential Revision: D8372455
fbshipit-source-id: d02a37e33e1ccd4dd1f695e38dbb40851dd51cd6
Summary:
Mostly this was about adding support for file stores to
`new_blobimport`.
Reviewed By: StanislavGlebik
Differential Revision: D8372063
fbshipit-source-id: 2e3791c6222ec430015008f038e1df0464d3f0ba
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary: Lack of proper error context makes it hard to understand errors returned from Mononoke
Reviewed By: StanislavGlebik
Differential Revision: D8298154
fbshipit-source-id: 57fe5df7d891b5215fba783255178f14122199cf
Summary: the newly added context should add much more visibility into why an error was returned
Reviewed By: jsgf
Differential Revision: D8286343
fbshipit-source-id: d65387f40da2c14964e85552ae1ec7e75a135c47
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: Factor out common stuff for local state.
Reviewed By: StanislavGlebik
Differential Revision: D8310823
fbshipit-source-id: e1ce7eebd76d37688e830a5df0486bdfd1e3361c
Summary: This log is by far the most common one and it makes reading logs much harder. It should probably be changed to ODS counters, but for now lets just make it trace!
Reviewed By: farnz
Differential Revision: D8235663
fbshipit-source-id: 3685b260f1c6c43c1fde8501731583debc8d063b
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary: It only needs to borrow them.
Reviewed By: kulshrax
Differential Revision: D8244267
fbshipit-source-id: 2a24a3b7c6eb65177e4e26c57650dd7e096b4202
Summary:
I got confused myself, so decided to rename it.
dirname here means "all the path without the last path component". So for directory "a/b/c", "a/b" is a dirname. For file "a/b.txt", "a" is a dirname For root dirname is empty.
Reviewed By: jsgf
Differential Revision: D8207274
fbshipit-source-id: 0744ee020d3cf1ecc185efee445f6d40183d6366
Summary:
This is a follow-on from D8014913 (Make in-memory manifests load lazily), addressing jsgf's review comments.
I've made the following stylistic changes:
1. Use `impl Future` for all private APIs instead of `BoxFuture` (making use of `Either` instead of boxing where possible).
2. Use Rust 1.26 style `match` statements with inferred `ref` and `&` whenever possible.
3. Rework the logic in `MemoryManifestEntry::save` to not need `filter_map`.
4. Remove `future::result` and exploit `impl IntoFuture for Result` where possible to reduce code
5. Improve a couple of comments
Reviewed By: jsgf
Differential Revision: D8209186
fbshipit-source-id: c759ad8894fc25616dc6a291d46c487191f96382
Summary: Since signal_parent_ready purpose is to ensure that the child Changesets might perform their own checks it is good enough to send signal_parent_ready after all blobs have been written out, but before the parent did it's own checks
Reviewed By: farnz
Differential Revision: D8202108
fbshipit-source-id: 15ac85bd18bcf9ded61363a6380ad05462c189d6
Summary:
Memcache provides us with a low-latency out-of-process cache, shared
between multiple machines. If we use this to provide a write-through cache,
then we get a couple of benefits:
1. Mononoke servers accessing the same memcache backend get low latency access
to commonly requested blobs.
2. Recently written blobs are available at low latency to the server that just
wrote them, hopefully speeding up new_blobimport.
Reviewed By: StanislavGlebik
Differential Revision: D8124623
fbshipit-source-id: 5af085aa8bb63c1366740edfda9020d72b8a9015
Summary:
instead of doing for_each which causes the entries to be processed one-by-one use the map + buffer_unordered technique
This has speeden up new_blobimoprt of 2 commits from >2m to <1m30s
Reviewed By: farnz
Differential Revision: D8184176
fbshipit-source-id: 4a0c3124d2398ed41f8b93785bc3c890a23a88aa
Summary:
In previous diffs we've added a function that does a bulk prefetch of
filenodes. Now we can use it in the getfiles request handling code.
Before constructing file history we do a bulk prefetch. Then we use
results of the bulk prefetch as a cache. It means that we first check
whether we have file history node in the prefetched data, and if not it will be
fetched again. It allows us to later plugin a memcache or some other kind of
cache.
repo.rs file is getting bigger and bigger. I'm planning to split it, however
only after I land this diff in order not to deal with merges
Reviewed By: farnz
Differential Revision: D8185701
fbshipit-source-id: 6ca37aeb029236db51d2b5a03cb7053f969cf47e
Summary:
I'm about to introduce a MemoizedBlobstore - let's ensure that
ambiguity is kept to a minimum by renaming CachingBlobstore
Reviewed By: jsgf
Differential Revision: D8124625
fbshipit-source-id: 301fbb31c5a772c11e84566889b8a6ac86cdae19
Summary:
In practice, caching blobstores are always wrapped around a known
concrete blob store. Make it generic in the type of the Blobstore, so that
rustc can reach in and inline appropriately if it sees an optimization
opportunity.
This is a micro-optimization, hence nice to have
Reviewed By: StanislavGlebik
Differential Revision: D8124627
fbshipit-source-id: 0d5d9b11fdede062ceaa949ace574b3d1c75e6b5
Summary:
We want to be able to go to and from Bonsai Changesets in Mononoke, so
that we can operate with Mercurial clients that haven't yet been updated to
support Bonsai form. This RFC commit shows a shell of doing so, as an
illustration for the hackamonth task
Reviewed By: jsgf
Differential Revision: D8014909
fbshipit-source-id: 28adf18ecf80e0116290662c117731b4c1632ff9
Summary:
Loading the entire manifest tree just for a limited set of conflicts
and changes is wasteful. Lazily load the referenced parts of the tree instead
Reviewed By: jsgf
Differential Revision: D8014913
fbshipit-source-id: 07678bee39de02414fdc062cf680fcd049a28415
Summary:
Those flags will let us control the subset of commits that the new_blobimport is supposed to import.
It's a temporary solution to be used instead of the old blobimport but before "hg push" works
Reviewed By: farnz
Differential Revision: D8122318
fbshipit-source-id: a1ac0824020341cd4bb18ec46d91caee50d9606e
Summary:
This could never fail - so don't allow for failure, but use the type
system to deal with the error case.
Reviewed By: jsgf
Differential Revision: D8014910
fbshipit-source-id: 8f5f0a3ff55a96b57cd4c246d072793c986724d5
Summary:
We want to handle conflicts here - do so, but with asserts, by merging
two manifests together.
The asserts will be removed once we make this all lazy
Reviewed By: jsgf
Differential Revision: D8014912
fbshipit-source-id: 7a09186f7e24a3af93d15a859c877e3a319fb110
Summary: Once we have a lazy loading version, we'll want a way to cope with multiple accessors all trying to update the same `children` array with identical data; make it immutable in use, so that overwriting it multiple times is harmless
Reviewed By: jsgf
Differential Revision: D8014908
fbshipit-source-id: 9a2750fb1fca54601051fede1d9a37de8cfc2a74
Summary:
We're going to need to be able to edit memory manifests. Provide
remove and set operations, to match the Bonsai Changeset data structures
Reviewed By: StanislavGlebik
Differential Revision: D7620527
fbshipit-source-id: e85459c5dbfa8855267fd2cf6578c9fc39f223f8
Summary:
Removing entries can leave us with empty manifests. Rather than clean
that up mid-flow, simply skip empty manifests when saving
Reviewed By: StanislavGlebik
Differential Revision: D7620526
fbshipit-source-id: 2f0799eba305a5295eaad28cd4ad90c9de04306f
Summary:
To do Bonsai Changesets, we're going to need to perform surgery on
manifests. Provide a mechanism to get the Tree nodes only into memory, and
write them back out, so that we can do the surgery the easy way.
Reviewed By: StanislavGlebik
Differential Revision: D7557271
fbshipit-source-id: 7afdc3ef464fc042eb758af863ade8938c4e9fc5
Summary:
Previously, we assumed that all content hashes came from Mercurial;
this is not going to remain true, as we will want to be able to upload manifests
that have been synthesised from Bonsai Changesets. Turn the previous boolean
into a tri-state, and fix up all callers to get the behaviour they expect.
Reviewed By: StanislavGlebik
Differential Revision: D8014911
fbshipit-source-id: 9156b9fab4542ceb269626ad005e1b28392b5329
Summary:
Rust 1.26 adds many new language features. In particular `impl Trait` is now
stable, so we no longer need `conservative_impl_trait`.
There also seems to have been changed in the (unstable) TryFrom with respect to
usize, and the behaviour of the never type `!`.
There are still a few deprecation warnings, but they don't cause the build to
fail.
Path remapping is now stable, so the buck config needs to change to use it
rather than the unstable command line option.
TODO:
- get aarch64 rust-crates-io build (can defer to a later update)
Reviewed By: Imxset21
Differential Revision: D7966091
fbshipit-source-id: 2e61e262c21eb01c852a36f49c6a6369cdaddcdb
Summary:
This is a (hopefully) short term hack to overcome the problem of overloading
Manifold.
Ideally manifold client has to adjust dynamically to the load. However
implementing it is
not trivial, so for now let's configure via config option.
Reviewed By: jsgf
Differential Revision: D7910979
fbshipit-source-id: c2dc32b592747732e7e6574e0fecf2d0aaef447e
Summary:
Simple precaching. Reads all the manifests for a bookmark and up to
`commit_warmup_limit` of ancestors.
Warming up file content can be slow, so we don't do it now.
Reviewed By: jsgf
Differential Revision: D7863728
fbshipit-source-id: bed1508b01e4e002a399d00ea45faf8a8e228d0a
Summary:
This will make it easier to change the "real" bookmark type from AsciiString to
String if we decide to do that.
BookmarkPrefix is a separate type because we may want to change it from
AsciiString to String. Also we don't want to confuse a bookmark prefix with a
bookmark name.
Reviewed By: jsgf
Differential Revision: D7909992
fbshipit-source-id: 3d4d075c204ed5ef1114a743430982c2836bac04
Summary:
We don't need to explicitly create timers since the environment has
one set up by default.
Reviewed By: StanislavGlebik
Differential Revision: D7873576
fbshipit-source-id: bfcdc27a46397bff0730f64ad4f3de3865c7cfa1
Summary:
Let's use the new feature in SendWrapper to use many io threads. That will help
us mitigate the high cpu usage issues we were having with blobstore requests.
Manifold blobstore now creates the io threads itself.
Reviewed By: kulshrax
Differential Revision: D7831420
fbshipit-source-id: ec9f3327347ca6bfbd23c482e69a6fee663b1da5
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary:
Specialized revsets to make pull faster.
Previous Union + Intersect combination was extremely slow because it fetched a
lot of stuff that wasn't used.
Reviewed By: farnz
Differential Revision: D7829394
fbshipit-source-id: c038f184c305e48e18b6fcb0f83bab9e9a42b098
Summary:
The commits that are blobimported have out of order or simply incorrect lists of changed files.
Because we have to persists Changesets as is we are passing the untouched list of files here to be used by Changeset.
Reviewed By: farnz
Differential Revision: D7830310
fbshipit-source-id: 56adec2c317896decaa9176b3a6bfb0cab187ed0
Summary: The deltas are meant to be applied on raw filenode content that has incorporated copy/move information. Make it so
Reviewed By: jsgf
Differential Revision: D7777329
fbshipit-source-id: 23ce90269cdf5dbb8a16aab026c031c2af68fccb
Summary: the idea of Mercurial heads in Mononoke will be represented by bookmarks, so there is no need to have them around
Reviewed By: StanislavGlebik
Differential Revision: D7775032
fbshipit-source-id: 1618a1e51862d7c115b2955082f40ee890a045f1
Summary: Although this change is breaking with the traditional definition of heads in Mercurial, we decided that using bookmarks as heads is sufficient for fbsource use case as well as it matches the git model where every unnamed branch is garbage collected.
Reviewed By: jsgf
Differential Revision: D7745541
fbshipit-source-id: 2ec716db7799bf7bab8b2b77a85d1521a880f3df
Summary: For on-disk-rocksdb use cases we should persist bookmarks like any other table we use
Reviewed By: farnz
Differential Revision: D7728717
fbshipit-source-id: f63a6410f5ed254a719a16a7504d1b31da5a20a8
Summary: The eden integration test contains a commit with no content which new_blobimport couldn't import. With this changes the commit API is capable of handling such commits.
Reviewed By: jsgf
Differential Revision: D7709243
fbshipit-source-id: 7d55eb2ec421820d189ab05b0f8cb4411f850a7b
Summary: The problem in compute_changed_files is that for 2 parents given instead of producing a list of files that were changed in the merge commit from perspective of both parents it produced the opposite of that. This should fix it
Reviewed By: jsgf
Differential Revision: D7708215
fbshipit-source-id: d58457054625beb853f61cb060d25b09279ecceb
Summary:
Use asyncmemo to cache Changesets.
Unfortunately currently we are using separate asyncmemo cache, so we have to
specify the size for the caches separately. Later we'll have a single cache for
everything, and the number of config knobs will go down.
Reviewed By: lukaspiatkowski
Differential Revision: D7685376
fbshipit-source-id: efe8a3a95fcc72fab4f4af93564e706cd1540c2f
Summary:
Let's use it! Pass config option that set's the cache max memory usage (don't
put a limit on the number of entries, it's useless in that case).
Currently we'll set a separate size for each of the caches that we use
(blobstore, changesets, filenodes, etc). Later we'll have just one single option that
sets the cache size for all of them.
Reviewed By: lukaspiatkowski
Differential Revision: D7671814
fbshipit-source-id: f9571078e6faaa80ea4c31c76a9eebcc24d8a68a
Summary:
We know that the hashes for non-root-tree-manifests and filenodes
should always be consistent. Verify that.
Reviewed By: farnz
Differential Revision: D7704087
fbshipit-source-id: 7f6207878c5cd372b272aa6970506dd63b5a3c7c
Summary:
As the comment explains, sometimes the hashes don't match the
contents. Accept such pushes.
Reviewed By: farnz
Differential Revision: D7699930
fbshipit-source-id: 376f01b6cf03f6cad84c2c878d192d55f8d81812
Summary:
* `Into<Bytes>`, not `Into<String>`
* Centralize in one place because I'm going to be adding more non-trivial code to this in upcoming diffs
Reviewed By: farnz
Differential Revision: D7694764
fbshipit-source-id: e6373cf1562b9a464894466ecf3be59e9892ac78
Summary:
We're going to keep this around for now as part of double-writing.
All the hashes here are definitely Mercurial hashes, so use them that way.
Reviewed By: lukaspiatkowski
Differential Revision: D7683890
fbshipit-source-id: 270091cd11f3cec7ef4cf565de5ef913fcf7adea
Summary:
file::File works entirely in the Mercurial domain, so these
conversions are good.
Reviewed By: StanislavGlebik
Differential Revision: D7665973
fbshipit-source-id: 8a192c5d1886492ad21593693b080c8e5ddf8f7e
Summary:
This is because these Mercurial entries are (at least currently) going
to be stored as they come in, and this data structure is entirely in the
Mercurial domain.
Reviewed By: lukaspiatkowski
Differential Revision: D7664972
fbshipit-source-id: 9de5475eed0d7ab7085c29fd0282f205043cfe5a
Summary:
This is the first step to getting bonsai changesets into the blob store.
So far the code just waits for content blob uploads to be done. In future diffs Mononoke will synthesize this information into a bonsai changeset as well.
Reviewed By: lukaspiatkowski
Differential Revision: D7627784
fbshipit-source-id: da0c96772ce4d18aed579cecca6135137a8dbe18
Summary:
I was trying to debug something with the new blobimport, and this was
getting in the way.
Reviewed By: StanislavGlebik
Differential Revision: D7664660
fbshipit-source-id: 2ec4ee79fbe13584f35e7dcd9e8df2b8bdf181c0
Summary:
This confused me at first -- took me a while to figure out what's going on
here. Hopefully these comments will help the next person who stares at this
code.
Reviewed By: StanislavGlebik
Differential Revision: D7659993
fbshipit-source-id: a009041501f9108d2c01532e9df793cfcecb5496
Summary:
The list of arguments is becoming too long, and I need to add even
more here.
Reviewed By: StanislavGlebik, farnz
Differential Revision: D7652096
fbshipit-source-id: 62a4631e163e95cf5c950a949e72facab629ea54
Summary:
The base type is better because it can represent dates from before
1970 as well.
Reviewed By: StanislavGlebik
Differential Revision: D7652095
fbshipit-source-id: 6d66a06e18ba28e13e70b9f0e921acbd3d55baaf
Summary:
Currently, any sort of `Bytes` can be stored in the blobstore. That
caused me to make several mistakes while writing the code to store bonsai
changesets, because I'd just end up storing the wrong set of `Bytes`.
Introduce stronger typing so that only types that explicitly implement
`BlobStorable` can be stored in the blobstore.
Currently, these sorts of blobs can be stored in the blob store:
* `ChangesetBlob` and `ContentBlob` in `mononoke-types` (these are Thrift-serialized structures)
* The envelope `RawNodeBlob` and `RawCSBlob` types in `blobrepo`, once converted to `EnvelopeBlob` instances
* `HgBlob`, which contains revlog data (manifests or files) exactly as serialized by Mercurial
Reviewed By: StanislavGlebik
Differential Revision: D7627290
fbshipit-source-id: d1bcbde8881e365dec99618556e7b054985bccf7
Summary:
Pass mysql tier name to the BlobRepo, so that we can use it to connect to mysql
based storages like mysql changeset storage, filenodes storage etc.
Note that currently Filenodes storage only connects to master region. This will
be fixed in the later diffs
Reviewed By: lukaspiatkowski
Differential Revision: D7585191
fbshipit-source-id: 168082abfeb7ccba549c7a49e6269cc01c490c14
Summary:
This helps out with the stricter-types-in-blobstore work I'm doing,
and is nice cleanup anyway.
Reviewed By: StanislavGlebik
Differential Revision: D7625562
fbshipit-source-id: c27901ef94a03a716368403b63a6afd74a1c2a90
Summary:
Now that `BlobNode` no longer returns `None`:
* don't expose the `BlobNode` API outside the crate because it turns out to not be very useful (it should probably go away eventually?)
* make the `File` API not return `Option` types
* Add a new `file_contents` that returns a brand-new `FileContents` (this is the first time we're tying together Mercurial and Mononoke data structures!)
Also remove a `Symlink` API that isn't really correct honestly.
Reviewed By: StanislavGlebik
Differential Revision: D7624729
fbshipit-source-id: 38443093b8bfea91384c959f3425cf355fac9f65
Summary:
This is not only the newer, more specific type -- it also makes a couple
of upcoming diffs more straightforward.
Reviewed By: StanislavGlebik
Differential Revision: D7622906
fbshipit-source-id: 4e453b827512c538f4f9777ae4d24627f3b124cf
Summary: Having an implicit `From` parsing makes it hard to track the exceptional places where RevlogChangest can be directly translated to BlobChangeset. Make it explicit for better tracking of this behavior
Reviewed By: StanislavGlebik
Differential Revision: D7637247
fbshipit-source-id: 781341315102ea6b2265c33bb09a89aae3d0c329
Summary:
I'm going to put in some stronger typing around what can be stored in
the blob store. Centralizing the management here makes that much easier.
Reviewed By: StanislavGlebik
Differential Revision: D7619519
fbshipit-source-id: a428679018f0a1571e54bc01bb5483ba9fdf1cb5
Summary: mercurial_types::DBlobNode should be replaced by types from mononoke_types or mercurial in most cases. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619793
fbshipit-source-id: 261fd92acae825dc4bc8011c3716c5585eb0413c
Summary: mercurial_types::DParent should be replaced by types from mononoke_types or mercurial in most cases. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619686
fbshipit-source-id: 5ad105113779387f1408c806860483e06ed5fb3d
Summary: mercurial_types::DEntryId should be replaced by types from mononoke_types or mercurial in most cases. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619571
fbshipit-source-id: bf8d81ec9ffe6a5525d923d7ee67d8e92498aa4d
Summary: mercurial_types::DFileNodeId should be replaced by types from mononoke_types in most cases. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619290
fbshipit-source-id: aa6a8e55ae3810c4531028c3b3db2e5730fe7846
Summary: mercurial_types::DManifestId should be replaced by types from mononoke_types in most cases and by mercurial::HgManifestId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619062
fbshipit-source-id: 447224194c6555334b64dc29ebabe3ef0d0cb87e
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary: mercurial_types::NodeHash should be replaced by types from mononoke_types in most cases and by mercurial::NodeHash in others. This rename should help with tracking this fact.
Reviewed By: sid0
Differential Revision: D7618389
fbshipit-source-id: a876e723d911df626c7851fba56a056843b4e049
Summary:
I haven't actually fixed the bug here because that's quite a bit of work that
will hopefully soon be somewhat moot.
Reviewed By: StanislavGlebik
Differential Revision: D7594278
fbshipit-source-id: b29861c71628a7dc903d6405c3348a9e41002fba
Summary:
This is a cleanup of NodeHash API. There were few unused methods and few ways to convert between mercurial and mercurial_types hashes. With this diff it is very easy to identify the places where this converstion happens.
A followup of this diff will be to use this new API to easily replace the NodeHash convertions in places where it requires remapping.
Reviewed By: sid0
Differential Revision: D7592876
fbshipit-source-id: 6875aa6df1a3708ce54ca5724f6eb960d179192b
Summary: They are replaced by filenodes
Reviewed By: farnz
Differential Revision: D7443320
fbshipit-source-id: 13c7d07bc00dcbaa991663c8da8a07fcb0de1332
Summary:
This will probably go away soon, but for now I want to be able to
disambiguate the new Thrift-encoded blobs in Mononoke from these.
Reviewed By: StanislavGlebik
Differential Revision: D7565808
fbshipit-source-id: d61f3096fa368b934a923dee54a0ea1e3469ae0d
Summary:
`left` and `right` are too vague in this context, since there's
already `Added` and `Removed` which imply an explicit direction.
Reviewed By: StanislavGlebik
Differential Revision: D7560975
fbshipit-source-id: 847f600508434d62857ff60a6e6678a4426ffa88
Summary:
There's no guarantee that a symlink is a valid `MPath` -- in fact, it
frequently will be an absolute symlink which is not a valid `MPath`.
Reviewed By: lukaspiatkowski
Differential Revision: D7541672
fbshipit-source-id: c7eca54cd06e811cf54932ab8f671c13a62b5f71
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary: Previously, the `.timed()` combinator exported by the `futures-stats` would accept a synchronous callback to log the recorded stats. In practice, logging the stats may require blocking IO, so this combinator should be modified to accept a closure that returns a Future instead.
Reviewed By: farnz
Differential Revision: D7492957
fbshipit-source-id: c634b511e5a93a18787ab3e11ab8e1c19de7566e
Summary:
filenodes are faster than blobstore in answering `get_parents` and
`get_copy_info` queries. So let's use them instead of going to blobstore.
Reviewed By: jsgf
Differential Revision: D7485151
fbshipit-source-id: f9e2ce9dd42b3e9b18140cc3de2c9718f20ab6d4
Summary:
Use filenodes in BlobRepo instead of linknodes.
That involves changes in commit API because we need to write filenodes not linknodes.
Also it involves changes to memblob test repos.
Note that in this diff they are basically "broken" - linknodes were replaced with empty filenodes. However looks like our unittests haven't exercised this functionality, so no unit-tests has failed. In the next diffs the correct filenodes will be added.
Reviewed By: farnz
Differential Revision: D7413481
fbshipit-source-id: 7f994ea55887b96a0eacf8e6ffdebd1f8c5f025d
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
Summary:
For historical reasons (back when we thought Mononoke might need to handle both flat and tree manifests natively), we decided to make the entry type in a Manifest be `MPath`, not `MPathElement`.
In the tree manifest only world we're now in, it's a bug if a Manifest contains a path instead of a path element - enforce this through the type system, so that we can't backslide accidentally.
Reviewed By: StanislavGlebik
Differential Revision: D7413870
fbshipit-source-id: fa13b676475ea8a3265fc962910542f2f555d7de
Summary: Make it possible to use a simulated remote backend as a blobstore. This allows me to look at the test results and be happy that Mononoke is at fault for any slowness.
Reviewed By: kulshrax
Differential Revision: D7353229
fbshipit-source-id: 57528af704b517a70570bb2b9b140caa9120a956
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary:
There's no point passing it by reference since callers don't need to
retain it, and the async implementation needs to move it into another context.
Reviewed By: farnz
Differential Revision: D7350001
fbshipit-source-id: 5947557a84621afae801dc20e3994496244e3a10
Summary:
This codemod tries not to change the existing behavior of system, only introduce new types specific to Mercurial Revlogs.
It introduces a lot of copypasta intentionally and it will be cleaned in following diffs.
Reviewed By: farnz
Differential Revision: D7367191
fbshipit-source-id: 0a915f427dff431065e903b5f6fbd3cba6bc22a7
Summary: Remove usage of deprecated `time` crate in `futures-stats`, and fix all callsites using the new `time-ext` crate.
Reviewed By: farnz
Differential Revision: D7349956
fbshipit-source-id: 10ef86c4942b8533a734c7daadfa895f5ef92f23
Summary:
A lot of Mononoke expects MPath instances to be non-empty, and some
Mononoke code wants empty MPaths. Have formerly-empty `MPath` instances be
represented as `None` instead.
There's a bit of a wart where we have to check for some instances being empty
even though they should never be. This is not a regression -- this is actually
a deficiency in our current type hierarchy being exposed. `Entry` can sometimes
represent a root manifest, but a root path and a root manifest will never be
joined together in practice. We may want to figure out a better way to
represent that in the future.
Also clean up a bunch of code that checks for empty `MPath` instances. That
just isn't possible any more -- yay.
Reviewed By: StanislavGlebik
Differential Revision: D7359048
fbshipit-source-id: 684f908dabaa06cb25b1dd33f4fd7bb021cb5e1b
Summary:
Lots of places specifically want a non-empty `MPath`, and earlier those cases
weren't type-checked. Now `Option<MPath>` stands for an empty `MPath`. In the
next diff `MPath::empty()` will go away and the only way to represent an
`MPath` that doesn't exist will be with `Option<MPath>`.
Reviewed By: farnz
Differential Revision: D7350970
fbshipit-source-id: 1612aec67134e7a0ebad15dbaa93b5ea972f8ddf
Summary:
Actually update the bookmarks using pushkey part. Note that bookmark deletion
doesn't currently work. It will be fixed in the next diffs
Reviewed By: farnz
Differential Revision: D7271600
fbshipit-source-id: fd13d1adfd3ea490174c31e23289e8560dc2b737
Summary:
The `Option<&MPathElement>` type is more general -- it's easy to
convert from `&Option<MPathElement>` to it, but the other way around can
require a clone.
Reviewed By: farnz
Differential Revision: D7339161
fbshipit-source-id: 0c8ab57a19bc330245c612e3e0e3651e368ab8cb
Summary: just to excersise our RocksDb blobstore which might come in handy more often than file based as it should be more efficient
Reviewed By: StanislavGlebik
Differential Revision: D7290069
fbshipit-source-id: ce776cfa14e43dc45cca796ef187655ba665d177
Summary:
To diagnose slow changeset creation, we need to know which chunk was
slow. As with upload_blob and process_entries in past diffs, measure key
phases.
The idea is that we can combine all these times (5 per changeset, 2 per blob)
to work out what needs tracing when we're looking for slowness
Reviewed By: StanislavGlebik
Differential Revision: D7195013
fbshipit-source-id: 9769ead981f02d6ecf0258190763e90762936387
Summary:
Processing entries does two things:
1. Polls all the upload futures until they're all complete.
2. Works out what entries are required to be present for this changeset.
This is the bulk of the Blobstore operations in creating a changeset, so we
know that if this is slow, then we either have a slow Blobstore, *or* we have a
bug. Further, we can combine this with the metrics for upload_blob to know
whether we're doing uploads nicely in parallel, or whether there's
serialization we didn't intended (time taken here greater than sum of time
taken in blob uploads)
Reviewed By: StanislavGlebik
Differential Revision: D7182445
fbshipit-source-id: 205b43700f28ff7e5461235e16a6955b8c597a52
Summary:
While writing Thrift deserialization code I realized there was nothing
that actually checked that MPathElement instances don't have embedded nulls or
slashes.
Reviewed By: farnz
Differential Revision: D7296838
fbshipit-source-id: 6a23d559da11e5e935e23d7b9a13f58894efaf62
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary:
I'm going to reuse this for unit-testing changeset timings. Make it a
macro so that I don't keep repeating myself.
Reviewed By: StanislavGlebik
Differential Revision: D7182442
fbshipit-source-id: de40e0f10892b2268c4d39cf771b7a8be6e1cf76
Summary:
I want to be able to work in a private Manifold namespace when testing
that my hunches about Manifold/Mononoke interactions are good, and when testing
changes to the ManifoldBlob that should make life better. Make that possible.
In the future, once I have a Blobstore that simulates Manifold locally, this
also makes it possible to compare the simulation to Manifold without tripping
over other users.
Reviewed By: StanislavGlebik
Differential Revision: D7194754
fbshipit-source-id: 601bf1c2ded1af5db6a123fdd05600bc3eb5d7cf
Summary: SharedError has a default implementation for std Error, but the parameter must implement std Error, that is why I wrapped failure::Error in failure::Compat. Thanks to that we can use SharedError as any other normal std Error (including the fact that it implements failure::Fail)
Reviewed By: farnz
Differential Revision: D7213459
fbshipit-source-id: 54899c64c2627dfdba276630d986a3d6007ea59a
Summary:
We want to be able to measure the time it takes to upload individual
blobs, to confirm that we don't have a concurrency issue to chase down (e.g.
blobs accidentally uploaded in series).
Measure content upload time separately, so that we know not to dive down a
rabbit hole if the measured slowness is just the time spent uploading content
Reviewed By: StanislavGlebik
Differential Revision: D7172154
fbshipit-source-id: 08729a8ffaa69a364a64f6277edfa591a8712592
Summary:
I'm going to be adding more to test the timing features, so split this
up to make code sharing easier
Reviewed By: StanislavGlebik
Differential Revision: D7172156
fbshipit-source-id: 056be70268dd1c8a37aff8e8d53342b8cea4a355
