Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
This proves to be extremely useful for the upcoming bonsai
verification code.
The in-memory stuff is more complicated for the database backends, so punt on
that for now with some warnings.
Reviewed By: farnz
Differential Revision: D8909426
fbshipit-source-id: 1d66d877cfa48ef06bbe614f37c66cf6c2f0446c
Summary:
The thrift client should return false or None for missing objects, but instead
errors out.
Reviewed By: StanislavGlebik
Differential Revision: D8899842
fbshipit-source-id: 81e5433ee63b7c8e6a4e898443a2a24bc784d014
Summary:
I don't like this because particularly the empty string for regular
files looks weird.
Reviewed By: StanislavGlebik
Differential Revision: D8888553
fbshipit-source-id: 20a9048a19b3fdfe681160a637bc2dfc8932c113
Summary: Also make it possible to have mutable clones
Reviewed By: farnz
Differential Revision: D8889900
fbshipit-source-id: 77d304ec5e07c84398e328b361ebf062cd18de45
Summary: Let's pass just one field instead of 4
Reviewed By: farnz
Differential Revision: D8889899
fbshipit-source-id: 8b30496a86950ed534439f5469f8740ee32345b8
Summary: This commit removes the usage of the old manifold client so we can get rid of `tokio_io::Remote`;
Reviewed By: jsgf
Differential Revision: D8751815
fbshipit-source-id: 41ab75076f1b2591496919ea91eb6b1d4d7ae506
Summary:
Bonsai verification tool fails
`buck run mode/opt scm/mononoke:bonsai-verify -- --repo-id 1001 867ca5df5dd87de7a98ffd4495bc97ea57a8af61 --debug --limit 1 --changes`
After debuggin this run with a lot of printfs it turned out we only keeping last entry, `manifest_lookup` was to blame
Reviewed By: farnz
Differential Revision: D8875499
fbshipit-source-id: 0df4c766d55657c8c4cd7005facce715b31e733e
Summary: tokio::runtime can handle multithreading, no need for all this bolierplate
Reviewed By: StanislavGlebik
Differential Revision: D8861170
fbshipit-source-id: 2c489068a55f8cba1854f8a748df1e6efe8b47b7
Summary:
Removed all references to RepoGenCache from publically callable functions in the revset package. This involved:
- Modifying blobrepo so that its get_generation_number method returned a Generation wrapper instead of a raw usize, to allow it to be used in a cleaner manner in the revset code.
- Simultaneously changing the constructors of all the structures in revset. This seems like a big change, but many of them call each other, passing a RepoGenCache object down the line, so eliminating them all at once made for the cleanest update.
- Modifying helper functions in the revset structures which would create streams of nodes by taking ownership of a RepoGenCache object within a closure. Instead they now take ownership of a clone of the repo. This strategy was already done earlier in the same helper functions, so I am assuming the cost of cloning a repo into a closure is small.
- Modifying the only external usage of revset within the mononoke server code.
This is part of a several step process to completely remove RepoGenCache from the code base. The next steps should be:
- Remove all references to RepoGenCache in the testing macros for revset.
- Delete RepoGenCache and clean up any dangling references to it.
Reviewed By: StanislavGlebik
Differential Revision: D8743560
fbshipit-source-id: 125f851075d836d40224d339e1daee912a39f7e4
Summary:
When we add cachelib bindings to Rust, we're going to want to implement a
cachelib blobstore that's more or less the same as the memcache version, but
backed by a cachelib pool instead of a memcache instance.
Split this code up so that we don't duplicate functionality
Reviewed By: StanislavGlebik
Differential Revision: D8523713
fbshipit-source-id: 882298abab8c208103f6d8c74fee60a768c877f6
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
This diff implements hooks which act upon individual files. The results of the hooks are cached using Asyncmemo.
The cache is currently keyed on (changeset_id, hook_name, file path) but this will change to file content hash once we move to Bonsai changesets.
Reviewed By: StanislavGlebik
Differential Revision: D8707726
fbshipit-source-id: ceaf94abd09e1dd7f6b2d8f9c87a9a221439a252
Summary:
Correctly resolve file/directory conflict in `MemoryManifest`
- when we have file/directory conflict and have entry in `BonsaiChangeset` that indicates that file should be deleted we need resolve conflict by keeping directory
- more details in [Bonsai changeset actions](https://fb.quip.com/A2kqArd9Nb90#WHBACA4Vitd)
Reviewed By: lukaspiatkowski
Differential Revision: D8658958
fbshipit-source-id: 815a91b847c5c0c9586eb5d317ebb9412196d5f4
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: This will be pretty useful while debugging.
Reviewed By: jsgf
Differential Revision: D8667770
fbshipit-source-id: 1c776741844d74529415124ab826b013414678c7
Summary:
It's easier to use scuba when you separate the part of log that identifies the log from the part that can have arbitrary data in it.
It's also easier to search for code after a sample was found
Reviewed By: jsgf
Differential Revision: D8625612
fbshipit-source-id: 7d7e382530dd5d7e5d69c6d34caccda4b6d2921b
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary: This code handles "trivial merge" (when file blob contains same data but different parents) of `MemoryManifest`
Reviewed By: farnz
Differential Revision: D8420775
fbshipit-source-id: dd0c640b8cc822d2dc42eed8f0e9a8e0a00f2cdc
Summary: more logging more fun
Reviewed By: StanislavGlebik
Differential Revision: D8577655
fbshipit-source-id: 92a160ea8f8c0b8e012a1461fbd3f5d71b4bd171
Summary:
Manifests are always able to return entries immediately, and never
fail.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D8556499
fbshipit-source-id: e21a2522f1219e47db9b55b24b6ac6c0c463933e
Summary: Going to master adds a latency. Let's go to master only if we haven't found anything in the replica
Reviewed By: lukaspiatkowski
Differential Revision: D8548598
fbshipit-source-id: b0632ccf7f148d20e66763981c8ef4e615fd7f6c
Summary: we want most of the read traffic to be handled by replicas
Reviewed By: lukaspiatkowski
Differential Revision: D8548027
fbshipit-source-id: 297b070b5b33ec114026ecf3f1079fdb26f54b19
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.
Reviewed By: farnz
Differential Revision: D8440330
fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary:
Since file and manifest nodes are going to be stored differently, rename this to
get_hg_raw_content to make it clear that it'll work for both files and
manifests.
I tried allowing a `Chain` to be used, but ran into https://github.com/carllerche/bytes/issues/213. Boo.
Reviewed By: farnz
Differential Revision: D8413001
fbshipit-source-id: 2f06affa265beaed2ac0a48ee4fec8ce922307da
Summary:
Pretty straightforward. Also using this opportunity to add per-repo
prefixes, since all the hashes are going to break anyway.
Note for reviewers: almost all the change is regenerated test fixtures (unfortunately necessary to make atomic). The actual substantive changes are all in the first few files.
Reviewed By: farnz
Differential Revision: D8392234
fbshipit-source-id: c93fc8c6388cb00fe5cff95646ad8c853581cb8c
Summary:
This diff introduces the ChangesetStore trait so the HookManager does not directly depend on the BlobRepo. This means the manager can obtain changesets from a simple in memory implementation of ChangesetStore which can be populated with changesets before they are committed to the server. This means the exact same hook manager logic can be used irrespective of whether the changesets are committed or not.
The diff includes two implementations of ChangesetStore - A simple in memory implementation backed by a HashMap, and one that backs onto BlobStore.
Reviewed By: StanislavGlebik
Differential Revision: D8446430
fbshipit-source-id: 8d14e48cb562fcd10a17370e34f13a662af827df
Summary:
It would be suboptimal if memcache could be abused to examine objects
you should not have access to. Change the key used so that (a) it's a regional
cluster for wider caching, and (b) it's got the blobstore details in it so that
you cannot extract objects not from your blob store.
Reviewed By: StanislavGlebik
Differential Revision: D8424298
fbshipit-source-id: 78f9a1a7302b4a60575f257bda665f719dc1a7b6
Summary: Added logic to save `FileChange` as a Mercurial format `HgBlobEntry`
Reviewed By: sunshowers
Differential Revision: D8187792
fbshipit-source-id: 4714c81ab23ebac528cfec15c4a9e66083d4fb6c
Summary:
Update to Rust 1.26.2 toolchain, which has a fix for a soundness
problem in match ergonomics. There was one instance of code affected by this.
Reviewed By: farnz
Differential Revision: D8401773
fbshipit-source-id: 9dfdd933b1e0cf92cdc179a84ea1b67064585ba1
Summary:
Note that no prefix is actually prepended at the moment -- there's an
XXX marking the spots we'll need to update. We'll probably add a prefix once Thrift serialization is turned on.
Reviewed By: farnz
Differential Revision: D8387761
fbshipit-source-id: 0fe2005692183fa91f9787b4c80f600df21d1d93
Summary:
Implementation of generic `store|fetch` for bonsai types.
- bonsai types have unique typed hashes associated with each bonsai type, I'm leveraging this fact to implement generic `store|fetch` methods on `BlobRepo`
Reviewed By: farnz
Differential Revision: D8254810
fbshipit-source-id: 5f798fade4cb8d1ac851f94c7ad7e64636bbca65
Summary:
Unfortunately `HgParents` can't represent all valid parents, because
it can't represent the semantically important case where `p1` is `None` and
`p2` is not. (For incoming changesets we'd like to keep full fidelity with
Mercurial.)
All the Thrift definitions store `p1` and `p2` separately anyway, so just make
that change throughout `RevlogChangeset` and `BlobChangeset`.
Reviewed By: StanislavGlebik
Differential Revision: D8374125
fbshipit-source-id: 63674daaad05d4d4cae3778744dbf1c14b3c2e3b
Summary:
The old blobimport tool will not be able to import commits with the new Thrift serialization they'll be switching to.
`blobrepo::utils::RawNodeBlob` is also used by the admin tool, and it will go away once we start using Thrift serialization.
Reviewed By: farnz
Differential Revision: D8372455
fbshipit-source-id: d02a37e33e1ccd4dd1f695e38dbb40851dd51cd6
Summary:
Mostly this was about adding support for file stores to
`new_blobimport`.
Reviewed By: StanislavGlebik
Differential Revision: D8372063
fbshipit-source-id: 2e3791c6222ec430015008f038e1df0464d3f0ba
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary: Lack of proper error context makes it hard to understand errors returned from Mononoke
Reviewed By: StanislavGlebik
Differential Revision: D8298154
fbshipit-source-id: 57fe5df7d891b5215fba783255178f14122199cf
Summary: the newly added context should add much more visibility into why an error was returned
Reviewed By: jsgf
Differential Revision: D8286343
fbshipit-source-id: d65387f40da2c14964e85552ae1ec7e75a135c47
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: Factor out common stuff for local state.
Reviewed By: StanislavGlebik
Differential Revision: D8310823
fbshipit-source-id: e1ce7eebd76d37688e830a5df0486bdfd1e3361c
Summary: This log is by far the most common one and it makes reading logs much harder. It should probably be changed to ODS counters, but for now lets just make it trace!
Reviewed By: farnz
Differential Revision: D8235663
fbshipit-source-id: 3685b260f1c6c43c1fde8501731583debc8d063b
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
