Summary:
Implemented the methods which query for reachability on the SkiplistIndex, and calls to the appropriate tests on the test fixtures. All the functionality for this exists in the new method `query_reachability_with_generation_hints`. This method operates as follows:
* If there are skip list edges, traverse the one that goes the closest to the generation number of dst without passing it, then recurse for reachability from the new node..
* If there are only parent edges, recurse on each parent, and return the logical or of the answers.
* If the node is unindexed, perform `lazy_index_node` on it, and take the difference in generation numbers between the src and dst nodes as the maximum depth to index to.
Reviewed By: StanislavGlebik
Differential Revision: D9009060
fbshipit-source-id: ec3d372bd3ce99d9d9853972f76dc4a39e316b19
Summary: See the doc comment for what this state represents and why it is necessary.
Reviewed By: StanislavGlebik
Differential Revision: D9025772
fbshipit-source-id: b0588d037365194bbf2d9889ead60237ef097359
Summary:
Now bonsai changesets are created at the same time as hg changesets, and
the mapping between bonsai and hg changesets is recorded
One important piece is missing. At the moment copy info information is ignored.
I'll add it in the next diffs.
Before diff is landed, I re-run the blobimport to prefill missing bonsai changesets.
Reviewed By: farnz
Differential Revision: D8893504
fbshipit-source-id: 1cc4bbcca2f489a0ef6990d6c04d5b3fd8bef92c
Summary:
* Moved the field `skip_edges_per_node` from `SkiplistIndex` to `SkiplistEdgeMapping`.
* Removed the `Arc<BlobRepo>` field from `SkiplistIndex`.
The first change was to avoid having to pass around the `skip_edges_per_node` value into all the closures. Since these closures include an `Arc<SkiplistEdgeMapping>`, they can just get the value from that.
The second change is to be more in line with the `ReachabilityIndex` trait that I defined and to fit more easily into how I wrote the tests which take a generic `ReachabilityIndex` object. One other possibility, if I decide that the object should hold onto a reference to the repo, and not receive it as input at query time, is to add another trait method `use_repo`, which would start using a given `Arc<BlobRepo>`. A single instance of `ReachabilityIndex` shouldn't be used for different repos, and this could be a way to check that at compiler time. But right now I'm assuming it's passed at query time.
Reviewed By: StanislavGlebik
Differential Revision: D9009059
fbshipit-source-id: 9d63acf8a690816b3a17c84108a6feaa98ebe2ba
Summary:
Added a helper function which attempts to fetch the Generation number for a node hash, and if successful, returns the (HgNodeHash, Generation) tuple. I found myself needing this in several places while working on the SkiplistIndex, and will probably continue to need it, so I gave it its own method.
Refactored existing helper methods to use `impl Future` instead of `BoxFuture`, since it was pointed out that this could be a performance issue in lower level helpers. This didn't change any functionality or require refactoring elsewhere, it just involved changing the signatures and removing any calls to `boxify()`.
Reviewed By: StanislavGlebik
Differential Revision: D9009057
fbshipit-source-id: 6b30bb92513213787fc92bdca1507ed117f6e715
Summary: Previously there was no way to get at the underlying `u64` in a Generation object. I want to use this in order to determine how deep into the commit graph I need to lazily index nodes while answering queries. This will help keep the indexing limited to only what is relevant to incoming queries.
Reviewed By: StanislavGlebik
Differential Revision: D9009058
fbshipit-source-id: 1b4ec44b8245ead75f3097048e85b0a1aafe84f6
Summary:
The tests for ReachabilityIndex now accept a closure that creates a trait object, instead of a trait object directly.
I was having problems later in this stack where I couldn't pass a SkiplistIndex object into these tests, because it help Arcs and hence wasnt 'UnwindSafe'. Modifying the tests to take a closure that creates the index object inside of the tokio async unit test fixes this.
Reviewed By: StanislavGlebik
Differential Revision: D9009053
fbshipit-source-id: de5e698b4d27aec1ab47fa7fda73da8ad46aab95
Summary:
Implemented the method lazy_index_node, which takes a starting node and a maximum depth value, and returns a future representing the computation to index all nodes beneath the start, up until previously indexed nodes, or nodes at depth at least the maximum desired depth. This will allow for partial indexing during warmup and during queries, in an environment where indexing the entire set of nodes reachable from the current master will be prohibitively expensive. This also includes a few synchronous helper methods, such as computing the skip list edges assuming a reasonable number of nodes below the desired node have been indexed.
This also includes tests to make sure that the behavior is as expected for things like, confirming the nodes you would expect to be indexed are actually indexed after a lazy_index_node call. The real test will be if this is consistent for reachability queries. It's possible that reachability query tests, and in particular how these methods are used by reachability queries, will expose some bugs. These tests will be introduced when I add the actual reachability query to this index. But I feel confident that the methods in this diff will require at most minimal modification going forward.
Reviewed By: StanislavGlebik
Differential Revision: D8959089
fbshipit-source-id: a025034dfac11215a412114de70a0233d7598f30
Summary: Someone imported `slog-async` so API Server can get rid of the `Mutex`.
Reviewed By: farnz
Differential Revision: D9031672
fbshipit-source-id: 1525707899f29826c363496459b2a9bb246f3e99
Summary:
Make it return manifest hashes. This is not very useful right now, but it will
be useful in the later diffs where we add BonsaiChangeset generation.
Reviewed By: farnz
Differential Revision: D9013976
fbshipit-source-id: 85f0c78bc7b944d300d537dd29cfd66f6f6530c6
Summary:
It's useful to get the hash of the changeset. Looks like we have to clone, but
I'm happy to avoid doing it if that's possible.
Reviewed By: farnz
Differential Revision: D9013977
fbshipit-source-id: bdf06b457fb11c222afddc623c71f3b6b6be30fc
Summary:
Extras value may be non-utf8, for example, hg-git extension generate binary
diff which may be non-utf8.
At the same time it seems that it's better to leave extras key as a utf8
string. However, I'm open to changing it if there is a big pushback
Reviewed By: sunshowers, farnz
Differential Revision: D8991340
fbshipit-source-id: 8f9f33ab3ea77281ae33e0bc735b15201720b761
Summary: This seems to be the most straightforward way to do this.
Reviewed By: farnz
Differential Revision: D9025771
fbshipit-source-id: 241768b8171730dcc674989a0c31ae54bff46528
Summary: This is a workaround for buggy commits produced by older versions of treemanifest.
Reviewed By: farnz
Differential Revision: D9018502
fbshipit-source-id: 71f2aeeaee01500a1a73814dc9c4979ef2c79746
Summary:
RFC3339 is a standard way to represent dates. See https://tools.ietf.org/html/rfc3339
We're going to use this in code to check for commits older than a certain time.
Reviewed By: farnz
Differential Revision: D9017006
fbshipit-source-id: a4e4325f32d84d4be9b160adf428411d014fb62b
Summary: Going to use this in an upcoming diff.
Reviewed By: farnz
Differential Revision: D9017005
fbshipit-source-id: f2dac81c5349a0715fd68aa4584389de35e85b01
Summary: Because get_all_filenodes can go above 1MB (but still less than 3MB) I added a chunking logic to speed up fetching of even the long histories
Reviewed By: farnz
Differential Revision: D9029774
fbshipit-source-id: c2d46a4b49bfebd3c778a0b9c5cf226857f34768
Summary: The blake2_path_hash is used only in MySql right now, but I want to use it in caching as well, so moving it to a more accessible place.
Reviewed By: jsgf
Differential Revision: D9028799
fbshipit-source-id: b055550fbfca5c6c419702490504f08812592470
Summary:
This information is useful to decide if get_all_filenodes is small enough to fit into memcache.
Later it will be used to track how many elements are fitting into memcache nicely.
Reviewed By: jsgf
Differential Revision: D9028395
fbshipit-source-id: ae71486f9e9d60d7b54ce28b8c591c09b8925947
Summary: a bit of cleanup for the upcoming diffs
Reviewed By: farnz
Differential Revision: D9014639
fbshipit-source-id: d03e4969426eced70fd19b91a2c15b607bf52b39
Summary: Those structures will be used in next diffs to store the FilenodeInfo inside memcache for caching purposes
Reviewed By: farnz
Differential Revision: D9014213
fbshipit-source-id: 4952a90415d4b8ab903387fd5cdfaf08d9870c07
Summary: I was surprised these didn't exist, but they work fine.
Reviewed By: Imxset21
Differential Revision: D9016243
fbshipit-source-id: 8405009f536b2e1aa0c9c28be59bb8c1d9ab7a4f
Summary:
I had enough seeing compiler giving me error messages with `blobrepo::Error` whereas it should be `failure::Error`.
`pub type` is exporting a type alias whereas `pub use` is re-exporting. The later apparently will fool the compiler to think that they own the thing they are re-exporting which in this case, failure::Error. If it is for the purpose of declaring the error type the crate is using, type alias is obviously more suitable as it won't take the ownership of the type.
Reviewed By: lukaspiatkowski
Differential Revision: D8926275
fbshipit-source-id: 6b1346adf52bec6b666a3f8b23e7a4ea4e0ab070
Summary: This commit changes MononokeBackingStore to use new APIs provided by Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D8882789
fbshipit-source-id: 0f06ca5f850af9fb52f1d593b9abd715a541488a
Summary: This commit adds support for changeset information retriving at `/<repo>/changeset/<commit hash>`.
Reviewed By: StanislavGlebik
Differential Revision: D8880547
fbshipit-source-id: ed68c577316693e0c685c347405b5d344d1bc87e
Summary: This commit adds support for `/<repo>/tree/<treehash>` (retrieving tree content by hash) .
Reviewed By: StanislavGlebik
Differential Revision: D8870870
fbshipit-source-id: 8b3271c819e47d112a8b44097f626360a05540d1
Summary: This commit implements the `<repo>/blob/<blobhash>` API that Eden needs.
Reviewed By: StanislavGlebik
Differential Revision: D8870300
fbshipit-source-id: eca9dc434c8fb584dfba1542c5242fbee18e6619
Summary: This commit adds support to ls operation that lists files in a folder at some commit.
Reviewed By: StanislavGlebik
Differential Revision: D8729389
fbshipit-source-id: cad6d02da075e94b5269cc18052a5a3916ddac86
Summary: Canaries are better used with ODS counters, so the important metrics (like latency) need to be logged to both scuba and ODS
Reviewed By: jsgf
Differential Revision: D9007402
fbshipit-source-id: 43c057dfdccc5639051d2adee90704de71fc2df7
Summary:
Fix crate names for where the crate name doesn't match the package
name. This affected a few crates, but in practice only rust-crypto/crypto was
used.
Reviewed By: Imxset21
Differential Revision: D9002131
fbshipit-source-id: d9591e4b6da9a00029054785b319a6584958f043
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
Previously incorrect filenodes were recorded. Instead of filenode "dir/file",
"file" was recorded as a path in filenodes table.
The next diff that adds creating file copies in bonsai changesets relies on
correct filenodes being present in the table.
Reviewed By: farnz
Differential Revision: D8915297
fbshipit-source-id: dcbde286d935bc726847f788779cfdc7c888c023
Summary: Makes it easier to understand the progress
Reviewed By: farnz
Differential Revision: D8989178
fbshipit-source-id: dcfebc12cf70871b3353a2cab456486c74829dd8
Summary:
Added support for queries which use bookmark names in place of node hashes. This involved:
* Creating a method `string_to_bookmark_changeset_id`, which takes a string, treats it as a bookmark, and tries to find the corresponding changeset id in the repo.
* Modifying the `is_ancestor call` in `MononokeRepoActor` to try to interpret the query strings as bookmarks if they can't be interpretted as node hashes.
* Introducing the `cloned` crate from `//common/rust` into the API server to make the above methods cleaner.
* Modifying the integration test to add a bookmark to the test repo and attempt querying using the bookmark name.
Reviewed By: fanzeyi
Differential Revision: D8976793
fbshipit-source-id: 3a2b58cac0fb80ee18fad8529bd58af5b54f85ef
Summary:
The test was working fine, however, with the next diff in the stack
it started to fail under stress runs. And in that case I assume the problem in
the test itself.
The test creates two commits, that create file with the same content.
However, one filenode wasn't ancestor of another, and that's semantically
incorrect.
Reviewed By: farnz
Differential Revision: D8913129
fbshipit-source-id: 1b3838c6cf5060d768e71ef056cb7765ebbabc99
Summary:
These types are Hg specific. Since we are going to add bonsai changeset
creation soon, let's make it clear in the types
Reviewed By: farnz
Differential Revision: D8911359
fbshipit-source-id: 8b6cc45122402d7b7e074e66d904d979030de705
Summary: The tests for GenerationBFS only test the ReachabilityIndex interface, so they should be reusable for future indexes. I made them generic and moved them to a separate module, so that they can be reused later on.
Reviewed By: StanislavGlebik
Differential Revision: D8919884
fbshipit-source-id: f5c4668f71e0ee51b72cc1e7e760eda0e0afef4b
Summary: Update actix and actix-web to the latest version
Reviewed By: sunshowers
Differential Revision: D8965698
fbshipit-source-id: 18324161c832ccce4d908799a703368cd615996c
Summary: There was a change in hg replycaps in D8958866, let's update the tests
Reviewed By: farnz
Differential Revision: D8966069
fbshipit-source-id: 72606943d9751b66705b36fbf39ad7dc0702627c
Summary:
The goal of this tool is to verify that bonsai changesets roundtrip with the same hashes. This uses the `ChangesetVisitor` stuff introduced in the previous diff to walk over a specific number of changesets.
I've been using this tool for a few days, and it's shaken out a number of bugs in the Bonsai support in Mononoke, and even found old bugs in Mercurial. So it's been a pretty useful exercise overall.
Reviewed By: farnz
Differential Revision: D8817725
fbshipit-source-id: b6976e8ff9024098c695e82383d98d95b2701e1b
Summary:
Moved two types of functionality to a shared 'helpers' file so that they can be used by other indexes:
* Getting the Generation number of a changeset. The BlobRepo method currently returns an Option<Generation> as the success type, so putting the combinator calls to get the underlying Generation or map to an Error as a separate method will help keep the code more readible, and allow this logic to be reused in other parts.
* Checking if a node exists in the repo. This wraps the changeset_exists method from BlobRepo and returns an error if it was false or an error itself, else success with a void item. This just helps with code readability, so it will be obvious if the result of a future is being used, or if its success is just a prereq for the rest of the operations.
* Convert a collection of HgChangesetId to a collection of (HgNodeHash, Generation). Again, will help with code readability in more complicated functions, since the combinators of this method are, in my opinion, cluttering up the other methods using this functionality and making them more difficult to follow.
Reviewed By: StanislavGlebik
Differential Revision: D8919874
fbshipit-source-id: fc6cdf6e3a1f0dfa73c74ec94f0abac4a7860794
Summary:
This allows parallel walks across thousands of changesets.
In my testing with the Bonsai verification tool this seemed to work well,
keeping my cores saturated almost all the time.
Something to pay attention to is the way cancellation works -- it seems to make
sense, but I could have gotten something subtly wrong :)
Reviewed By: farnz
Differential Revision: D8909719
fbshipit-source-id: f7f47df479431bca05f2cd1508c9c9a4e8a2af50
Summary: Makes structured logging a bit easier.
Reviewed By: farnz
Differential Revision: D8945574
fbshipit-source-id: 12209a99fe33ada1050fda681814b432116f6ca6
Summary: This proved to be pretty helpful while debugging some issues.
Reviewed By: farnz
Differential Revision: D8942186
fbshipit-source-id: c21da97c36c1379048f2b2d572e02a9e0166ac7f
