Summary:
It's used only in a very few places, and most likely that's by accident. We
pass logger via CoreContext now
Reviewed By: krallin
Differential Revision: D16336953
fbshipit-source-id: 36ea4678b3c3df448591c606628b93ff834fae45
Summary: As per request in previous diff, adding stats for requests to phases sql table so it would be easier to debug in the future
Reviewed By: farnz
Differential Revision: D15601643
fbshipit-source-id: a65509ad64d0a0d4948434f771274e4e17d7f133
Summary: Bulk fetch all public heads in `mark_rechable_as_public`, and remove all public heads from traversal, this would avoid initial burst of requests as most of the bookmarks never moves
Reviewed By: krallin
Differential Revision: D15600915
fbshipit-source-id: b6a5faf16d5c5465be53a575c4cc966ed8cdd875
Summary:
This adds support for recording server-side whether a given bookmark is publishing and / or pull-default. This is a change on the way towards supporting Infinite Push in Mononoke. This diff will require schema changes on `xdb.mononoke_production`.
There isn't a whole lot of new user-facing functionality in this particular diff. For starters, nobody can edit this flag on bookmarks, and pushes that create a new bookmark will always result in setting a bookmark as publishing AND pull_default.
What this change does however introduce is the notion of `BookmarkHgKind`, which is represents the behavior of this bookmark as far as Mercurial operations are concerned.
There are 3 such kinds, which are relevant in different parts of the codebase:
- PullDefault - this is useful when we want to respond to listkeys queries.
- Publishing — this is useful when we want to identify commit Phases.
- All - this is useful when we want to respond to listkeyspatterns.
Note that only the first two groups are cached in CachedBookmarks.
---
There are a few things going on in this diff (which logically have to happen together):
- First, I updated the `Bookmarks` trait and its various implementations to expose new methods to select Bookmarks based on their hg kind. There's one method per hg kind, and all the methods use a `Freshness` parameter to determine whether to hit cache or not.
- Second, I refactored the various bookmark-access methods in blobrepo. A few of them were duplicative of each other, and some were unused, which became a bigger problem now that we had more (for publishing, pull default, etc.). We are now down to just one method that doesn't hit cache (which is used only by the blobimport script and some tests — perhaps this could be changed?).
- Third, I updated the call sites for the methods that were udpated in step 2 to use the proper method for their use case.
---
Here's a summary of where each method is used (I'm only listing stuff that isn't unit tests):
- `get_bonsai_heads_maybe_stale`:
- `SqlPhases`'s `get_public`
- `build_skiplist_index`
- `get_bonsai_publishing_bookmarks_most_recent`:
- Blobimport (perhaps we can update this to a `maybe_stale` read?)
- `get_pull_default_bookmarks_maybe_stale`
- `listkeys` implementations
- `get_publishing_bookmarks_maybe_stale`
- API Server's `get_branches`
- `get_bookmarks_by_prefix_maybe_stale`:
- `listkeyspatterns` implementation
---
As an aside, note that a lot of the code changes in this diff are actually in CacheBookmark's tests — I had to update those to minimize sleeps: it was fine to sleep in the earlier tests, but I introduced new quickcheck-based tests, and sleeping in those isn't ideal.
Reviewed By: StanislavGlebik
Differential Revision: D15298598
fbshipit-source-id: 4f0fe80ea98dea8af0c8289db4f6f4e4070af622
Summary:
As part of adding support for infinitepush in Mononoke, we'll include additional server-side metadata on Bookmarks (specifically, whether they are publishing and pull-default).
However, we do use the name `Bookmark` right now to just reference a Bookmark name. This patch updates all reference to `Bookmark` to `BookmarkName` in order to free up `Bookmark`.
Reviewed By: StanislavGlebik
Differential Revision: D15364674
fbshipit-source-id: 126142e24e4361c19d1a6e20daa28bc793fb8686
Summary:
`Phases` currently have very ugly API, which is constant source of confusion. I've made following changes
- only return/cache public phases
- do not require `public_heads` and always request them from `BlobRepo::get_bonsai_heads_maybe_stale`
- consolidated `HintPhases` and `SqlPhases`
- removed `Hint` from types which does not carry any meaning
- fixed all effected callsites
Reviewed By: StanislavGlebik
Differential Revision: D15344092
fbshipit-source-id: 848245a58a4e34e481706dbcea23450f3c43810b
Summary: Instead of calculating phases of each request, this diff makes sure we are keeping all public phases up to date in database
Reviewed By: StanislavGlebik
Differential Revision: D15045340
fbshipit-source-id: 1ee95eabce4ff517925d5d2b2705e26e68474d92
Summary: Before this change we were only marking head of the pushrebase, this change fixes this problem and marks all reachable changesets as public
Reviewed By: StanislavGlebik
Differential Revision: D15063835
fbshipit-source-id: 2a360684fc01cec0f639c1789eff8150e5ba5ebb
Summary: - convert phases crate to rust-2018, in preparation for the next diff in a stack
Reviewed By: StanislavGlebik
Differential Revision: D14995610
fbshipit-source-id: 30327508cee5154a8a788e8c5a2f914f7ccacfec
Summary:
Add a LABEL constant to the SqlConstructors trait to make it easier to identify
which table is being used, for stats and logging.
Reviewed By: HarveyHunt
Differential Revision: D13457488
fbshipit-source-id: a061a9582bc1783604f249d5b7dcede4b1e1d3c5
Summary:
Using `lca_hint` is more efficient then LeastCommonAncestorsHint because
`lca_hint` can check if a node is ancestor of any of the bookmarks.
Reviewed By: aslpavel
Differential Revision: D14684418
fbshipit-source-id: 0de14832996f90697116da9fb2f478448eb21b46
Summary:
Together with logging bookmark moves, let's also log bundle handle. It will be
used during replay from Mononoke to mercurial.
Reviewed By: ikostia
Differential Revision: D13990929
fbshipit-source-id: 4039322903b13e84fb31c8e65cc2e097ca765213
Summary:
This is the first step in adding support for tracking all bookmark moves. They
will be recorded in the separate mysql table in the same transaction as
bookmark is updated.
That gives us two things:
1) Ability to inspect all bookmark moves and debug issues with them
2) Also record which mercurial bundle moved a bookmark if any so that we could
later replay these bundles in correct order on hg
Add a struct that let us track bookmark moves.
Reviewed By: ikostia
Differential Revision: D13958872
fbshipit-source-id: 9adfee6d977457db5af4ad5d3a6734c73fcbcd76
Summary:
These are **not** the schemas that we use in production.
So at the moment they are not used for anything and they just create confusion.
Reviewed By: aslpavel
Differential Revision: D13986001
fbshipit-source-id: 7aae0a5da474f579c9cdf1bbf5dfe183835cae2d
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
`blobrepo_factory` is a crate that knows how to create blobrepo given
a configuration i.e. it creates blobstores, filenodes, changesets etc and
initializes blobrepo with them.
`post_commit` is a small part of blobrepo which can also be extracted from
blobrepo crate.
There are a few upsides with this approach
1) Less dependencies on Blobrepo, meaning we have to rebuild it fewer times
2) BlobRepo compilation is faster
Reviewed By: jsgf
Differential Revision: D13896334
fbshipit-source-id: 1f5701341f01fcefff4e5f9430ddc914b9496064
Summary:
Let's split reachability index crate. The main goal is to reduce compilation
time. Now crates like revsets will only depend on traits definition but not on
the actual implementation (skiplist of genbfs).
Reviewed By: lukaspiatkowski
Differential Revision: D13878403
fbshipit-source-id: 022eca50ac4bc7416e9fe5f3104f0a9a65195b26
Summary:
Some crates, namely revsets and reachabilityindex, currently depend on
blobrepo, while all they need is the ability to fetch commits.
By moving changeset_fetcher outside this dependency will be removed. That may
make builds faster
Reviewed By: lukaspiatkowski
Differential Revision: D13878369
fbshipit-source-id: 9ee8973a9170557a4dede5404dd374aa4a000405
Summary:
This will reduce max potential number of fetching bookmarks from O(len(longest stack)) to O(1) (with is 0 or 1) if there are draft commits.
Previously, the refetching could happen on every iteration while we are
searching for public roots.
This diff allows us to reuse the bookmarks from the first time they were needed
and has been fetched in the whole phases calculation process. Fetch just once!
If there are no draft commits bookmarks were fetched only once already (and only if needed).
Reviewed By: StanislavGlebik
Differential Revision: D13753520
fbshipit-source-id: d96a6cf434cb4a1fe95e51ae734afb1671124336
Summary:
this is required to cover corner cases when client has some stacks and part of those became public
calculation for public roots happen for draft heads only, it doesn't change performance of hg pull
Reviewed By: StanislavGlebik
Differential Revision: D13742685
fbshipit-source-id: d8c8bc357628b9b513bbfad4a82a7220d143f364
Summary:
The bulk api makes less queries to mysql and therefore is more efficient.
This is especially important for `hg pull` requests where the list of heads is very large.
Reviewed By: lukaspiatkowski
Differential Revision: D13677298
fbshipit-source-id: 3dec1b3462c520c11481325e82523ef7a6ae6516
Summary:
the remaining part to implement bulk phases fetch and update
this is required to optimize number of MySql queries we are using to look up at the db for phases and bookmarks.
the single get api has been removed and reimplemented by calling bulk ones
Reviewed By: aslpavel
Differential Revision: D13664900
fbshipit-source-id: 29342e86c057b92e331fadcebe51f452d9569e09
Summary:
this is required to optimize number of MySql queries we are using to look up at the db for phases and bookmarks.
the next step is to add the same with Memcache to caching.rs
implementations for the single get is replaced with just calling the implementation for multiple get
Reviewed By: aslpavel
Differential Revision: D13658610
fbshipit-source-id: e3876044e2cbbefb156175c51ab7051db3885eb8
Summary:
use the correct skip index
sorry for some rustfmt.
Reviewed By: StanislavGlebik
Differential Revision: D13636059
fbshipit-source-id: 2815d82b63b86bda053f5a3a9a1b8a3b72abbf82
Summary:
We have decided that this will be used to transport phases to the client
Hg client already supports this part.
Reviewed By: StanislavGlebik
Differential Revision: D13507921
fbshipit-source-id: 621e93bb6e1a0c87d4f4963ba7fa635b77a5b6ec
Summary:
Basically if any of the bookmarks is reachable, all other bookmarks are not
interesting to check, so those futures should be skipped.
Reviewed By: StanislavGlebik
Differential Revision: D13538511
fbshipit-source-id: 193a7ea8d505690aeb96247a07c8f2688cd7a59f
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary:
New HintPhases api that will be used for tests vs CachingHintPhases that will be used in prod.
The api will be owned by RepoClient.
We will check what type of BlobRepo we have (i.e. blob:rocks, blob:files, blob:remote etc) and build the phases depending on the type of the BlobRepo.
We will use either CachingHintPhases backed by real db / MyRouter or HintPhases backed by Sqlite.
Reviewed By: StanislavGlebik
Differential Revision: D13466225
fbshipit-source-id: 06ea565171d8ea8d7335fbbd91d86fbdcc01c8fc
Summary: Calculation is based on beeing ancestor of a public bookmark
Reviewed By: StanislavGlebik
Differential Revision: D13441622
fbshipit-source-id: e20df656847913bc124b491aaeb2660d21c85da1
Summary:
This diff includes the logic on how we will receive a phase for a given commit based on memcahe / db lookup and refresh, and slow path calculation.
It has a blank place of slow path (not found in the memcache, not found in the db => we have to calculate the phase based on being ancestor of public bookmark).
Collecting the stats should be added separately.
Reviewed By: StanislavGlebik
Differential Revision: D13415481
fbshipit-source-id: 6a4cb5b8dfbb0d7b2535d903c653bbf7a088c422
Summary: add/get api for adding phases to mysql phases table
Reviewed By: lukaspiatkowski
Differential Revision: D13376701
fbshipit-source-id: b71e52db2c30b59b0070f49327bfdd189c28d6cc
