Summary: Implement a backing store that uses thrift protocol to communicate with Mononoke API Server.
Reviewed By: strager
Differential Revision: D13966575
fbshipit-source-id: 66f66dda6b17aecd6c6b4475ab6b004c608f457f
Summary:
Removed references to HgNodeHash in repo_client in the specified functions. In
addition, updated other files due to type related dependencies.
Reviewed By: StanislavGlebik
Differential Revision: D14543934
fbshipit-source-id: b0d860fe7085ed4b91a62ab1e27fb2907a642a1d
Summary: Add some logging to the API server during repo intiialization, since startup of the API server in non-opt modes takes a long time.
Reviewed By: quark-zju
Differential Revision: D14605643
fbshipit-source-id: 4b3c46f16cbeb3bf571e553eadc4cb002ce044d9
Summary:
The main fix is in speeding up sql query that returns entries to heal.
The sql query was slow in the case when there are a lot of entries for one
repo and few entries for another repo. Selecting entries for smaller repo can
become too slow because mysql has to scan the whole table in order to sort
entries. Since ordering by id doesn't look necessary I suggest to just remove
them.
Also waiting for 1 minute between heal attemps is too slow
There are a few more smaller fixes - replacing join_all with more efficient
futures_unordered and doing batch delete of entries from the sync queue
Reviewed By: aslpavel
Differential Revision: D14598578
fbshipit-source-id: e8d302aab7b5a4bc16c63e14228713b75295e97a
Summary: Slim down the blobstore trait crate as much as possible.
Reviewed By: aslpavel
Differential Revision: D14542675
fbshipit-source-id: faf09255f7fe2236a491742cd836226474f5967c
Summary:
- This type of pushrebase should not succeed but it should not timeout either, as it does now
```
$ hg push -r. --to master
o < -master (public)
|
o @ <- commit we are trying to rebase
| |
: o <- master2
|/
o
```
- this code uses `get_changeset_parents_by_bonsai` for traversal which makes it much faster, and it is correctly fails even if we are running mononoke in lla region
- I've also added additional logging to scuba to indicate begging of pushrebase, which will simplify debugging in the future
Reviewed By: StanislavGlebik
Differential Revision: D14598576
fbshipit-source-id: 25e792996aa08ca977678bd61ffe8cb51f386bcf
Summary:
This is the test to cover tricky case in the discovery logic.
Previously Mononoke's known() wireproto method returned `true` for both public
and draft commits. The problem was in that it affects pushrebase.
There are a few problems with the current setup. A push command like `hg push
-r HASH --to BOOK` may actually do two things - it can either move a bookmark
on the server or do a pushrebase. What it does depends on how discovery phase
of the push finishes.
Each `hg push` starts with a discovery algorithm that tries to figure out what commits
to send to the server. If client decides that server already has all the
commits then it'll just move the bookmark, otherwise it'll run the pushrebase.
During discovery client sends wireproto `known()` method to the server
with a list of commit hashes, and server returns a list of booleans telling if
a server knows the commit or not. Before this diff Mononoke returned true for
both draft commits and public commits, while Mercurial returned true it only
for public commits.
So if Mononoke already has a draft commit (it might have it because the commit was created
via `hg pushbackup` or was created in the previous unsuccessful push attempt),
then hg client discovery will decide to move a bookmark instead of
pushrebasing, which in the case of master bookmark might have disastrous
consequences.
To fix it let's return false for draft commits, and also implement `knownnodes` which return true for draft commits (a better name for these methods would be `knownpublic` and `known`).
Note though that in order to trigger the problem the position of the bookmark on the server
should be different from the position of the bookmark on the client. This is
because of short-circuting in the hg client discovery logic (see
https://fburl.com/s5r76yle).
The potential downside of the change is that we'll fetch bookmarks more often,
but we'll add bookmark cache later if necessary.
Reviewed By: ikostia
Differential Revision: D14560355
fbshipit-source-id: b943714199576e14a32e87f325ae8059d95cb8ed
Summary: To verify that BookmarkCache is actually needed we want to collect data on how bookmarks are queried
Reviewed By: StanislavGlebik
Differential Revision: D14560748
fbshipit-source-id: ce08511b98c3566cc6ed9052180d73f6076c68fe
Summary:
Is_ancestor was inexplicably slow so I added a few perf counters to it. The slowness should be fixed with D14540318.
This will only add the most important perf counters as I have seen some slow down when adding more.
I also changed the param naming from maybe_ancestor to ancestor. The maybe is implied through the method name being is_ancestor. This change also brings skiplist in line with apiserver
What will we log:
Ancestor/Descendant Generation
skip/noskip iterations
skipped generations
Reviewed By: StanislavGlebik
Differential Revision: D14085659
fbshipit-source-id: e0c7d34ed218e94c2c5a39f98d3c0b3d4265214f
Summary:
Pointing the configs to a new package allows us to control the rollout more easily than overriding the existing skiplist in manifold.
This skiplist features a single skip entry for each node with a depth of up to 512 commits. Previously depth was 32k. This means on average we will have to do 16k iterations to find the commit we want after doing the last skiplist jump.
If we were to jump to the very beginning of the repo that would mean before we had to do ~130 skips and then on average 16k steps. With this skiplist we would do ~8k skips and 256 steps. As skips and steps are equally expensive this should be faster in almost every scenario. The total size of the skiplist is not affected by this change (as each node still contains one skip entry) but the new skiplist also covers the complete history, not only the 2kk most recent commits. This leads to size increase of ~20%.
Reviewed By: StanislavGlebik, HarveyHunt
Differential Revision: D14540318
fbshipit-source-id: 86fda5b6c57a06bc4a77e30625014ec119e7a155
Summary: Should be in sync with D14424208
Reviewed By: markbt
Differential Revision: D14541101
fbshipit-source-id: 2e1d544081cd7dd336a76d0490ee91e9137cef55
Summary:
- healer runs on all repositories at once, and queries for some repositories are timing out
- It is now possible to run healer just for specified repository
Reviewed By: HarveyHunt
Differential Revision: D14539978
fbshipit-source-id: 9139999da97b2655ae9312c33c9e8c15f0b24016
Summary:
Basically we should check that the commits have been backed up.
If this is not true and the commits are local we can just back them up.
If they are not known by this repo, pull from the old one and then back them
up.
Reviewed By: markbt
Differential Revision: D14508239
fbshipit-source-id: 3fdd83335cb937b153510ec3c7510ecd1167d0ca
Summary:
As part of the mononoke lock testing, we realised that it would
be helpful to see why a repo is locked. Add the ability to express this
to the RepoReadOnly enum entry.
Reviewed By: aslpavel
Differential Revision: D14544801
fbshipit-source-id: ecae1460f8e6f0a8a31f4e6de3bd99c87fba8944
Summary:
- convert to 2018 edition, and removed all `extern crate`
- wait for `myrouter` to be available before actually doing anything
Reviewed By: HarveyHunt
Differential Revision: D14524247
fbshipit-source-id: ebe2e2e74935f00c87945129370f268c794fcab7
Summary: All of the API server handlers were unnecessarily cloning URL parameters. This diff eliminates these clones.
Reviewed By: singhsrb
Differential Revision: D14537692
fbshipit-source-id: 02ea9dccae02b2813d04dd95feb8225e35870b6c
Summary:
This is particulary useful for `hg cloud sync` when it calls pushbackup in the
background to the secondary storage at the end of cloud sync.
Pushbackup is not smart enough, so it will back up again what we just pulled from cloud sync.
However, all those commits are probably already backed up on the secondary
storage on another machine while cloud sync.
Reviewed By: singhsrb
Differential Revision: D14386616
fbshipit-source-id: e62ed0afb89c28fe6880346077c279e6705da602
Summary:
Previously it was counting all stream entries, and for each file we have more
than one entry. This diff fixes it
Reviewed By: aslpavel
Differential Revision: D14519710
fbshipit-source-id: faf31f92933d63c3d4015efdc71eabb6c21888d7
Summary: To verify that slow downloads are caused by the client connection we log the total amount of time spent downloading files from Manifold.
Reviewed By: HarveyHunt
Differential Revision: D14502779
fbshipit-source-id: 9d6e1fa18faa4689680ed39087aefd418ac2bf62
Summary: `LooseHistoryEntry` and `PackHistoryEntry` aren't the best names for these types, since the latter is what most users should use, whereas the former should only typically used for data transmission. As such, we should rename these to clarify the intent.
Differential Revision: D14512749
fbshipit-source-id: 5293df89766825077b2ba07224297b958bf46002
Summary: Let's validate the content we return to users in the same way we do it for getfiles
Reviewed By: farnz
Differential Revision: D14420148
fbshipit-source-id: e109f6586210858e26334c1547d374c1c9b9e441
Summary: In Mononoke we want to be able to block merge commits on a repo per repo basis.
Reviewed By: aslpavel
Differential Revision: D14455502
fbshipit-source-id: 400e85834c20df811674405bc0c391860cf677dd
Summary:
Allow using a database entry to determine if a repository
is in read-only mode.
If a repo is marked read/write in the config then we will communicate with the db.
Reviewed By: ikostia
Differential Revision: D14279170
fbshipit-source-id: 57abd597f939e57f274079011d2fdcc7e7037854
Summary: Bookmarks that are set to be non fastforward moveable should also not be deleteable
Reviewed By: StanislavGlebik
Differential Revision: D14420457
fbshipit-source-id: a10231466350c0b25437972c66472b46044fc625
Summary:
This is a hook in mercurial, in Mononoke it will be part of the implementation. By default all non fastforward pushes are blocked, except when using the NON_FAST_FORWARD pushvar (--non-forward-move is also needed to circumvent client side restrictions). Additionally certain bookmarks (e.g. master) shouldn't be able to be moved in a non fastforward manner at all. This can be done by setting block_non_fast_forward field in config.
Pushrebase can only move the bookmark that is actually being pushrebased so we do not need to check whether it is a fastforward move (it always is)
Reviewed By: StanislavGlebik
Differential Revision: D14405696
fbshipit-source-id: 782b49c26a753918418e02c06dcfab76e3394dc1
Summary: There is no need to have an Option<Vec<XYZ>> as None can simply be represented by an empty vector. This makes these fields easier to use.
Reviewed By: StanislavGlebik
Differential Revision: D14405687
fbshipit-source-id: e4c5ba12a1e3c6a18130026af6814d54952da4d2
Summary: This is needed to run the scmadmin tool and lock the repo.
Reviewed By: StanislavGlebik
Differential Revision: D14497608
fbshipit-source-id: 5865b90375db29a17d462044ca4cdb87242a8209
Summary: To learn how far behind are we in the absolute bundle numbers.
Reviewed By: StanislavGlebik
Differential Revision: D14491672
fbshipit-source-id: 31d16f115b2b6fe4b88c25a847ce229e123b048b
Summary: We want to be able to manipulate hg-sync counters from the Mononoke admin.
Reviewed By: StanislavGlebik
Differential Revision: D14477676
fbshipit-source-id: 11218390bf469d4f297f7f13e9daee2d5f9bb35b
Summary:
Before adding hash validation to getpackv1 let's do this refactoring to make it
easier.
This diff also make hash validation more reliable. Previously we were
refetching the same file content again during validation instead of verifying the actual content
that was sent to the client. Since the content was in cache it was fine, but
it's better to check the same content that's sent to the client.
This diff also adds a integration test
Reviewed By: jsgf
Differential Revision: D14407292
fbshipit-source-id: b0667cb3dd6a7e0cee0b02cf87a61d43926d6058
Summary: Let's log it to scuba and to scribe just as we do with getfiles requests.
Reviewed By: jsgf
Differential Revision: D14404236
fbshipit-source-id: 079140372c128ee30e152c5626ef8f1127da36b1
Summary:
The diff adds support for getpackv1 request. Supporting this request is
necessary because hg client will stop using getfiles soon.
There are two main differences between getfiles and getpackv1. getfiles returns
loose files formats, while getpackv1 returns wirepack pack file (same format as
used by gettreepack). getpackv1 supports fetching more than one filenode per
file.
Differential Revision: D14404234
fbshipit-source-id: cfaef6a2ebb76da2df68c05e838d89690f56a9f0
Summary: The order was changed after ec00921aece0adc6aaca49e5580bff52784c4ca5
Reviewed By: quark-zju
Differential Revision: D14482440
fbshipit-source-id: d13327ed16387e597ca6fc1cfab0787a13f38d00
Summary:
We've see failures during the attempts to deploy this job to TW without such
waits. After adding the wait, things look better.
Reviewed By: StanislavGlebik
Differential Revision: D14456207
fbshipit-source-id: 8469932d7387060c026164ac97fa604453b0d296
Summary: Every packman-managed RPM in fbcode should now have a `packager: ONCALL` line in its packman.yml file. This diff adds one for fb-mononoke-admin. Assuming I found the right oncall, would you Accept+Ship this change?
Reviewed By: sunshowers
Differential Revision: D14399325
fbshipit-source-id: 5dd03b3f8722b4fb13b85acbe7702bc780b49f76
Summary: Increase timeout in line with observed operations.
Differential Revision: D14421000
fbshipit-source-id: 68941a5188e41c6dd7fbb3b59af0a912327f76a4
Summary: See D14279065, this diff is simply to clean up the deprecated code
Reviewed By: StanislavGlebik
Differential Revision: D14279210
fbshipit-source-id: 10801fb04ad533a80bb7a2f9dcdf3ee5906aa68d
Summary:
Currently we are logging new commits from BlobRepo. This will lead to issues once CommitCloud starts using Mononoke as we cannot differentiate between phases at that level. The solution is to log commits when they are pushrebased as this guarantees that they are public commits.
Note: This only introduces the new logic, cleaning up the existing implementation is part of D14279210
Reviewed By: StanislavGlebik
Differential Revision: D14279065
fbshipit-source-id: d714fae7164a8af815fc7716379ff0b7eb4826fb
Summary:
Fixes the below type error when pulling from a git repo using hg-git on my laptop:
File "edenscm/hgext/remotenames.py", line 225, in expull
pullremotenames(repo, remote, bookmarks)
File "edenscm/hgext/remotenames.py", line 314, in pullremotenames
path = activepath(repo.ui, remote)
File "edenscm/hgext/remotenames.py", line 1464, in activepath
rpath = _normalizeremote(remote.url)
File "edenscm/hgext/remotenames.py", line 1439, in _normalizeremote
u = util.url(remote)
File "edenscm/hgext/hggit/__init__.py", line 164, in _url
if not (path.startswith(pycompat.ossep) and ":" in path):
AttributeError: 'function' object has no attribute 'startswith'
Basically, `peer.url()` is the API, `peer._url` is a private field that does
not always exist somehow.
Besides, further remove named branches that can crash hg-git with
NotImplementedError:
File "edenscm/hgext/remotenames.py", line 225, in expull
pullremotenames(repo, remote, bookmarks)
File "edenscm/hgext/remotenames.py", line 322, in pullremotenames
for branch, nodes in remote.branchmap().iteritems():
File "edenscm/hgext/hggit/gitrepo.py", line 73, in branchmap
raise NotImplementedError
Reviewed By: DurhamG
Differential Revision: D14144462
fbshipit-source-id: 2e886c639cf6689480f84626eaf0d5ec25512ea0
Summary:
Apparently Gluster really can't copy with renames, so write to a
unique name, then symlink the canonical key to it. If the symlink already
exists, then we'll assume that the file already does, and remove the unique
name.
Reviewed By: StanislavGlebik
Differential Revision: D14014167
fbshipit-source-id: 1e5e2ce989652232d67d2aaac776e35127f58fb0
Summary: Keep track of error causes so we can print better errors.
Reviewed By: StanislavGlebik
Differential Revision: D14014165
fbshipit-source-id: e9a7846256bbfbfd689e0d78f01b1ac50cf64c1b
