Summary:
Saw a few timeout errors during infinitepush test.
Previous timeout was 10 secs, it makes sense to increase it.
There was a unit test failures because of the config options.
So this diff also adds missing config docs and copyrights headers.
Test Plan: Build the package, install it locally and run integrationtests
Reviewers: durham, rmcelroy, mitrandir, andrasbelo
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4103689
Tasks: 12479677
Summary:
Add an option to list remote and scratch bookmarks by pattern.
Only very simple pattern is allowed: either the literal string
or a prefix (like 'scratch/stash/*'). It was made intentionally
to make sure that pattern requests are fast in mysql.
Mysql tests will be added to the integration tests
Test Plan: Run `test-infinitepush-*`
Reviewers: mitrandir, andrasbelo, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4074409
Tasks: 12479677
Signature: t1:4074409:1477500968:e91cd2505d61a2f1db30c7f00cdcfc949e433507
Summary:
Tests were broken because there were internal API changes in
upstream mercurial:
1) `hg book` outputs nothing instead of `no bookmarks set`
2) `getbundle` function was removed, new `getbundlechunks` function was added
Test Plan: Run `test-infinitepush-*` and run integration tests
Reviewers: durham, rmcelroy, mitrandir, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4067223
Tasks: 14035546
Signature: t1:4067223:1477305056:f705764ff3cdd70fcc61e677ccb184870077ec86
Summary:
User should see no difference between scratch and non-scratch remote bookmarks.
So let's make scratch bookmarks remote if remotenames is enabled.
Unfortunately it requires saving scratch remote bookmarks before push/pull and
restoring it afterwards because `remotenames` extension doesn't know about
them.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4036653
Tasks: 12479677
Signature: t1:4036653:1476985129:b5c1afdab4c7585c8b12c2962a4b95a56751091b
Summary:
If param is set to '0', then `params.get(PARAMNAME)` is True.
This is not what's expected and can lead to subtle bugs in future.
Instead do not send anything if param is not set.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4036504
Tasks: 12479677
Signature: t1:4036504:1476797842:112bae5c318922795c8be5bb5e65458fe9cf034e
Summary:
If `branchpattern` is not set then it matches empty bookmark.
It causes regressions like this t13808497. Let's ignore empty `branchpattern`
Test Plan: Run `test-infinitepush-*`
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4031365
Tasks: 12479677
Signature: t1:4031365:1476755541:3d3781ef92f1701500d74ce7b98b4d82b5a1228e
Summary:
Remotefilelog can't find file revisions outside of the repo.
Because of it user can't update to the revisions that was pulled from
bundlestore. This diff downloads all filelogs for revisions
from bundlestore.
There were two options to pass info about changed files to remotefilelog:
through config or through bundlecaps. I decided to use config because
it's not really a bundle capability.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4030065
Tasks: 12479677
Signature: t1:4030065:1476755932:4aa02e540f13d3221163d906524fb9ce97f9c7e8
Summary:
Tests doesn't require mysql since it uses fileindexapi.
Let's move sqlindexapi and fileindexapi to different files and include
only necessary file.
Test Plan: Run test-infinitepush-*
Reviewers: durham, simonfar, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4029460
Tasks: 12479677
Signature: t1:4029460:1476709161:556f80605db9bd3c18dce1e16e996b13bbe96229
Summary:
Previously `--to` bookmark was not set locally.
This diff fixes it by setting this bookmark to the head after push happened.
This is correct because it's guaranteed that we'll push only one head.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001168
Tasks: 12479677
Summary: It will be used in a few places (see next diff in stack)
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001105
Tasks: 12479677
Signature: t1:4001105:1476330730:0ba513a3bab131b8eadadd4e90576880663f0d49
Summary:
Let's use a function that saves bookmarks, nodes and bundle
in the same time. For fileindexapi this call is equivalent
to addbundle() and addbookmarks() call. For sqlindexapi this
call makes two calls in one transaction.
Test Plan:
1) Run `test-infinitepush-bundlestore.t`
2) Test sql index locally
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001243
Tasks: 12479677
Signature: t1:4001243:1476227921:9d449b743686762edce4c07d95266ef7ae69c2b5
Summary:
Non-fast-forward pushes were disallowed. But having a
mutable shared history for scratch branches is a very
nice feature. Let's use --force option to force
non-fast forward push.
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: durham, rmcelroy, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3994813
Tasks: 12479677
Summary:
Previously error was thrown if mysql.connector wasn't present on client hosts.
But there is no need for mysql.connector to be installed on clients.
Remove global imports to fix it and create bundlestore() only for server repo.
P. S.
Changes in test is required because bundlestore() is not created if
infinitepush.server=True is not present in config.
Test Plan:
Uninstall mysql.connector: `sudo yum remove mysql-connector-python`
Enable infinitepush extension in any repo and run any hg command (hg st,
for example). Make sure it doesn't throw exceptions.
Install mysql.connector: `sudo yum install mysql-connector-python`.
Run `test-infinitepush-bundlestore.t`
Reviewers: durham, mitrandir, quark, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3967189
Tasks: 12479677
Signature: t1:3967189:1475581802:fbaafceac298242d0a449267ec0d68e290c3f262
Summary:
Hack to fix interaction with remotenames. Remotenames push '--to' bookmark to the server but we don't want to push scratch bookmark to the server. Let's delete '--to' and '--create' and also set allow_anon to True (because if --to is not set remotenames will think that we are pushing anonymoush head).
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3937202
Tasks: 12479677
Signature: t1:3937202:1475170335:fa4d39e1e17bea67b62b99dfc4cd2ffa4e5d6bea
Summary:
It's possible to push new scratch bookmark which points to existing scratch
commit. Previously it has failed with a stack trace.
Unfortunately it still doesn't allow to push scratch bookmark that points
to non-scratch commit. It requires more complex changes so
it will be fixed later if necessary.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3936728
Tasks: 12479677
Signature: t1:3936728:1475078059:ac4588e086b0344070a90f806b6f518c6e187c3b
Summary:
infinitepush bundle2 part should be run before pushrebase bundle2 part. This is a bit tricky since both pushrebase and infinitepush change the `extensions._order` and both of them try to load before bundle2 `changeset` part.
I suggest to load before 'changeset' or pushrebase part whichever is earlier. Then we have two situations:
1) pushrebase comes first. It sets it's part before 'changeset' part, then infinitepush sets it's part before pushrebase part.
2) infinitepush comes first. It sets part before 'changeset' and pushrebase part, then pushrebase sets it's part before 'changeset' part. Since infinitepush part comes before 'changeset' part it still be before pushrebase part.
This solution is not robust but it should be fine for now. The proper solution requires changes upstream that will introduce dependencies between bundle2 parts.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3930697
Tasks: 12479677
Signature: t1:3930697:1475079064:1b5851df4d2067ec21ec01e7510ddf83ff1684ad
Summary:
--bundle-store option makes push go to bundle store
even if bookmark is not specified or bookmark doesn't match scratch branch pattern.
The main reason for the `--bundle-store` is to be able to push not a scratch bookmark, but a scratch commit, for example, `hg push -r . --bundle-store`.
It will be useful when we will save all the local commits in the bundlestore.
To do this:
1) Wrap `discovery.checkheads()` to allow push to scratch branch even if new head is created.
2) Make scratchbranchpat params advisory because now bookmark may not be specified.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3862665
Tasks: 12479677
Signature: t1:3862665:1475077612:7e02a5bdc1da28161e45029b1e3a38fe8223a305
Summary:
We are going to add `push` option to force commits to go to bundle store.
This is a preliminary refactoring. Add another experimental config param that sets whether this is a scratch or usual push
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3862604
Tasks: 12479677
Signature: t1:3862604:1473934819:3ead319a885d99dd9a215941b317540121739b32
Summary:
There is no point in disallowing it.
The next step is to allow to fetch arbitrary many scratch bookmarks with one command.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3835502
Tasks: 12479677
Summary:
scratchbookmarks are not automatically pulled (because they are not showed in
listkeys). Let's record them after the actual pull. It introduces a race
condition. But this race condition is unlikely and shouldn't be harmful.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: mjpieters, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3835414
Tasks: 12479677
Signature: t1:3835414:1473703998:554c890be50f732f34b2793160ee1abb26561e39
Summary:
Diff introduces `listkeyspatterns` wireproto method. In future it will be able to filter keys by pattern, but for now it has limited functionality:
1) it is able to find scratch bookmarks only by it's full name (no patterns)
2) and it doesn't filters non-scratch bookmarks.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797800
Tasks: 12479677
Signature: t1:3797800:1473440073:61db6803235c53afbb3cdc72084aa35ce7d0ee63
Summary:
Infinitepush index implementation that uses mysql db to store data.
Three tables:
1) Node to bundle
2) Bookmark to node
3) All bundles (this is necessary to
find unused bundles.
Test Plan: Code was tested but tests are not included in the diff - see comments below
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3811247
Tasks: 12479677
Signature: t1:3811247:1473378520:bc23b3436aa2e618a7bfefd2d571257dfe8ace94
Summary:
Use external bundles to store bundles.
Put and get binaries should be provided to upload/download bundles to/from external store (see `put_binary` and `get_binary` config options). Put binary should output a handle, that can be used with `get_binary` to retrieve bundle. `put_args` and `get_args` config options can be used to pass additional params to the binaries.
Test Plan:
Change `test-infinitepush-bundlestore.t` test file to point to external store and run it
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803936
Tasks: 12479677
Signature: t1:3803936:1473123987:c34987ee459b2a66ee831574a32cdc8231043cb2
Test Plan: No need to test - comment change
Reviewers: rmcelroy, durham, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3791555
Tasks: 12479677
Summary: --create and --to options work with non-scratch bookmarks too
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, durham, rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797183
Tasks: 12479677
Summary: It's safer to don't have bundlestore then to have uninitialized bundlestore
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, rmcelroy, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797175
Tasks: 12479677
Summary:
Copy of hackbranch hackathon project with a few changes.
The goal is to allow saving commits under certain bookmarks in the separate store. These bookmarks are called 'scratch' bookmarks and are defined in config file (for example, 're:scratch/.+').
Scratch commits are sent in a separate bundle2 part during push. During pull new bundle repo is created and requests are served using this bundle repo.
There are a few changes from hackathon project implementation.
# It requires no changes in mercurial core. It was done by wrapping `findcommonincoming()` function and changing `lookup()` wireproto method (see below).
# Instead of introducing `listkeypattern()` wireproto method, `lookup()` method and `pull()` command were wrapped. I did it because I thought that it will make code easier. It probably haven't made it easier, it's almost the same. I can change it back to `listkeypattern` if it's necessary.
The bad thing about current implementation is that scratch bookmarks are not downloaded. But in the previous implementation also haven't downloaded bookmark if remotenames is enabled.
There are a couple of problems:
1) As I mentioned before, scratch bookmarks are not downloaded
2) It's impossible to pull both scratch and non-scratch bookmark
Test Plan: python ../../../hg-crew/tests/run-tests.py test-bundlestore.t
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3750999
Tasks: 12479677
Signature: t1:3750999:1472052396:5d7828566863f0aa437cdc34439ec0759503c6c9
Summary:
Code was written by Laurent, Durham and Mateusz on hackathon.
Index stores metadata, store stores bundles.
This diff adds interfaces and simple filesystem implementations
Test Plan:
cd tests
py.test *.py
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3741963
Tasks: 12479677
Signature: t1:3741963:1472207457:c9e7fb6163ef3cd6fdc1c4bd5113c623169c3a7c
