Summary: It will be used in a few places (see next diff in stack)
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001105
Tasks: 12479677
Signature: t1:4001105:1476330730:0ba513a3bab131b8eadadd4e90576880663f0d49
Summary:
Let's use a function that saves bookmarks, nodes and bundle
in the same time. For fileindexapi this call is equivalent
to addbundle() and addbookmarks() call. For sqlindexapi this
call makes two calls in one transaction.
Test Plan:
1) Run `test-infinitepush-bundlestore.t`
2) Test sql index locally
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001243
Tasks: 12479677
Signature: t1:4001243:1476227921:9d449b743686762edce4c07d95266ef7ae69c2b5
Summary: This check was missing before
Test Plan: It's difficult to test and the change should be safe
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001229
Tasks: 12479677
Signature: t1:4001229:1476227703:9e40d6efa9892e1e0d7fbcb24fdce3d607a3a7fe
Summary:
Non-fast-forward pushes were disallowed. But having a
mutable shared history for scratch branches is a very
nice feature. Let's use --force option to force
non-fast forward push.
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: durham, rmcelroy, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3994813
Tasks: 12479677
Summary:
Previously error was thrown if mysql.connector wasn't present on client hosts.
But there is no need for mysql.connector to be installed on clients.
Remove global imports to fix it and create bundlestore() only for server repo.
P. S.
Changes in test is required because bundlestore() is not created if
infinitepush.server=True is not present in config.
Test Plan:
Uninstall mysql.connector: `sudo yum remove mysql-connector-python`
Enable infinitepush extension in any repo and run any hg command (hg st,
for example). Make sure it doesn't throw exceptions.
Install mysql.connector: `sudo yum install mysql-connector-python`.
Run `test-infinitepush-bundlestore.t`
Reviewers: durham, mitrandir, quark, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3967189
Tasks: 12479677
Signature: t1:3967189:1475581802:fbaafceac298242d0a449267ec0d68e290c3f262
Summary:
Hack to fix interaction with remotenames. Remotenames push '--to' bookmark to the server but we don't want to push scratch bookmark to the server. Let's delete '--to' and '--create' and also set allow_anon to True (because if --to is not set remotenames will think that we are pushing anonymoush head).
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3937202
Tasks: 12479677
Signature: t1:3937202:1475170335:fa4d39e1e17bea67b62b99dfc4cd2ffa4e5d6bea
Summary:
It's possible to push new scratch bookmark which points to existing scratch
commit. Previously it has failed with a stack trace.
Unfortunately it still doesn't allow to push scratch bookmark that points
to non-scratch commit. It requires more complex changes so
it will be fixed later if necessary.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3936728
Tasks: 12479677
Signature: t1:3936728:1475078059:ac4588e086b0344070a90f806b6f518c6e187c3b
Summary:
Bookmark can be moved to another node and node can be moved to another bundle.
In this case INSERT will fail because of the PRIMARY KEY violation.
Let's add ON DUPLICATE KEY UPDATE which will update existing row instead of
creating a new one.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3936281
Tasks: 12479677
Signature: t1:3936281:1475078339:d2ff0a01e1d753570c60172c5ef34ceeb7af1b96
Summary:
infinitepush bundle2 part should be run before pushrebase bundle2 part. This is a bit tricky since both pushrebase and infinitepush change the `extensions._order` and both of them try to load before bundle2 `changeset` part.
I suggest to load before 'changeset' or pushrebase part whichever is earlier. Then we have two situations:
1) pushrebase comes first. It sets it's part before 'changeset' part, then infinitepush sets it's part before pushrebase part.
2) infinitepush comes first. It sets part before 'changeset' and pushrebase part, then pushrebase sets it's part before 'changeset' part. Since infinitepush part comes before 'changeset' part it still be before pushrebase part.
This solution is not robust but it should be fine for now. The proper solution requires changes upstream that will introduce dependencies between bundle2 parts.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3930697
Tasks: 12479677
Signature: t1:3930697:1475079064:1b5851df4d2067ec21ec01e7510ddf83ff1684ad
Summary:
--bundle-store option makes push go to bundle store
even if bookmark is not specified or bookmark doesn't match scratch branch pattern.
The main reason for the `--bundle-store` is to be able to push not a scratch bookmark, but a scratch commit, for example, `hg push -r . --bundle-store`.
It will be useful when we will save all the local commits in the bundlestore.
To do this:
1) Wrap `discovery.checkheads()` to allow push to scratch branch even if new head is created.
2) Make scratchbranchpat params advisory because now bookmark may not be specified.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3862665
Tasks: 12479677
Signature: t1:3862665:1475077612:7e02a5bdc1da28161e45029b1e3a38fe8223a305
Summary:
We are going to add `push` option to force commits to go to bundle store.
This is a preliminary refactoring. Add another experimental config param that sets whether this is a scratch or usual push
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3862604
Tasks: 12479677
Signature: t1:3862604:1473934819:3ead319a885d99dd9a215941b317540121739b32
Summary:
There is no point in disallowing it.
The next step is to allow to fetch arbitrary many scratch bookmarks with one command.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3835502
Tasks: 12479677
Summary:
scratchbookmarks are not automatically pulled (because they are not showed in
listkeys). Let's record them after the actual pull. It introduces a race
condition. But this race condition is unlikely and shouldn't be harmful.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: mjpieters, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3835414
Tasks: 12479677
Signature: t1:3835414:1473703998:554c890be50f732f34b2793160ee1abb26561e39
Summary:
Diff introduces `listkeyspatterns` wireproto method. In future it will be able to filter keys by pattern, but for now it has limited functionality:
1) it is able to find scratch bookmarks only by it's full name (no patterns)
2) and it doesn't filters non-scratch bookmarks.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797800
Tasks: 12479677
Signature: t1:3797800:1473440073:61db6803235c53afbb3cdc72084aa35ce7d0ee63
Summary:
Infinitepush index implementation that uses mysql db to store data.
Three tables:
1) Node to bundle
2) Bookmark to node
3) All bundles (this is necessary to
find unused bundles.
Test Plan: Code was tested but tests are not included in the diff - see comments below
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3811247
Tasks: 12479677
Signature: t1:3811247:1473378520:bc23b3436aa2e618a7bfefd2d571257dfe8ace94
Summary:
Use external bundles to store bundles.
Put and get binaries should be provided to upload/download bundles to/from external store (see `put_binary` and `get_binary` config options). Put binary should output a handle, that can be used with `get_binary` to retrieve bundle. `put_args` and `get_args` config options can be used to pass additional params to the binaries.
Test Plan:
Change `test-infinitepush-bundlestore.t` test file to point to external store and run it
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803936
Tasks: 12479677
Signature: t1:3803936:1473123987:c34987ee459b2a66ee831574a32cdc8231043cb2
Summary: Add exceptions that will be thrown from bundle stores
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803928
Tasks: 12479677
Signature: t1:3803928:1473073448:32f6191ea0548c3d83cbaad100b6c8c48cbd8131
Test Plan: No need to test - comment change
Reviewers: rmcelroy, durham, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3791555
Tasks: 12479677
Summary: --create and --to options work with non-scratch bookmarks too
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, durham, rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797183
Tasks: 12479677
Summary: It's safer to don't have bundlestore then to have uninitialized bundlestore
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, rmcelroy, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797175
Tasks: 12479677
Summary:
Copy of hackbranch hackathon project with a few changes.
The goal is to allow saving commits under certain bookmarks in the separate store. These bookmarks are called 'scratch' bookmarks and are defined in config file (for example, 're:scratch/.+').
Scratch commits are sent in a separate bundle2 part during push. During pull new bundle repo is created and requests are served using this bundle repo.
There are a few changes from hackathon project implementation.
# It requires no changes in mercurial core. It was done by wrapping `findcommonincoming()` function and changing `lookup()` wireproto method (see below).
# Instead of introducing `listkeypattern()` wireproto method, `lookup()` method and `pull()` command were wrapped. I did it because I thought that it will make code easier. It probably haven't made it easier, it's almost the same. I can change it back to `listkeypattern` if it's necessary.
The bad thing about current implementation is that scratch bookmarks are not downloaded. But in the previous implementation also haven't downloaded bookmark if remotenames is enabled.
There are a couple of problems:
1) As I mentioned before, scratch bookmarks are not downloaded
2) It's impossible to pull both scratch and non-scratch bookmark
Test Plan: python ../../../hg-crew/tests/run-tests.py test-bundlestore.t
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3750999
Tasks: 12479677
Signature: t1:3750999:1472052396:5d7828566863f0aa437cdc34439ec0759503c6c9
Summary:
Code was written by Laurent, Durham and Mateusz on hackathon.
Index stores metadata, store stores bundles.
This diff adds interfaces and simple filesystem implementations
Test Plan:
cd tests
py.test *.py
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3741963
Tasks: 12479677
Signature: t1:3741963:1472207457:c9e7fb6163ef3cd6fdc1c4bd5113c623169c3a7c
Summary: I'll later add most of your hackhaton hackbranch code to this project with a few small fixes
Test Plan: No need
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: rmcelroy, mitrandir
Subscribers: quark, rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D3736089
Signature: t1:3736089:1471876504:98c4efc3bdf66d60f98eed11a3f13bae56fdc716
