Commit Graph

63 Commits

Author SHA1 Message Date
Stanislau Hlebik
f0a4c560a1 infinitepush: use --non-forward-move option instead of --force
Summary: --force option is scary, let's use --non-forward-move instead.

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4362055

Tasks: 12479677

Signature: t1:4362055:1482426570:b57748835f5ec3b6916b9d29d67b7c3d9582a6e2
2016-12-22 09:20:49 -08:00
Stanislau Hlebik
6d29203ee5 infinitepush: use dest parameter in background backups
Summary:
Default path was used during background backup even if non-default path was
passed on the command line. So `hg push somepath --background` is
equivalent to `hg push --background`. This is not correct and this diff fixes it


Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4357295

Tasks: 12479677

Signature: t1:4357295:1482330125:d2b043f8035046ab43667b4387bf642e53261681
2016-12-21 07:44:51 -08:00
Stanislau Hlebik
8c14ed67f3 infinitepush: add new fields to the log
Summary: Add `errormsg` and `reponame`

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, andrasbelo

Reviewed By: andrasbelo

Subscribers: andrasbelo, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4339606

Tasks: 12479677

Signature: t1:4339606:1482312794:15c04789ba03059c9a178d1adcf2bd09d9cf7b57
2016-12-21 01:41:44 -08:00
Durham Goode
08a77d4294 infinitepush: fix regex formating
In Python 2.6 we need to explicitly number all the '{}' format entries. This was
breaking the Centos6 build.
2016-12-16 17:29:04 -08:00
Stanislau Hlebik
feecb32f3a infinitepush: pullbackup command
Summary:
As the name suggest it will restore backup made by `hg pushbackup`.

If user has only one backup for the `dest` repo then it will be restored.
But user may have backed up many local repos that points to `dest` repo.
These local repos may reside on different hosts or in different
repo roots. It makes restore ambiguous; `--reporoot` and `--hostname`
options are used to disambiguate.

Example situation:
1) User has only one laptop with mercurial repo `repo` that was cloned
from remote server. He or she run `hg pushbackup`. Then laptop breaks
and user gets a new one, clones the `repo` again and runs `hg restore`.
It automatically restores the backup.
2) User has devserver and laptop and backups were made from both.
Then if user decides to switch devserver and run `hg restore` on the new
devserver he or she has to specify `--hostname`.


Future plans:
1) Add `--user` option to make it possible to restore another user's backup

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4280832

Tasks: 12479677

Signature: t1:4280832:1481565335:2a21ceafa2ff80242076a79693046514434afb40
2016-12-16 08:29:08 -08:00
Stanislau Hlebik
db0892583c infinitepush: send cg v2
Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4299163

Tasks: 12479677

Signature: t1:4299163:1481212693:21120d98330b9a4fa0aced0523cc389eaa54551b
2016-12-15 10:02:59 -08:00
Stanislau Hlebik
06231c630a infinitepush: add cleanup
Summary:
`bundlerepo` should be closed to ensure that temp file is deleted.
`bundlerepo` creates temp file when bundle is compressed. This is *not* the
case in infinitepush (all our bundle are uncompressed). But seems that
upstream cg1unpacker.compressed() returns True even if bundle is uncompressed.

Also this diff makes `_getoutputbundleraw()` read output bundle in memory
instead of returning a generator. The reason for doing it is because generator
becomes invalid as soon as `bundlerepo` is closed.
This approach will obviously increase memory usage but it shouldn't be a
problem since bundles are small (no more than 10s Mb).

#thanks @quark for reporting it

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, quark

Reviewed By: quark

Subscribers: quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4327186

Tasks: 12479677

Signature: t1:4327186:1481738559:2c6b22c305da4d572da9de21dfdf1179f7281744
2016-12-14 12:26:30 -08:00
Stanislau Hlebik
3cbecabdb8 infinitepush: add logging
Summary: Let's add logging to infinitepush.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4306193

Tasks: 12479677

Signature: t1:4306193:1481312610:1c3f6af7fedbaca51ef81291a471047c7e5931ab
2016-12-13 10:44:40 -08:00
Stanislau Hlebik
c014561bb5 infinitepush: accept different cg versions server-side
Summary:
infinitepush has been storing bundle1 with changegroups v1. Turned out that in
some cases generated bundles can become huge because v1 changegroups don't
support generaldelta. It makes sense to support all changegroup versions (v1 for BC,
v2 for new infinitepush bundles, and maybe later we'll switch to v3).
To do this we have to store bundle2 instead of bundle1 because bundle1
supports only changegroups v1.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, quark, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4299151

Tasks: 12479677

Signature: t1:4299151:1481564714:810be69447d0b35aa57328c60aab72ad374e994d
2016-12-13 06:32:31 -08:00
Stanislau Hlebik
e44e5d8bc8 infinitepush: move writing to store outside of indexapi transaction
Summary:
Writing to bundlestore can take a lot of time. It doesn't make sense to do it
inside indexapi transaction

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4305903

Tasks: 12479677

Signature: t1:4305903:1481312165:6aca8c908367d9715f8ba3f5643cd56650b4456f
2016-12-10 13:37:48 -08:00
Stanislau Hlebik
01528f5840 infinitepush: rename debugbackup to pushbackup
Summary:
@simon suggested to rename `debugbackup` because `debug-` prefix
implies that command is internal only and it's not safe for user to run it.
This is not the case for the backup because user can safely run it.

Test Plan: Run `test-infinitepush-*`

Reviewers: simon, rmcelroy, mitrandir, durham, mjpieters

Reviewed By: mjpieters

Subscribers: simon, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4292516

Tasks: 12479677

Signature: t1:4292516:1481213170:a03c3eaebd3459887bbb3ff1c80f20977139cc90
2016-12-08 09:24:19 -08:00
Stanislau Hlebik
09d6fda2a0 infinitepush: avoid using push during debugbackup
Summary:
`hg push` takes a lock because it updates phases and may receive other bundle2
parts from remote server. We don't need it in debugbackup. And since
debugbackup is intended to run in background we want to avoid taking locks.
Instead let's use raw `unbundle()` wireproto method and disable pushback
bundle2 parts.

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, quark

Reviewed By: quark

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4291740

Tasks: 12479677

Signature: t1:4291740:1481118369:1b20f771c5db6e4fed4fc18681cc66ef4e91dd3a
2016-12-07 06:05:29 -08:00
Stanislau Hlebik
83c0fd8ccd infinitepush: support pull from many scratch bookmarks
Summary:
We could've avoided creating this diff if bundlerepo supported many bundles.
Unfortunately it doesn't. Adding support requires changes in upstream mercurial and it can take a lot of time.
This implementation wraps `listkeys` to set phases properly and `exchange._changegrouppart` to send more
than one bundle at a time.
It has unsolved problem with phases. For example in this situation


  # Publishing server creates commit A (it is marked as draft on the server).
  # Client pulls commit A from publishing server (in this case commit A is marked on the client but it's still draft on the server).
  # Another client make pull from scratch bookmark. Commit A will be draft on client because we delete publishing = True from listkeys.

We assume that this case is quite rare.

Test Plan: Run `test-infinitepush-*`

Reviewers: mitrandir, rmcelroy, quark, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4074889

Tasks: 12479677

Signature: t1:4074889:1480446669:97e7f1d8ad23d457d7984a4cde0efb2b6e89eb2e
2016-12-06 07:41:04 -08:00
Stanislau Hlebik
f0a4bf7c7b infinitepush: do not create transcations if there are no bookmarks to update
Summary:
There are lots of breakages if upstream tests are run with infinitepush
enabled. This patch fixes one issue in test-pull-update.t test.

Test Plan:
Run `test-infinitepush-*`
Run test-pull-update.t with --extra-config-opt=extensions.infinitepush=PATH_TO_INFINITEPUSH.
Make sure that it works fine up to line 115 (later lines may still be broken)

Reviewers: durham, rmcelroy, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4250678

Tasks: 12479677

Signature: t1:4250678:1480689565:792f3d77e08c969a9ffed904815fe7e9aa72f61d
2016-12-05 03:31:06 -08:00
Stanislau Hlebik
bfe26123d5 infinitepush: abort if no default path
Summary:
There are lots of breakages if upstream tests are run with infinitepush.
It fixes test-default-push.t upstream test.

Test Plan:
Run `test-infinitepush-*`
Run `test-default-push.t` upstream test

Reviewers: durham, rmcelroy, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4250684

Tasks: 12479677

Signature: t1:4250684:1480689556:7d6bc4788cfebb5aa62c631a104e9d735eca3e47
2016-12-05 03:28:23 -08:00
Stanislau Hlebik
8121ee7328 infinitepush: backup bookmarks even if no commits were created
Summary:
We want to backup bookmarks even if no new commits were added. To do this let's
save the hash of all the backuped bookmarks and backup only if saved hash is
different from the hash of the current bookmarks.

In the case when no new commits need to be backuped revs=['null'] is passed
(remotenames does the same in `push --delete`).

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, quark

Reviewed By: quark

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4251589

Tasks: 12479677

Signature: t1:4251589:1480611524:6faaa9d6d1b1aba9f7a45a4bf9a4c0df3d039ebe
2016-12-05 03:25:34 -08:00
Stanislau Hlebik
7dc24c7c82 infinitepush: backup local heads and bookmarks
Summary:
Backuping changesets is not enougn for restore. We also want to backup heads
and bookmarks to completely capture the state of the repo.

This diff is the first step in implementing this functionality. It adds new
bundle2 part which contains encoded dict (simple json encoding is used).
If value in the dict is empty then key is the bookmark pattern to delete.
If value is not empty then key is the bookmark name to save and value is a
node hash. The reason to put them in the same part is to make it possible to
delete and insert into indexapi in one transaction. It's also possible to pass
patterns to delete in part parameters but there is a bug in upstream hg that
limits parameters' size to 256 and we can potentially have longer bookmarks.

Local bookmarks are saved in infinitepush in the following form:
  infinitepush/backups/USERNAME/HOSTNAME/REPOROOT/bookmarks/LOCAL_BOOKMARK_NAME

Local heads are saved in infintiepush in the following form:
  infinitepush/backups/USERNAME/HOSTNAME/REPOROOTheads/HEAD_HASH

Hostname, username and repo root is necessary to distinguish different backups.

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, quark

Reviewed By: quark

Subscribers: quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4245080

Tasks: 12479677

Signature: t1:4245080:1480518959:aa199d67fac4e2cd2f543651ff56fdd649dac729
2016-12-01 08:44:57 -08:00
Stanislau Hlebik
c2e54d52a7 infinitepush: make indexapi a context manager
Summary:
As @quark suggested let's make `indexapi` a contextmanager.
It greatly simplifies code

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy, mitrandir, quark

Reviewed By: quark

Subscribers: quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4258082

Tasks: 12479677

Signature: t1:4258082:1480596008:32c39150dbb1966e76064ad8d93a808c89d78b71
2016-12-01 08:40:09 -08:00
Stanislau Hlebik
35b49c7546 infinitepush: set push defaults in debugbackup
Summary:
Not setting all default options may result in KeyValue errors.
For example pushvars extensions does
  repo._shellvars = opts['pushvars']

It results in failure if 'pushvars' is not set.
Let's fix it by explicitly setting default values.

Also we need to set `allow_anon` to remotenames because otherwise
debugbackup will fail.

Test Plan:
Run `test-infinitepush-*`

Run `hg debugbackup` inside fbsource make sure there were no failures

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4237230

Tasks: 12479677

Signature: t1:4237230:1480446802:36e0630ce3e60c947be47d83bbc6deff8624048e
2016-11-30 02:11:28 -08:00
Stanislau Hlebik
6257caad35 infinitepush: address comments from D4175280
Summary:
I forgot to fix inline comments in D4175280.
Fix'em now

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4213794

Tasks: 12479677

Signature: t1:4213794:1479750999:a90f265413881450220f3cf4769f73baf6b393de
2016-11-22 04:10:08 -08:00
Stanislau Hlebik
024d254256 infinitepush: increase test timeout and add logging
Summary:
Tests failed on mac. My guess is that timeout is too low.
Let's increase it to 10 seconds (it's fine because we want wait for 10 sec
if debugbackup finishes faster).

Also let's add logging

Test Plan: Run `test-infinitepush-*`

Reviewers: #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4213330

Tasks: 12479677
2016-11-21 08:25:10 -08:00
Stanislau Hlebik
24dc1a5878 infinitepush: fix tests
Summary:
`backup` was renamed to `debugbackup` a few commits ago.

Rename it there too

Test Plan: Run `test-infinitepush-*`

Reviewers: #sourcecontrol, andrasbelo

Reviewed By: andrasbelo

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4212539

Tasks: 12479677

Signature: t1:4212539:1479721307:1ac25c7dc8951d655910e2b0c41b2c9555353f74
2016-11-21 01:44:54 -08:00
Stanislau Hlebik
0a536f70fd infinitepush: add background backup
Summary:
`hg backup --background` will be used as a `txnclose` hook to backup all of the
local commits to infinitepush.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4175953

Tasks: 12479677

Signature: t1:4175953:1479145307:e698903b519361b376f6e182db7c49869c992617
2016-11-21 00:53:37 -08:00
Stanislau Hlebik
565b70f23e infinitepush: support bundle with many heads
Summary:
During `hg backup` bundle with many heads may be pushed. Let's support it too.
Many heads bundle is not allowed only when we are pushing it with `--to`
because in this case we don't know the node for the bookmark.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4175280

Tasks: 12479677

Signature: t1:4175280:1479231056:fb0c47a7752319d77a6cfc83a29c57e9e2dced16
2016-11-21 00:48:09 -08:00
Stanislau Hlebik
fd8f0625ef infinitepush: add hg backup command
Summary:
Backups all new non-extinct [1] commits to bundlestore.

When it is called for the first time `hg backup` will backup all draft
visible commits. Next backups will save only new commits since the last
backup (it is recorded in `.hg/store/infinitepushbackuptip`[2]).

It's an initial implementation. Later the following features will be added:
1) It will be called automatically whenever user creates or strips commits
or even creates or deletes bookmarks
2) It will also save all local bookmarks and all local heads
(probably only visible).

Note: calling `pushcmd` directly does not set default values for opts. That means that `--to` will be None and `_scratchbranchmatcher` will throw exception. Let's add a check to ensure that `--to` is never None.

[1] I also want to backup extinct commits (i.e. obsolete invisible commits).
But it will require bigger changes in discovery algorithm, so I'd leave it
for later.
[2] The name is a bit verbose. But I want to keep `infinitepush` part to make
it easier to debug problems.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4147651

Tasks: 12479677

Signature: t1:4147651:1479229440:3eb38880c14f18e9a2fb4eaba44bedf079bca506
2016-11-16 02:13:17 -08:00
Stanislau Hlebik
1b8b5987b2 infinitepush: overwrite bundle for old revs
Summary:
Previously if bundle has contained nodes that are already saved in bundlestore
then we wouldn't update bundleid value for these nodes.
That means that we can never delete a bundle because there will always be a
node that points to it.

I suggest to always update bundleid value for all nodes in the bundle.

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, mitrandir, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4130662

Tasks: 12479677

Signature: t1:4130662:1478565392:682fe39c5832d5edc84c84541f0ba6c1f2e1daf8
2016-11-08 02:27:25 -08:00
Stanislau Hlebik
f4f6b2411a infinitepush: inhibit pulled revs
Summary:
Pulling scratch commits that were stripped before results in
abort: 00changelog.i@c153241c1ea22cbce8a229dccdf7f46de827d962: filtered node!

The reason is that upstream pull does
checkout = str(repo.changelog.rev(checkout))

and it fails if revision is filtered.
I suggest to inhibit filtered revisions that are present in the local repo.
It makes sense because if user tries to pull scratch branch he or she
probably expects to see them in the repo even though these revisions were
stripped before.

Also we can avoid doing any pull if all revisions are present in the repo.
I decided not to do this because it complicates code (for example,
we need to ensure that bookmarks are updated correctly) and it doesn't give
any significant benefits.

Test Plan: arc unit

Reviewers: durham, rmcelroy, mitrandir

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4103869

Tasks: 12479677
2016-11-01 03:42:26 -07:00
Stanislau Hlebik
acdd58a41e infinitepush: possible fix to prevent scratch bookmarks disappearing
Summary:
Sometimes remote scratch bookmarks disappear during `hg pull`.
I can't reliably reproduce it, but the best guess I had so far is
that `repo.names['remotebookmarks']` is not up-to-date.
Let's add `clearnames()` before.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4104227

Tasks: 12479677

Signature: t1:4104227:1477954723:70e1ab2b4a90f26046a1dd05e5bb4aab68777e65
2016-11-01 01:53:41 -07:00
Stanislau Hlebik
cf1214923e infinitepush: disable pushes to svn repo
Summary:
During infinitepush test we noticed that `hg push -r . --to SCRATCHBRANCH`
fails with stack trace in www. This is expected because infinitepush doesn't work
with svn servers. But it's better to print a nice error message instead of stack trace.

Test Plan:
Try to push to scratch bookmark from www without specifying path:
stash@dev1918 ~/www (344c6bf) $ hg push -r . --to hack/durhamstash
abort: infinite push does not work with svn repo
(Did you forget to `hg push default`?)
[Exit: 255]

Got nice error instead of stack trace.

Push to scratch bookmark from www with path specified works fine

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4103745

Tasks: 12479677

Signature: t1:4103745:1477950448:14b5ae8484f3e65cb88c83b2628f73f3a670b1a0
2016-11-01 01:50:34 -07:00
Stanislau Hlebik
563a57128e infinitepush: add mysql waittimeout server-side
Summary:
Saw a few timeout errors during infinitepush test.
Previous timeout was 10 secs, it makes sense to increase it.

There was a unit test failures because of the config options.
So this diff also adds missing config docs and copyrights headers.

Test Plan: Build the package, install it locally and run integrationtests

Reviewers: durham, rmcelroy, mitrandir, andrasbelo

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4103689

Tasks: 12479677
2016-11-01 01:44:38 -07:00
Stanislau Hlebik
005a9ace91 infinitepush: bookmarks remote-list
Summary:
Add an option to list remote and scratch bookmarks by pattern.
Only very simple pattern is allowed: either the literal string
or a prefix (like 'scratch/stash/*'). It was made intentionally
to make sure that pattern requests are fast in mysql.

Mysql tests will be added to the integration tests

Test Plan: Run `test-infinitepush-*`

Reviewers: mitrandir, andrasbelo, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4074409

Tasks: 12479677

Signature: t1:4074409:1477500968:e91cd2505d61a2f1db30c7f00cdcfc949e433507
2016-10-26 10:03:35 -07:00
Stanislau Hlebik
6086c2f0cb infinitepush fix: pass dest to the orig
Test Plan: Run `test-infinitepush-*`

Reviewers: durham, mitrandir, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4081739

Tasks: 12479677

Signature: t1:4081739:1477498757:7e105e400fd99e0a6dcb80165a2ed9d9c6d1a548
2016-10-26 09:29:23 -07:00
Stanislau Hlebik
0ead2362af infinitepush: fix tests
Summary:
Tests were broken because there were internal API changes in
upstream mercurial:

1) `hg book` outputs nothing instead of `no bookmarks set`
2) `getbundle` function was removed, new `getbundlechunks` function was added

Test Plan: Run `test-infinitepush-*` and run integration tests

Reviewers: durham, rmcelroy, mitrandir, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4067223

Tasks: 14035546

Signature: t1:4067223:1477305056:f705764ff3cdd70fcc61e677ccb184870077ec86
2016-10-24 06:07:35 -07:00
Stanislau Hlebik
88085025a6 infinitepush: make scratch bookmark remote
Summary:
User should see no difference between scratch and non-scratch remote bookmarks.
So let's make scratch bookmarks remote if remotenames is enabled.
Unfortunately it requires saving scratch remote bookmarks before push/pull and
restoring it afterwards because `remotenames` extension doesn't know about
them.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4036653

Tasks: 12479677

Signature: t1:4036653:1476985129:b5c1afdab4c7585c8b12c2962a4b95a56751091b
2016-10-21 06:36:14 -07:00
Stanislau Hlebik
af148df6e7 infinitepush: make scratchbranchpart parameters cleaner
Summary:
If param is set to '0', then `params.get(PARAMNAME)` is True.
This is not what's expected and can lead to subtle bugs in future.
Instead do not send anything if param is not set.

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, mitrandir, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4036504

Tasks: 12479677

Signature: t1:4036504:1476797842:112bae5c318922795c8be5bb5e65458fe9cf034e
2016-10-18 07:43:52 -07:00
Stanislau Hlebik
c15797e1dd infinitepush: ignore scranch branch if not set
Summary:
If `branchpattern` is not set then it matches empty bookmark.
It causes regressions like this t13808497. Let's ignore empty `branchpattern`

Test Plan: Run `test-infinitepush-*`

Reviewers: mitrandir, rmcelroy, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4031365

Tasks: 12479677

Signature: t1:4031365:1476755541:3d3781ef92f1701500d74ce7b98b4d82b5a1228e
2016-10-18 01:21:18 -07:00
Stanislau Hlebik
b9fab6bc04 infinitepush: fix remotefilelog interaction
Summary:
Remotefilelog can't find file revisions outside of the repo.
Because of it user can't update to the revisions that was pulled from
bundlestore. This diff downloads all filelogs for revisions
from bundlestore.

There were two options to pass info about changed files to remotefilelog:
through config or through bundlecaps. I decided to use config because
it's not really a bundle capability.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4030065

Tasks: 12479677

Signature: t1:4030065:1476755932:4aa02e540f13d3221163d906524fb9ce97f9c7e8
2016-10-18 01:20:12 -07:00
stash
b421695c98 infinitepush: do not require mysql for tests
Summary:
Tests doesn't require mysql since it uses fileindexapi.
Let's move sqlindexapi and fileindexapi to different files and include
only necessary file.

Test Plan: Run test-infinitepush-*

Reviewers: durham, simonfar, quark

Reviewed By: quark

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4029460

Tasks: 12479677

Signature: t1:4029460:1476709161:556f80605db9bd3c18dce1e16e996b13bbe96229
2016-10-17 14:23:56 +01:00
Stanislau Hlebik
8bdaa46014 infinitepush: set scratch bookmark locally during push
Summary:
Previously `--to` bookmark was not set locally.
This diff fixes it by setting this bookmark to the head after push happened.
This is correct because it's guaranteed that we'll push only one head.

Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: durham, rmcelroy, mitrandir

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4001168

Tasks: 12479677
2016-10-13 00:11:56 -07:00
Stanislau Hlebik
1b98921c48 infinitepush: add function _savebookmarks()
Summary: It will be used in a few places (see next diff in stack)

Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4001105

Tasks: 12479677

Signature: t1:4001105:1476330730:0ba513a3bab131b8eadadd4e90576880663f0d49
2016-10-13 00:12:24 -07:00
Stanislau Hlebik
edf67f905d infinitepush: use addbookmarkandbundle
Summary:
Let's use a function that saves bookmarks, nodes and bundle
in the same time. For fileindexapi this call is equivalent
to addbundle() and addbookmarks() call. For sqlindexapi this
call makes two calls in one transaction.

Test Plan:
1) Run `test-infinitepush-bundlestore.t`
2) Test sql index locally

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4001243

Tasks: 12479677

Signature: t1:4001243:1476227921:9d449b743686762edce4c07d95266ef7ae69c2b5
2016-10-12 02:59:23 -07:00
Stanislau Hlebik
a4b6a2a1bb infinitepush: better error message
Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4001218

Tasks: 12479677

Signature: t1:4001218:1476227692:c60b3def4e82911a40fe577bc44cf7d24cba9438
2016-10-12 02:57:53 -07:00
Stanislau Hlebik
f1e2014203 infinitepush: add option to force non-fast-forward push
Summary:
Non-fast-forward pushes were disallowed. But having a
mutable shared history for scratch branches is a very
nice feature. Let's use --force option to force
non-fast forward push.

Test Plan: Run test-infinitepush-bundlestore.t

Reviewers: durham, rmcelroy, mitrandir

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3994813

Tasks: 12479677
2016-10-10 03:48:21 -07:00
Stanislau Hlebik
d63ba71acf infinitepush: remove defaults
Summary: I think it's better throw if misconfigured then silently proceed

Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: durham, mitrandir, quark, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3967202

Tasks: 12479677

Signature: t1:3967202:1475581664:9d7b3e21fce48a49eda35bde473915cb64dfdfa0
2016-10-04 04:52:31 -07:00
Stanislau Hlebik
e42fe1fdc5 infinitepush: do not require mysql.connector to be present on client
Summary:
Previously error was thrown if mysql.connector wasn't present on client hosts.
But there is no need for mysql.connector to be installed on clients.
Remove global imports to fix it and create bundlestore() only for server repo.

P. S.
Changes in test is required because bundlestore() is not created if
infinitepush.server=True is not present in config.

Test Plan:
Uninstall mysql.connector: `sudo yum remove mysql-connector-python`
Enable infinitepush extension in any repo and run any hg command (hg st,
for example). Make sure it doesn't throw exceptions.

Install mysql.connector: `sudo yum install mysql-connector-python`.
Run `test-infinitepush-bundlestore.t`

Reviewers: durham, mitrandir, quark, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3967189

Tasks: 12479677

Signature: t1:3967189:1475581802:fbaafceac298242d0a449267ec0d68e290c3f262
2016-10-04 04:52:03 -07:00
Stanislau Hlebik
ddd4873227 infinitepush: temp remotenames fix
Summary:
Hack to fix interaction with remotenames. Remotenames push '--to' bookmark to the server but we don't want to push scratch bookmark to the server. Let's delete '--to' and '--create' and also set allow_anon to True (because if --to is not set remotenames will think that we are pushing anonymoush head).


Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: durham, mitrandir, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3937202

Tasks: 12479677

Signature: t1:3937202:1475170335:fa4d39e1e17bea67b62b99dfc4cd2ffa4e5d6bea
2016-10-02 05:13:06 -07:00
Stanislau Hlebik
387f18b269 infinitepush: fix push of scratch bookmark on existing commit
Summary:
It's possible to push new scratch bookmark which points to existing scratch
commit. Previously it has failed with a stack trace.

Unfortunately it still doesn't allow to push scratch bookmark that points
to non-scratch commit. It requires more complex changes so
it will be fixed later if necessary.

Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3936728

Tasks: 12479677

Signature: t1:3936728:1475078059:ac4588e086b0344070a90f806b6f518c6e187c3b
2016-10-02 03:05:50 -07:00
Stanislau Hlebik
ba3ebe099e infinitepush: fix interaction with pushrebase
Summary:
infinitepush bundle2 part should be run before pushrebase bundle2 part. This is a bit tricky since both pushrebase and infinitepush change the `extensions._order` and both of them try to load before bundle2 `changeset` part.

I suggest to load before 'changeset' or pushrebase part whichever is earlier. Then we have two situations:
1) pushrebase comes first. It sets it's part before 'changeset' part, then infinitepush sets it's part before pushrebase part.
2) infinitepush comes first. It sets part before 'changeset' and pushrebase part, then pushrebase sets it's part before 'changeset' part. Since infinitepush part comes before 'changeset' part it still be before pushrebase part.

This solution is not robust but it should be fine for now. The proper solution requires changes upstream that will introduce dependencies between bundle2 parts.

Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3930697

Tasks: 12479677

Signature: t1:3930697:1475079064:1b5851df4d2067ec21ec01e7510ddf83ff1684ad
2016-10-02 02:54:03 -07:00
Stanislau Hlebik
24be1a2da5 infinitepush: add bundle-store option to push
Summary:
--bundle-store option makes push go to bundle store
even if bookmark is not specified or bookmark doesn't match scratch branch pattern.
The main reason for the `--bundle-store` is to be able to push not a scratch bookmark, but a scratch commit, for example, `hg push -r . --bundle-store`.
It will be useful when we will save all the local commits in the bundlestore.

To do this:
1) Wrap `discovery.checkheads()` to allow push to scratch branch even if new head is created.
2) Make scratchbranchpat params advisory because now bookmark may not be specified.

Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: mitrandir, rmcelroy, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3862665

Tasks: 12479677

Signature: t1:3862665:1475077612:7e02a5bdc1da28161e45029b1e3a38fe8223a305
2016-09-28 13:17:25 -07:00
Stanislau Hlebik
23ab46339b infinitepush: use configscratchpush setconfig param
Summary:
We are going to add `push` option to force commits to go to bundle store.
This is a preliminary refactoring. Add another experimental config param that sets whether this is a scratch or usual push

Test Plan: Run `test-infinitepush-bundlestore.t`

Reviewers: durham, rmcelroy, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3862604

Tasks: 12479677

Signature: t1:3862604:1473934819:3ead319a885d99dd9a215941b317540121739b32
2016-09-15 03:36:24 -07:00