Summary:
This small change makes a way better user experience. Users usually have one
big repo per host, but they may have many hosts (for example, laptop and
devserver). If we ask for reporoot first then it's very likely that next
`pullbackup` invocation will fail because there will be ambigious hosts.
Let's switch the order.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4649946
Tasks: 15389402
Signature: t1:4649946:1488561645:330590fcf1dcd4af7fb572c5d4ccfd8a5ab78c60
Summary:
Next diff will add special option to specify whose backup to restore.
This diff does a necessary refactoring
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4650282
Tasks: 15389402
Signature: t1:4650282:1488562203:632a8c84d6d537663bd7d94ce9dfd18a4498ccfb
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Finally reuse the same bundlerepo.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612094
Tasks: 15389402
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Now close all bundlerepos at once not one by one. This is necessary because
bundlerepos will be reused.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612089
Tasks: 15389402
Signature: t1:4612089:1488501776:e9fc2863adebc69bf44fa5dcb1610612027508f6
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Test Plan: arc unti
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612073
Tasks: 15389402
Signature: t1:4612073:1487955965:03af1a25acc21d0b6d913a60dd5ef82fa7f033ad
Summary:
There are a couple of reasons to avoid using ui.username():
1) If config option ui.username is set on the server then it will use it
instead of the name of the user that does push
2) It prints confusing warning `no username found, using '....' instead`
3) In some cases it fails (probably because it calls socket.getfqdn() which
does network request).
Let's use simple util.getuser() function and set username to 'unknown' if
it fails.
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: azich, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4642457
Signature: t1:4642457:1488458040:5dfac435bf44dfa6ab7725d2f5800963e8c860ec
Summary:
Previously --list-remote wasn't user friendly. User have to always specify a
pattern, and specifying * just won't work because of bash globbing. Also
it wasn't possible to specify many patterns at once.
This diff allows specifying many patterns, it allows to not specify
patterns at all and also it allows to specify default scratch patterns to fetch
if no patterns were specified. This is useful if there are many auto-generated
bookmarks under the same scratch bookmark pattern.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4605548
Signature: t1:4605548:1487934838:066a12c28dda16fa8f90674d736d21345631ef7e
Summary:
discovery.findcommonoutgoing() explicitly skips secret commits. Because of it
we also need to skip bookmarks that point to secret commits, otherwise they
will point to non-existent nodes.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4611393
Tasks: 15389402
Signature: t1:4611393:1487933210:72e79923c944b13204f4cde64d415076703bbe47
Summary:
hasscratchbookmarks is True if scratchbookmarks is not empty and vice-versa.
No need to use this variable
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4590917
Signature: t1:4590917:1487682597:c2a6e0accbd8e539a885dd11e6819986e57d24df
Summary:
listkeyspatterns support batching. Let's use and avoid making many network
calls to the hg server.
I had to modify tests because scratchbookmarks are passed to the listkeyspatterns,
and scratchbookmarks is dict and the order of the keys is unpredictable.
Because of it bundles from the server will be sent in unpredictable order, and log
output may be different.
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4590914
Signature: t1:4590914:1487788586:dd5ba0dd41b0cd19f790755d7b3c1fc5ba4ade61
Summary:
Let's pull node during update if not found locally.
This is a part of selectivepull functionality.
See remotenames extensions for details about selectivepull.
Test Plan: arc unit
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: indragie, mjpieters, sergeyb
Differential Revision: https://phabricator.intern.facebook.com/D4536129
Tasks: 12479658
Signature: t1:4536129:1486667537:3d1df30cb5d1db0dd7451756102ccafee20789d5
Summary:
Default lock timeout value is very small (around couple of secs).
Let's set it for something bigger by default
Test Plan: arc unit
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4435236
Tasks: 15389402
Signature: t1:4435236:1484880747:d46276a82782ef551cda0165dea686f267c13ff9
Summary:
On a big repo with many old heads bundle that is sent to the server can become
big (in some cases even 500 Mb). That looks like a waste of bundlestore space
and will probably make backup and restore slower. Most of the space is taken
by manifest deltas because it prefers to diff manifest against previous commit
in the bundle. There are two possible approaches to reduce the size:
1) Send many small bundles (for example, one bundle per head)
2) Wrap deltaparent function and diff against actual parent previous commit in the bundle.
I chose the second approach for the following reasons:
1) It's easier to implement (main reason)
2) Many bundles probably means slower restore because there will be many requests to the bundlestore instead of just one
With this diff bundle size was reduced from 500 Mb to 8 Mb.
It can potentially increase CPU usage. I'm not sure how bad is it and will investigate it more.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: simpkins, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4384053
Tasks: 12479677
Signature: t1:4384053:1483664446:62ec30fad433e8d279758926199a8330cb73ed2b
Summary:
If client pulled a commit without filelogs (because of remotefilelog) and
this commit was later stripped on the server, then attempt to backup this
commit fails because of missing filelogs. It's a rare issue but unfortunately
it happens sometimes. To fix it let's exclude from the backup all the
commits and their descendants that doesn't have all necessary filelogs.
Test Plan: Run infinitepush tests
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4365747
Signature: t1:4365747:1483466164:560cdf5cd2369cd2603dfd0fe6b30d7a70951f00
Summary:
`pushbackup` command became huge. This diff splits it into a few
smaller functions
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4384008
Tasks: 12479677
Signature: t1:4384008:1483659549:a97b63a38b702a55d19cf6b47a6fd6b2547b9168
Summary: __init__.py became huge and I suggest to split it
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4383661
Tasks: 12479677
Signature: t1:4383661:1483659134:eaf3d7e217633895a65ad568831d9c21ea0f18d8
Summary:
`pushbackup` failed when there are hidden heads in the repo but no visible
heads (see test for example). This diff fixes it.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4365677
Tasks: 12479677
Signature: t1:4365677:1483465909:fd1d96211909d3eeccab4fd17a364af4a1997f85
Summary:
The '&>>' operator only works in linux, not OSX. So let's replace it with a more
verbose equivalent. 'man bash' says these two are equivalent.
Test Plan:
Running the tests normally on OSX don't actually catch this, because
this particular test is skipped because it doesn't see that evolve is installed.
We need to run the tests with:
PYTHONPATH=/opt/facebook/hg/lib/python2.7/site-packages/:/opt/homebrew/lib/python2.7/site-packages/ /opt/homebrew/opt/python27/bin/python2.7 ../../facebook-hg-rpms/hg-crew/tests/run-tests.py test-infinitepush-backup.t
To get coverage. And the test is now passing on OSX and linux.
Reviewers: stash, #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4379593
Signature: t1:4379593:1483488383:ce8a045bbc29d719eafa372db56453b83f1d8df8
Summary:
Default path was used during background backup even if non-default path was
passed on the command line. So `hg push somepath --background` is
equivalent to `hg push --background`. This is not correct and this diff fixes it
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4357295
Tasks: 12479677
Signature: t1:4357295:1482330125:d2b043f8035046ab43667b4387bf642e53261681
Summary:
As the name suggest it will restore backup made by `hg pushbackup`.
If user has only one backup for the `dest` repo then it will be restored.
But user may have backed up many local repos that points to `dest` repo.
These local repos may reside on different hosts or in different
repo roots. It makes restore ambiguous; `--reporoot` and `--hostname`
options are used to disambiguate.
Example situation:
1) User has only one laptop with mercurial repo `repo` that was cloned
from remote server. He or she run `hg pushbackup`. Then laptop breaks
and user gets a new one, clones the `repo` again and runs `hg restore`.
It automatically restores the backup.
2) User has devserver and laptop and backups were made from both.
Then if user decides to switch devserver and run `hg restore` on the new
devserver he or she has to specify `--hostname`.
Future plans:
1) Add `--user` option to make it possible to restore another user's backup
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4280832
Tasks: 12479677
Signature: t1:4280832:1481565335:2a21ceafa2ff80242076a79693046514434afb40
Summary:
`bundlerepo` should be closed to ensure that temp file is deleted.
`bundlerepo` creates temp file when bundle is compressed. This is *not* the
case in infinitepush (all our bundle are uncompressed). But seems that
upstream cg1unpacker.compressed() returns True even if bundle is uncompressed.
Also this diff makes `_getoutputbundleraw()` read output bundle in memory
instead of returning a generator. The reason for doing it is because generator
becomes invalid as soon as `bundlerepo` is closed.
This approach will obviously increase memory usage but it shouldn't be a
problem since bundles are small (no more than 10s Mb).
#thanks @quark for reporting it
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4327186
Tasks: 12479677
Signature: t1:4327186:1481738559:2c6b22c305da4d572da9de21dfdf1179f7281744
Summary:
infinitepush has been storing bundle1 with changegroups v1. Turned out that in
some cases generated bundles can become huge because v1 changegroups don't
support generaldelta. It makes sense to support all changegroup versions (v1 for BC,
v2 for new infinitepush bundles, and maybe later we'll switch to v3).
To do this we have to store bundle2 instead of bundle1 because bundle1
supports only changegroups v1.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, quark, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4299151
Tasks: 12479677
Signature: t1:4299151:1481564714:810be69447d0b35aa57328c60aab72ad374e994d
Summary:
Writing to bundlestore can take a lot of time. It doesn't make sense to do it
inside indexapi transaction
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4305903
Tasks: 12479677
Signature: t1:4305903:1481312165:6aca8c908367d9715f8ba3f5643cd56650b4456f
Summary:
@simon suggested to rename `debugbackup` because `debug-` prefix
implies that command is internal only and it's not safe for user to run it.
This is not the case for the backup because user can safely run it.
Test Plan: Run `test-infinitepush-*`
Reviewers: simon, rmcelroy, mitrandir, durham, mjpieters
Reviewed By: mjpieters
Subscribers: simon, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4292516
Tasks: 12479677
Signature: t1:4292516:1481213170:a03c3eaebd3459887bbb3ff1c80f20977139cc90
Summary:
`hg push` takes a lock because it updates phases and may receive other bundle2
parts from remote server. We don't need it in debugbackup. And since
debugbackup is intended to run in background we want to avoid taking locks.
Instead let's use raw `unbundle()` wireproto method and disable pushback
bundle2 parts.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4291740
Tasks: 12479677
Signature: t1:4291740:1481118369:1b20f771c5db6e4fed4fc18681cc66ef4e91dd3a
Summary:
We could've avoided creating this diff if bundlerepo supported many bundles.
Unfortunately it doesn't. Adding support requires changes in upstream mercurial and it can take a lot of time.
This implementation wraps `listkeys` to set phases properly and `exchange._changegrouppart` to send more
than one bundle at a time.
It has unsolved problem with phases. For example in this situation
# Publishing server creates commit A (it is marked as draft on the server).
# Client pulls commit A from publishing server (in this case commit A is marked on the client but it's still draft on the server).
# Another client make pull from scratch bookmark. Commit A will be draft on client because we delete publishing = True from listkeys.
We assume that this case is quite rare.
Test Plan: Run `test-infinitepush-*`
Reviewers: mitrandir, rmcelroy, quark, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4074889
Tasks: 12479677
Signature: t1:4074889:1480446669:97e7f1d8ad23d457d7984a4cde0efb2b6e89eb2e
Summary:
There are lots of breakages if upstream tests are run with infinitepush
enabled. This patch fixes one issue in test-pull-update.t test.
Test Plan:
Run `test-infinitepush-*`
Run test-pull-update.t with --extra-config-opt=extensions.infinitepush=PATH_TO_INFINITEPUSH.
Make sure that it works fine up to line 115 (later lines may still be broken)
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4250678
Tasks: 12479677
Signature: t1:4250678:1480689565:792f3d77e08c969a9ffed904815fe7e9aa72f61d
Summary:
There are lots of breakages if upstream tests are run with infinitepush.
It fixes test-default-push.t upstream test.
Test Plan:
Run `test-infinitepush-*`
Run `test-default-push.t` upstream test
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4250684
Tasks: 12479677
Signature: t1:4250684:1480689556:7d6bc4788cfebb5aa62c631a104e9d735eca3e47
Summary:
We want to backup bookmarks even if no new commits were added. To do this let's
save the hash of all the backuped bookmarks and backup only if saved hash is
different from the hash of the current bookmarks.
In the case when no new commits need to be backuped revs=['null'] is passed
(remotenames does the same in `push --delete`).
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4251589
Tasks: 12479677
Signature: t1:4251589:1480611524:6faaa9d6d1b1aba9f7a45a4bf9a4c0df3d039ebe
Summary:
Backuping changesets is not enougn for restore. We also want to backup heads
and bookmarks to completely capture the state of the repo.
This diff is the first step in implementing this functionality. It adds new
bundle2 part which contains encoded dict (simple json encoding is used).
If value in the dict is empty then key is the bookmark pattern to delete.
If value is not empty then key is the bookmark name to save and value is a
node hash. The reason to put them in the same part is to make it possible to
delete and insert into indexapi in one transaction. It's also possible to pass
patterns to delete in part parameters but there is a bug in upstream hg that
limits parameters' size to 256 and we can potentially have longer bookmarks.
Local bookmarks are saved in infinitepush in the following form:
infinitepush/backups/USERNAME/HOSTNAME/REPOROOT/bookmarks/LOCAL_BOOKMARK_NAME
Local heads are saved in infintiepush in the following form:
infinitepush/backups/USERNAME/HOSTNAME/REPOROOTheads/HEAD_HASH
Hostname, username and repo root is necessary to distinguish different backups.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4245080
Tasks: 12479677
Signature: t1:4245080:1480518959:aa199d67fac4e2cd2f543651ff56fdd649dac729
Summary:
Right now `hg debugbackup` backups only changesets. We also want to backup
bookmarks. Since user may have many bookmarks let's add a separate method
that will save many bookmarks at once.
Test Plan: Will be tested in next diffs
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4244823
Tasks: 12479677
Signature: t1:4244823:1480596116:7114128dafe2b8a10f599d05d68c1a6dce522f4a
Summary:
This methods accepts list of bookmark patterns to delete (patterns are not
supported yet, but they will be supported).
Test Plan: Will be tested in subsquent diffs
Reviewers: andrasbelo, rmcelroy, durham
Reviewed By: rmcelroy, durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4176016
Tasks: 12479677
Signature: t1:4176016:1479144865:8c9121970878d4d462cc2c679a414999dfdce1f4
Summary:
Not setting all default options may result in KeyValue errors.
For example pushvars extensions does
repo._shellvars = opts['pushvars']
It results in failure if 'pushvars' is not set.
Let's fix it by explicitly setting default values.
Also we need to set `allow_anon` to remotenames because otherwise
debugbackup will fail.
Test Plan:
Run `test-infinitepush-*`
Run `hg debugbackup` inside fbsource make sure there were no failures
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4237230
Tasks: 12479677
Signature: t1:4237230:1480446802:36e0630ce3e60c947be47d83bbc6deff8624048e
Summary:
Tests failed on mac. My guess is that timeout is too low.
Let's increase it to 10 seconds (it's fine because we want wait for 10 sec
if debugbackup finishes faster).
Also let's add logging
Test Plan: Run `test-infinitepush-*`
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4213330
Tasks: 12479677
Summary:
`backup` was renamed to `debugbackup` a few commits ago.
Rename it there too
Test Plan: Run `test-infinitepush-*`
Reviewers: #sourcecontrol, andrasbelo
Reviewed By: andrasbelo
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4212539
Tasks: 12479677
Signature: t1:4212539:1479721307:1ac25c7dc8951d655910e2b0c41b2c9555353f74
Summary:
`hg backup --background` will be used as a `txnclose` hook to backup all of the
local commits to infinitepush.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4175953
Tasks: 12479677
Signature: t1:4175953:1479145307:e698903b519361b376f6e182db7c49869c992617
Summary:
During `hg backup` bundle with many heads may be pushed. Let's support it too.
Many heads bundle is not allowed only when we are pushing it with `--to`
because in this case we don't know the node for the bookmark.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4175280
Tasks: 12479677
Signature: t1:4175280:1479231056:fb0c47a7752319d77a6cfc83a29c57e9e2dced16
Summary:
Backups all new non-extinct [1] commits to bundlestore.
When it is called for the first time `hg backup` will backup all draft
visible commits. Next backups will save only new commits since the last
backup (it is recorded in `.hg/store/infinitepushbackuptip`[2]).
It's an initial implementation. Later the following features will be added:
1) It will be called automatically whenever user creates or strips commits
or even creates or deletes bookmarks
2) It will also save all local bookmarks and all local heads
(probably only visible).
Note: calling `pushcmd` directly does not set default values for opts. That means that `--to` will be None and `_scratchbranchmatcher` will throw exception. Let's add a check to ensure that `--to` is never None.
[1] I also want to backup extinct commits (i.e. obsolete invisible commits).
But it will require bigger changes in discovery algorithm, so I'd leave it
for later.
[2] The name is a bit verbose. But I want to keep `infinitepush` part to make
it easier to debug problems.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4147651
Tasks: 12479677
Signature: t1:4147651:1479229440:3eb38880c14f18e9a2fb4eaba44bedf079bca506
Summary:
Previously if bundle has contained nodes that are already saved in bundlestore
then we wouldn't update bundleid value for these nodes.
That means that we can never delete a bundle because there will always be a
node that points to it.
I suggest to always update bundleid value for all nodes in the bundle.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4130662
Tasks: 12479677
Signature: t1:4130662:1478565392:682fe39c5832d5edc84c84541f0ba6c1f2e1daf8
Summary:
Pulling scratch commits that were stripped before results in
abort: 00changelog.i@c153241c1ea22cbce8a229dccdf7f46de827d962: filtered node!
The reason is that upstream pull does
checkout = str(repo.changelog.rev(checkout))
and it fails if revision is filtered.
I suggest to inhibit filtered revisions that are present in the local repo.
It makes sense because if user tries to pull scratch branch he or she
probably expects to see them in the repo even though these revisions were
stripped before.
Also we can avoid doing any pull if all revisions are present in the repo.
I decided not to do this because it complicates code (for example,
we need to ensure that bookmarks are updated correctly) and it doesn't give
any significant benefits.
Test Plan: arc unit
Reviewers: durham, rmcelroy, mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4103869
Tasks: 12479677
Summary:
Sometimes remote scratch bookmarks disappear during `hg pull`.
I can't reliably reproduce it, but the best guess I had so far is
that `repo.names['remotebookmarks']` is not up-to-date.
Let's add `clearnames()` before.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4104227
Tasks: 12479677
Signature: t1:4104227:1477954723:70e1ab2b4a90f26046a1dd05e5bb4aab68777e65
Summary:
During infinitepush test we noticed that `hg push -r . --to SCRATCHBRANCH`
fails with stack trace in www. This is expected because infinitepush doesn't work
with svn servers. But it's better to print a nice error message instead of stack trace.
Test Plan:
Try to push to scratch bookmark from www without specifying path:
stash@dev1918 ~/www (344c6bf) $ hg push -r . --to hack/durhamstash
abort: infinite push does not work with svn repo
(Did you forget to `hg push default`?)
[Exit: 255]
Got nice error instead of stack trace.
Push to scratch bookmark from www with path specified works fine
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4103745
Tasks: 12479677
Signature: t1:4103745:1477950448:14b5ae8484f3e65cb88c83b2628f73f3a670b1a0
Summary:
Saw a few timeout errors during infinitepush test.
Previous timeout was 10 secs, it makes sense to increase it.
There was a unit test failures because of the config options.
So this diff also adds missing config docs and copyrights headers.
Test Plan: Build the package, install it locally and run integrationtests
Reviewers: durham, rmcelroy, mitrandir, andrasbelo
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4103689
Tasks: 12479677
Summary:
Add an option to list remote and scratch bookmarks by pattern.
Only very simple pattern is allowed: either the literal string
or a prefix (like 'scratch/stash/*'). It was made intentionally
to make sure that pattern requests are fast in mysql.
Mysql tests will be added to the integration tests
Test Plan: Run `test-infinitepush-*`
Reviewers: mitrandir, andrasbelo, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4074409
Tasks: 12479677
Signature: t1:4074409:1477500968:e91cd2505d61a2f1db30c7f00cdcfc949e433507
Summary:
Tests were broken because there were internal API changes in
upstream mercurial:
1) `hg book` outputs nothing instead of `no bookmarks set`
2) `getbundle` function was removed, new `getbundlechunks` function was added
Test Plan: Run `test-infinitepush-*` and run integration tests
Reviewers: durham, rmcelroy, mitrandir, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4067223
Tasks: 14035546
Signature: t1:4067223:1477305056:f705764ff3cdd70fcc61e677ccb184870077ec86
Summary:
User should see no difference between scratch and non-scratch remote bookmarks.
So let's make scratch bookmarks remote if remotenames is enabled.
Unfortunately it requires saving scratch remote bookmarks before push/pull and
restoring it afterwards because `remotenames` extension doesn't know about
them.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4036653
Tasks: 12479677
Signature: t1:4036653:1476985129:b5c1afdab4c7585c8b12c2962a4b95a56751091b
Summary:
If param is set to '0', then `params.get(PARAMNAME)` is True.
This is not what's expected and can lead to subtle bugs in future.
Instead do not send anything if param is not set.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4036504
Tasks: 12479677
Signature: t1:4036504:1476797842:112bae5c318922795c8be5bb5e65458fe9cf034e
Summary:
If `branchpattern` is not set then it matches empty bookmark.
It causes regressions like this t13808497. Let's ignore empty `branchpattern`
Test Plan: Run `test-infinitepush-*`
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4031365
Tasks: 12479677
Signature: t1:4031365:1476755541:3d3781ef92f1701500d74ce7b98b4d82b5a1228e
Summary:
Remotefilelog can't find file revisions outside of the repo.
Because of it user can't update to the revisions that was pulled from
bundlestore. This diff downloads all filelogs for revisions
from bundlestore.
There were two options to pass info about changed files to remotefilelog:
through config or through bundlecaps. I decided to use config because
it's not really a bundle capability.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4030065
Tasks: 12479677
Signature: t1:4030065:1476755932:4aa02e540f13d3221163d906524fb9ce97f9c7e8
Summary:
Tests doesn't require mysql since it uses fileindexapi.
Let's move sqlindexapi and fileindexapi to different files and include
only necessary file.
Test Plan: Run test-infinitepush-*
Reviewers: durham, simonfar, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4029460
Tasks: 12479677
Signature: t1:4029460:1476709161:556f80605db9bd3c18dce1e16e996b13bbe96229
Summary:
Previously `--to` bookmark was not set locally.
This diff fixes it by setting this bookmark to the head after push happened.
This is correct because it's guaranteed that we'll push only one head.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001168
Tasks: 12479677
Summary: It will be used in a few places (see next diff in stack)
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001105
Tasks: 12479677
Signature: t1:4001105:1476330730:0ba513a3bab131b8eadadd4e90576880663f0d49
Summary:
Let's use a function that saves bookmarks, nodes and bundle
in the same time. For fileindexapi this call is equivalent
to addbundle() and addbookmarks() call. For sqlindexapi this
call makes two calls in one transaction.
Test Plan:
1) Run `test-infinitepush-bundlestore.t`
2) Test sql index locally
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001243
Tasks: 12479677
Signature: t1:4001243:1476227921:9d449b743686762edce4c07d95266ef7ae69c2b5
Summary: This check was missing before
Test Plan: It's difficult to test and the change should be safe
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4001229
Tasks: 12479677
Signature: t1:4001229:1476227703:9e40d6efa9892e1e0d7fbcb24fdce3d607a3a7fe
Summary:
Non-fast-forward pushes were disallowed. But having a
mutable shared history for scratch branches is a very
nice feature. Let's use --force option to force
non-fast forward push.
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: durham, rmcelroy, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3994813
Tasks: 12479677
Summary:
Previously error was thrown if mysql.connector wasn't present on client hosts.
But there is no need for mysql.connector to be installed on clients.
Remove global imports to fix it and create bundlestore() only for server repo.
P. S.
Changes in test is required because bundlestore() is not created if
infinitepush.server=True is not present in config.
Test Plan:
Uninstall mysql.connector: `sudo yum remove mysql-connector-python`
Enable infinitepush extension in any repo and run any hg command (hg st,
for example). Make sure it doesn't throw exceptions.
Install mysql.connector: `sudo yum install mysql-connector-python`.
Run `test-infinitepush-bundlestore.t`
Reviewers: durham, mitrandir, quark, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3967189
Tasks: 12479677
Signature: t1:3967189:1475581802:fbaafceac298242d0a449267ec0d68e290c3f262
Summary:
Hack to fix interaction with remotenames. Remotenames push '--to' bookmark to the server but we don't want to push scratch bookmark to the server. Let's delete '--to' and '--create' and also set allow_anon to True (because if --to is not set remotenames will think that we are pushing anonymoush head).
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3937202
Tasks: 12479677
Signature: t1:3937202:1475170335:fa4d39e1e17bea67b62b99dfc4cd2ffa4e5d6bea
Summary:
It's possible to push new scratch bookmark which points to existing scratch
commit. Previously it has failed with a stack trace.
Unfortunately it still doesn't allow to push scratch bookmark that points
to non-scratch commit. It requires more complex changes so
it will be fixed later if necessary.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3936728
Tasks: 12479677
Signature: t1:3936728:1475078059:ac4588e086b0344070a90f806b6f518c6e187c3b
Summary:
Bookmark can be moved to another node and node can be moved to another bundle.
In this case INSERT will fail because of the PRIMARY KEY violation.
Let's add ON DUPLICATE KEY UPDATE which will update existing row instead of
creating a new one.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3936281
Tasks: 12479677
Signature: t1:3936281:1475078339:d2ff0a01e1d753570c60172c5ef34ceeb7af1b96
Summary:
infinitepush bundle2 part should be run before pushrebase bundle2 part. This is a bit tricky since both pushrebase and infinitepush change the `extensions._order` and both of them try to load before bundle2 `changeset` part.
I suggest to load before 'changeset' or pushrebase part whichever is earlier. Then we have two situations:
1) pushrebase comes first. It sets it's part before 'changeset' part, then infinitepush sets it's part before pushrebase part.
2) infinitepush comes first. It sets part before 'changeset' and pushrebase part, then pushrebase sets it's part before 'changeset' part. Since infinitepush part comes before 'changeset' part it still be before pushrebase part.
This solution is not robust but it should be fine for now. The proper solution requires changes upstream that will introduce dependencies between bundle2 parts.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3930697
Tasks: 12479677
Signature: t1:3930697:1475079064:1b5851df4d2067ec21ec01e7510ddf83ff1684ad
Summary:
--bundle-store option makes push go to bundle store
even if bookmark is not specified or bookmark doesn't match scratch branch pattern.
The main reason for the `--bundle-store` is to be able to push not a scratch bookmark, but a scratch commit, for example, `hg push -r . --bundle-store`.
It will be useful when we will save all the local commits in the bundlestore.
To do this:
1) Wrap `discovery.checkheads()` to allow push to scratch branch even if new head is created.
2) Make scratchbranchpat params advisory because now bookmark may not be specified.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3862665
Tasks: 12479677
Signature: t1:3862665:1475077612:7e02a5bdc1da28161e45029b1e3a38fe8223a305
Summary:
We are going to add `push` option to force commits to go to bundle store.
This is a preliminary refactoring. Add another experimental config param that sets whether this is a scratch or usual push
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3862604
Tasks: 12479677
Signature: t1:3862604:1473934819:3ead319a885d99dd9a215941b317540121739b32
Summary:
There is no point in disallowing it.
The next step is to allow to fetch arbitrary many scratch bookmarks with one command.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: durham, rmcelroy, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3835502
Tasks: 12479677
Summary:
scratchbookmarks are not automatically pulled (because they are not showed in
listkeys). Let's record them after the actual pull. It introduces a race
condition. But this race condition is unlikely and shouldn't be harmful.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: mjpieters, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3835414
Tasks: 12479677
Signature: t1:3835414:1473703998:554c890be50f732f34b2793160ee1abb26561e39
Summary:
Diff introduces `listkeyspatterns` wireproto method. In future it will be able to filter keys by pattern, but for now it has limited functionality:
1) it is able to find scratch bookmarks only by it's full name (no patterns)
2) and it doesn't filters non-scratch bookmarks.
Test Plan: Run `test-infinitepush-bundlestore.t`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797800
Tasks: 12479677
Signature: t1:3797800:1473440073:61db6803235c53afbb3cdc72084aa35ce7d0ee63
Summary:
Infinitepush index implementation that uses mysql db to store data.
Three tables:
1) Node to bundle
2) Bookmark to node
3) All bundles (this is necessary to
find unused bundles.
Test Plan: Code was tested but tests are not included in the diff - see comments below
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3811247
Tasks: 12479677
Signature: t1:3811247:1473378520:bc23b3436aa2e618a7bfefd2d571257dfe8ace94
Summary:
Use external bundles to store bundles.
Put and get binaries should be provided to upload/download bundles to/from external store (see `put_binary` and `get_binary` config options). Put binary should output a handle, that can be used with `get_binary` to retrieve bundle. `put_args` and `get_args` config options can be used to pass additional params to the binaries.
Test Plan:
Change `test-infinitepush-bundlestore.t` test file to point to external store and run it
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803936
Tasks: 12479677
Signature: t1:3803936:1473123987:c34987ee459b2a66ee831574a32cdc8231043cb2
Summary: Add exceptions that will be thrown from bundle stores
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803928
Tasks: 12479677
Signature: t1:3803928:1473073448:32f6191ea0548c3d83cbaad100b6c8c48cbd8131
Test Plan: No need to test - comment change
Reviewers: rmcelroy, durham, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3791555
Tasks: 12479677
Summary: --create and --to options work with non-scratch bookmarks too
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, durham, rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797183
Tasks: 12479677
Summary: It's safer to don't have bundlestore then to have uninitialized bundlestore
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, rmcelroy, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797175
Tasks: 12479677
Summary:
Copy of hackbranch hackathon project with a few changes.
The goal is to allow saving commits under certain bookmarks in the separate store. These bookmarks are called 'scratch' bookmarks and are defined in config file (for example, 're:scratch/.+').
Scratch commits are sent in a separate bundle2 part during push. During pull new bundle repo is created and requests are served using this bundle repo.
There are a few changes from hackathon project implementation.
# It requires no changes in mercurial core. It was done by wrapping `findcommonincoming()` function and changing `lookup()` wireproto method (see below).
# Instead of introducing `listkeypattern()` wireproto method, `lookup()` method and `pull()` command were wrapped. I did it because I thought that it will make code easier. It probably haven't made it easier, it's almost the same. I can change it back to `listkeypattern` if it's necessary.
The bad thing about current implementation is that scratch bookmarks are not downloaded. But in the previous implementation also haven't downloaded bookmark if remotenames is enabled.
There are a couple of problems:
1) As I mentioned before, scratch bookmarks are not downloaded
2) It's impossible to pull both scratch and non-scratch bookmark
Test Plan: python ../../../hg-crew/tests/run-tests.py test-bundlestore.t
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3750999
Tasks: 12479677
Signature: t1:3750999:1472052396:5d7828566863f0aa437cdc34439ec0759503c6c9
Summary:
Code was written by Laurent, Durham and Mateusz on hackathon.
Index stores metadata, store stores bundles.
This diff adds interfaces and simple filesystem implementations
Test Plan:
cd tests
py.test *.py
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3741963
Tasks: 12479677
Signature: t1:3741963:1472207457:c9e7fb6163ef3cd6fdc1c4bd5113c623169c3a7c
Summary: I'll later add most of your hackhaton hackbranch code to this project with a few small fixes
Test Plan: No need
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: rmcelroy, mitrandir
Subscribers: quark, rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D3736089
Signature: t1:3736089:1471876504:98c4efc3bdf66d60f98eed11a3f13bae56fdc716
