Summary: Infinitepush didn't clean up temp files during pulls.
Test Plan:
Choose any hg server that's not in vip (for example, hg014.lla2).
Run `hg pull ssh://hg014.lla2.facebook.com//data/scm/fbsource -r INFINITEPUSH_COMMIT_HASH`
Check that new /tmp/tmp* file has appeared.
Install fixed infinitepush extension on the server.
Run the same command again, make sure it works fine.
Note: no new unittests, because temp filenames are random and that's difficult to test.
Reviewers: #mercurial, dschatzberg, tja
Reviewed By: tja
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5077767
Tasks: 17599765
Signature: t1:5077767:1495025042:d209d1e5342442ac8b05a4c80748780bbf36846c
Summary:
`bundlefile` variable wasn't initialized if `head in nodestobundle` were true.
Even worse, the previous version of `bundlefile` were used. That means that in
some cases infinitepush sends back a wrong bundle.
Adding a unittest is quite tricky because problem shows up only in one rare
case. Example: we have a stack of 2 commits A and B and each commit in the
stack has a bookmark that points to it. Then `getbundlechunks()` `heads`
parameter contains two heads: commit A and commit B in undefined order.
If it happens that first element in the list is commit A we'll create a bundle
that contains only commit A. And we'll reuse the same bundle when we process
commit B. That means that we'll send the same bundle with commit A twice,
but no bundles with commit B.
I refactored the code to make sure we won't get the same bug in
the future.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4969547
Tasks: 17719083, 15389402
Signature: t1:4969547:1493749313:587f9e4446a3c21b47c081a0fe4cd9e200dab5cd
Summary:
`exchange.readbundle()` can return bundle2 unbundler or changegroup cg1unpacker.
In case of cg1unpacker let's just read it from the stream
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4913313
Tasks: 15389402
Signature: t1:4913313:1492623174:72abe4d1e449ae31d78a6c98b554c0406e8a2ea2
Summary:
Profiles show that rebundling is the slowest part in infinitepush that can
take up to 10 secs to run. In many cases rebundling is not needed at all.
For example, if bundle contains only one head and that head was requested
during pull then rebundling can be avoided at all.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4834982
Tasks: 15389402
Signature: t1:4834982:1491450862:fe6984e6384676bcbab1c3fb9f0a47cb93902d0d
Summary: Let's add a table that will store metadata about infinitepush nodes.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4778867
Tasks: 16899722
Summary:
Previously we don't set findcommonincoming to True, and that bug was unnoticed
because we send this info anyway in server-side `getbundlechunks()` function.
But next diff in the stack uses a fast path server-side which won't work if
fincommonincoming is not set to True.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4770830
Tasks: 15389402
Signature: t1:4770830:1490719209:c251002d992e244580b7dbfeca0c30dff95734d2
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Finally reuse the same bundlerepo.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612094
Tasks: 15389402
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Now close all bundlerepos at once not one by one. This is necessary because
bundlerepos will be reused.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612089
Tasks: 15389402
Signature: t1:4612089:1488501776:e9fc2863adebc69bf44fa5dcb1610612027508f6
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Test Plan: arc unti
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612073
Tasks: 15389402
Signature: t1:4612073:1487955965:03af1a25acc21d0b6d913a60dd5ef82fa7f033ad
Summary:
There are a couple of reasons to avoid using ui.username():
1) If config option ui.username is set on the server then it will use it
instead of the name of the user that does push
2) It prints confusing warning `no username found, using '....' instead`
3) In some cases it fails (probably because it calls socket.getfqdn() which
does network request).
Let's use simple util.getuser() function and set username to 'unknown' if
it fails.
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: azich, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4642457
Signature: t1:4642457:1488458040:5dfac435bf44dfa6ab7725d2f5800963e8c860ec
Summary:
Previously --list-remote wasn't user friendly. User have to always specify a
pattern, and specifying * just won't work because of bash globbing. Also
it wasn't possible to specify many patterns at once.
This diff allows specifying many patterns, it allows to not specify
patterns at all and also it allows to specify default scratch patterns to fetch
if no patterns were specified. This is useful if there are many auto-generated
bookmarks under the same scratch bookmark pattern.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4605548
Signature: t1:4605548:1487934838:066a12c28dda16fa8f90674d736d21345631ef7e
Summary:
hasscratchbookmarks is True if scratchbookmarks is not empty and vice-versa.
No need to use this variable
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4590917
Signature: t1:4590917:1487682597:c2a6e0accbd8e539a885dd11e6819986e57d24df
Summary:
listkeyspatterns support batching. Let's use and avoid making many network
calls to the hg server.
I had to modify tests because scratchbookmarks are passed to the listkeyspatterns,
and scratchbookmarks is dict and the order of the keys is unpredictable.
Because of it bundles from the server will be sent in unpredictable order, and log
output may be different.
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4590914
Signature: t1:4590914:1487788586:dd5ba0dd41b0cd19f790755d7b3c1fc5ba4ade61
Summary:
Let's pull node during update if not found locally.
This is a part of selectivepull functionality.
See remotenames extensions for details about selectivepull.
Test Plan: arc unit
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: indragie, mjpieters, sergeyb
Differential Revision: https://phabricator.intern.facebook.com/D4536129
Tasks: 12479658
Signature: t1:4536129:1486667537:3d1df30cb5d1db0dd7451756102ccafee20789d5
Summary:
Default lock timeout value is very small (around couple of secs).
Let's set it for something bigger by default
Test Plan: arc unit
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4435236
Tasks: 15389402
Signature: t1:4435236:1484880747:d46276a82782ef551cda0165dea686f267c13ff9
Summary: __init__.py became huge and I suggest to split it
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4383661
Tasks: 12479677
Signature: t1:4383661:1483659134:eaf3d7e217633895a65ad568831d9c21ea0f18d8
Summary:
`pushbackup` failed when there are hidden heads in the repo but no visible
heads (see test for example). This diff fixes it.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4365677
Tasks: 12479677
Signature: t1:4365677:1483465909:fd1d96211909d3eeccab4fd17a364af4a1997f85
Summary:
The '&>>' operator only works in linux, not OSX. So let's replace it with a more
verbose equivalent. 'man bash' says these two are equivalent.
Test Plan:
Running the tests normally on OSX don't actually catch this, because
this particular test is skipped because it doesn't see that evolve is installed.
We need to run the tests with:
PYTHONPATH=/opt/facebook/hg/lib/python2.7/site-packages/:/opt/homebrew/lib/python2.7/site-packages/ /opt/homebrew/opt/python27/bin/python2.7 ../../facebook-hg-rpms/hg-crew/tests/run-tests.py test-infinitepush-backup.t
To get coverage. And the test is now passing on OSX and linux.
Reviewers: stash, #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4379593
Signature: t1:4379593:1483488383:ce8a045bbc29d719eafa372db56453b83f1d8df8
Summary:
Default path was used during background backup even if non-default path was
passed on the command line. So `hg push somepath --background` is
equivalent to `hg push --background`. This is not correct and this diff fixes it
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4357295
Tasks: 12479677
Signature: t1:4357295:1482330125:d2b043f8035046ab43667b4387bf642e53261681
Summary:
As the name suggest it will restore backup made by `hg pushbackup`.
If user has only one backup for the `dest` repo then it will be restored.
But user may have backed up many local repos that points to `dest` repo.
These local repos may reside on different hosts or in different
repo roots. It makes restore ambiguous; `--reporoot` and `--hostname`
options are used to disambiguate.
Example situation:
1) User has only one laptop with mercurial repo `repo` that was cloned
from remote server. He or she run `hg pushbackup`. Then laptop breaks
and user gets a new one, clones the `repo` again and runs `hg restore`.
It automatically restores the backup.
2) User has devserver and laptop and backups were made from both.
Then if user decides to switch devserver and run `hg restore` on the new
devserver he or she has to specify `--hostname`.
Future plans:
1) Add `--user` option to make it possible to restore another user's backup
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4280832
Tasks: 12479677
Signature: t1:4280832:1481565335:2a21ceafa2ff80242076a79693046514434afb40
Summary:
`bundlerepo` should be closed to ensure that temp file is deleted.
`bundlerepo` creates temp file when bundle is compressed. This is *not* the
case in infinitepush (all our bundle are uncompressed). But seems that
upstream cg1unpacker.compressed() returns True even if bundle is uncompressed.
Also this diff makes `_getoutputbundleraw()` read output bundle in memory
instead of returning a generator. The reason for doing it is because generator
becomes invalid as soon as `bundlerepo` is closed.
This approach will obviously increase memory usage but it shouldn't be a
problem since bundles are small (no more than 10s Mb).
#thanks @quark for reporting it
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4327186
Tasks: 12479677
Signature: t1:4327186:1481738559:2c6b22c305da4d572da9de21dfdf1179f7281744
Summary:
infinitepush has been storing bundle1 with changegroups v1. Turned out that in
some cases generated bundles can become huge because v1 changegroups don't
support generaldelta. It makes sense to support all changegroup versions (v1 for BC,
v2 for new infinitepush bundles, and maybe later we'll switch to v3).
To do this we have to store bundle2 instead of bundle1 because bundle1
supports only changegroups v1.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, quark, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4299151
Tasks: 12479677
Signature: t1:4299151:1481564714:810be69447d0b35aa57328c60aab72ad374e994d
Summary:
Writing to bundlestore can take a lot of time. It doesn't make sense to do it
inside indexapi transaction
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4305903
Tasks: 12479677
Signature: t1:4305903:1481312165:6aca8c908367d9715f8ba3f5643cd56650b4456f
Summary:
@simon suggested to rename `debugbackup` because `debug-` prefix
implies that command is internal only and it's not safe for user to run it.
This is not the case for the backup because user can safely run it.
Test Plan: Run `test-infinitepush-*`
Reviewers: simon, rmcelroy, mitrandir, durham, mjpieters
Reviewed By: mjpieters
Subscribers: simon, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4292516
Tasks: 12479677
Signature: t1:4292516:1481213170:a03c3eaebd3459887bbb3ff1c80f20977139cc90
Summary:
`hg push` takes a lock because it updates phases and may receive other bundle2
parts from remote server. We don't need it in debugbackup. And since
debugbackup is intended to run in background we want to avoid taking locks.
Instead let's use raw `unbundle()` wireproto method and disable pushback
bundle2 parts.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4291740
Tasks: 12479677
Signature: t1:4291740:1481118369:1b20f771c5db6e4fed4fc18681cc66ef4e91dd3a
Summary:
We could've avoided creating this diff if bundlerepo supported many bundles.
Unfortunately it doesn't. Adding support requires changes in upstream mercurial and it can take a lot of time.
This implementation wraps `listkeys` to set phases properly and `exchange._changegrouppart` to send more
than one bundle at a time.
It has unsolved problem with phases. For example in this situation
# Publishing server creates commit A (it is marked as draft on the server).
# Client pulls commit A from publishing server (in this case commit A is marked on the client but it's still draft on the server).
# Another client make pull from scratch bookmark. Commit A will be draft on client because we delete publishing = True from listkeys.
We assume that this case is quite rare.
Test Plan: Run `test-infinitepush-*`
Reviewers: mitrandir, rmcelroy, quark, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4074889
Tasks: 12479677
Signature: t1:4074889:1480446669:97e7f1d8ad23d457d7984a4cde0efb2b6e89eb2e
Summary:
There are lots of breakages if upstream tests are run with infinitepush
enabled. This patch fixes one issue in test-pull-update.t test.
Test Plan:
Run `test-infinitepush-*`
Run test-pull-update.t with --extra-config-opt=extensions.infinitepush=PATH_TO_INFINITEPUSH.
Make sure that it works fine up to line 115 (later lines may still be broken)
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4250678
Tasks: 12479677
Signature: t1:4250678:1480689565:792f3d77e08c969a9ffed904815fe7e9aa72f61d
Summary:
There are lots of breakages if upstream tests are run with infinitepush.
It fixes test-default-push.t upstream test.
Test Plan:
Run `test-infinitepush-*`
Run `test-default-push.t` upstream test
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4250684
Tasks: 12479677
Signature: t1:4250684:1480689556:7d6bc4788cfebb5aa62c631a104e9d735eca3e47
Summary:
We want to backup bookmarks even if no new commits were added. To do this let's
save the hash of all the backuped bookmarks and backup only if saved hash is
different from the hash of the current bookmarks.
In the case when no new commits need to be backuped revs=['null'] is passed
(remotenames does the same in `push --delete`).
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4251589
Tasks: 12479677
Signature: t1:4251589:1480611524:6faaa9d6d1b1aba9f7a45a4bf9a4c0df3d039ebe
Summary:
Backuping changesets is not enougn for restore. We also want to backup heads
and bookmarks to completely capture the state of the repo.
This diff is the first step in implementing this functionality. It adds new
bundle2 part which contains encoded dict (simple json encoding is used).
If value in the dict is empty then key is the bookmark pattern to delete.
If value is not empty then key is the bookmark name to save and value is a
node hash. The reason to put them in the same part is to make it possible to
delete and insert into indexapi in one transaction. It's also possible to pass
patterns to delete in part parameters but there is a bug in upstream hg that
limits parameters' size to 256 and we can potentially have longer bookmarks.
Local bookmarks are saved in infinitepush in the following form:
infinitepush/backups/USERNAME/HOSTNAME/REPOROOT/bookmarks/LOCAL_BOOKMARK_NAME
Local heads are saved in infintiepush in the following form:
infinitepush/backups/USERNAME/HOSTNAME/REPOROOTheads/HEAD_HASH
Hostname, username and repo root is necessary to distinguish different backups.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4245080
Tasks: 12479677
Signature: t1:4245080:1480518959:aa199d67fac4e2cd2f543651ff56fdd649dac729
Summary:
Not setting all default options may result in KeyValue errors.
For example pushvars extensions does
repo._shellvars = opts['pushvars']
It results in failure if 'pushvars' is not set.
Let's fix it by explicitly setting default values.
Also we need to set `allow_anon` to remotenames because otherwise
debugbackup will fail.
Test Plan:
Run `test-infinitepush-*`
Run `hg debugbackup` inside fbsource make sure there were no failures
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4237230
Tasks: 12479677
Signature: t1:4237230:1480446802:36e0630ce3e60c947be47d83bbc6deff8624048e
Summary:
Tests failed on mac. My guess is that timeout is too low.
Let's increase it to 10 seconds (it's fine because we want wait for 10 sec
if debugbackup finishes faster).
Also let's add logging
Test Plan: Run `test-infinitepush-*`
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4213330
Tasks: 12479677
Summary:
`backup` was renamed to `debugbackup` a few commits ago.
Rename it there too
Test Plan: Run `test-infinitepush-*`
Reviewers: #sourcecontrol, andrasbelo
Reviewed By: andrasbelo
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4212539
Tasks: 12479677
Signature: t1:4212539:1479721307:1ac25c7dc8951d655910e2b0c41b2c9555353f74
Summary:
`hg backup --background` will be used as a `txnclose` hook to backup all of the
local commits to infinitepush.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4175953
Tasks: 12479677
Signature: t1:4175953:1479145307:e698903b519361b376f6e182db7c49869c992617
Summary:
During `hg backup` bundle with many heads may be pushed. Let's support it too.
Many heads bundle is not allowed only when we are pushing it with `--to`
because in this case we don't know the node for the bookmark.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4175280
Tasks: 12479677
Signature: t1:4175280:1479231056:fb0c47a7752319d77a6cfc83a29c57e9e2dced16
Summary:
Backups all new non-extinct [1] commits to bundlestore.
When it is called for the first time `hg backup` will backup all draft
visible commits. Next backups will save only new commits since the last
backup (it is recorded in `.hg/store/infinitepushbackuptip`[2]).
It's an initial implementation. Later the following features will be added:
1) It will be called automatically whenever user creates or strips commits
or even creates or deletes bookmarks
2) It will also save all local bookmarks and all local heads
(probably only visible).
Note: calling `pushcmd` directly does not set default values for opts. That means that `--to` will be None and `_scratchbranchmatcher` will throw exception. Let's add a check to ensure that `--to` is never None.
[1] I also want to backup extinct commits (i.e. obsolete invisible commits).
But it will require bigger changes in discovery algorithm, so I'd leave it
for later.
[2] The name is a bit verbose. But I want to keep `infinitepush` part to make
it easier to debug problems.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4147651
Tasks: 12479677
Signature: t1:4147651:1479229440:3eb38880c14f18e9a2fb4eaba44bedf079bca506
