This makes the client side pushes also include tree packs in the push.
Also adds a test to see that the tree parts round trip between two tree-only
clients.
This was reverted in D921 because clients couldn't push old non-tree commits to
infinitepush anymore. I've fixed the issue and added a test to this commit.
Differential Revision: https://phab.mercurial-scm.org/D1017
Summary:
Instead of making an autobackup, make hg pullbackup simply
write the backup state from the recently restored backup.
Test Plan:
cd ~/facebook-hg-rpms/fb-hgext/tests
source ../../hg-dev
rt test-infinitepush-*.t
Reviewers: #mercurial, cdown, stash
Reviewed By: stash
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D6186148
Tasks: T22856338
Tags: bootcamp
Signature: 6186148:1509457565:e66cefc6e95d9b1cfb91e9bf8fa5c40b05e4fb82
With shared working directories, infinitepush must always read the backup state
from the source repo. Currently it does that for pushing backups, but not for
commands that just query the backup state.
Differential Revision: https://phab.mercurial-scm.org/D1052
Summary:
This diff caused a regression - clients may not have a tree manifests entries
for old commits. That causes infinitepush fail with
```
KeyError: ('', 'f6db63080ed0d9a81adf2cb266646cf5cf73bb79')
```
Test Plan: Run infinitepush with this diff reverted, make sure it doesn't fail
Reviewers: #fbhgext, ryanmce, durham
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D921
This makes the client side pushes also include tree packs in the push.
Also adds a test to see that the tree parts round trip between two tree-only
clients.
Differential Revision: https://phab.mercurial-scm.org/D721
Upstream Mercurial now has a function that handles the iteration over bundle
parts. As part of making infinitepush handle more part types, let's override
processparts and completely take over part processing.
Initially the part processing will just mimic the existing handlers, but there's
a config flag that causes it to simply redirect the received parts into the new
bundle. This will be useful later for storing all bundle parts (like tree packs
and obsmarkers).
Differential Revision: https://phab.mercurial-scm.org/D717
The notbackedup revset predicate still has performance problems when the set of
not-backed-up changesets includes some that are completely disjoint from the
set of backed-up changesets.
Rather than trying to express it as a combination of other expressions,
implement the revset calculation directly. This allows us to also stop as soon
as a public commit is encountered, which should be faster in all cases.
Test Plan:
Expand tests to cover more complex scenarios. Test perf on a larger and older
checkout of a big repo.
Differential Revision: https://phab.mercurial-scm.org/D764
Summary:
Previously we ran auto backups in txnclose hook. But commands like histedit and
rebase create many transactions, and that means that lots of backup processes
are started at the same time. That means that sometimes backup processes use
too much memory.
Instead let's start it once per command if any transaction was opened
during the command.
Test Plan:
Run unit-test.
Run histedit with autobackup enabled, and make sure just one backup process
was started.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D678
Summary:
The infinitepush backup smartlog summary uses only(a, b) to determine which
revisions have not been backed up. However, if the set of backup heads is
empty, only() behaves as if it were called as only(a), which is wrong and slow.
Instead, use ::a - ::b, which is always correct, even if b is empty.
Test Plan: Re-run UTs. Test perf on a repo with no backup heads.
Reviewers: #fbhgext, stash
Differential Revision: https://phab.mercurial-scm.org/D676
It seems `debugcheckbackup` could get a new kind of error with the latest
upstream code:
Traceback (most recent call last):
File "hg/mercurial/scmutil.py", line 150, in callcatch
return func()
File "hg/mercurial/dispatch.py", line 292, in _runcatchfunc
return _dispatch(req)
File "hg/mercurial/dispatch.py", line 896, in _dispatch
cmdpats, cmdoptions)
File "hg/mercurial/dispatch.py", line 658, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "hg/mercurial/dispatch.py", line 904, in _runcommand
return cmdfunc()
File "hg/mercurial/dispatch.py", line 893, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "hg/mercurial/extensions.py", line 322, in closure
return func(*(args + a), **kw)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "hg/hgext/mq.py", line 3542, in mqcommand
return orig(ui, repo, *args, **kwargs)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "fb-hgext/infinitepush/backupcommands.py", line 272, in checkbackup
if not _dobackupcheck(bkpstate, ui, repo, dest, **opts):
File "fb-hgext/infinitepush/backupcommands.py", line 469, in _dobackupcheck
for r in lookupresults:
File "hg/mercurial/wireproto.py", line 137, in results
next(batchable)
File "hg/mercurial/wireproto.py", line 238, in lookup
self._abort(error.RepoError(data))
File "hg/mercurial/sshpeer.py", line 235, in _abort
raise exception
RepoError: 0e1a088ff2825213eaa838a82a842bc186f10dd5
abort: 0e1a088ff2825213eaa838a82a842bc186f10dd5!
The error message only contains a commit hash, which is not friendly to be
printed out. Therefore make sure we have control about the error message and
update the test accordingly.
Differential Revision: https://phab.mercurial-scm.org/D485
Summary:
The first version of infinitepush smartlog summary included some false
positives (in particular, not-backed-up but obsolete revs), and took a long
time to compute the set if there were many heads in the repo.
Improve the revset that is used by taking into account the heads that *should*
be backed up. Ancestors of these heads, that are not ancestors of the actual
backup heads are the ones that should be considered "not backed up".
Test Plan:
Re-run unit tests.
Manual test on devvm.
Perf test against stash's repo which has more heads than mine.
Reviewers: #fbhgext, stash
Differential Revision: https://phab.mercurial-scm.org/D475
Summary:
- change the infinitepush extension to record additional state when the a backup occurs.
- useful for other tools (scm-prompt, nuclide) to indicate when a backup hasn't occurred recently
Test Plan:
- cd facebook-hg-rpms/
- source hg-dev
- cd fb-hgext
- arc unit
Reviewers: rmcelroy, stash
Reviewed By: stash
Subscribers: rmcelroy, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5321221
Tasks: 18349390
Signature: t1:5321221:1501271966:6698785e0714e3ecf7e1e2684d1e62e52ac0f1a0
Summary:
Allow summarisation of commits that have not been backed up
by infinitepush.
1. Add a template keyword (backupstatus) which evaluates to
one of "Public", "Backed up", "Backup pending", or
"Not backed up" depending on the backup state of the
commit. Commits are pending for 10 minutes, after which
they are declared not backed up.
2. Add a summary to the end of smartlog that shows the
number of commits that are not backed up.
Configuration options allow the addition of an education
message to inform users what to do about their failing
backups, and the message can also be suppressed.
Test Plan:
New unit test for both the keyword and the summary.
Reviewers: #fbhgext, stash
Subscribers: mitrandir, quark, akushner
Differential Revision: https://phab.mercurial-scm.org/D184
Summary:
This option was intended as temporary and it was enabled for long enough time.
Now let's delete it.
Test Plan: Run unit-tests
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D195
Summary:
We've seen problems when public nodes were pushed to the server.
While the real reason is unknown it's a good idea to add a safety net to prevent
sending too many commits to the server because it's usually means that we are
pushing public commits. We limit number of nodes to 1000
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D143
Summary:
Infinitepush has too many options and many of them are unused or has never
been changed from the default value. Let's remove some of them.
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D141
This is the result of running a codemod script:
python ~/hg/contrib/codemod/codemod_nestedwith.py **/*.py
Plus a manual cleanup that removes the comment in `absorb/__init.py`.
See D76 for the codemod script.
Differential Revision: https://phab.mercurial-scm.org/D78
Summary:
Previously pushbackups failed when it tried to unlink non-existent infinitepush
backup state file. This diff fixes it.
Test Plan: arc unit
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: medson, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5309684
Signature: t1:5309684:1498210504:49af716dbb4e7e1875e8a2b7ee4b40fe28ceefc1
Summary:
There is a problem with socket.gethostname() because it can return
inconsistent names. See attached task for details.
Instead this diff lets us specify any hostname.
Depends on D5180633
Test Plan: arc unit
Reviewers: #mercurial, simpkins, tja
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5180665
Tasks: 18528165
Summary:
Previously there were many small functions that were used to create backup
bookmark name. Changing the logic was quite onerous because one would need to
change function signature in a few places.
This diff introduces special class that incapsulates this behaviour.
With this class making changes to naming scheme would be less problematic.
Depends on D5180390
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5180633
Tasks: 18528165
Summary:
Let's add an option that can be used to trigger full clean backup.
It can be useful in many situations, for example, in case of bug in
infinitepush backup or if the naming scheme of backup bookmarks were changed.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5180390
Tasks: 15389402
Summary:
Previously `_getrevstobackup()` would load manifest for each commit in memory
only to check if a file was deleted in this commit or not. Manifest can be
quite big and since every loaded flatmanifest is cached in fastmanifest then
memory usage can be huge.
Since we are doing this only to check if there are any commits that were
downloaded to the client without filelogs and then were stripped server-side.
This cases are rare and it would be easier to track these commits in the
config file.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5111372
Tasks: 15389402
Signature: t1:5111372:1495848867:4028bd48313ac0e2022c3695195bedc6eda80abf
Summary: Upstream has deprecated cmdutil.commands() in favor of registrar.commands()
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5106486
Signature: t1:5106486:1495485074:0e20f00622cc651e8c9dda837f84dd84cc51099e
Summary:
One of the heads may point to filtered commit and isbackedup command fails in
this case. Let's use repo.unfiltered()
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5001977
Signature: t1:5001977:1493907725:f6a138bd4e2cae48b64152fceaf73660ea91f43b
Summary:
As @quark suggested, it's better to use `infinitepush` path instead of
default-push, because `default-push` is facebook-specific configuration.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: quark, rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4962324
Tasks: 15389402
Signature: t1:4962324:1493727112:35c45c57a527d2de5a35ea83e1031dc1908ac28e
Summary:
Command that checks if a commit was backed up.
It does it by checking if revisions are in local backup state.
Test Plan: Run unit tests and lints
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4938072
Tasks: 17519836
Signature: t1:4938072:1493072854:291765bb59d327db8504feb47d6089818ae1e11a
Summary:
infinitepushbackup.tempcleanworkingcopiesbackups option sends special command
to clean backup bookmarks from the server for the shared working copy (i.e.
if we have main repo `fbsource` and shared working copy `fbsource2` we want
to have backups only for `fbsource` not for `fbsource` and `fbsource2`).
Before this diff cleanup commands were send for every backup.
This diff makes them send only if backup state file is present.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4929489
Signature: t1:4929489:1493075761:a408300338a10864043b538540d03880a49c4e1a
Summary:
There was a bug that caused commits with file deletions to not be backed up.
This diff fixes it by first checking if file exists in the commit and only
then downloading context of the file.
Note:
In tests I had to ignore stdout of `hg pushbackup` because the output was
different on macs and linux.
On linux there was an additional line
remote: abort: data/committostripfirst.i@091b63e5e4: no match found!
Probably mac's remotefilelog closes stdout/stderr earlier, but I wasn't
able to find a root cause.
Test Plan: Run unit tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4929244
Tasks: 15389402
Signature: t1:4929244:1492791804:77b2baa9eb54a53120a955e72e6c132be5db6b44
Summary:
Previously we had separate backup per working copy. That's very confusing
since all these working copies shares the same repo. This diff fixes it and
also adds config option to clean unnecessary working copy server-side.
Test Plan: Run infinitepush unittest
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: rmcelroy, quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4876230
Signature: t1:4876230:1492025747:3579e5046efc2ed309044fc3335c36ac4f7bdd04
Summary: Check owner of a log file before writing to it. See comments for details
Test Plan: arc unit
Reviewers: #mercurial, simpkins
Reviewed By: simpkins
Subscribers: net-systems-diffs@fb.com, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4842933
Tasks: 17155924
Signature: t1:4842933:1491861137:e8a027fb7a930c0c5f553c75cb84214d24f66ce3
Summary:
It's a good way to check if backup is consistent. If backup is not consistent
then we set special exit code that will be logged. It let's us easily find users
with inconsistent backups and fix it.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4810442
Tasks: 15389402
Summary:
Give better names to the variables to underline what these variables really
store. And a bit of code movement.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4803405
Tasks: 15389402
Summary:
`headstobackup` won't point to bad nodes and they are only draft.
No need to check one more time
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4803376
Tasks: 15389402
Summary:
Previously we backed up extinct node if bookmark pointed to them. But this not
is not present in the bundle because `findcommonoutgoing()` intentionally skips
extinct and secret commits. In this case we'll have infinitepush bookmark that
points to non-existent infinitepush commit. To fix it let's filter extinct heads
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4803322
Tasks: 15389402
Summary:
Command to list available backups for the user.
--json will be used by automation
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4728729
Tasks: 15389402
Signature: t1:4728729:1490005762:41b9683cb7dfc9d84ae1032c807d1f0c3fe60dbf
Summary:
ui.shortuser() doesn't call util.shortuser() if ui.verbose equals to True.
But in our case it's very important to always use short username because
we'll get broken backups otherwise. Let's call util.shortuser() directly.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4721949
Tasks: 15389402
Signature: t1:4721949:1489717973:b7e59929f530578e060b8cdef94ef92b54fa2647
Summary:
On macos sshpeer flushes remote output later and it causes tests to fail. Let's
add a call to cleanup() to make sure the output is flushed.
This diff also makes remotefilelog test more robust.
Test Plan: Run tests on macos
Reviewers: #mercurial, tja, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4721276
Tasks: 15389402
Signature: t1:4721276:1489669109:4ce59f4a1d224d57dbb7c1eb341c4e6a659d2e8c
Summary:
By default debugcheckbackup checks only one backup for the user.
With --all option it checks all backups for the user from all host/reporoots
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700993
Tasks: 15389402
Signature: t1:4700993:1489548921:800d08f420acc8ef4f807ffd17b31837dbb3fe82
Summary:
Backing up very big repo can cause problems. And very often old commits are
not necessary at all. Let's add a config option to limit the number of heads
that are being backed up.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700965
Tasks: 15389402
Signature: t1:4700965:1489549163:4e2c121f01debd7b495486a1f3b062926873399d
Summary:
Previously we saved bookmark hash and revision number as state of
the last backup. Storing last backed up revision is error-prone. I saw
a few corrupted backups where bookmarks pointed to non-backed up
nodes. Also this approach puts more pressure on mysql bundle index.
On every backup it first deletes all existing backup bookmarks and then it
writes new bookmarks. Even if user changed one head or one bookmark
(and that is the usual case) current approach still deletes bookmarks first
and then rewrites them.
Instead let's store all backed up heads and bookmarks locally in json format.
Json was chosen because it's simple and human-readable. Then during backup
let's compare last backed up state to the current state of the repo, and send
to the server only the difference.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700930
Tasks: 15389402
Signature: t1:4700930:1489613249:a34369bf53e0718c22258304493dfa27b578857f
Summary:
Let's rename _getbackupstate() to _downloadbackupstate() because it actually
downloads it from the server. Also return class instead of tuple
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4713633
Tasks: 15389402
Signature: t1:4713633:1489612870:3439a6a99ef311bf96784b3b1e53d7e9b94c0713
