It seems `debugcheckbackup` could get a new kind of error with the latest
upstream code:
Traceback (most recent call last):
File "hg/mercurial/scmutil.py", line 150, in callcatch
return func()
File "hg/mercurial/dispatch.py", line 292, in _runcatchfunc
return _dispatch(req)
File "hg/mercurial/dispatch.py", line 896, in _dispatch
cmdpats, cmdoptions)
File "hg/mercurial/dispatch.py", line 658, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "hg/mercurial/dispatch.py", line 904, in _runcommand
return cmdfunc()
File "hg/mercurial/dispatch.py", line 893, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "hg/mercurial/extensions.py", line 322, in closure
return func(*(args + a), **kw)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "hg/hgext/mq.py", line 3542, in mqcommand
return orig(ui, repo, *args, **kwargs)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "fb-hgext/infinitepush/backupcommands.py", line 272, in checkbackup
if not _dobackupcheck(bkpstate, ui, repo, dest, **opts):
File "fb-hgext/infinitepush/backupcommands.py", line 469, in _dobackupcheck
for r in lookupresults:
File "hg/mercurial/wireproto.py", line 137, in results
next(batchable)
File "hg/mercurial/wireproto.py", line 238, in lookup
self._abort(error.RepoError(data))
File "hg/mercurial/sshpeer.py", line 235, in _abort
raise exception
RepoError: 0e1a088ff2825213eaa838a82a842bc186f10dd5
abort: 0e1a088ff2825213eaa838a82a842bc186f10dd5!
The error message only contains a commit hash, which is not friendly to be
printed out. Therefore make sure we have control about the error message and
update the test accordingly.
Differential Revision: https://phab.mercurial-scm.org/D485
Summary:
The first version of infinitepush smartlog summary included some false
positives (in particular, not-backed-up but obsolete revs), and took a long
time to compute the set if there were many heads in the repo.
Improve the revset that is used by taking into account the heads that *should*
be backed up. Ancestors of these heads, that are not ancestors of the actual
backup heads are the ones that should be considered "not backed up".
Test Plan:
Re-run unit tests.
Manual test on devvm.
Perf test against stash's repo which has more heads than mine.
Reviewers: #fbhgext, stash
Differential Revision: https://phab.mercurial-scm.org/D475
Summary:
--list-remote can return too many nodes if used without a pattern.
Let's require a pattern.
Test Plan: Run unittests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D221
Summary:
- change the infinitepush extension to record additional state when the a backup occurs.
- useful for other tools (scm-prompt, nuclide) to indicate when a backup hasn't occurred recently
Test Plan:
- cd facebook-hg-rpms/
- source hg-dev
- cd fb-hgext
- arc unit
Reviewers: rmcelroy, stash
Reviewed By: stash
Subscribers: rmcelroy, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5321221
Tasks: 18349390
Signature: t1:5321221:1501271966:6698785e0714e3ecf7e1e2684d1e62e52ac0f1a0
Summary:
Allow summarisation of commits that have not been backed up
by infinitepush.
1. Add a template keyword (backupstatus) which evaluates to
one of "Public", "Backed up", "Backup pending", or
"Not backed up" depending on the backup state of the
commit. Commits are pending for 10 minutes, after which
they are declared not backed up.
2. Add a summary to the end of smartlog that shows the
number of commits that are not backed up.
Configuration options allow the addition of an education
message to inform users what to do about their failing
backups, and the message can also be suppressed.
Test Plan:
New unit test for both the keyword and the summary.
Reviewers: #fbhgext, stash
Subscribers: mitrandir, quark, akushner
Differential Revision: https://phab.mercurial-scm.org/D184
Summary:
This option was intended as temporary and it was enabled for long enough time.
Now let's delete it.
Test Plan: Run unit-tests
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D195
Summary:
There was a report from an infinitepush user who complained about
`hg update remote/scratch/book` failing to pull the bookmark, while
`hg update scratch/book` pulling fine. It confuses users because they see
`remote/scratch/book` in smartlog.
On the other hand let's not allow `hg pull -B remote/scratch/book` to stay
consistent with remotenames.
Test Plan: Slowly run tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D172
Summary:
We've seen problems when public nodes were pushed to the server.
While the real reason is unknown it's a good idea to add a safety net to prevent
sending too many commits to the server because it's usually means that we are
pushing public commits. We limit number of nodes to 1000
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D143
Summary:
We've had problems when users sent huge bundles to the server.
Let's limit the size of the bundle.
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D142
Summary:
Infinitepush has too many options and many of them are unused or has never
been changed from the default value. Let's remove some of them.
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D141
Summary:
D17 was reverted because it broke creating bundle for non-general delta repos.
The reason is the following: D17 made infinitepush extension override
changegroup.supportedoutgoingversion() function and discard '01' version.
For non-general delta repositories it resulted in broken `hg bundle ...`
command.
`abort: repository does not support bundle version 01`
This diff fixes it by not overriding supportedoutgoingversion(). Instead
getscratchbranchpart() has its own logic of selecting changegroup version.
Test Plan:
Run unit-test
Run `hg bundle -r . --base .^ somefile` in non-general delta repo,
make sure it works fine.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D81
This is the result of running a codemod script:
python ~/hg/contrib/codemod/codemod_nestedwith.py **/*.py
Plus a manual cleanup that removes the comment in `absorb/__init.py`.
See D76 for the codemod script.
Differential Revision: https://phab.mercurial-scm.org/D78
Summary:
Upstream has added warnings when writing files outside the wlock, so let's put
this write in the wlock.
Test Plan: The tests now pass
Reviewers: #fbhgext, stash
Reviewed By: #fbhgext, stash
Subscribers: stash, simonfar
Differential Revision: https://phab.mercurial-scm.org/D104
Summary:
Upstream has added devel warnings when writing store files outside of a lock.
Certain files allow this though and there's a whitelist of which files are
allowed. Since backups happen behind the scenes and use their own lock, it's
safe to add our files to that list.
Test Plan: Ran the tests against latest upstream hg. They now pass.
Reviewers: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D87
Summary: This broke hg bundle on non-generaldelta repositories.
Test Plan: Ran the tests
Reviewers: #fbhgext, stash, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D74
Summary:
Make sure that infinitepush work fine with lfs. To achieve it we try to use
changegroup3 if it's available and also run prepush hook before infinitepush.
Test Plan: Run `arc unit`
Reviewers: #fbhgext, quark, dsp
Reviewed By: #fbhgext, dsp
Differential Revision: https://phab.mercurial-scm.org/D17
Summary:
debuginhibit was removed since we no longer have a separate inhibit state.
smartlog hack about changing "o" to "x" was removed since "obsolete()"
revset is correct and the hack is unnecessary now.
directaccess was removed from tests since inhibit does not depend on it.
`- obsolete()` was added to some revsets to avoid divergence and other
surprises.
Use `inhibit.revive` API in infinitepush and reset to revive changesets
properly.
Remove various hacky code that mangle inhibit state in corner cases.
Most test changes are `o` changed to `x` in output since we draw
`obsolete()` state correctly now. `test-infinitepush-backup-remotefilelog.t`
change was because output could be `bytes/sec` instead of `KB/sec`.
Test Plan: arc unit
Reviewers: #mercurial, kulshrax
Reviewed By: kulshrax
Subscribers: kulshrax, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5391361
Signature: t1:5391361:1499722618:d3c1cf629f0c59ecdf1dfd5e653c1eb6176646b8
Summary: It's a useful piece of information for Mononoke (see attached task).
Test Plan: arc unit
Reviewers: #mercurial, jsgf
Reviewed By: jsgf
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5310026
Tasks: 18812931
Signature: t1:5310026:1498234999:0c06a30bd659d6a1d3cedc28252f17dd1fc93dc8
Summary:
Previously pushbackups failed when it tried to unlink non-existent infinitepush
backup state file. This diff fixes it.
Test Plan: arc unit
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: medson, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5309684
Signature: t1:5309684:1498210504:49af716dbb4e7e1875e8a2b7ee4b40fe28ceefc1
Summary: Currently seconds are sent to logging for elapsed time. Given the that most values are 0, 1, 2, or 3 seconds, this change adds some better signal. Not reusing the elapsed field, instead introduced elapsed ms.
Test Plan:
arc unit
modified test-infinitepush-backup-logging.t to include the new elapsedms field
Reviewers: stash
Reviewed By: stash
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5292576
Tasks: 19557691
Signature: t1:5292576:1498059638:8461a71398a41bf741384ddcda15d5c016633d66
Summary:
Final part: launching debugfillinfinitepushmetadata in the background. We
intentionally don't wait for it's completion in order not to slow down pushes.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mitrandir, quark, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5265325
Tasks: 19103180
Signature: t1:5265325:1497886939:6c306c08b240373ea43a7fc868b17ef776258cf9
Summary:
Add an option to limit the number of file entries in the metadata field. This
is to avoid huge codemods to eat up all the space.
Depends on D5255883
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5264202
Tasks: 19103180
Signature: t1:5264202:1497630109:05d684db34578e309e0bbad314165e66d1f23ce9
Summary:
We may want to fill infinitepush metadata for many commits at once.
Let's make it possible to pass many nodes to debugfillinfinitepushmetadata
Depends on D5255670
Test Plan: arc unit
Reviewers: #mercurial, mitrandir, quark
Reviewed By: quark
Subscribers: mitrandir, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5255883
Tasks: 19103180
Signature: t1:5255883:1497629860:26332e3cc115881bca158664014ce48087d99215
Summary:
Also need to use `with index` to make sure that transaction closes.
Depends on D5255664
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mitrandir, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5255670
Tasks: 19103180
Signature: t1:5255670:1497557000:906a39810931d92f30c6d6b8e5bfc28e0c888594
Summary:
I've forgot to add it in the previous diff. We also want to know copies
information and status (added, modified, removed).
Test Plan: arc unit
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, medson, azich, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5255664
Tasks: 19103180
Signature: t1:5255664:1497542963:b552c07d261067f426d7af7a29d2f02c1917394c
Summary:
Special command that fills optional metadata about infinitepush nodes. For now
it's info about changed files + info about how many lines were
added/removed. Since information is saved in json format it's easily
extensible.
It's a separate command so that we will be able to run it asynchronously:
during infinitepush push separate `hg debugfillinfinitepushmetadata` process
will be started. Push process won't wait for `debugfillinfinitepushmetadata`
process.
sqlindexapi will be implemented in the next diffs.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, medson, azich, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5237547
Tasks: 19103180
Signature: t1:5237547:1497397211:569d1ffe3be37c8d14fb7e07e42cd085ad5686d4
Summary:
Previously infinitepush lookup function didn't call original lookup function.
It made it impossible to work with extension like gitlookup that also
overwrite lookup. This diff fixes it.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5246683
Tasks: 17733144
Summary:
Let's add a special field that would store arbitrary metadata in json format.
It can be used to store, for example, lists of changes files, file diffs etc.
Depends on D5236375
Test Plan: arc unit
Reviewers: #mercurial, azich
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5236394
Tasks: 19103180
Summary:
Remotenames can return a name that has no nodes (it may exist in the remotename
store, but when we go to resolve the node the node doesn't exist). So we need to
check for existence after fetching the nodes.
Test Plan: This was crashing for a user. After the fix they no longer crash.
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D5194317
Signature: t1:5194317:1496847686:ab05335284b83a2060c2bfea4beec6bf0453905c
Summary:
There is a problem with socket.gethostname() because it can return
inconsistent names. See attached task for details.
Instead this diff lets us specify any hostname.
Depends on D5180633
Test Plan: arc unit
Reviewers: #mercurial, simpkins, tja
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5180665
Tasks: 18528165
Summary:
Previously there were many small functions that were used to create backup
bookmark name. Changing the logic was quite onerous because one would need to
change function signature in a few places.
This diff introduces special class that incapsulates this behaviour.
With this class making changes to naming scheme would be less problematic.
Depends on D5180390
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5180633
Tasks: 18528165
Summary:
Let's add an option that can be used to trigger full clean backup.
It can be useful in many situations, for example, in case of bug in
infinitepush backup or if the naming scheme of backup bookmarks were changed.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5180390
Tasks: 15389402
Summary:
Previously `_getrevstobackup()` would load manifest for each commit in memory
only to check if a file was deleted in this commit or not. Manifest can be
quite big and since every loaded flatmanifest is cached in fastmanifest then
memory usage can be huge.
Since we are doing this only to check if there are any commits that were
downloaded to the client without filelogs and then were stripped server-side.
This cases are rare and it would be easier to track these commits in the
config file.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5111372
Tasks: 15389402
Signature: t1:5111372:1495848867:4028bd48313ac0e2022c3695195bedc6eda80abf
Summary: Upstream has deprecated cmdutil.commands() in favor of registrar.commands()
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5106486
Signature: t1:5106486:1495485074:0e20f00622cc651e8c9dda837f84dd84cc51099e
Summary:
One of the heads may point to filtered commit and isbackedup command fails in
this case. Let's use repo.unfiltered()
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5001977
Signature: t1:5001977:1493907725:f6a138bd4e2cae48b64152fceaf73660ea91f43b
Summary: Infinitepush didn't clean up temp files during pulls.
Test Plan:
Choose any hg server that's not in vip (for example, hg014.lla2).
Run `hg pull ssh://hg014.lla2.facebook.com//data/scm/fbsource -r INFINITEPUSH_COMMIT_HASH`
Check that new /tmp/tmp* file has appeared.
Install fixed infinitepush extension on the server.
Run the same command again, make sure it works fine.
Note: no new unittests, because temp filenames are random and that's difficult to test.
Reviewers: #mercurial, dschatzberg, tja
Reviewed By: tja
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5077767
Tasks: 17599765
Signature: t1:5077767:1495025042:d209d1e5342442ac8b05a4c80748780bbf36846c
Summary:
`bundlefile` variable wasn't initialized if `head in nodestobundle` were true.
Even worse, the previous version of `bundlefile` were used. That means that in
some cases infinitepush sends back a wrong bundle.
Adding a unittest is quite tricky because problem shows up only in one rare
case. Example: we have a stack of 2 commits A and B and each commit in the
stack has a bookmark that points to it. Then `getbundlechunks()` `heads`
parameter contains two heads: commit A and commit B in undefined order.
If it happens that first element in the list is commit A we'll create a bundle
that contains only commit A. And we'll reuse the same bundle when we process
commit B. That means that we'll send the same bundle with commit A twice,
but no bundles with commit B.
I refactored the code to make sure we won't get the same bug in
the future.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4969547
Tasks: 17719083, 15389402
Signature: t1:4969547:1493749313:587f9e4446a3c21b47c081a0fe4cd9e200dab5cd
Summary:
As @quark suggested, it's better to use `infinitepush` path instead of
default-push, because `default-push` is facebook-specific configuration.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: quark, rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4962324
Tasks: 15389402
Signature: t1:4962324:1493727112:35c45c57a527d2de5a35ea83e1031dc1908ac28e
Summary:
Command that checks if a commit was backed up.
It does it by checking if revisions are in local backup state.
Test Plan: Run unit tests and lints
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4938072
Tasks: 17519836
Signature: t1:4938072:1493072854:291765bb59d327db8504feb47d6089818ae1e11a
Summary:
infinitepushbackup.tempcleanworkingcopiesbackups option sends special command
to clean backup bookmarks from the server for the shared working copy (i.e.
if we have main repo `fbsource` and shared working copy `fbsource2` we want
to have backups only for `fbsource` not for `fbsource` and `fbsource2`).
Before this diff cleanup commands were send for every backup.
This diff makes them send only if backup state file is present.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4929489
Signature: t1:4929489:1493075761:a408300338a10864043b538540d03880a49c4e1a
Summary:
There was a bug that caused commits with file deletions to not be backed up.
This diff fixes it by first checking if file exists in the commit and only
then downloading context of the file.
Note:
In tests I had to ignore stdout of `hg pushbackup` because the output was
different on macs and linux.
On linux there was an additional line
remote: abort: data/committostripfirst.i@091b63e5e4: no match found!
Probably mac's remotefilelog closes stdout/stderr earlier, but I wasn't
able to find a root cause.
Test Plan: Run unit tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4929244
Tasks: 15389402
Signature: t1:4929244:1492791804:77b2baa9eb54a53120a955e72e6c132be5db6b44
Summary:
`exchange.readbundle()` can return bundle2 unbundler or changegroup cg1unpacker.
In case of cg1unpacker let's just read it from the stream
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4913313
Tasks: 15389402
Signature: t1:4913313:1492623174:72abe4d1e449ae31d78a6c98b554c0406e8a2ea2
Summary:
Previously we had separate backup per working copy. That's very confusing
since all these working copies shares the same repo. This diff fixes it and
also adds config option to clean unnecessary working copy server-side.
Test Plan: Run infinitepush unittest
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: rmcelroy, quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4876230
Signature: t1:4876230:1492025747:3579e5046efc2ed309044fc3335c36ac4f7bdd04
