Treemanifest now has a unified spot to check if it can send trees. Infinitepush
needs to respect that, otherwise we're uploading trees to infinitepush that
might not be readable on other systems.
Summary:
Add support to hg book -d to delete scratch infinitepush bookmarks
Uses functions from remotenames to rewrite the remotenames cache omitting
the specified scratch bookmarks
Test Plan:
cd ~/facebook-hg-rpms/fb-hgext/tests
source ../../hg-dev
rt test-infinitepush-*.t --extra-config-opt=devel.all-warnings=False
Reviewers: #mercurial, cdown, stash, durham
Differential Revision: https://phabricator.intern.facebook.com/D6221853
Tasks: T22615396
Summary:
Since reponame is part of the primary key, it also makes sense to add it to
WHERE clause. Otherwise query may trigger full table scan and it can be very
slow.
Test Plan: Run unittest
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Differential Revision: https://phab.mercurial-scm.org/D1256
This makes the client side pushes also include tree packs in the push.
Also adds a test to see that the tree parts round trip between two tree-only
clients.
This was reverted in D921 because clients couldn't push old non-tree commits to
infinitepush anymore. I've fixed the issue and added a test to this commit.
Differential Revision: https://phab.mercurial-scm.org/D1017
Summary:
Instead of making an autobackup, make hg pullbackup simply
write the backup state from the recently restored backup.
Test Plan:
cd ~/facebook-hg-rpms/fb-hgext/tests
source ../../hg-dev
rt test-infinitepush-*.t
Reviewers: #mercurial, cdown, stash
Reviewed By: stash
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D6186148
Tasks: T22856338
Tags: bootcamp
Signature: 6186148:1509457565:e66cefc6e95d9b1cfb91e9bf8fa5c40b05e4fb82
In 272dd7d8a1bb1b2e93697b5b65183b08c8bf0853 upstream hg commit phase pushes
started to use separate bundle2 part. Because of that:
1) pushkey hook is not called when updating phases.
2) pushkey bundle2 part doesn't process phases anymore, 'phase-heads' part do
it instead.
It caused breakages in infinitepush, because phases handling is not skipped even during
infinitepush. This diff fixes it.
Test Plan:
Run infinitepush tests
Differential Revision: https://phab.mercurial-scm.org/D1225
With shared working directories, infinitepush must always read the backup state
from the source repo. Currently it does that for pushing backups, but not for
commands that just query the backup state.
Differential Revision: https://phab.mercurial-scm.org/D1052
Summary:
This diff caused a regression - clients may not have a tree manifests entries
for old commits. That causes infinitepush fail with
```
KeyError: ('', 'f6db63080ed0d9a81adf2cb266646cf5cf73bb79')
```
Test Plan: Run infinitepush with this diff reverted, make sure it doesn't fail
Reviewers: #fbhgext, ryanmce, durham
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D921
This makes the client side pushes also include tree packs in the push.
Also adds a test to see that the tree parts round trip between two tree-only
clients.
Differential Revision: https://phab.mercurial-scm.org/D721
When serving data from a stored bundle, if we need to rebundle the data first,
we need to make sure that the tree data stored within the bundle gets packaged
up as well.
If we're not rebundling, then things work fine as is since it just forwards the
parts along.
Differential Revision: https://phab.mercurial-scm.org/D720
Previously, when serving the contents of a stored bundle, we would explicitly
pick out the changegroup and serve only that. Let's also send along all other
parts that are stored in the bundle.
At the moment the only other parts that are stored in the bundle are the ones in
the partforwardingwhitelist in processparts, which is only treemanifest packs
right now. Later we'll serve bookmarks using this same pattern.
Differential Revision: https://phab.mercurial-scm.org/D719
Makes the infinitepush server store the received treepack parts in the bundle.
Future patches will enable serving the trees from the bundle, and sending the
trees from the client.
Differential Revision: https://phab.mercurial-scm.org/D718
Upstream Mercurial now has a function that handles the iteration over bundle
parts. As part of making infinitepush handle more part types, let's override
processparts and completely take over part processing.
Initially the part processing will just mimic the existing handlers, but there's
a config flag that causes it to simply redirect the received parts into the new
bundle. This will be useful later for storing all bundle parts (like tree packs
and obsmarkers).
Differential Revision: https://phab.mercurial-scm.org/D717
Previously the logic that created a bundle from a changegroup part and put it in
the store was all in the same function. Let's split it up.
This will be useful in a later part where we stop using the default bundle2 part
handling logic and instead handle each part manually (allowing us to store other
parts in the bundle).
Differential Revision: https://phab.mercurial-scm.org/D716
The notbackedup revset predicate still has performance problems when the set of
not-backed-up changesets includes some that are completely disjoint from the
set of backed-up changesets.
Rather than trying to express it as a combination of other expressions,
implement the revset calculation directly. This allows us to also stop as soon
as a public commit is encountered, which should be faster in all cases.
Test Plan:
Expand tests to cover more complex scenarios. Test perf on a larger and older
checkout of a big repo.
Differential Revision: https://phab.mercurial-scm.org/D764
Summary:
Previously we ran auto backups in txnclose hook. But commands like histedit and
rebase create many transactions, and that means that lots of backup processes
are started at the same time. That means that sometimes backup processes use
too much memory.
Instead let's start it once per command if any transaction was opened
during the command.
Test Plan:
Run unit-test.
Run histedit with autobackup enabled, and make sure just one backup process
was started.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D678
Summary:
The infinitepush backup smartlog summary uses only(a, b) to determine which
revisions have not been backed up. However, if the set of backup heads is
empty, only() behaves as if it were called as only(a), which is wrong and slow.
Instead, use ::a - ::b, which is always correct, even if b is empty.
Test Plan: Re-run UTs. Test perf on a repo with no backup heads.
Reviewers: #fbhgext, stash
Differential Revision: https://phab.mercurial-scm.org/D676
It seems `debugcheckbackup` could get a new kind of error with the latest
upstream code:
Traceback (most recent call last):
File "hg/mercurial/scmutil.py", line 150, in callcatch
return func()
File "hg/mercurial/dispatch.py", line 292, in _runcatchfunc
return _dispatch(req)
File "hg/mercurial/dispatch.py", line 896, in _dispatch
cmdpats, cmdoptions)
File "hg/mercurial/dispatch.py", line 658, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "hg/mercurial/dispatch.py", line 904, in _runcommand
return cmdfunc()
File "hg/mercurial/dispatch.py", line 893, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "hg/mercurial/extensions.py", line 322, in closure
return func(*(args + a), **kw)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "hg/hgext/mq.py", line 3542, in mqcommand
return orig(ui, repo, *args, **kwargs)
File "hg/mercurial/util.py", line 1118, in check
return func(*args, **kwargs)
File "fb-hgext/infinitepush/backupcommands.py", line 272, in checkbackup
if not _dobackupcheck(bkpstate, ui, repo, dest, **opts):
File "fb-hgext/infinitepush/backupcommands.py", line 469, in _dobackupcheck
for r in lookupresults:
File "hg/mercurial/wireproto.py", line 137, in results
next(batchable)
File "hg/mercurial/wireproto.py", line 238, in lookup
self._abort(error.RepoError(data))
File "hg/mercurial/sshpeer.py", line 235, in _abort
raise exception
RepoError: 0e1a088ff2825213eaa838a82a842bc186f10dd5
abort: 0e1a088ff2825213eaa838a82a842bc186f10dd5!
The error message only contains a commit hash, which is not friendly to be
printed out. Therefore make sure we have control about the error message and
update the test accordingly.
Differential Revision: https://phab.mercurial-scm.org/D485
Summary:
The first version of infinitepush smartlog summary included some false
positives (in particular, not-backed-up but obsolete revs), and took a long
time to compute the set if there were many heads in the repo.
Improve the revset that is used by taking into account the heads that *should*
be backed up. Ancestors of these heads, that are not ancestors of the actual
backup heads are the ones that should be considered "not backed up".
Test Plan:
Re-run unit tests.
Manual test on devvm.
Perf test against stash's repo which has more heads than mine.
Reviewers: #fbhgext, stash
Differential Revision: https://phab.mercurial-scm.org/D475
Summary:
--list-remote can return too many nodes if used without a pattern.
Let's require a pattern.
Test Plan: Run unittests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D221
Summary:
- change the infinitepush extension to record additional state when the a backup occurs.
- useful for other tools (scm-prompt, nuclide) to indicate when a backup hasn't occurred recently
Test Plan:
- cd facebook-hg-rpms/
- source hg-dev
- cd fb-hgext
- arc unit
Reviewers: rmcelroy, stash
Reviewed By: stash
Subscribers: rmcelroy, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5321221
Tasks: 18349390
Signature: t1:5321221:1501271966:6698785e0714e3ecf7e1e2684d1e62e52ac0f1a0
Summary:
Allow summarisation of commits that have not been backed up
by infinitepush.
1. Add a template keyword (backupstatus) which evaluates to
one of "Public", "Backed up", "Backup pending", or
"Not backed up" depending on the backup state of the
commit. Commits are pending for 10 minutes, after which
they are declared not backed up.
2. Add a summary to the end of smartlog that shows the
number of commits that are not backed up.
Configuration options allow the addition of an education
message to inform users what to do about their failing
backups, and the message can also be suppressed.
Test Plan:
New unit test for both the keyword and the summary.
Reviewers: #fbhgext, stash
Subscribers: mitrandir, quark, akushner
Differential Revision: https://phab.mercurial-scm.org/D184
Summary:
This option was intended as temporary and it was enabled for long enough time.
Now let's delete it.
Test Plan: Run unit-tests
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D195
Summary:
There was a report from an infinitepush user who complained about
`hg update remote/scratch/book` failing to pull the bookmark, while
`hg update scratch/book` pulling fine. It confuses users because they see
`remote/scratch/book` in smartlog.
On the other hand let's not allow `hg pull -B remote/scratch/book` to stay
consistent with remotenames.
Test Plan: Slowly run tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D172
Summary:
We've seen problems when public nodes were pushed to the server.
While the real reason is unknown it's a good idea to add a safety net to prevent
sending too many commits to the server because it's usually means that we are
pushing public commits. We limit number of nodes to 1000
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D143
Summary:
We've had problems when users sent huge bundles to the server.
Let's limit the size of the bundle.
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D142
Summary:
Infinitepush has too many options and many of them are unused or has never
been changed from the default value. Let's remove some of them.
Test Plan: Run infinitepush tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D141
Summary:
D17 was reverted because it broke creating bundle for non-general delta repos.
The reason is the following: D17 made infinitepush extension override
changegroup.supportedoutgoingversion() function and discard '01' version.
For non-general delta repositories it resulted in broken `hg bundle ...`
command.
`abort: repository does not support bundle version 01`
This diff fixes it by not overriding supportedoutgoingversion(). Instead
getscratchbranchpart() has its own logic of selecting changegroup version.
Test Plan:
Run unit-test
Run `hg bundle -r . --base .^ somefile` in non-general delta repo,
make sure it works fine.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D81
This is the result of running a codemod script:
python ~/hg/contrib/codemod/codemod_nestedwith.py **/*.py
Plus a manual cleanup that removes the comment in `absorb/__init.py`.
See D76 for the codemod script.
Differential Revision: https://phab.mercurial-scm.org/D78
Summary:
Upstream has added warnings when writing files outside the wlock, so let's put
this write in the wlock.
Test Plan: The tests now pass
Reviewers: #fbhgext, stash
Reviewed By: #fbhgext, stash
Subscribers: stash, simonfar
Differential Revision: https://phab.mercurial-scm.org/D104
Summary:
Upstream has added devel warnings when writing store files outside of a lock.
Certain files allow this though and there's a whitelist of which files are
allowed. Since backups happen behind the scenes and use their own lock, it's
safe to add our files to that list.
Test Plan: Ran the tests against latest upstream hg. They now pass.
Reviewers: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D87
Summary: This broke hg bundle on non-generaldelta repositories.
Test Plan: Ran the tests
Reviewers: #fbhgext, stash, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D74
Summary:
Make sure that infinitepush work fine with lfs. To achieve it we try to use
changegroup3 if it's available and also run prepush hook before infinitepush.
Test Plan: Run `arc unit`
Reviewers: #fbhgext, quark, dsp
Reviewed By: #fbhgext, dsp
Differential Revision: https://phab.mercurial-scm.org/D17
Summary:
debuginhibit was removed since we no longer have a separate inhibit state.
smartlog hack about changing "o" to "x" was removed since "obsolete()"
revset is correct and the hack is unnecessary now.
directaccess was removed from tests since inhibit does not depend on it.
`- obsolete()` was added to some revsets to avoid divergence and other
surprises.
Use `inhibit.revive` API in infinitepush and reset to revive changesets
properly.
Remove various hacky code that mangle inhibit state in corner cases.
Most test changes are `o` changed to `x` in output since we draw
`obsolete()` state correctly now. `test-infinitepush-backup-remotefilelog.t`
change was because output could be `bytes/sec` instead of `KB/sec`.
Test Plan: arc unit
Reviewers: #mercurial, kulshrax
Reviewed By: kulshrax
Subscribers: kulshrax, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5391361
Signature: t1:5391361:1499722618:d3c1cf629f0c59ecdf1dfd5e653c1eb6176646b8
Summary: It's a useful piece of information for Mononoke (see attached task).
Test Plan: arc unit
Reviewers: #mercurial, jsgf
Reviewed By: jsgf
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5310026
Tasks: 18812931
Signature: t1:5310026:1498234999:0c06a30bd659d6a1d3cedc28252f17dd1fc93dc8
Summary:
Previously pushbackups failed when it tried to unlink non-existent infinitepush
backup state file. This diff fixes it.
Test Plan: arc unit
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: medson, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5309684
Signature: t1:5309684:1498210504:49af716dbb4e7e1875e8a2b7ee4b40fe28ceefc1
Summary: Currently seconds are sent to logging for elapsed time. Given the that most values are 0, 1, 2, or 3 seconds, this change adds some better signal. Not reusing the elapsed field, instead introduced elapsed ms.
Test Plan:
arc unit
modified test-infinitepush-backup-logging.t to include the new elapsedms field
Reviewers: stash
Reviewed By: stash
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5292576
Tasks: 19557691
Signature: t1:5292576:1498059638:8461a71398a41bf741384ddcda15d5c016633d66
Summary:
Final part: launching debugfillinfinitepushmetadata in the background. We
intentionally don't wait for it's completion in order not to slow down pushes.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mitrandir, quark, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5265325
Tasks: 19103180
Signature: t1:5265325:1497886939:6c306c08b240373ea43a7fc868b17ef776258cf9
Summary:
Add an option to limit the number of file entries in the metadata field. This
is to avoid huge codemods to eat up all the space.
Depends on D5255883
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5264202
Tasks: 19103180
Signature: t1:5264202:1497630109:05d684db34578e309e0bbad314165e66d1f23ce9
Summary:
We may want to fill infinitepush metadata for many commits at once.
Let's make it possible to pass many nodes to debugfillinfinitepushmetadata
Depends on D5255670
Test Plan: arc unit
Reviewers: #mercurial, mitrandir, quark
Reviewed By: quark
Subscribers: mitrandir, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5255883
Tasks: 19103180
Signature: t1:5255883:1497629860:26332e3cc115881bca158664014ce48087d99215
Summary:
Also need to use `with index` to make sure that transaction closes.
Depends on D5255664
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mitrandir, mjpieters, medson, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5255670
Tasks: 19103180
Signature: t1:5255670:1497557000:906a39810931d92f30c6d6b8e5bfc28e0c888594
Summary:
I've forgot to add it in the previous diff. We also want to know copies
information and status (added, modified, removed).
Test Plan: arc unit
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, medson, azich, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5255664
Tasks: 19103180
Signature: t1:5255664:1497542963:b552c07d261067f426d7af7a29d2f02c1917394c
Summary:
Special command that fills optional metadata about infinitepush nodes. For now
it's info about changed files + info about how many lines were
added/removed. Since information is saved in json format it's easily
extensible.
It's a separate command so that we will be able to run it asynchronously:
during infinitepush push separate `hg debugfillinfinitepushmetadata` process
will be started. Push process won't wait for `debugfillinfinitepushmetadata`
process.
sqlindexapi will be implemented in the next diffs.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, medson, azich, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5237547
Tasks: 19103180
Signature: t1:5237547:1497397211:569d1ffe3be37c8d14fb7e07e42cd085ad5686d4
