add an extension to upgrade manifests in-place to generaldelta
Summary:
We actually realized that converting a manifest to generaldelta is not as easy as just flipping a bit -- the meaning of 'base' is different for normal vs generaldelta revlogs.
Instead, open a new generaldelta revlog, write out every node from the manifest, and then close the revlog. Finally, move over the revlog index and data files.
Test Plan: Ran it on local non-generaldelta fbandroid and fbcode repos. It's pretty fast: under two minutes each for both.
Reviewers: davidsp, rmcelroy, akushner, daviser, mpm, pyd, durham
Reviewed By: pyd
Subscribers: hannesr
Differential Revision: https://phabricator.fb.com/D1738808
Signature: t1:1738808:1418522355:31e424dc523d40f8194490d75160e993f48a0f27
2014-12-14 04:37:14 +03:00
|
|
|
# upgradegeneraldelta.py
|
|
|
|
#
|
|
|
|
# Copyright 2014 Facebook, Inc.
|
|
|
|
#
|
|
|
|
# This software may be used and distributed according to the terms of the
|
|
|
|
# GNU General Public License version 2 or any later version.
|
|
|
|
|
|
|
|
'''upgrade manifests to generaldelta in-place
|
|
|
|
|
upgradegeneraldelta: only upgrade before pulls
Summary: This is nicer because it only does the upgrade when an operation that'll probably take a while anyway is run.
Test Plan: Enabled the extension on a non-generaldelta repo, then ran `hg status`. Saw that it wasn't upgraded. Next, ran `hg pull`, and saw that it was.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1749026
Signature: t1:1749026:1418940170:6fdecc074294b26b7c3e9bd8c1463eefd13a8fc1
2014-12-18 23:43:32 +03:00
|
|
|
Performs an upgrade of this repo's manifest to generaldelta on pull, without
|
|
|
|
needing to reclone. This tries to be as safe as possible but is inherently not a
|
|
|
|
completely safe operation.
|
add an extension to upgrade manifests in-place to generaldelta
Summary:
We actually realized that converting a manifest to generaldelta is not as easy as just flipping a bit -- the meaning of 'base' is different for normal vs generaldelta revlogs.
Instead, open a new generaldelta revlog, write out every node from the manifest, and then close the revlog. Finally, move over the revlog index and data files.
Test Plan: Ran it on local non-generaldelta fbandroid and fbcode repos. It's pretty fast: under two minutes each for both.
Reviewers: davidsp, rmcelroy, akushner, daviser, mpm, pyd, durham
Reviewed By: pyd
Subscribers: hannesr
Differential Revision: https://phabricator.fb.com/D1738808
Signature: t1:1738808:1418522355:31e424dc523d40f8194490d75160e993f48a0f27
2014-12-14 04:37:14 +03:00
|
|
|
|
|
|
|
Filelogs are not touched -- the manifest is often the revlog that benefits the
|
|
|
|
most from generaldelta. New filelogs will be generaldelta, though.
|
|
|
|
|
|
|
|
The following configuration options are available:
|
|
|
|
|
|
|
|
:upgradegeneraldelta.upgrade: Set to True to actually perform the
|
|
|
|
upgrade. Default is False.
|
|
|
|
|
|
|
|
:upgradegeneraldelta.dryrun: Write out the generaldelta manifest, but do not
|
|
|
|
move it into place. This will cause the generaldelta manifest to be
|
|
|
|
rewritten every single time a Mercurial command is run. Default is False.
|
|
|
|
|
|
|
|
:upgradegeneraldelta.backup: Set to False to disable backing up the old manifest
|
|
|
|
as 00manifestold.{i,d}. Default is True.
|
|
|
|
|
|
|
|
'''
|
|
|
|
|
upgradegeneraldelta: only upgrade before pulls
Summary: This is nicer because it only does the upgrade when an operation that'll probably take a while anyway is run.
Test Plan: Enabled the extension on a non-generaldelta repo, then ran `hg status`. Saw that it wasn't upgraded. Next, ran `hg pull`, and saw that it was.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1749026
Signature: t1:1749026:1418940170:6fdecc074294b26b7c3e9bd8c1463eefd13a8fc1
2014-12-18 23:43:32 +03:00
|
|
|
from mercurial import commands, extensions, revlog, util
|
add an extension to upgrade manifests in-place to generaldelta
Summary:
We actually realized that converting a manifest to generaldelta is not as easy as just flipping a bit -- the meaning of 'base' is different for normal vs generaldelta revlogs.
Instead, open a new generaldelta revlog, write out every node from the manifest, and then close the revlog. Finally, move over the revlog index and data files.
Test Plan: Ran it on local non-generaldelta fbandroid and fbcode repos. It's pretty fast: under two minutes each for both.
Reviewers: davidsp, rmcelroy, akushner, daviser, mpm, pyd, durham
Reviewed By: pyd
Subscribers: hannesr
Differential Revision: https://phabricator.fb.com/D1738808
Signature: t1:1738808:1418522355:31e424dc523d40f8194490d75160e993f48a0f27
2014-12-14 04:37:14 +03:00
|
|
|
import os, struct, weakref
|
|
|
|
|
upgradegeneraldelta: only upgrade before pulls
Summary: This is nicer because it only does the upgrade when an operation that'll probably take a while anyway is run.
Test Plan: Enabled the extension on a non-generaldelta repo, then ran `hg status`. Saw that it wasn't upgraded. Next, ran `hg pull`, and saw that it was.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1749026
Signature: t1:1749026:1418940170:6fdecc074294b26b7c3e9bd8c1463eefd13a8fc1
2014-12-18 23:43:32 +03:00
|
|
|
def wrappull(orig, ui, repo, *args, **kwargs):
|
|
|
|
_upgrade(ui, repo)
|
|
|
|
orig(ui, repo, *args, **kwargs)
|
|
|
|
|
|
|
|
def _upgrade(ui, repo):
|
2014-12-18 08:09:18 +03:00
|
|
|
if not util.safehasattr(repo, 'svfs'):
|
|
|
|
return
|
add an extension to upgrade manifests in-place to generaldelta
Summary:
We actually realized that converting a manifest to generaldelta is not as easy as just flipping a bit -- the meaning of 'base' is different for normal vs generaldelta revlogs.
Instead, open a new generaldelta revlog, write out every node from the manifest, and then close the revlog. Finally, move over the revlog index and data files.
Test Plan: Ran it on local non-generaldelta fbandroid and fbcode repos. It's pretty fast: under two minutes each for both.
Reviewers: davidsp, rmcelroy, akushner, daviser, mpm, pyd, durham
Reviewed By: pyd
Subscribers: hannesr
Differential Revision: https://phabricator.fb.com/D1738808
Signature: t1:1738808:1418522355:31e424dc523d40f8194490d75160e993f48a0f27
2014-12-14 04:37:14 +03:00
|
|
|
if not ui.configbool('upgradegeneraldelta', 'upgrade'):
|
|
|
|
return
|
|
|
|
|
|
|
|
f = repo.svfs('00manifest.i')
|
|
|
|
i = f.read(4)
|
|
|
|
f.close()
|
|
|
|
if len(i) < 4:
|
|
|
|
# empty manifest
|
|
|
|
return
|
|
|
|
|
|
|
|
# probably only works with revlogng -- it became the default years ago so
|
|
|
|
# that's fine
|
|
|
|
v = struct.unpack(revlog.versionformat, i)[0]
|
|
|
|
isgeneraldelta = v & revlog.REVLOGGENERALDELTA
|
|
|
|
if isgeneraldelta:
|
|
|
|
return
|
|
|
|
|
|
|
|
# write out a new revlog, this time with generaldelta
|
|
|
|
oldopts = repo.svfs.options
|
|
|
|
lock = repo.lock()
|
|
|
|
tr = repo.transaction('upgradegeneraldelta')
|
|
|
|
try:
|
upgradegeneraldelta: only upgrade before pulls
Summary: This is nicer because it only does the upgrade when an operation that'll probably take a while anyway is run.
Test Plan: Enabled the extension on a non-generaldelta repo, then ran `hg status`. Saw that it wasn't upgraded. Next, ran `hg pull`, and saw that it was.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1749026
Signature: t1:1749026:1418940170:6fdecc074294b26b7c3e9bd8c1463eefd13a8fc1
2014-12-18 23:43:32 +03:00
|
|
|
ui.warn('upgrading repository to generaldelta\n')
|
add an extension to upgrade manifests in-place to generaldelta
Summary:
We actually realized that converting a manifest to generaldelta is not as easy as just flipping a bit -- the meaning of 'base' is different for normal vs generaldelta revlogs.
Instead, open a new generaldelta revlog, write out every node from the manifest, and then close the revlog. Finally, move over the revlog index and data files.
Test Plan: Ran it on local non-generaldelta fbandroid and fbcode repos. It's pretty fast: under two minutes each for both.
Reviewers: davidsp, rmcelroy, akushner, daviser, mpm, pyd, durham
Reviewed By: pyd
Subscribers: hannesr
Differential Revision: https://phabricator.fb.com/D1738808
Signature: t1:1738808:1418522355:31e424dc523d40f8194490d75160e993f48a0f27
2014-12-14 04:37:14 +03:00
|
|
|
trp = weakref.proxy(tr)
|
|
|
|
|
|
|
|
newopts = oldopts.copy()
|
|
|
|
newopts['generaldelta'] = 1
|
|
|
|
repo.svfs.options = newopts
|
|
|
|
# remove 00manifestgd.i if present
|
|
|
|
repo.svfs.unlinkpath('00manifestgd.i', ignoremissing=True)
|
|
|
|
newmf = revlog.revlog(repo.svfs, '00manifestgd.i')
|
|
|
|
oldmf = repo.manifest
|
|
|
|
i = oldmf.index
|
|
|
|
chunk = oldmf._chunk
|
|
|
|
|
|
|
|
for rev in oldmf:
|
|
|
|
ui.progress('upgrading', rev, total=len(oldmf))
|
|
|
|
e = i[rev]
|
|
|
|
# if the delta base is the rev, this rev is a fulltext
|
|
|
|
isdelta = (rev != e[3])
|
|
|
|
revchunk = chunk(rev)
|
|
|
|
if isdelta:
|
|
|
|
newmf.addrevision(None, trp, e[4], i[e[5]][7], i[e[6]][7],
|
|
|
|
cachedelta=(rev - 1, revchunk),
|
|
|
|
node=e[7])
|
|
|
|
else:
|
|
|
|
newmf.addrevision(revchunk, trp, e[4], i[e[5]][7], i[e[6]][7],
|
|
|
|
node=e[7])
|
|
|
|
tr.close()
|
|
|
|
if not ui.configbool('upgradegeneraldelta', 'dryrun', default=False):
|
2014-12-16 00:28:00 +03:00
|
|
|
manifesti = repo.sjoin('00manifest.i')
|
|
|
|
manifestd = repo.sjoin('00manifest.d')
|
|
|
|
manifestgdi = repo.sjoin('00manifestgd.i')
|
|
|
|
manifestgdd = repo.sjoin('00manifestgd.d')
|
|
|
|
manifestoldi = repo.sjoin('00manifestold.i')
|
|
|
|
manifestoldd = repo.sjoin('00manifestold.d')
|
add an extension to upgrade manifests in-place to generaldelta
Summary:
We actually realized that converting a manifest to generaldelta is not as easy as just flipping a bit -- the meaning of 'base' is different for normal vs generaldelta revlogs.
Instead, open a new generaldelta revlog, write out every node from the manifest, and then close the revlog. Finally, move over the revlog index and data files.
Test Plan: Ran it on local non-generaldelta fbandroid and fbcode repos. It's pretty fast: under two minutes each for both.
Reviewers: davidsp, rmcelroy, akushner, daviser, mpm, pyd, durham
Reviewed By: pyd
Subscribers: hannesr
Differential Revision: https://phabricator.fb.com/D1738808
Signature: t1:1738808:1418522355:31e424dc523d40f8194490d75160e993f48a0f27
2014-12-14 04:37:14 +03:00
|
|
|
# move the newly created manifests over
|
|
|
|
if ui.configbool('upgradegeneraldelta', 'backup', default=True):
|
2014-12-16 00:28:00 +03:00
|
|
|
os.rename(manifesti, manifestoldi)
|
|
|
|
if os.path.exists(manifestd):
|
|
|
|
os.rename(manifestd, manifestoldd)
|
|
|
|
os.rename(manifestgdi, manifesti)
|
|
|
|
if os.path.exists(manifestgdd):
|
|
|
|
os.rename(manifestgdd, manifestd)
|
add an extension to upgrade manifests in-place to generaldelta
Summary:
We actually realized that converting a manifest to generaldelta is not as easy as just flipping a bit -- the meaning of 'base' is different for normal vs generaldelta revlogs.
Instead, open a new generaldelta revlog, write out every node from the manifest, and then close the revlog. Finally, move over the revlog index and data files.
Test Plan: Ran it on local non-generaldelta fbandroid and fbcode repos. It's pretty fast: under two minutes each for both.
Reviewers: davidsp, rmcelroy, akushner, daviser, mpm, pyd, durham
Reviewed By: pyd
Subscribers: hannesr
Differential Revision: https://phabricator.fb.com/D1738808
Signature: t1:1738808:1418522355:31e424dc523d40f8194490d75160e993f48a0f27
2014-12-14 04:37:14 +03:00
|
|
|
with repo.opener('requires', 'a+') as f:
|
|
|
|
f.write('generaldelta\n')
|
|
|
|
repo.invalidate()
|
|
|
|
finally:
|
|
|
|
ui.progress('upgrading', None)
|
|
|
|
if tr:
|
|
|
|
tr.release()
|
|
|
|
lock.release()
|
|
|
|
repo.svfs.options = oldopts
|
upgradegeneraldelta: only upgrade before pulls
Summary: This is nicer because it only does the upgrade when an operation that'll probably take a while anyway is run.
Test Plan: Enabled the extension on a non-generaldelta repo, then ran `hg status`. Saw that it wasn't upgraded. Next, ran `hg pull`, and saw that it was.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1749026
Signature: t1:1749026:1418940170:6fdecc074294b26b7c3e9bd8c1463eefd13a8fc1
2014-12-18 23:43:32 +03:00
|
|
|
|
|
|
|
def extsetup(ui):
|
|
|
|
extensions.wrapcommand(commands.table, 'pull', wrappull)
|