facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2025-01-04 03:06:30 +03:00
Code Issues Projects Releases Wiki Activity

treemanifest: set a hint commit hash for resolving base trees

Browse Source 
Summary:
When requesting trees from the server we try to provide a base tree
that we already have.  In the hybrid manifest days, when we were fetching
manifest X, we would look up the linknode for X in the flat manifest revlog,
then scan the changelog up and down from that linknode. In the treeonly world we
don't have that revlog anymore though, so the current algorithm just searches from
the tip. This ended up being a really bad algorithm, since if you have tip and you
request tip~1000, then tip~2000, then tip~3000, you end up basing them all
against tip and redownloading a lot of data. This makes hg blame and hg log -p
super slow.

Let's change the algorithm to base off of a hint linknode, and let's change
changectx to set that hint every time it attempts to read a manifest. In manual
testing this significantly sped up hg log -p

Reviewed By: phillco

Differential Revision: D8399613

fbshipit-source-id: 771a94ee8b82be682ea0091b8d6c0fcd5f4e6646
This commit is contained in:
Durham Goode 2018-06-13 11:42:16 -07:00 committed by Facebook Github Bot
parent 65f35cf9bd
commit 93eb7cc993
8 changed files with 97 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
hgext/treemanifest/__init__.py
View File

@ -536,6 +536,7 @@ class basetreemanifestlog(object):
def __init__(self):
self._mutabledatapack = None
self._mutablehistorypack = None
self.recentlinknode = None
def add(
self,
@ -2156,8 +2157,14 @@ class remotetreestore(generatingdatastore):
raise KeyError((name, node))
if linkrev is None:
# TODO: improve linkrev guessing when the revlog isn't available
linkrev = self._repo["tip"].rev()
# Recentlinknode is set any time a changectx accesses a
# manifest. This let's us get an approximate linknode to use for
# finding nearby manifests.
recentlinknode = self._repo.manifestlog.recentlinknode
if recentlinknode is not None:
linkrev = self._repo[recentlinknode].rev()
else:
linkrev = self._repo["tip"].rev()
# Find a recent tree that we already have
basemfnodes = _findrecenttree(self._repo, linkrev)

2
mercurial/context.py
View File

@ -609,10 +609,12 @@ class changectx(basectx):
@propertycache
def _manifest(self):
self._repo.manifestlog.recentlinknode = self.node()
return self._manifestctx.read()
@property
def _manifestctx(self):
self._repo.manifestlog.recentlinknode = self.node()
return self._repo.manifestlog[self._changeset.manifest]
@propertycache

1
mercurial/manifest.py
View File

@ -1389,6 +1389,7 @@ class manifestlog(object):
self._dirmancache[""] = util.lrucachedict(cachesize)
self.cachesize = cachesize
self.recentlinknode = None
def __nonzero__(self):
return bool(self._revlog)

12
tests/test-fb-hgext-treemanifest-convertflat.t
View File

@ -118,10 +118,10 @@ has tree manifest. In this case, this implies that tree manifest will be
generated for the commit 'b9b574be2f5d' and its parent commit '9055b56f3916'.
$ hg log -vpr 'b9b574be2f5d'
fetching tree '' 40f43426c87ba597f0d9553077c72fe06d4e2acb, found via 3795bd66ca70
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 3795bd66ca70
fetching tree '' 40f43426c87ba597f0d9553077c72fe06d4e2acb, found via 9055b56f3916
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 9055b56f3916
2 trees fetched over * (glob)
fetching tree '' 53c631458e338938766702d93d8bdd5c5d89b69f, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 3795bd66ca70
fetching tree '' 53c631458e338938766702d93d8bdd5c5d89b69f, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via b9b574be2f5d
changeset: 2:b9b574be2f5d
user: test
date: Thu Jan 01 00:00:00 1970 +0000
@ -217,7 +217,7 @@ able to view it even with 'treemanifest.demandgenerate' being False.
manifest for commit 'f7febcf0f689'.
$ hg log -vpr 'f7febcf0f689'
fetching tree '' a6875e5fbf695d5fe67e03b8a5df2bab0e6dd045, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 3795bd66ca70
fetching tree '' a6875e5fbf695d5fe67e03b8a5df2bab0e6dd045, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via f7febcf0f689
changeset: 3:f7febcf0f689
parent: 1:9055b56f3916
user: test
@ -269,8 +269,8 @@ manifest for commit 'f7febcf0f689'.
$ hg rebase -d '9055b56f3916' -s '3795bd66ca70'
rebasing 4:3795bd66ca70 "flat only commit 1 over flat only commit 2 at level 2" (tip)
fetching tree '' 40f43426c87ba597f0d9553077c72fe06d4e2acb, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 3795bd66ca70
fetching tree '' a6875e5fbf695d5fe67e03b8a5df2bab0e6dd045, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 3795bd66ca70
fetching tree '' 40f43426c87ba597f0d9553077c72fe06d4e2acb, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 9055b56f3916
fetching tree '' a6875e5fbf695d5fe67e03b8a5df2bab0e6dd045, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via f7febcf0f689
fetching tree '' b7db2b1fa98f78057c528e8dad908e05ef80aeb4, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 3795bd66ca70
merging subdir/x
warning: conflicts while merging subdir/x! (edit, then use 'hg resolve --mark')

2
tests/test-fb-hgext-treemanifest-infinitepush.t
View File

@ -372,6 +372,6 @@ treemanifest data for the public commits.
# Verify the non-infinitepush commit tree data is downloaded on demand
$ hg manifest -r 0
fetching tree '' bc0c2c938b929f98b1c31a8c5994396ebb096bf0, found via 02c12aef64ff
fetching tree '' bc0c2c938b929f98b1c31a8c5994396ebb096bf0
1 trees fetched over * (glob)
x

10
tests/test-fb-hgext-treemanifest-server.t
View File

@ -296,7 +296,7 @@ Test fetching from the server populates the cache
$ rm -rf .hg/store/00manifest*
$ clearcache
$ hg status --change tip > /dev/null
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd
2 trees fetched over * (glob)
fetching tree '' 7e680cec965bd202ea244b3c4869181424ca5fe8, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
2 trees fetched over * (glob)
@ -309,7 +309,7 @@ Test fetching from the server populates the cache
$ mv ../master/.hg/store/meta ../master/.hg/store/meta.bak
$ clearcache
$ hg status --change tip > /dev/null
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd
2 trees fetched over * (glob)
fetching tree '' 7e680cec965bd202ea244b3c4869181424ca5fe8, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
2 trees fetched over * (glob)
@ -319,7 +319,7 @@ Test fetching from the server populates the cache
$ cp ../master/.hg/cache/trees/v2/get/0b/0fa4abc415aa6a46e003c61283b182ccc989b6 ../master/.hg/cache/trees/v2/get/d4/395b5ffa18499864439ac2b1a731ff7b7491fa
$ clearcache
$ hg status --change tip > /dev/null
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd
2 trees fetched over * (glob)
fetching tree '' 7e680cec965bd202ea244b3c4869181424ca5fe8, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
abort: "unable to find the following nodes locally or on the server: ('', 7e680cec965bd202ea244b3c4869181424ca5fe8)"
@ -330,7 +330,7 @@ Test fetching from the server populates the cache
$ clearcache
$ mv ../master/.hg/store/meta.bak ../master/.hg/store/meta
$ hg status --change tip > /dev/null
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd
2 trees fetched over * (glob)
fetching tree '' 7e680cec965bd202ea244b3c4869181424ca5fe8, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
2 trees fetched over * (glob)
@ -348,7 +348,7 @@ Test fetching from the server populates the cache
\s*16 (re)
$ clearcache
$ hg status --change tip
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd
2 trees fetched over * (glob)
fetching tree '' 7e680cec965bd202ea244b3c4869181424ca5fe8, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via a30b520ebf7a
2 trees fetched over * (glob)

70
tests/test-fb-hgext-treemanifest-treeonly-fetching.t Normal file
View File

@ -0,0 +1,70 @@
$ . "$TESTDIR/library.sh"
Setup the server
$ hginit master
$ cd master
$ cat >> .hg/hgrc <<EOF
> [extensions]
> treemanifest=
> [treemanifest]
> server=True
> treeonly=True
> [remotefilelog]
> server=True
> shallowtrees=True
> EOF
Setup the client
$ cd ..
$ hgcloneshallow ssh://user@dummy/master client -q --config treemanifest.treeonly=True
$ cd client
$ cat >> .hg/hgrc <<EOF
> [extensions]
> treemanifest=
> fastmanifest=
>
> [treemanifest]
> sendtrees=True
> treeonly=True
>
> [fastmanifest]
> usetree=True
> usecache=False
>
> [remotefilelog]
> reponame=treeonlyrepo
> EOF
Make some commits
$ cd ../master
$ mkdir subdir
$ echo a >> subdir/foo
$ hg commit -Aqm 'a > subdir/foo'
$ echo b >> subdir/foo
$ hg commit -Aqm 'b >> subdir/foo'
$ echo c >> subdir/foo
$ hg commit -Aqm 'c >> subdir/foo'
$ echo d >> subdir/foo
$ hg commit -Aqm 'd >> subdir/foo'
Test that log -p downloads each tree using the prior tree as a base
$ cd ../client
$ hg pull -q
$ hg up tip
fetching tree '' f50e2ff15ddef5802543b56b0b84d742512e90f0, found via 3b158baa90a6
2 trees fetched over * (glob)
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg log -p 1>/dev/null
fetching tree '' 88dd1b582645feb893f44bd3b20947ff2d275360, based on f50e2ff15ddef5802543b56b0b84d742512e90f0, found via 2f885f603416
2 trees fetched over * (glob)
fetching tree '' d09a7a1172be7d3c00d4bc16831b6394d11ce33f, based on 88dd1b582645feb893f44bd3b20947ff2d275360, found via 6bfefea56efe
2 trees fetched over * (glob)
fetching tree '' 1b3e02c1b4460e2d6264781579eb163e76cffad4, based on d09a7a1172be7d3c00d4bc16831b6394d11ce33f
2 trees fetched over * (glob)
3 files fetched over 3 fetches - (3 misses, 0.00% hit ratio) over * (glob)

6
tests/test-fb-hgext-treemanifest-treeonly.t
View File

@ -329,7 +329,7 @@ Test peer-to-peer push/pull of tree only commits
| y | 1 +
| 1 files changed, 1 insertions(+), 0 deletions(-)
|
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, based on d9920715ba88cbc7962c4dac9f20004aafd94ac8, found via 5b483416c8aa
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd, based on d9920715ba88cbc7962c4dac9f20004aafd94ac8
2 trees fetched over * (glob)
o modify subdir/x
| subdir/x | 1 +
@ -455,7 +455,7 @@ Switch back to hybrid mode
$ hg log -r 'last(public())' --stat
fetching tree '' 85b359fdb09e9b8d7ac4a74551612b277345e8fd
2 trees fetched over * (glob)
fetching tree '' d9920715ba88cbc7962c4dac9f20004aafd94ac8, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 2f8e443c6ba8
fetching tree '' d9920715ba88cbc7962c4dac9f20004aafd94ac8, based on 85b359fdb09e9b8d7ac4a74551612b277345e8fd, found via 2937cde31c19
2 trees fetched over * (glob)
changeset: 4:2937cde31c19
parent: 0:2278cc8c6ce6
@ -572,7 +572,7 @@ Test pulling to a treeonly client from a treeonly server
new changesets 4f84204095e0:5b1ec8639460
(run 'hg heads' to see heads, 'hg merge' to merge)
$ hg log -r tip -T '{desc}\n' --stat
fetching tree '' 83b03df1c9d62b8a2dedf46629e3262423af655c, based on d9920715ba88cbc7962c4dac9f20004aafd94ac8, found via 5b1ec8639460
fetching tree '' 83b03df1c9d62b8a2dedf46629e3262423af655c, based on d9920715ba88cbc7962c4dac9f20004aafd94ac8, found via 4f84204095e0
1 trees fetched over * (glob)
fetching tree '' bd5ff58fa887770ff0ea29dde0b91f5804cdeff0, based on 83b03df1c9d62b8a2dedf46629e3262423af655c, found via 5b1ec8639460
2 trees fetched over * (glob)