sapling/eden/scm/tests/test-fb-hgext-treemanifest-prefetch.t
Durham Goode 9c1b611dff indexedlog: make writing to indexedlog the default
Summary:
We've rolled both of these out to 100%. Let's make this the default so
we can delete those configs.

Reviewed By: quark-zju

Differential Revision: D26233645

fbshipit-source-id: cd7a08c404483f78ab714763870f5bf0fa801e7a
2021-02-11 09:34:55 -08:00

541 lines
21 KiB
Perl

#chg-compatible
$ CACHEDIR=`pwd`/hgcache
$ . "$TESTDIR/library.sh"
$ setconfig remotefilelog.write-hgcache-to-indexedlog=False remotefilelog.write-local-to-indexedlog=False
$ enable remotenames
$ hginit master
$ cd master
$ cat >> .hg/hgrc <<EOF
> [treemanifest]
> server=True
> [remotefilelog]
> server=True
> EOF
$ mkdir dir
$ echo x > dir/x
$ hg commit -qAm 'add x'
$ mkdir subdir
$ echo z > subdir/z
$ hg commit -qAm 'add subdir/z'
$ echo x >> dir/x
$ hg commit -Am 'modify x'
$ hg bookmark master
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> server=True
> name=master
> cachepath=$CACHEDIR
>
> [treemanifest]
> server=True
> sendtrees=True
> EOF
$ cd ..
$ hgcloneshallow ssh://user@dummy/master client
streaming all changes
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
3 files to transfer, 752 bytes of data
transferred 752 bytes in 0.0 seconds (734 KB/sec)
searching for changes
no changes found
updating to branch default
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via bd6f9b289c01
3 trees fetched over * (glob)
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd master
$ cd ../client
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> reponame = master
> prefetchdays=0
> cachepath = $CACHEDIR
> EOF
Test prefetch
$ hg prefetch -r '0 + 1 + 2'
4 trees fetched over * (glob)
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
$ ls $CACHEDIR/master/packs/manifests
214f2046312905a44188b27497625a36ffaa4c3d.histidx
214f2046312905a44188b27497625a36ffaa4c3d.histpack
5363bd2b98ea8fb0987f683c0cae80e0313e95a6.dataidx
5363bd2b98ea8fb0987f683c0cae80e0313e95a6.datapack
9695a88823f13dc82d06280705d9b37f7c25c50d.histidx
9695a88823f13dc82d06280705d9b37f7c25c50d.histpack
9c4f8bc12fc0d3a742847e75687c57cc0cafa334.dataidx
9c4f8bc12fc0d3a742847e75687c57cc0cafa334.datapack
$ hg debugdatapack --config extensions.remotefilelog= \
> --long $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/5363bd2b98ea8fb0987f683c0cae80e0313e95a6:
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a648a43a997aef53123bce309c794fd 0000000000000000000000000000000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
1be4ab2126dd2252dcae6be2aac2561dd3ddcda0 0000000000000000000000000000000000000000 95 (missing)
dir:
Node Delta Base Delta Length Blob Size
bc0c2c938b929f98b1c31a8c5994396ebb096bf0 0000000000000000000000000000000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
ef362f8bbe8aa457b0cfc49f200cbeb7747984ed 0000000000000000000000000000000000000000 46 (missing)
$TESTTMP/hgcache/master/packs/manifests/9c4f8bc12fc0d3a742847e75687c57cc0cafa334:
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76d6aab2edb46810b20fbdbd10fb4b 0000000000000000000000000000000000000000 43 (missing)
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a648a43a997aef53123bce309c794fd 0000000000000000000000000000000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5 0000000000000000000000000000000000000000 95 (missing)
$ hg debughistorypack --config extensions.remotefilelog= \
> $CACHEDIR/master/packs/manifests/*.histidx
Node P1 Node P2 Node Link Node Copy From
60a7f7acb6bb 1be4ab2126dd 000000000000 bd6f9b289c01
dir
Node P1 Node P2 Node Link Node Copy From
a18d21674e76 bc0c2c938b92 000000000000 bd6f9b289c01
subdir
Node P1 Node P2 Node Link Node Copy From
ddb35f099a64 000000000000 000000000000 f15c65c6e9bd
Node P1 Node P2 Node Link Node Copy From
1be4ab2126dd ef362f8bbe8a 000000000000 f15c65c6e9bd
ef362f8bbe8a 000000000000 000000000000 ecfb693caff5
dir
Node P1 Node P2 Node Link Node Copy From
bc0c2c938b92 000000000000 000000000000 ecfb693caff5
subdir
Node P1 Node P2 Node Link Node Copy From
ddb35f099a64 000000000000 000000000000 f15c65c6e9bd
$ hg debugdatapack --config extensions.remotefilelog= \
> --node-delta ef362f8bbe8aa457b0cfc49f200cbeb7747984ed $CACHEDIR/master/packs/manifests/5363bd2b98ea8fb0987f683c0cae80e0313e95a6.dataidx
$TESTTMP/hgcache/master/packs/manifests/5363bd2b98ea8fb0987f683c0cae80e0313e95a6:
Node Delta Base Delta SHA1 Delta Length
ef362f8bbe8aa457b0cfc49f200cbeb7747984ed 0000000000000000000000000000000000000000 3b295111780498d177793f9228bf736b915f0255 46
$ hg -R ../master debugindex ../master/.hg/store/00manifesttree.i
rev offset length delta linkrev nodeid p1 p2
0 0 47 -1 0 ef362f8bbe8a 000000000000 000000000000
1 47 61 0 1 1be4ab2126dd ef362f8bbe8a 000000000000
2 108 58 1 2 60a7f7acb6bb 1be4ab2126dd 000000000000
Test prefetch with base node (subdir/ shouldn't show up in the pack)
$ rm -rf $CACHEDIR/master
Multiple trees are fetched in this case because the file prefetching code path
requires tree manifest for the base commit.
$ hg prefetch -r '2' --base '1'
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via bd6f9b289c01
3 trees fetched over 0.00s
fetching tree '' 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, based on 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via bd6f9b289c01
2 trees fetched over * (glob)
$ ls $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/b3bd3b918c8eb05705e84b9c46b6be6cc2934f64.dataidx
$ hg debugdatapack $TESTTMP/hgcache/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/b3bd3b918c8eb05705e84b9c46b6be6cc2934f64:
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
dir:
Node Delta Base Delta Length Blob Size
bc0c2c938b92 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
1be4ab2126dd 000000000000 95 (missing)
Test prefetching when a draft commit is marked public
$ mkdir $TESTTMP/cachedir.bak
$ mv $CACHEDIR/* $TESTTMP/cachedir.bak
- Create a draft commit, and force it to be public
$ hg prefetch -r .
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via bd6f9b289c01
3 trees fetched over * (glob)
$ echo foo > foo
$ hg commit -Aqm 'add foo'
$ hg debugmakepublic -r .
$ hg log -G -T '{phase} {manifest}'
@ public 5cf0d3bd4f40594eff7f0c945bec8baa8d115d01
o public 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5
o public 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0
o public ef362f8bbe8aa457b0cfc49f200cbeb7747984ed
- Add remotenames for the remote heads
$ hg pull --config extensions.remotenames=
pulling from ssh://user@dummy/master
searching for changes
no changes found
- Attempt to download the latest server commit. Verify there's no error about a
- missing manifest from the server.
$ clearcache
$ hg status --change 2 --config extensions.remotenames=
fetching tree '' 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, based on 5cf0d3bd4f40594eff7f0c945bec8baa8d115d01, found via f15c65c6e9bd
3 trees fetched over * (glob)
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, based on 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, found via bd6f9b289c01
2 trees fetched over * (glob)
M dir/x
$ hg debugstrip -r 3
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ clearcache
$ mv $TESTTMP/cachedir.bak/* $CACHEDIR
Test auto prefetch during normal access
$ rm -rf $CACHEDIR/master
|| ( exit 1 ) is needed because ls on OSX and Linux exits differently
$ ls $CACHEDIR/master/packs/manifests || ( exit 1 )
* $ENOENT$ (glob)
[1]
$ hg log -r tip --stat --pager=off
fetching tree '' 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, found via f15c65c6e9bd
3 trees fetched over * (glob)
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, based on 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, found via bd6f9b289c01
2 trees fetched over * (glob)
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
commit: bd6f9b289c01
bookmark: default/master
hoistedname: master
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: modify x
dir/x | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
$ ls $CACHEDIR/master/packs/manifests
0ac61f8fd04a6623ecbde9812036089c557f07fa.histidx
0ac61f8fd04a6623ecbde9812036089c557f07fa.histpack
93d481677f2b09bd1cec155608977cb9806df077.dataidx
93d481677f2b09bd1cec155608977cb9806df077.datapack
$ hg debugdatapack --config extensions.remotefilelog= $CACHEDIR/master/packs/manifests/*.datapack
$TESTTMP/hgcache/master/packs/manifests/93d481677f2b09bd1cec155608977cb9806df077:
dir:
Node Delta Base Delta Length Blob Size
bc0c2c938b92 000000000000 43 (missing)
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
1be4ab2126dd 000000000000 95 (missing)
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
Test that auto prefetch scans up the changelog for base trees
$ rm -rf $CACHEDIR/master
$ hg prefetch -r 'tip^'
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, found via bd6f9b289c01
3 trees fetched over * (glob)
$ rm -rf $CACHEDIR/master
$ hg prefetch -r tip
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via bd6f9b289c01
3 trees fetched over * (glob)
- Only 2 of the 3 trees from tip^ are downloaded as part of --stat's fetch
$ hg log -r tip --stat --pager=off > /dev/null
fetching tree '' 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, based on 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via f15c65c6e9bd
2 trees fetched over * (glob)
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
Test auto prefetch during pull
- Prefetch everything
$ echo a >> a
$ hg commit -Aqm 'draft commit that shouldnt affect prefetch'
$ rm -rf $CACHEDIR/master
$ hg pull --config treemanifest.pullprefetchcount=10 --traceback
pulling from ssh://user@dummy/master
searching for changes
no changes found
prefetching trees for 3 commits
6 trees fetched over * (glob)
$ hg debugdatapack --config extensions.remotefilelog= \
> $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/adc41dd93f904447812dd994d49bce59ea7c4360:
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
1be4ab2126dd 000000000000 95 (missing)
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
dir:
Node Delta Base Delta Length Blob Size
bc0c2c938b92 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
ef362f8bbe8a 000000000000 46 (missing)
$ hg debugstrip -q -r 'draft()'
- Prefetch just the top manifest (but the full one)
$ rm -rf $CACHEDIR/master
$ hg pull --config treemanifest.pullprefetchcount=1 --traceback
pulling from ssh://user@dummy/master
searching for changes
no changes found
prefetching tree for bd6f9b289c01
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via bd6f9b289c01
3 trees fetched over * (glob)
$ hg debugdatapack --config extensions.remotefilelog= \
> $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/9c4f8bc12fc0d3a742847e75687c57cc0cafa334:
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
- Prefetch commit 1 then minimally prefetch commit 2
$ rm -rf $CACHEDIR/master
$ hg prefetch -r 1
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, found via bd6f9b289c01
3 trees fetched over * (glob)
$ ls $CACHEDIR/master/packs/manifests/*dataidx
$TESTTMP/hgcache/master/packs/manifests/c3ae0c4afc5f96ac510fd7ea3dddd0720a6d4dfb.dataidx
$ hg pull --config treemanifest.pullprefetchcount=1 --traceback
pulling from ssh://user@dummy/master
searching for changes
no changes found
prefetching tree for bd6f9b289c01
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, based on 1be4ab2126dd2252dcae6be2aac2561dd3ddcda0, found via bd6f9b289c01
2 trees fetched over * (glob)
$ ls $CACHEDIR/master/packs/manifests/*dataidx
$TESTTMP/hgcache/master/packs/manifests/4113a1ecc22f9f280deb722133d462720d3d7a9d.dataidx
$TESTTMP/hgcache/master/packs/manifests/c3ae0c4afc5f96ac510fd7ea3dddd0720a6d4dfb.dataidx
$ hg debugdatapack --config extensions.remotefilelog= \
> $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/4113a1ecc22f9f280deb722133d462720d3d7a9d:
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
$TESTTMP/hgcache/master/packs/manifests/c3ae0c4afc5f96ac510fd7ea3dddd0720a6d4dfb:
dir:
Node Delta Base Delta Length Blob Size
bc0c2c938b92 000000000000 43 (missing)
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
1be4ab2126dd 000000000000 95 (missing)
Test prefetching certain revs during pull
$ cd ../master
$ echo x >> dir/x
$ hg commit -qm "modify dir/x a third time"
$ echo x >> dir/x
$ hg commit -qm "modify dir/x a fourth time"
$ cd ../client
$ rm -rf $CACHEDIR/master
$ hg pull --config treemanifest.pullprefetchrevs='tip~2'
pulling from ssh://user@dummy/master
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 0 changes to 0 files
prefetching tree for bd6f9b289c01
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via cfacdcc4cee5
3 trees fetched over * (glob)
$ hg debugdatapack --config extensions.remotefilelog= \
> $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/9c4f8bc12fc0d3a742847e75687c57cc0cafa334:
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
- Test prefetching only the new tree parts for a commit who's parent tree is not
- downloaded already. Note that subdir/z was not downloaded this time.
$ hg pull --config treemanifest.pullprefetchrevs='tip'
pulling from ssh://user@dummy/master
searching for changes
no changes found
prefetching tree for cfacdcc4cee5
fetching tree '' aa52a49be5221fd6fb50743e0641040baa96ba89, based on 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via cfacdcc4cee5
2 trees fetched over * (glob)
$ hg debugdatapack --config extensions.remotefilelog= \
> $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/59658c2bcbdcbfd3c836edc4cdca4f0297acc287:
dir:
Node Delta Base Delta Length Blob Size
bf22bc15398b 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
aa52a49be522 000000000000 95 (missing)
$TESTTMP/hgcache/master/packs/manifests/9c4f8bc12fc0d3a742847e75687c57cc0cafa334:
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
subdir:
Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
Test that prefetch refills just part of a tree when the cache is deleted
$ echo >> dir/x
$ hg commit -m 'edit x locally'
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
$ rm -rf $CACHEDIR/master/*
$ hg cat subdir/z
fetching tree 'subdir' ddb35f099a648a43a997aef53123bce309c794fd
1 trees fetched over * (glob)
z
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
Test prefetch non-parent commits with no base node (should fetch minimal
trees - in this case 3 trees for commit 2, and 2 for commit 4 despite it having
3 directories)
$ rm -rf $CACHEDIR/master
$ hg prefetch -r '2 + 4'
5 trees fetched over * (glob)
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over * (glob) (?)
Test repack option
$ rm -rf $CACHEDIR/master
$ hg prefetch -r '0'
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' ef362f8bbe8aa457b0cfc49f200cbeb7747984ed, found via 7609b5c63072
2 trees fetched over * (glob)
$ hg prefetch -r '2'
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, based on ef362f8bbe8aa457b0cfc49f200cbeb7747984ed, found via 7609b5c63072
3 trees fetched over * (glob)
$ hg prefetch -r '4' --repack
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
fetching tree '' aa52a49be5221fd6fb50743e0641040baa96ba89, based on 60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5, found via 7609b5c63072
2 trees fetched over 0.00s
(running background incremental repack)
$ sleep 3
$ hg debugwaitonrepack
$ ls_l $CACHEDIR/master/packs/manifests | grep datapack | wc -l
\s*1 (re)
Test prefetching with no options works. The expectation is to prefetch the stuff
required for working with the draft commits which happens to be only revision 5
in this case.
$ rm -rf $CACHEDIR/master
The tree prefetching code path fetches no trees for revision 5. However, the
file prefetching code path fetches 1 file for revision 5 and while doing so,
also fetches 3 trees dealing with the tree manifest of the base revision 2.
$ hg prefetch
fetching tree 'subdir' ddb35f099a648a43a997aef53123bce309c794fd
1 trees fetched over * (glob)
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) (?)
Prefetching with treemanifest.ondemandfetch=True should fall back to normal
fetch is the server doesn't support it.
$ rm -rf $CACHEDIR/master
$ hg prefetch --config treemanifest.ondemandfetch=True
fetching tree 'subdir' ddb35f099a648a43a997aef53123bce309c794fd
1 trees fetched over 0.00s
Running prefetch in the master repository should exit gracefully
$ cd ../master
$ hg prefetch