facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-11 09:17:30 +03:00
Code Issues Projects Releases Wiki Activity
8b3bb637b4
sapling/tests/test-treemanifest-prefetch.t

244 lines
8.1 KiB
Perl
Raw Normal View History

treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
$ CACHEDIR=`pwd`/hgcache
$ PYTHONPATH=$TESTDIR/..:$PYTHONPATH
$ export PYTHONPATH
$ . "$TESTDIR/library.sh"
$ hg init master
$ cd master
$ mkdir dir
$ echo x > dir/x
$ hg commit -qAm 'add x'
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
$ mkdir subdir
$ echo z > subdir/z
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
$ hg commit -qAm 'add subdir/z'
$ echo x >> dir/x
$ hg commit -Am 'modify x'
$ cat >> .hg/hgrc <<EOF
> [extensions]
> treemanifest=
>
> [remotefilelog]
> name=master
> cachepath=$CACHEDIR
> usefastdatapack=True
>
> [fastmanifest]
> usetree=True
> usecache=False
>
> [treemanifest]
> server=True
> EOF
$ cd ..
$ hg clone ssh://user@dummy/master client
streaming all changes
4 files to transfer, 952 bytes of data
transferred 952 bytes in * seconds (*) (glob)
searching for changes
no changes found
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd master
$ hg backfilltree
$ cd ../client
$ cat >> .hg/hgrc <<EOF
> [extensions]
> treemanifest =
> fastmanifest =
> [remotefilelog]
> reponame = master
> cachepath = $CACHEDIR
> [fastmanifest]
> usetree = True
> usecache = False
> EOF
Test prefetchtrees
$ hg prefetchtrees -r '0 + 1 + 2'
$ ls $CACHEDIR/master/packs/manifests
treemanifest: add support for minimal tree pack sending Summary: The gettreepack protocol already had the arguments for the client to specify what nodes they want and what nodes they have, but it wasn't implemented (we just always returned the entire tree). This patch adds support for only sending parts of the tree that are new. In addition, if the client requests many nodes, we will only send the portion of each node that is not contained in the other nodes. For instance if you ask for X and Y, where X is the parent of Y, then we will send all of X and only the part of Y that is new. The order changed a bit, probably because we stopped over delivering data. Test Plan: Ran and updated the tests. The next patch will actually add a test for this code. Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911531 Signature: t1:4911531:1492643897:58aabe5a32f156dd7424cc07c990301dbd5dc1c7
2017-04-20 07:14:04 +03:00
8adc618d23082c0a5311a4bbf9ac08b9b9672471.dataidx
8adc618d23082c0a5311a4bbf9ac08b9b9672471.datapack
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
c0a571385df83107268172e0ab75abb30a12079a.histidx
c0a571385df83107268172e0ab75abb30a12079a.histpack
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
$ hg debugdatapack --long $CACHEDIR/master/packs/manifests/*.dataidx
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
subdir
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
Node Delta Base Delta Length
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
ddb35f099a648a43a997aef53123bce309c794fd 0000000000000000000000000000000000000000 43
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
Node Delta Base Delta Length
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
1be4ab2126dd2252dcae6be2aac2561dd3ddcda0 0000000000000000000000000000000000000000 95
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
treemanifest: chop off trailing slash when requesting manifests Summary: The name being passed to the store was wrong, because it had a trailing slash. This wasn't caught before because the code was reading and writing the paths with a slash at the end, so it matched as long as we only interacted with packs produced by the code. The issue became more obvious when I tried to have packs generated from revlogs interact with this code. All the tests are affected since the entry keys changed. Also use 'const ManifestFetcher &x' to pass the ref to avoid the copy. Test Plan: Tests updated Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4901274 Signature: t1:4901274:1492638476:ff28f8976657baec99effbd82ecd436f6282ea5b
2017-04-20 07:14:04 +03:00
dir
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
Node Delta Base Delta Length
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
a18d21674e76d6aab2edb46810b20fbdbd10fb4b 0000000000000000000000000000000000000000 43
treemanifest: change server to use ctreemanifest for reading revlogs Summary: The server treemanifest reading code was super slow because it used the pure python treemanifest implementation. Let's switch it to use the ctreemanifest implementation on top of a revlog store class. The tests change because the output is now in reverse topological order (deepest trees first). Test Plan: Tests updated Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4901268 Signature: t1:4901268:1492637601:371a2303ac4f6e2b81f69a85ca657849150abcf7
2017-04-20 07:14:04 +03:00
Node Delta Base Delta Length
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5 0000000000000000000000000000000000000000 95
treemanifest: add support for minimal tree pack sending Summary: The gettreepack protocol already had the arguments for the client to specify what nodes they want and what nodes they have, but it wasn't implemented (we just always returned the entire tree). This patch adds support for only sending parts of the tree that are new. In addition, if the client requests many nodes, we will only send the portion of each node that is not contained in the other nodes. For instance if you ask for X and Y, where X is the parent of Y, then we will send all of X and only the part of Y that is new. The order changed a bit, probably because we stopped over delivering data. Test Plan: Ran and updated the tests. The next patch will actually add a test for this code. Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911531 Signature: t1:4911531:1492643897:58aabe5a32f156dd7424cc07c990301dbd5dc1c7
2017-04-20 07:14:04 +03:00
dir
Node Delta Base Delta Length
bc0c2c938b929f98b1c31a8c5994396ebb096bf0 0000000000000000000000000000000000000000 43
Node Delta Base Delta Length
treemanifest: change server to use ctreemanifest for reading revlogs Summary: The server treemanifest reading code was super slow because it used the pure python treemanifest implementation. Let's switch it to use the ctreemanifest implementation on top of a revlog store class. The tests change because the output is now in reverse topological order (deepest trees first). Test Plan: Tests updated Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4901268 Signature: t1:4901268:1492637601:371a2303ac4f6e2b81f69a85ca657849150abcf7
2017-04-20 07:14:04 +03:00
ef362f8bbe8aa457b0cfc49f200cbeb7747984ed 0000000000000000000000000000000000000000 46
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
$ hg debughistorypack $CACHEDIR/master/packs/manifests/*.histidx
Node P1 Node P2 Node Link Node Copy From
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
60a7f7acb6bb 1be4ab2126dd 000000000000 bd6f9b289c01
1be4ab2126dd ef362f8bbe8a 000000000000 f15c65c6e9bd
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
ef362f8bbe8a 000000000000 000000000000 ecfb693caff5
treemanifest: chop off trailing slash when requesting manifests Summary: The name being passed to the store was wrong, because it had a trailing slash. This wasn't caught before because the code was reading and writing the paths with a slash at the end, so it matched as long as we only interacted with packs produced by the code. The issue became more obvious when I tried to have packs generated from revlogs interact with this code. All the tests are affected since the entry keys changed. Also use 'const ManifestFetcher &x' to pass the ref to avoid the copy. Test Plan: Tests updated Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4901274 Signature: t1:4901274:1492638476:ff28f8976657baec99effbd82ecd436f6282ea5b
2017-04-20 07:14:04 +03:00
dir
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
Node P1 Node P2 Node Link Node Copy From
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
a18d21674e76 bc0c2c938b92 000000000000 bd6f9b289c01
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
bc0c2c938b92 000000000000 000000000000 ecfb693caff5
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
subdir
Node P1 Node P2 Node Link Node Copy From
ddb35f099a64 000000000000 000000000000 f15c65c6e9bd
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
$ hg debugdatapack --node ef362f8bbe8aa457b0cfc49f200cbeb7747984ed $CACHEDIR/master/packs/manifests/*.dataidx
Node Delta Base Delta SHA1 Delta Length
ef362f8bbe8aa457b0cfc49f200cbeb7747984ed 0000000000000000000000000000000000000000 3b295111780498d177793f9228bf736b915f0255 46
$ hg -R ../master debugindex ../master/.hg/store/00manifesttree.i
rev offset length delta linkrev nodeid p1 p2
0 0 47 -1 0 ef362f8bbe8a 000000000000 000000000000
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
1 47 61 0 1 1be4ab2126dd ef362f8bbe8a 000000000000
2 108 58 1 2 60a7f7acb6bb 1be4ab2126dd 000000000000
treemanifest: add prefetchtrees for receiving trees from the server Summary: This adds a new prefetchtrees command and gettrees wireprotocol command for receiving trees from the server. The actual wire protocol is a custom request format, with a normal pack file response format (same as used in remotefilelog). The request args can specify which manifest nodes are wanted, for which directory and which subdirectories. I attempted to use the changegroup3 response format instead of the pack format, but changegroup is extremely intertwined with revlogs and with producing changegroups for linear swaths of the repo history, instead of random parts of just the manifest. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: stash, quark, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864435 Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-20 07:14:03 +03:00
$ hg -R ../master debugindex ../master/.hg/store/00manifest.i
rev offset length delta linkrev nodeid p1 p2
0 0 48 -1 0 ef362f8bbe8a 000000000000 000000000000
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
1 48 62 0 1 1be4ab2126dd ef362f8bbe8a 000000000000
2 110 59 1 2 60a7f7acb6bb 1be4ab2126dd 000000000000
treemanifest: automatically download missing trees on demand Summary: This adds a remote tree datastore to the store, so if we don't have a tree on local disk, we can reach out to the server for it. This also makes a small change to fastmanifest to use get() instead of getmissing() to test if the tree is available. getmissing() would indicate the tree is missing if it wasn't local, while get() will attempt to fetch it from the server. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, stash, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864645 Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-20 07:14:03 +03:00
treemanifest: add test for prefetch with base nodes Summary: Now that prefetching supports base nodes, let's add a simple test covering it. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911537 Signature: t1:4911537:1492641400:752c52acfb264ec42c5d05865bf87ebfc2558801
2017-04-20 07:14:04 +03:00
Test prefetch with base node (subdir/ shouldn't show up in the pack)
$ rm -rf $CACHEDIR/master
$ hg prefetchtrees -r '2' --base '1'
$ ls $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx
$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx
dir
Node Delta Base Delta Length
a18d21674e76 000000000000 43
Node Delta Base Delta Length
60a7f7acb6bb 000000000000 95
treemanifest: automatically download missing trees on demand Summary: This adds a remote tree datastore to the store, so if we don't have a tree on local disk, we can reach out to the server for it. This also makes a small change to fastmanifest to use get() instead of getmissing() to test if the tree is available. getmissing() would indicate the tree is missing if it wasn't local, while get() will attempt to fetch it from the server. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, stash, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864645 Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-20 07:14:03 +03:00
Test auto prefetch during normal access
$ rm -rf $CACHEDIR/master
tests_: fix osx compatibility in ls output Summary: BSD and GNU ls behave differently in case of a missing dir: - exit code is [1] vs [2] - message is also slightly different We want this to be unified for both platforms. Test Plan: - run the test on Linux - hope for the best on OSX Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4921369
2017-04-20 15:46:20 +03:00
|| ( exit 1 ) is needed because ls on OSX and Linux exits differently
$ ls $CACHEDIR/master/packs/manifests || ( exit 1 )
*No such file or directory (glob)
[1]
treemanifest: automatically download missing trees on demand Summary: This adds a remote tree datastore to the store, so if we don't have a tree on local disk, we can reach out to the server for it. This also makes a small change to fastmanifest to use get() instead of getmissing() to test if the tree is available. getmissing() would indicate the tree is missing if it wasn't local, while get() will attempt to fetch it from the server. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, stash, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864645 Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-20 07:14:03 +03:00
$ hg log -r tip --stat --pager=off
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
changeset: 2:bd6f9b289c01
treemanifest: automatically download missing trees on demand Summary: This adds a remote tree datastore to the store, so if we don't have a tree on local disk, we can reach out to the server for it. This also makes a small change to fastmanifest to use get() instead of getmissing() to test if the tree is available. getmissing() would indicate the tree is missing if it wasn't local, while get() will attempt to fetch it from the server. Test Plan: Added a test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: quark, stash, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4864645 Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-20 07:14:03 +03:00
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: modify x
dir/x | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
$ ls $CACHEDIR/master/packs/manifests
treemanifest: change test to use multiple directories Summary: A future patch will make tree prefetching more efficient. Let's change the prefetch test to have changes in two separate directories so we can observe that one directory is not downloaded when doing an efficient prefetch. Test Plan: Ran the test Reviewers: #mercurial, quark Reviewed By: quark Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4911523 Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-20 07:14:04 +03:00
148e9eb32f473ea522c591c95be0f9e772be9675.dataidx
148e9eb32f473ea522c591c95be0f9e772be9675.datapack
4ee15de76c068ec1c80e3e61f2c3c476a779078a.dataidx
4ee15de76c068ec1c80e3e61f2c3c476a779078a.datapack
6fbf331dc6290577b48479f94c2746754f4a898a.histidx
6fbf331dc6290577b48479f94c2746754f4a898a.histpack
a2b37afce1a72987e098a62a03ef89f3f208bc70.histidx
a2b37afce1a72987e098a62a03ef89f3f208bc70.histpack
treemanifest: enable auto prefetching trees during hg pull Summary: Adds a config option that will automatically download the last N trees from the server during pull. It efficiently decides what bases to set, based on what commit parents it already has. Test Plan: Added a test Reviewers: #mercurial, rmcelroy Reviewed By: rmcelroy Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4923211 Signature: t1:4923211:1493200730:39872db6aabb48c12131ff8501a950c8ee60e2b0
2017-04-27 20:44:33 +03:00
Test auto prefetch during pull
- Prefetch everything
$ rm -rf $CACHEDIR/master
$ hg pull --config treemanifest.pullprefetchcount=10 --traceback
pulling from ssh://user@dummy/master
searching for changes
no changes found
prefetching trees
$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx
subdir
Node Delta Base Delta Length
ddb35f099a64 000000000000 43
Node Delta Base Delta Length
1be4ab2126dd 000000000000 95
dir
Node Delta Base Delta Length
a18d21674e76 000000000000 43
Node Delta Base Delta Length
60a7f7acb6bb 000000000000 95
dir
Node Delta Base Delta Length
bc0c2c938b92 000000000000 43
Node Delta Base Delta Length
ef362f8bbe8a 000000000000 46
- Prefetch just the top manifest (but the full one)
$ rm -rf $CACHEDIR/master
$ hg pull --config treemanifest.pullprefetchcount=1 --traceback
pulling from ssh://user@dummy/master
searching for changes
no changes found
prefetching trees
$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx
dir
Node Delta Base Delta Length
a18d21674e76 000000000000 43
subdir
Node Delta Base Delta Length
ddb35f099a64 000000000000 43
Node Delta Base Delta Length
60a7f7acb6bb 000000000000 95
- Prefetch commit 1 then minimally prefetch commit 2
$ rm -rf $CACHEDIR/master
$ hg prefetchtrees -r 1
$ ls $CACHEDIR/master/packs/manifests/*dataidx
$TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx
$ hg pull --config treemanifest.pullprefetchcount=1 --traceback
pulling from ssh://user@dummy/master
searching for changes
no changes found
prefetching trees
$ ls $CACHEDIR/master/packs/manifests/*dataidx
$TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx
$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx
$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx
dir
Node Delta Base Delta Length
a18d21674e76 000000000000 43
Node Delta Base Delta Length
60a7f7acb6bb 000000000000 95
Reference in New Issue Copy Permalink