2019-12-10 02:24:31 +03:00
|
|
|
#chg-compatible
|
|
|
|
|
|
2020-01-20 13:42:49 +03:00
|
|
|
$ disable treemanifest
|
treemanifest: include all trees when infinitepush rebundle converts trees
Summary:
If an infinitepush bundle contains flat manifests and is served from a
treemanifest repository, it can potentially fail to send all the needed data to
the client.
Understanding the bug requires two bits of context:
1. When sending commits from a tree server to a tree client, we generally don't
send any trees because they can be fetched by the client ondemand later. The one
exception to this is for infinitepush bundles, where the trees inside the bundle
cannot be served ondemand, and therefore must be served at pull time. To do this
we check if a given manifest node exists in the repositories permanent storage,
and if it doesn't, we assume it came from an infinitepush bundle and serve it
with the pull.
2. When we lookup a manifest and fail to find a tree, our last resort is the
ondemandstore which knows how to convert a flat manifest into a tree manifest.
On the server, this is responsible for converting each of the flat bundle's
manifests into treemanifests before we serve the bundle to the client. As part
of converting the flat manifests into treemanifests, it writes the new tree
data into a pack file.
The bug is then, when serving a stack of commits, if we try to package up the
top tree first (i.e. the most recent tree), we end up converting the entire
stack from flat into trees, which inserts the bottom most trees into the
temporary pack file. Because they exist in the temporary pack file, when we
later check if they are part of the repositories store we end up finding them,
which causes us to treat them as not-infinitepush-trees which means we don't
serve them to the client.
The fix is to change the infinitepush tree-serving code to not consider the
mutable packs when checking if it should send trees.
Reviewed By: mitrandir77
Differential Revision: D14403925
fbshipit-source-id: 38043dfc49df5ff9ea2fae1d3cac341c4936509c
2019-03-11 15:35:47 +03:00
|
|
|
$ . "$TESTDIR/library.sh"
|
2019-05-24 19:57:15 +03:00
|
|
|
$ setconfig treemanifest.treeonly=False
|
treemanifest: include all trees when infinitepush rebundle converts trees
Summary:
If an infinitepush bundle contains flat manifests and is served from a
treemanifest repository, it can potentially fail to send all the needed data to
the client.
Understanding the bug requires two bits of context:
1. When sending commits from a tree server to a tree client, we generally don't
send any trees because they can be fetched by the client ondemand later. The one
exception to this is for infinitepush bundles, where the trees inside the bundle
cannot be served ondemand, and therefore must be served at pull time. To do this
we check if a given manifest node exists in the repositories permanent storage,
and if it doesn't, we assume it came from an infinitepush bundle and serve it
with the pull.
2. When we lookup a manifest and fail to find a tree, our last resort is the
ondemandstore which knows how to convert a flat manifest into a tree manifest.
On the server, this is responsible for converting each of the flat bundle's
manifests into treemanifests before we serve the bundle to the client. As part
of converting the flat manifests into treemanifests, it writes the new tree
data into a pack file.
The bug is then, when serving a stack of commits, if we try to package up the
top tree first (i.e. the most recent tree), we end up converting the entire
stack from flat into trees, which inserts the bottom most trees into the
temporary pack file. Because they exist in the temporary pack file, when we
later check if they are part of the repositories store we end up finding them,
which causes us to treat them as not-infinitepush-trees which means we don't
serve them to the client.
The fix is to change the infinitepush tree-serving code to not consider the
mutable packs when checking if it should send trees.
Reviewed By: mitrandir77
Differential Revision: D14403925
fbshipit-source-id: 38043dfc49df5ff9ea2fae1d3cac341c4936509c
2019-03-11 15:35:47 +03:00
|
|
|
|
|
|
|
|
Start with a server that has flat manifests
|
|
|
|
|
|
|
|
|
|
$ newrepo master
|
|
|
|
|
$ enable infinitepush
|
|
|
|
|
$ setconfig remotefilelog.server=true infinitepush.server=true
|
|
|
|
|
$ setconfig infinitepush.branchpattern=re:scratch/.+
|
|
|
|
|
$ setconfig infinitepush.indextype=disk infinitepush.storetype=disk
|
|
|
|
|
$ mkdir dir1
|
|
|
|
|
$ echo base > dir1/base
|
|
|
|
|
$ hg commit -Aqm base
|
|
|
|
|
|
|
|
|
|
Make a remotefilelog client
|
|
|
|
|
|
|
|
|
|
$ cd $TESTTMP
|
|
|
|
|
$ hgcloneshallow ssh://user@dummy/master client1
|
|
|
|
|
streaming all changes
|
2019-08-31 00:56:12 +03:00
|
|
|
3 files to transfer, * of data (glob)
|
treemanifest: include all trees when infinitepush rebundle converts trees
Summary:
If an infinitepush bundle contains flat manifests and is served from a
treemanifest repository, it can potentially fail to send all the needed data to
the client.
Understanding the bug requires two bits of context:
1. When sending commits from a tree server to a tree client, we generally don't
send any trees because they can be fetched by the client ondemand later. The one
exception to this is for infinitepush bundles, where the trees inside the bundle
cannot be served ondemand, and therefore must be served at pull time. To do this
we check if a given manifest node exists in the repositories permanent storage,
and if it doesn't, we assume it came from an infinitepush bundle and serve it
with the pull.
2. When we lookup a manifest and fail to find a tree, our last resort is the
ondemandstore which knows how to convert a flat manifest into a tree manifest.
On the server, this is responsible for converting each of the flat bundle's
manifests into treemanifests before we serve the bundle to the client. As part
of converting the flat manifests into treemanifests, it writes the new tree
data into a pack file.
The bug is then, when serving a stack of commits, if we try to package up the
top tree first (i.e. the most recent tree), we end up converting the entire
stack from flat into trees, which inserts the bottom most trees into the
temporary pack file. Because they exist in the temporary pack file, when we
later check if they are part of the repositories store we end up finding them,
which causes us to treat them as not-infinitepush-trees which means we don't
serve them to the client.
The fix is to change the infinitepush tree-serving code to not consider the
mutable packs when checking if it should send trees.
Reviewed By: mitrandir77
Differential Revision: D14403925
fbshipit-source-id: 38043dfc49df5ff9ea2fae1d3cac341c4936509c
2019-03-11 15:35:47 +03:00
|
|
|
transferred * (glob)
|
|
|
|
|
searching for changes
|
|
|
|
|
no changes found
|
|
|
|
|
updating to branch default
|
|
|
|
|
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
2020-01-09 18:44:55 +03:00
|
|
|
1 files fetched over 1 fetches - * (glob) (?)
|
treemanifest: include all trees when infinitepush rebundle converts trees
Summary:
If an infinitepush bundle contains flat manifests and is served from a
treemanifest repository, it can potentially fail to send all the needed data to
the client.
Understanding the bug requires two bits of context:
1. When sending commits from a tree server to a tree client, we generally don't
send any trees because they can be fetched by the client ondemand later. The one
exception to this is for infinitepush bundles, where the trees inside the bundle
cannot be served ondemand, and therefore must be served at pull time. To do this
we check if a given manifest node exists in the repositories permanent storage,
and if it doesn't, we assume it came from an infinitepush bundle and serve it
with the pull.
2. When we lookup a manifest and fail to find a tree, our last resort is the
ondemandstore which knows how to convert a flat manifest into a tree manifest.
On the server, this is responsible for converting each of the flat bundle's
manifests into treemanifests before we serve the bundle to the client. As part
of converting the flat manifests into treemanifests, it writes the new tree
data into a pack file.
The bug is then, when serving a stack of commits, if we try to package up the
top tree first (i.e. the most recent tree), we end up converting the entire
stack from flat into trees, which inserts the bottom most trees into the
temporary pack file. Because they exist in the temporary pack file, when we
later check if they are part of the repositories store we end up finding them,
which causes us to treat them as not-infinitepush-trees which means we don't
serve them to the client.
The fix is to change the infinitepush tree-serving code to not consider the
mutable packs when checking if it should send trees.
Reviewed By: mitrandir77
Differential Revision: D14403925
fbshipit-source-id: 38043dfc49df5ff9ea2fae1d3cac341c4936509c
2019-03-11 15:35:47 +03:00
|
|
|
$ cd client1
|
|
|
|
|
$ enable infinitepush
|
|
|
|
|
$ setconfig infinitepush.server=false infinitepush.branchpattern=re:scratch/.+
|
|
|
|
|
|
|
|
|
|
Push a bundle with four commits
|
|
|
|
|
|
|
|
|
|
$ mkdir dir2
|
|
|
|
|
$ echo 1 > dir2/bundled
|
|
|
|
|
$ hg commit -Aqm bundled1
|
|
|
|
|
$ echo 2 > dir2/bundled
|
|
|
|
|
$ hg commit -Aqm bundled2
|
|
|
|
|
$ echo 3 > dir2/bundled
|
|
|
|
|
$ hg commit -Aqm bundled3
|
|
|
|
|
$ echo 4 > dir2/bundled
|
|
|
|
|
$ hg commit -Aqm bundled4
|
|
|
|
|
$ tglog
|
|
|
|
|
@ 4: d1944cedf06c 'bundled4'
|
|
|
|
|
|
|
|
|
|
|
o 3: 916baec915e2 'bundled3'
|
|
|
|
|
|
|
|
|
|
|
o 2: 9494660bae92 'bundled2'
|
|
|
|
|
|
|
|
|
|
|
o 1: f570e0648bfb 'bundled1'
|
|
|
|
|
|
|
|
|
|
|
o 0: f7e449aab27f 'base'
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
$ hg push -r . --to scratch/bundled --create
|
|
|
|
|
pushing to ssh://user@dummy/master
|
|
|
|
|
searching for changes
|
|
|
|
|
remote: pushing 4 commits:
|
|
|
|
|
remote: f570e0648bfb bundled1
|
|
|
|
|
remote: 9494660bae92 bundled2
|
|
|
|
|
remote: 916baec915e2 bundled3
|
|
|
|
|
remote: d1944cedf06c bundled4
|
|
|
|
|
|
|
|
|
|
Upgrade the server to treemanifest
|
|
|
|
|
|
|
|
|
|
$ cd $TESTTMP/master
|
|
|
|
|
$ enable treemanifest
|
|
|
|
|
$ setconfig treemanifest.server=true
|
|
|
|
|
$ setconfig fastmanifest.usetree=true fastmanifest.usecache=false
|
|
|
|
|
|
|
|
|
|
$ hg backfilltree
|
|
|
|
|
|
|
|
|
|
$ setconfig treemanifest.treeonly=true
|
|
|
|
|
|
|
|
|
|
Clone another client, this time treeonly
|
|
|
|
|
|
|
|
|
|
$ cd $TESTTMP
|
|
|
|
|
$ hgcloneshallow ssh://user@dummy/master client2 --config extensions.treemanifest= --config treemanifest.treeonly=true
|
|
|
|
|
streaming all changes
|
2019-08-31 00:56:12 +03:00
|
|
|
3 files to transfer, * of data (glob)
|
treemanifest: include all trees when infinitepush rebundle converts trees
Summary:
If an infinitepush bundle contains flat manifests and is served from a
treemanifest repository, it can potentially fail to send all the needed data to
the client.
Understanding the bug requires two bits of context:
1. When sending commits from a tree server to a tree client, we generally don't
send any trees because they can be fetched by the client ondemand later. The one
exception to this is for infinitepush bundles, where the trees inside the bundle
cannot be served ondemand, and therefore must be served at pull time. To do this
we check if a given manifest node exists in the repositories permanent storage,
and if it doesn't, we assume it came from an infinitepush bundle and serve it
with the pull.
2. When we lookup a manifest and fail to find a tree, our last resort is the
ondemandstore which knows how to convert a flat manifest into a tree manifest.
On the server, this is responsible for converting each of the flat bundle's
manifests into treemanifests before we serve the bundle to the client. As part
of converting the flat manifests into treemanifests, it writes the new tree
data into a pack file.
The bug is then, when serving a stack of commits, if we try to package up the
top tree first (i.e. the most recent tree), we end up converting the entire
stack from flat into trees, which inserts the bottom most trees into the
temporary pack file. Because they exist in the temporary pack file, when we
later check if they are part of the repositories store we end up finding them,
which causes us to treat them as not-infinitepush-trees which means we don't
serve them to the client.
The fix is to change the infinitepush tree-serving code to not consider the
mutable packs when checking if it should send trees.
Reviewed By: mitrandir77
Differential Revision: D14403925
fbshipit-source-id: 38043dfc49df5ff9ea2fae1d3cac341c4936509c
2019-03-11 15:35:47 +03:00
|
|
|
transferred * (glob)
|
|
|
|
|
searching for changes
|
|
|
|
|
no changes found
|
|
|
|
|
updating to branch default
|
|
|
|
|
fetching tree '' a8b0ba84fc9d10d4e1e5be15a0f2b83872021770
|
|
|
|
|
2 trees fetched over * (glob)
|
|
|
|
|
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
|
|
|
|
$ cd client2
|
|
|
|
|
$ enable infinitepush
|
|
|
|
|
$ enable treemanifest
|
|
|
|
|
$ setconfig fastmanifest.usetree=true fastmanifest.usecache=false
|
|
|
|
|
$ setconfig treemanifest.treeonly=true
|
|
|
|
|
|
|
|
|
|
Pull three of the commits, triggering a rebundle. The server must include all of the
|
|
|
|
|
trees for the infinitepush commits.
|
|
|
|
|
|
|
|
|
|
$ hg pull -r 916baec915e2
|
|
|
|
|
pulling from ssh://user@dummy/master
|
|
|
|
|
searching for changes
|
|
|
|
|
adding changesets
|
|
|
|
|
adding manifests
|
|
|
|
|
adding file changes
|
|
|
|
|
added 3 changesets with 3 changes to 1 files
|
|
|
|
|
|
|
|
|
|
Make sure we can check out the commit we pulled
|
|
|
|
|
|
|
|
|
|
$ hg update 916baec915e2
|
|
|
|
|
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
