facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-07 23:38:50 +03:00
Code Issues Projects Releases Wiki Activity
6945cf0f5b
sapling/tests/test-bundle-type.t

196 lines
4.4 KiB
Perl
Raw Normal View History

test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.
2015-10-20 12:48:49 +03:00
$ cat << EOF >> $HGRCPATH
> [format]
> usegeneraldelta=yes
> EOF
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
bundle w/o type option
$ hg init t1
$ hg init t2
$ cd t1
$ echo blablablablabla > file.txt
$ hg ci -Ama
adding file.txt
$ hg log | grep summary
summary: a
$ hg bundle ../b1 ../t2
searching for changes
1 changesets found
$ cd ../t2
$ hg pull ../b1
pulling from ../b1
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
(run 'hg update' to get a working copy)
$ hg up
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg log | grep summary
summary: a
$ cd ..
test bundle types
util: implement zstd compression engine Now that zstd is vendored and being built (in some configurations), we can implement a compression engine for zstd! The zstd engine is a little different from existing engines. Because it may not always be present, we have to defer load the module in case importing it fails. We facilitate this via a cached property that holds a reference to the module or None. The "available" method is implemented to reflect reality. The zstd engine declares its ability to handle bundles using the "zstd" human name and the "ZS" internal name. The latter was chosen because internal names are 2 characters (by only convention I think) and "ZS" seems reasonable. The engine, like others, supports specifying the compression level. However, there are no consumers of this API that yet pass in that argument. I have plans to change that, so stay tuned. Since all we need to do to support bundle generation with a new compression engine is implement and register the compression engine, bundle generation with zstd "just works!" Tests demonstrating this have been added. How does performance of zstd for bundle generation compare? On the mozilla-unified repo, `hg bundle --all -t <engine>-v2` yields the following on my i7-6700K on Linux: engine CPU time bundle size vs orig size throughput none 97.0s 4,054,405,584 100.0% 41.8 MB/s bzip2 (l=9) 393.6s 975,343,098 24.0% 10.3 MB/s gzip (l=6) 184.0s 1,140,533,074 28.1% 22.0 MB/s zstd (l=1) 108.2s 1,119,434,718 27.6% 37.5 MB/s zstd (l=2) 111.3s 1,078,328,002 26.6% 36.4 MB/s zstd (l=3) 113.7s 1,011,823,727 25.0% 35.7 MB/s zstd (l=4) 116.0s 1,008,965,888 24.9% 35.0 MB/s zstd (l=5) 121.0s 977,203,148 24.1% 33.5 MB/s zstd (l=6) 131.7s 927,360,198 22.9% 30.8 MB/s zstd (l=7) 139.0s 912,808,505 22.5% 29.2 MB/s zstd (l=12) 198.1s 854,527,714 21.1% 20.5 MB/s zstd (l=18) 681.6s 789,750,690 19.5% 5.9 MB/s On compression, zstd for bundle generation delivers: * better compression than gzip with significantly less CPU utilization * better than bzip2 compression ratios while still being significantly faster than gzip * ability to aggressively tune compression level to achieve significantly smaller bundles That last point is important. With clone bundles, a server can pre-generate a bundle file, upload it to a static file server, and redirect clients to transparently download it during clone. The server could choose to produce a zstd bundle with the highest compression settings possible. This would take a very long time - a magnitude longer than a typical zstd bundle generation - but the result would be hundreds of megabytes smaller! For the clone volume we do at Mozilla, this could translate to petabytes of bandwidth savings per year and faster clones (due to smaller transfer size). I don't have detailed numbers to report on decompression. However, zstd decompression is fast: >1 GB/s output throughput on this machine, even through the Python bindings. And it can do that regardless of the compression level of the input. By the time you have enough data to worry about overhead of decompression, you have plenty of other things to worry about performance wise. zstd is wins all around. I can't wait to implement support for it on the wire protocol and in revlogs.
2016-11-11 12:10:07 +03:00
$ testbundle() {
> echo % test bundle type $1
> hg init t$1
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
> cd t1
util: implement zstd compression engine Now that zstd is vendored and being built (in some configurations), we can implement a compression engine for zstd! The zstd engine is a little different from existing engines. Because it may not always be present, we have to defer load the module in case importing it fails. We facilitate this via a cached property that holds a reference to the module or None. The "available" method is implemented to reflect reality. The zstd engine declares its ability to handle bundles using the "zstd" human name and the "ZS" internal name. The latter was chosen because internal names are 2 characters (by only convention I think) and "ZS" seems reasonable. The engine, like others, supports specifying the compression level. However, there are no consumers of this API that yet pass in that argument. I have plans to change that, so stay tuned. Since all we need to do to support bundle generation with a new compression engine is implement and register the compression engine, bundle generation with zstd "just works!" Tests demonstrating this have been added. How does performance of zstd for bundle generation compare? On the mozilla-unified repo, `hg bundle --all -t <engine>-v2` yields the following on my i7-6700K on Linux: engine CPU time bundle size vs orig size throughput none 97.0s 4,054,405,584 100.0% 41.8 MB/s bzip2 (l=9) 393.6s 975,343,098 24.0% 10.3 MB/s gzip (l=6) 184.0s 1,140,533,074 28.1% 22.0 MB/s zstd (l=1) 108.2s 1,119,434,718 27.6% 37.5 MB/s zstd (l=2) 111.3s 1,078,328,002 26.6% 36.4 MB/s zstd (l=3) 113.7s 1,011,823,727 25.0% 35.7 MB/s zstd (l=4) 116.0s 1,008,965,888 24.9% 35.0 MB/s zstd (l=5) 121.0s 977,203,148 24.1% 33.5 MB/s zstd (l=6) 131.7s 927,360,198 22.9% 30.8 MB/s zstd (l=7) 139.0s 912,808,505 22.5% 29.2 MB/s zstd (l=12) 198.1s 854,527,714 21.1% 20.5 MB/s zstd (l=18) 681.6s 789,750,690 19.5% 5.9 MB/s On compression, zstd for bundle generation delivers: * better compression than gzip with significantly less CPU utilization * better than bzip2 compression ratios while still being significantly faster than gzip * ability to aggressively tune compression level to achieve significantly smaller bundles That last point is important. With clone bundles, a server can pre-generate a bundle file, upload it to a static file server, and redirect clients to transparently download it during clone. The server could choose to produce a zstd bundle with the highest compression settings possible. This would take a very long time - a magnitude longer than a typical zstd bundle generation - but the result would be hundreds of megabytes smaller! For the clone volume we do at Mozilla, this could translate to petabytes of bandwidth savings per year and faster clones (due to smaller transfer size). I don't have detailed numbers to report on decompression. However, zstd decompression is fast: >1 GB/s output throughput on this machine, even through the Python bindings. And it can do that regardless of the compression level of the input. By the time you have enough data to worry about overhead of decompression, you have plenty of other things to worry about performance wise. zstd is wins all around. I can't wait to implement support for it on the wire protocol and in revlogs.
2016-11-11 12:10:07 +03:00
> hg bundle -t $1 ../b$1 ../t$1
> f -q -B6 -D ../b$1; echo
> cd ../t$1
> hg debugbundle ../b$1
> hg debugbundle --spec ../b$1
test-bundle-type: replace unbundle with debugbundle We now have a convenient command to look at bundle contents, let's use it.
2015-10-02 06:15:00 +03:00
> echo
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
> cd ..
util: implement zstd compression engine Now that zstd is vendored and being built (in some configurations), we can implement a compression engine for zstd! The zstd engine is a little different from existing engines. Because it may not always be present, we have to defer load the module in case importing it fails. We facilitate this via a cached property that holds a reference to the module or None. The "available" method is implemented to reflect reality. The zstd engine declares its ability to handle bundles using the "zstd" human name and the "ZS" internal name. The latter was chosen because internal names are 2 characters (by only convention I think) and "ZS" seems reasonable. The engine, like others, supports specifying the compression level. However, there are no consumers of this API that yet pass in that argument. I have plans to change that, so stay tuned. Since all we need to do to support bundle generation with a new compression engine is implement and register the compression engine, bundle generation with zstd "just works!" Tests demonstrating this have been added. How does performance of zstd for bundle generation compare? On the mozilla-unified repo, `hg bundle --all -t <engine>-v2` yields the following on my i7-6700K on Linux: engine CPU time bundle size vs orig size throughput none 97.0s 4,054,405,584 100.0% 41.8 MB/s bzip2 (l=9) 393.6s 975,343,098 24.0% 10.3 MB/s gzip (l=6) 184.0s 1,140,533,074 28.1% 22.0 MB/s zstd (l=1) 108.2s 1,119,434,718 27.6% 37.5 MB/s zstd (l=2) 111.3s 1,078,328,002 26.6% 36.4 MB/s zstd (l=3) 113.7s 1,011,823,727 25.0% 35.7 MB/s zstd (l=4) 116.0s 1,008,965,888 24.9% 35.0 MB/s zstd (l=5) 121.0s 977,203,148 24.1% 33.5 MB/s zstd (l=6) 131.7s 927,360,198 22.9% 30.8 MB/s zstd (l=7) 139.0s 912,808,505 22.5% 29.2 MB/s zstd (l=12) 198.1s 854,527,714 21.1% 20.5 MB/s zstd (l=18) 681.6s 789,750,690 19.5% 5.9 MB/s On compression, zstd for bundle generation delivers: * better compression than gzip with significantly less CPU utilization * better than bzip2 compression ratios while still being significantly faster than gzip * ability to aggressively tune compression level to achieve significantly smaller bundles That last point is important. With clone bundles, a server can pre-generate a bundle file, upload it to a static file server, and redirect clients to transparently download it during clone. The server could choose to produce a zstd bundle with the highest compression settings possible. This would take a very long time - a magnitude longer than a typical zstd bundle generation - but the result would be hundreds of megabytes smaller! For the clone volume we do at Mozilla, this could translate to petabytes of bandwidth savings per year and faster clones (due to smaller transfer size). I don't have detailed numbers to report on decompression. However, zstd decompression is fast: >1 GB/s output throughput on this machine, even through the Python bindings. And it can do that regardless of the compression level of the input. By the time you have enough data to worry about overhead of decompression, you have plenty of other things to worry about performance wise. zstd is wins all around. I can't wait to implement support for it on the wire protocol and in revlogs.
2016-11-11 12:10:07 +03:00
> }
$ for t in "None" "bzip2" "gzip" "none-v2" "v2" "v1" "gzip-v1"; do
> testbundle $t
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
> done
% test bundle type None
searching for changes
1 changesets found
test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.
2015-10-20 12:48:49 +03:00
HG20\x00\x00 (esc)
Stream params: {}
bundle2: store changeset count when creating file bundles The bundle2 changegroup part has an advisory param saying how many changesets are in the part. Before this patch, we were setting this part when generating bundle2 parts via the wire protocol but not when generating local bundle2 files. A side effect of not setting the changeset count part is that progress bars don't work when applying changesets. As the tests show, this impacted clone bundles, shelve, backup bundles, `hg unbundle`, and anything touching bundle2 files. This patch adds a backdoor to allow us to pass state from changegroup generation into the unbundler. We store the number of changesets in the changegroup in this state and use it to populate the aforementioned advisory part parameter when generating the bundle2 bundle. I concede that I'm not thrilled by how state is being passed in changegroup.py (it feels a bit hacky). I would love to overhaul the rather confusing set of functions in changegroup.py with something that passes rich objects around instead of e.g. low-level generators. However, given the code freeze for 3.9 is imminent, I'd rather not undertake this endeavor right now. This feels like the easiest way to get the parameter added to the changegroup part.
2016-07-18 01:13:51 +03:00
changegroup -- "sortdict([('version', '02'), ('nbchanges', '1')])"
test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.
2015-10-20 12:48:49 +03:00
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.
2016-01-15 09:57:55 +03:00
none-v2
test-bundle-type: replace unbundle with debugbundle We now have a convenient command to look at bundle contents, let's use it.
2015-10-02 06:15:00 +03:00
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
% test bundle type bzip2
searching for changes
1 changesets found
test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.
2015-10-20 12:48:49 +03:00
HG20\x00\x00 (esc)
util: implement a deterministic __repr__ on sortdict `hg debugbundle` is calling repr() on bundle2 part params, which are now util.sortdict instances. Unfortunately, repr() doesn't appear to be deterministic for util.sortdict. So, we implement one. We include the type name because that's the common convention for __repr__ implementations. Having the type name in `hg debugbundle` is a bit ugly. But it's a debug command and I don't care enough to fix it.
2016-07-18 01:10:30 +03:00
Stream params: sortdict([('Compression', 'BZ')])
bundle2: store changeset count when creating file bundles The bundle2 changegroup part has an advisory param saying how many changesets are in the part. Before this patch, we were setting this part when generating bundle2 parts via the wire protocol but not when generating local bundle2 files. A side effect of not setting the changeset count part is that progress bars don't work when applying changesets. As the tests show, this impacted clone bundles, shelve, backup bundles, `hg unbundle`, and anything touching bundle2 files. This patch adds a backdoor to allow us to pass state from changegroup generation into the unbundler. We store the number of changesets in the changegroup in this state and use it to populate the aforementioned advisory part parameter when generating the bundle2 bundle. I concede that I'm not thrilled by how state is being passed in changegroup.py (it feels a bit hacky). I would love to overhaul the rather confusing set of functions in changegroup.py with something that passes rich objects around instead of e.g. low-level generators. However, given the code freeze for 3.9 is imminent, I'd rather not undertake this endeavor right now. This feels like the easiest way to get the parameter added to the changegroup part.
2016-07-18 01:13:51 +03:00
changegroup -- "sortdict([('version', '02'), ('nbchanges', '1')])"
test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.
2015-10-20 12:48:49 +03:00
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.
2016-01-15 09:57:55 +03:00
bzip2-v2
test-bundle-type: replace unbundle with debugbundle We now have a convenient command to look at bundle contents, let's use it.
2015-10-02 06:15:00 +03:00
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
% test bundle type gzip
searching for changes
1 changesets found
test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.
2015-10-20 12:48:49 +03:00
HG20\x00\x00 (esc)
util: implement a deterministic __repr__ on sortdict `hg debugbundle` is calling repr() on bundle2 part params, which are now util.sortdict instances. Unfortunately, repr() doesn't appear to be deterministic for util.sortdict. So, we implement one. We include the type name because that's the common convention for __repr__ implementations. Having the type name in `hg debugbundle` is a bit ugly. But it's a debug command and I don't care enough to fix it.
2016-07-18 01:10:30 +03:00
Stream params: sortdict([('Compression', 'GZ')])
bundle2: store changeset count when creating file bundles The bundle2 changegroup part has an advisory param saying how many changesets are in the part. Before this patch, we were setting this part when generating bundle2 parts via the wire protocol but not when generating local bundle2 files. A side effect of not setting the changeset count part is that progress bars don't work when applying changesets. As the tests show, this impacted clone bundles, shelve, backup bundles, `hg unbundle`, and anything touching bundle2 files. This patch adds a backdoor to allow us to pass state from changegroup generation into the unbundler. We store the number of changesets in the changegroup in this state and use it to populate the aforementioned advisory part parameter when generating the bundle2 bundle. I concede that I'm not thrilled by how state is being passed in changegroup.py (it feels a bit hacky). I would love to overhaul the rather confusing set of functions in changegroup.py with something that passes rich objects around instead of e.g. low-level generators. However, given the code freeze for 3.9 is imminent, I'd rather not undertake this endeavor right now. This feels like the easiest way to get the parameter added to the changegroup part.
2016-07-18 01:13:51 +03:00
changegroup -- "sortdict([('version', '02'), ('nbchanges', '1')])"
test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.
2015-10-20 12:48:49 +03:00
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.
2016-01-15 09:57:55 +03:00
gzip-v2
test-bundle-type: replace unbundle with debugbundle We now have a convenient command to look at bundle contents, let's use it.
2015-10-02 06:15:00 +03:00
bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.
2015-10-02 05:16:00 +03:00
% test bundle type none-v2
searching for changes
1 changesets found
HG20\x00\x00 (esc)
Stream params: {}
bundle2: store changeset count when creating file bundles The bundle2 changegroup part has an advisory param saying how many changesets are in the part. Before this patch, we were setting this part when generating bundle2 parts via the wire protocol but not when generating local bundle2 files. A side effect of not setting the changeset count part is that progress bars don't work when applying changesets. As the tests show, this impacted clone bundles, shelve, backup bundles, `hg unbundle`, and anything touching bundle2 files. This patch adds a backdoor to allow us to pass state from changegroup generation into the unbundler. We store the number of changesets in the changegroup in this state and use it to populate the aforementioned advisory part parameter when generating the bundle2 bundle. I concede that I'm not thrilled by how state is being passed in changegroup.py (it feels a bit hacky). I would love to overhaul the rather confusing set of functions in changegroup.py with something that passes rich objects around instead of e.g. low-level generators. However, given the code freeze for 3.9 is imminent, I'd rather not undertake this endeavor right now. This feels like the easiest way to get the parameter added to the changegroup part.
2016-07-18 01:13:51 +03:00
changegroup -- "sortdict([('version', '02'), ('nbchanges', '1')])"
bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.
2015-10-02 05:16:00 +03:00
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.
2016-01-15 09:57:55 +03:00
none-v2
bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.
2015-10-02 05:16:00 +03:00
% test bundle type v2
searching for changes
1 changesets found
HG20\x00\x00 (esc)
util: implement a deterministic __repr__ on sortdict `hg debugbundle` is calling repr() on bundle2 part params, which are now util.sortdict instances. Unfortunately, repr() doesn't appear to be deterministic for util.sortdict. So, we implement one. We include the type name because that's the common convention for __repr__ implementations. Having the type name in `hg debugbundle` is a bit ugly. But it's a debug command and I don't care enough to fix it.
2016-07-18 01:10:30 +03:00
Stream params: sortdict([('Compression', 'BZ')])
bundle2: store changeset count when creating file bundles The bundle2 changegroup part has an advisory param saying how many changesets are in the part. Before this patch, we were setting this part when generating bundle2 parts via the wire protocol but not when generating local bundle2 files. A side effect of not setting the changeset count part is that progress bars don't work when applying changesets. As the tests show, this impacted clone bundles, shelve, backup bundles, `hg unbundle`, and anything touching bundle2 files. This patch adds a backdoor to allow us to pass state from changegroup generation into the unbundler. We store the number of changesets in the changegroup in this state and use it to populate the aforementioned advisory part parameter when generating the bundle2 bundle. I concede that I'm not thrilled by how state is being passed in changegroup.py (it feels a bit hacky). I would love to overhaul the rather confusing set of functions in changegroup.py with something that passes rich objects around instead of e.g. low-level generators. However, given the code freeze for 3.9 is imminent, I'd rather not undertake this endeavor right now. This feels like the easiest way to get the parameter added to the changegroup part.
2016-07-18 01:13:51 +03:00
changegroup -- "sortdict([('version', '02'), ('nbchanges', '1')])"
bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.
2015-10-02 05:16:00 +03:00
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.
2016-01-15 09:57:55 +03:00
bzip2-v2
bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.
2015-10-02 05:16:00 +03:00
% test bundle type v1
searching for changes
1 changesets found
HG10BZ
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.
2016-01-15 09:57:55 +03:00
bzip2-v1
bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.
2015-10-02 05:16:00 +03:00
% test bundle type gzip-v1
searching for changes
1 changesets found
HG10GZ
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.
2016-01-15 09:57:55 +03:00
gzip-v1
bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.
2015-10-02 05:16:00 +03:00
commands: config option to control bundle compression level Currently, bundle compression uses the default compression level for the active compression engine. The default compression level is tuned as a compromise between speed and size. Some scenarios may call for a different compression level. For example, with clone bundles, bundles are generated once and used several times. Since the cost to generate is paid infrequently, server operators may wish to trade extra CPU time for better compression ratios. This patch introduces an experimental and undocumented config option to control the bundle compression level. As the inline comment says, this approach is a bit hacky. I'd prefer for the compression level to be encoded in the bundle spec. e.g. "zstd-v2;complevel=15." However, given that the 4.1 freeze is imminent, I'm not comfortable implementing this user-facing change without much time to test and consider the implications. So, we're going with the quick and dirty solution for now. Having this option in the 4.1 release will enable Mozilla to easily produce and test zlib and zstd bundles with non-default compression levels in production. This will help drive future development of the feature and zstd integration with Mercurial.
2017-01-10 22:20:32 +03:00
Compression level can be adjusted for bundle2 bundles
$ hg init test-complevel
$ cd test-complevel
$ cat > file0 << EOF
> this is a file
> with some text
> and some more text
> and other content
> EOF
$ cat > file1 << EOF
> this is another file
> with some other content
> and repeated, repeated, repeated, repeated content
> EOF
$ hg -q commit -A -m initial
$ hg bundle -a -t gzip-v2 gzip-v2.hg
1 changesets found
$ f --size gzip-v2.hg
gzip-v2.hg: size=427
$ hg --config experimental.bundlecomplevel=1 bundle -a -t gzip-v2 gzip-v2-level1.hg
1 changesets found
$ f --size gzip-v2-level1.hg
gzip-v2-level1.hg: size=435
$ cd ..
util: implement zstd compression engine Now that zstd is vendored and being built (in some configurations), we can implement a compression engine for zstd! The zstd engine is a little different from existing engines. Because it may not always be present, we have to defer load the module in case importing it fails. We facilitate this via a cached property that holds a reference to the module or None. The "available" method is implemented to reflect reality. The zstd engine declares its ability to handle bundles using the "zstd" human name and the "ZS" internal name. The latter was chosen because internal names are 2 characters (by only convention I think) and "ZS" seems reasonable. The engine, like others, supports specifying the compression level. However, there are no consumers of this API that yet pass in that argument. I have plans to change that, so stay tuned. Since all we need to do to support bundle generation with a new compression engine is implement and register the compression engine, bundle generation with zstd "just works!" Tests demonstrating this have been added. How does performance of zstd for bundle generation compare? On the mozilla-unified repo, `hg bundle --all -t <engine>-v2` yields the following on my i7-6700K on Linux: engine CPU time bundle size vs orig size throughput none 97.0s 4,054,405,584 100.0% 41.8 MB/s bzip2 (l=9) 393.6s 975,343,098 24.0% 10.3 MB/s gzip (l=6) 184.0s 1,140,533,074 28.1% 22.0 MB/s zstd (l=1) 108.2s 1,119,434,718 27.6% 37.5 MB/s zstd (l=2) 111.3s 1,078,328,002 26.6% 36.4 MB/s zstd (l=3) 113.7s 1,011,823,727 25.0% 35.7 MB/s zstd (l=4) 116.0s 1,008,965,888 24.9% 35.0 MB/s zstd (l=5) 121.0s 977,203,148 24.1% 33.5 MB/s zstd (l=6) 131.7s 927,360,198 22.9% 30.8 MB/s zstd (l=7) 139.0s 912,808,505 22.5% 29.2 MB/s zstd (l=12) 198.1s 854,527,714 21.1% 20.5 MB/s zstd (l=18) 681.6s 789,750,690 19.5% 5.9 MB/s On compression, zstd for bundle generation delivers: * better compression than gzip with significantly less CPU utilization * better than bzip2 compression ratios while still being significantly faster than gzip * ability to aggressively tune compression level to achieve significantly smaller bundles That last point is important. With clone bundles, a server can pre-generate a bundle file, upload it to a static file server, and redirect clients to transparently download it during clone. The server could choose to produce a zstd bundle with the highest compression settings possible. This would take a very long time - a magnitude longer than a typical zstd bundle generation - but the result would be hundreds of megabytes smaller! For the clone volume we do at Mozilla, this could translate to petabytes of bandwidth savings per year and faster clones (due to smaller transfer size). I don't have detailed numbers to report on decompression. However, zstd decompression is fast: >1 GB/s output throughput on this machine, even through the Python bindings. And it can do that regardless of the compression level of the input. By the time you have enough data to worry about overhead of decompression, you have plenty of other things to worry about performance wise. zstd is wins all around. I can't wait to implement support for it on the wire protocol and in revlogs.
2016-11-11 12:10:07 +03:00
#if zstd
$ for t in "zstd" "zstd-v2"; do
> testbundle $t
> done
% test bundle type zstd
searching for changes
1 changesets found
HG20\x00\x00 (esc)
Stream params: sortdict([('Compression', 'ZS')])
changegroup -- "sortdict([('version', '02'), ('nbchanges', '1')])"
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
zstd-v2
% test bundle type zstd-v2
searching for changes
1 changesets found
HG20\x00\x00 (esc)
Stream params: sortdict([('Compression', 'ZS')])
changegroup -- "sortdict([('version', '02'), ('nbchanges', '1')])"
c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
zstd-v2
#else
zstd is a valid engine but isn't available
$ hg -R t1 bundle -a -t zstd irrelevant.hg
abort: compression engine zstd could not be loaded
[255]
#endif
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
test garbage file
$ echo garbage > bgarbage
$ hg init tgarbage
$ cd tgarbage
$ hg pull ../bgarbage
pull: print "pulling from foo" before accessing the other repo 1. This is consistent with pushing. 2. This allows to see the URL of the other repo in case accessing the repo fails, e.g. wrong ssh path or issues with the https certificate, without using --debug or showconfig paths. Additionally add test for this in the context of ssh with a wrong path.
2015-02-24 12:55:24 +03:00
pulling from ../bgarbage
bundlerepo: remove duplication of bundle decompressors
2010-08-26 01:55:54 +04:00
abort: ../bgarbage: not a Mercurial bundle
tests: add exit codes to unified tests
2010-09-17 02:51:32 +04:00
[255]
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
$ cd ..
test invalid bundle type
$ cd t1
$ hg bundle -a -t garbage ../bgarbage
exchange: refactor bundle specification parsing The old code was tailored to `hg bundle` usage and not appropriate for use as a general API, which clone bundles will require. The code has been rewritten to make it more generally suitable. We introduce dedicated error types to represent invalid and unsupported bundle specifications. The reason we need dedicated error types (rather than error.Abort) is because clone bundles will want to catch these exception as part of filtering entries. We don't want to swallow error.Abort on principle.
2015-10-13 20:57:54 +03:00
abort: garbage is not a recognized bundle specification
bundle: use single quotes in use warning
2016-09-21 02:46:15 +03:00
(see 'hg help bundle' for supported values for --type)
tests: add exit codes to unified tests
2010-09-17 02:51:32 +04:00
[255]
tests: unify test-bundle-type
2010-08-14 05:23:56 +04:00
$ cd ..
Reference in New Issue Copy Permalink