facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-12-27 15:13:37 +03:00
Code Issues Projects Releases Wiki Activity
c7d12b648f
sapling/eden/scm/tests/test-fb-hgext-remotefilelog-repack-remove-old.t

48 lines
1.4 KiB
Perl
Raw Normal View History

tests: enable chg for 572 tests Summary: Add `#chg-compatible` to 572 tests that seem to pass with chg enabled. This should make them run faster. Reviewed By: xavierd Differential Revision: D18870507 fbshipit-source-id: fe895e733efffc9286cd3d17c7a156c803124395
2019-12-10 02:24:31 +03:00
#chg-compatible
tests: use disable treemanifest in place of setconfig Summary: Use `disable treemanifest` in place of `setconfig extensions.treemanifest=!` Reviewed By: quark-zju Differential Revision: D19457678 fbshipit-source-id: 47c8b3c4f4ed5cfc97275ca67afab4a86ffe7f0d
2020-01-20 13:42:49 +03:00
$ disable treemanifest
remotefilelog: enforce a limit on the packfiles size Summary: Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to limit the size of the hgcache. This will be particularly challenging for Sandcastle/Quicksand as they already see hgcache over 100GB. The long-term plan is switching to IndexedLog based stores with a log rotate functionality to control the cache size. In the meantime, we can implement a basic logic to enforce the size of the hgcache that simply remove packfiles once the cache is over the configured size. One complication of this method is that several concurrent Mercurial processes could be running and accessing the packfiles being removed. In this case, we can split the packfiles in 2 categories: ones created a while back, and new ones. Removing packfiles from the first case, lookups will simply raise a KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is acceptable. The second category belongs to ones that were just created by downloading them from Memcache/Mononoke, and the code strongly assume that they will stick around. A failure at this point will not be recovered. One way of fixing this would be to handle these failures properly and simply retry, the other is to not remove new packfiles. A time of 10 minutes was chosen to categorize the packfiles. Reviewed By: quark-zju Differential Revision: D15014076 fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd
2019-04-23 22:55:24 +03:00
$ . "$TESTDIR/library.sh"
$ cat >> $HGRCPATH <<EOF
> [remotefilelog]
> fetchpacks=True
> EOF
$ hginit master
$ cd master
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> server=True
> serverexpiration=-1
> EOF
$ echo x > x
$ hg commit -qAm x
$ echo x >> x
$ hg commit -qAm x2
$ cd ..
$ hgcloneshallow ssh://user@dummy/master shallow -q
tests: make file fetching messages optional Summary: The Rust ContentStore will not print these, and in order to better understand what tests are broken with it enabled, let's make them optional. Reviewed By: quark-zju Differential Revision: D19318848 fbshipit-source-id: 87a1da510b51adc40a0d08157925a990456d3e9c
2020-01-09 18:44:55 +03:00
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob) (?)
remotefilelog: enforce a limit on the packfiles size Summary: Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to limit the size of the hgcache. This will be particularly challenging for Sandcastle/Quicksand as they already see hgcache over 100GB. The long-term plan is switching to IndexedLog based stores with a log rotate functionality to control the cache size. In the meantime, we can implement a basic logic to enforce the size of the hgcache that simply remove packfiles once the cache is over the configured size. One complication of this method is that several concurrent Mercurial processes could be running and accessing the packfiles being removed. In this case, we can split the packfiles in 2 categories: ones created a while back, and new ones. Removing packfiles from the first case, lookups will simply raise a KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is acceptable. The second category belongs to ones that were just created by downloading them from Memcache/Mononoke, and the code strongly assume that they will stick around. A failure at this point will not be recovered. One way of fixing this would be to handle these failures properly and simply retry, the other is to not remove new packfiles. A time of 10 minutes was chosen to categorize the packfiles. Reviewed By: quark-zju Differential Revision: D15014076 fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd
2019-04-23 22:55:24 +03:00
$ cd shallow
$ cd ..
$ cd shallow
remotefilelog: adapt tests for remotefilelog to support LFS Summary: The hgcache will soon contain an LFS subdirectory, this is uninteresting for most of the tests, let's change them a bit so they only look at the packs subdirectory. With this, enabling remotefilelog.useruststore only has 2 failures. One due a difference in handling corrupt data, with the ruststore being more explicit that some data is missing instead of a generic "stream ended unexpectedly". And the other due to some ordering difference when dealing with LFS data. The latter will go away with the new LFS implementation. Reviewed By: DurhamG Differential Revision: D20543146 fbshipit-source-id: 09c76cacfb4687fd699b82cdf5057665ac6bd521
2020-03-27 04:59:47 +03:00
$ find $CACHEDIR/master/packs | sort
remotefilelog: enforce a limit on the packfiles size Summary: Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to limit the size of the hgcache. This will be particularly challenging for Sandcastle/Quicksand as they already see hgcache over 100GB. The long-term plan is switching to IndexedLog based stores with a log rotate functionality to control the cache size. In the meantime, we can implement a basic logic to enforce the size of the hgcache that simply remove packfiles once the cache is over the configured size. One complication of this method is that several concurrent Mercurial processes could be running and accessing the packfiles being removed. In this case, we can split the packfiles in 2 categories: ones created a while back, and new ones. Removing packfiles from the first case, lookups will simply raise a KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is acceptable. The second category belongs to ones that were just created by downloading them from Memcache/Mononoke, and the code strongly assume that they will stick around. A failure at this point will not be recovered. One way of fixing this would be to handle these failures properly and simply retry, the other is to not remove new packfiles. A time of 10 minutes was chosen to categorize the packfiles. Reviewed By: quark-zju Differential Revision: D15014076 fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd
2019-04-23 22:55:24 +03:00
$TESTTMP/hgcache/master/packs
$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histidx
$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histpack
$TESTTMP/hgcache/master/packs/887690f1138ae5b99c50d754ed02262874bf8ecb.dataidx
$TESTTMP/hgcache/master/packs/887690f1138ae5b99c50d754ed02262874bf8ecb.datapack
$ touch -m -t 200001010000 $TESTTMP/hgcache/master/packs/887690f1138ae5b99c50d754ed02262874bf8ecb.datapack
# Cleanup the old over the limit packfiles
$ hg repack --config remotefilelog.cleanoldpacks=True --config remotefilelog.cachelimit="10B"
remotefilelog: adapt tests for remotefilelog to support LFS Summary: The hgcache will soon contain an LFS subdirectory, this is uninteresting for most of the tests, let's change them a bit so they only look at the packs subdirectory. With this, enabling remotefilelog.useruststore only has 2 failures. One due a difference in handling corrupt data, with the ruststore being more explicit that some data is missing instead of a generic "stream ended unexpectedly". And the other due to some ordering difference when dealing with LFS data. The latter will go away with the new LFS implementation. Reviewed By: DurhamG Differential Revision: D20543146 fbshipit-source-id: 09c76cacfb4687fd699b82cdf5057665ac6bd521
2020-03-27 04:59:47 +03:00
$ find $CACHEDIR/master/packs | sort
remotefilelog: enforce a limit on the packfiles size Summary: Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to limit the size of the hgcache. This will be particularly challenging for Sandcastle/Quicksand as they already see hgcache over 100GB. The long-term plan is switching to IndexedLog based stores with a log rotate functionality to control the cache size. In the meantime, we can implement a basic logic to enforce the size of the hgcache that simply remove packfiles once the cache is over the configured size. One complication of this method is that several concurrent Mercurial processes could be running and accessing the packfiles being removed. In this case, we can split the packfiles in 2 categories: ones created a while back, and new ones. Removing packfiles from the first case, lookups will simply raise a KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is acceptable. The second category belongs to ones that were just created by downloading them from Memcache/Mononoke, and the code strongly assume that they will stick around. A failure at this point will not be recovered. One way of fixing this would be to handle these failures properly and simply retry, the other is to not remove new packfiles. A time of 10 minutes was chosen to categorize the packfiles. Reviewed By: quark-zju Differential Revision: D15014076 fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd
2019-04-23 22:55:24 +03:00
$TESTTMP/hgcache/master/packs
$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histidx
$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histpack
$TESTTMP/hgcache/master/packs/repacklock
Reference in New Issue Copy Permalink