facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-07 07:17:55 +03:00
Code Issues Projects Releases Wiki Activity
275ab15494
sapling/tests/test-fb-hgext-remotefilelog-corrupt-cache.t

111 lines
4.6 KiB
Perl
Raw Normal View History

remotefilelog: adjust PYTHONPATH during tests When remotefilelog moved from its own repo, the tests needed to be updated to adjust the PYTHONPATH to ensure the in-repo remotefilelog was loaded instead of the system one. This meant any local runs of remotefilelog tests would've been using the system remotefilelog unless the user had manually set the PYTHONPATH themselves.
2017-02-10 05:02:52 +03:00
Improve error message for corrupt cache files Summary: We've gotten reports of corrupt cache files, and the error message is pretty obtuse (ValueError for converting a string to an int). This refactors the size check into a function and provides a better error message. Test Plan: Added a test Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy Reviewed By: rmcelroy Differential Revision: https://phabricator.fb.com/D1774721 Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
2015-01-09 20:11:06 +03:00
$ . "$TESTDIR/library.sh"
$ hginit master
$ cd master
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> server=True
> EOF
$ echo x > x
$ echo y > y
$ echo z > z
$ hg commit -qAm xy
$ cd ..
$ hgcloneshallow ssh://user@dummy/master shallow -q
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
$ cd shallow
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
Verify corrupt cache handling repairs by default
$ hg up -q null
ioutil: move helper functions to ioutil Future patches will refactor the storage into more abstract APIs. Let's move these utility functions out to be on their own.
2016-04-05 02:26:12 +03:00
$ chmod u+w $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
$ echo x > $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
$ hg up tip
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
Improve error message for corrupt cache files Summary: We've gotten reports of corrupt cache files, and the error message is pretty obtuse (ValueError for converting a string to an int). This refactors the size check into a function and provides a better error message. Test Plan: Added a test Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy Reviewed By: rmcelroy Differential Revision: https://phabricator.fb.com/D1774721 Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
2015-01-09 20:11:06 +03:00
Verify corrupt cache error message
$ hg up -q null
tests: make test-fb-hgext-remotefilelog-corrupt-cache.t reliable Summary: This test has been extremely flaky in the last few weeks, causing builds to not being released, or just sandcastle jobs failing randomly. The reason for the flakyness was due to the mtime stored in the dirstate being -1 instead of the real mtime of the file. The reason for the mtime to be -1 is to fix a race where a file is modified in the same second the dirstate is updated. This can happen when a transaction spans less than a second. By forcing a transaction to always last at least a second, we can avoid this issue and make the test stable. Reviewed By: quark-zju Differential Revision: D14141924 fbshipit-source-id: ce151cd65d930f8c745633f7575ea006748af9b9
2019-02-20 03:19:56 +03:00
Enable delaywrite to avoid races when checking for corruption.
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> validatecache=off
tests: make test-fb-hgext-remotefilelog-corrupt-cache.t reliable Summary: This test has been extremely flaky in the last few weeks, causing builds to not being released, or just sandcastle jobs failing randomly. The reason for the flakyness was due to the mtime stored in the dirstate being -1 instead of the real mtime of the file. The reason for the mtime to be -1 is to fix a race where a file is modified in the same second the dirstate is updated. This can happen when a transaction spans less than a second. By forcing a transaction to always last at least a second, we can avoid this issue and make the test stable. Reviewed By: quark-zju Differential Revision: D14141924 fbshipit-source-id: ce151cd65d930f8c745633f7575ea006748af9b9
2019-02-20 03:19:56 +03:00
> [debug]
> dirstate.delaywrite=1
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
> EOF
ioutil: move helper functions to ioutil Future patches will refactor the storage into more abstract APIs. Let's move these utility functions out to be on their own.
2016-04-05 02:26:12 +03:00
$ chmod u+w $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
Improve error message for corrupt cache files Summary: We've gotten reports of corrupt cache files, and the error message is pretty obtuse (ValueError for converting a string to an int). This refactors the size check into a function and provides a better error message. Test Plan: Added a test Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy Reviewed By: rmcelroy Differential Revision: https://phabricator.fb.com/D1774721 Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
2015-01-09 20:11:06 +03:00
$ echo x > $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
remotefilelog: add revlog flags information to the protocol Summary: Make the unpacked file format to include the revlog flag information, and make the getfile(s) protocol support it. Note: The `getpackv1` protocol and packfile format is not changed yet. Test Plan: Run existing tests. Stronger tests and lfs integration test will be added when this area is more complete. Reviewers: #mercurial, rmcelroy, durham Reviewed By: durham Subscribers: rmcelroy, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4903772 Signature: t1:4903772:1493152451:ab393b0208f0eee199ffc4c8fcfdfd5dd6d0f3ac
2017-04-26 23:08:13 +03:00
$ hg up tip 2>&1 | egrep "^RuntimeError"
RuntimeError: unexpected remotefilelog header: illegal format
Add optional cache validation Summary: There are reports of the local cache becoming invalid when stored on disk. This adds an option that will do some basic validation and remediation for those entries, and log some data to disk. This is optional, since it incurs some performance overhead. We just want to use it long enough to track down the issue. Test Plan: Added a test Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir Reviewed By: mitrandir Differential Revision: https://phabricator.fb.com/D1774724 Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
2015-01-09 05:59:04 +03:00
Verify detection and remediation when remotefilelog.validatecachelog is set
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> validatecachelog=$PWD/.hg/remotefilelog_cache.log
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
> validatecache=strict
Add optional cache validation Summary: There are reports of the local cache becoming invalid when stored on disk. This adds an option that will do some basic validation and remediation for those entries, and log some data to disk. This is optional, since it incurs some performance overhead. We just want to use it long enough to track down the issue. Test Plan: Added a test Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir Reviewed By: mitrandir Differential Revision: https://phabricator.fb.com/D1774724 Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
2015-01-09 05:59:04 +03:00
> EOF
ioutil: move helper functions to ioutil Future patches will refactor the storage into more abstract APIs. Let's move these utility functions out to be on their own.
2016-04-05 02:26:12 +03:00
$ chmod u+w $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
$ echo x > $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
Add optional cache validation Summary: There are reports of the local cache becoming invalid when stored on disk. This adds an option that will do some basic validation and remediation for those entries, and log some data to disk. This is optional, since it incurs some performance overhead. We just want to use it long enough to track down the issue. Test Plan: Added a test Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir Reviewed By: mitrandir Differential Revision: https://phabricator.fb.com/D1774724 Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
2015-01-09 05:59:04 +03:00
$ hg up tip
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ cat .hg/remotefilelog_cache.log
corrupt $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0 during contains
remotefilelog: handle truncated server cache files Summary: If remotefilelog server cache files are truncated, when the server returns these files to the client, the client will crash trying to decompress them. Detect truncated files and treat them as cache misses. Test Plan: Add unit test for truncated server cache files. Reviewers: #fbhgext, simonfar Reviewed By: #fbhgext, simonfar Differential Revision: https://phab.mercurial-scm.org/D170
2017-07-21 18:07:09 +03:00
remotefilelog: add an option to verify all the hashes Reviewed By: DurhamG Differential Revision: D13838772 fbshipit-source-id: e7220ba8a0e07f3003693199086e3a3512023857
2019-02-06 01:50:37 +03:00
Verify that hashes are checked
$ rm .hg/remotefilelog_cache.log
$ hg up -q null
$ chmod u+w $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
remotefilelog: default file blobs to version 1 Summary: We're seeing cache corruption issues due to mixing and matching loose file versions. Let's just default to 1. Reviewed By: quark-zju Differential Revision: D13980892 fbshipit-source-id: 1e8cc9a53b8b92e05dc08b202505bb8f453a3074
2019-02-07 21:40:44 +03:00
$ printf 'z' | dd of=$CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0 bs=1 seek=9 count=1 conv=notrunc 2> /dev/null
remotefilelog: add an option to verify all the hashes Reviewed By: DurhamG Differential Revision: D13838772 fbshipit-source-id: e7220ba8a0e07f3003693199086e3a3512023857
2019-02-06 01:50:37 +03:00
$ hg up tip --config remotefilelog.validatecachehashes=False
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat x
z
$ hg st --config remotefilelog.validatecachehashes=False
Verify that hashes are checked
$ hg up -C -q null
$ hg up tip
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ cat .hg/remotefilelog_cache.log
corrupt $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0 during contains
$ cat x
x
remotefilelog: handle truncated server cache files Summary: If remotefilelog server cache files are truncated, when the server returns these files to the client, the client will crash trying to decompress them. Detect truncated files and treat them as cache misses. Test Plan: Add unit test for truncated server cache files. Reviewers: #fbhgext, simonfar Reviewed By: #fbhgext, simonfar Differential Revision: https://phab.mercurial-scm.org/D170
2017-07-21 18:07:09 +03:00
Verify handling of corrupt server cache
$ rm -f ../master/.hg/remotefilelogcache/y/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
$ touch ../master/.hg/remotefilelogcache/y/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
$ clearcache
$ hg prefetch -r .
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
$ test -s ../master/.hg/remotefilelogcache/y/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
$ hg debugremotefilelog $CACHEDIR/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
size: 2 bytes
path: $TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
key: 076f5e2225b3
remotefilelog: store loose file filenames alongside the data Summary: Loose files don't contain the original filename inside them, but rather the directory name is based on the sha1 hash of the filename. This means that given a remotefilelog data file, we can't find out the name of the file that it contains. Repack needs to know this (plus it's also useful for debug purposes), so add an additional file in the directory, `filename`, which contains the original filename of all the data files in the directory. It follows that `sha1(read(filename)) == basename(dirname(filename))` Reviewed By: DurhamG Differential Revision: D10516851 fbshipit-source-id: aeeb1a116e0d50a134c274c3bb31c7c26aa8015c
2018-10-26 11:13:41 +03:00
filename: y
remotefilelog: handle truncated server cache files Summary: If remotefilelog server cache files are truncated, when the server returns these files to the client, the client will crash trying to decompress them. Detect truncated files and treat them as cache misses. Test Plan: Add unit test for truncated server cache files. Reviewers: #fbhgext, simonfar Reviewed By: #fbhgext, simonfar Differential Revision: https://phab.mercurial-scm.org/D170
2017-07-21 18:07:09 +03:00
node => p1 p2 linknode copyfrom
076f5e2225b3 => 000000000000 000000000000 f3d0bb0d1e48
remotefilelog: handle corrupt loose files more gracefully Summary: If the history portion of a loose file was corrupt, users would often get errors like "ValueError: substring not found" when trying to split on the null separators. Let's catch that and handle it where appropriate. For cache stores, we move it out of the way and return a KeyError, so the union store will move on to try to fetch it from the server again. For local stores, we return a ValueError and surface it to the user since their local data may be corrupt. Reviewed By: quark-zju Differential Revision: D13976499 fbshipit-source-id: 39e63f19fc752ca0179fbc5b0908a28d46de3a3a
2019-02-08 00:31:07 +03:00
Verify some bad history data is caught and remediated even when validation is off
$ chmod u+w $CACHEDIR/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
test-fb-hgext-remotefilelog-corrupt-cache: do not use truncate Summary: It is unavailable on OSX, but we have a custom implementation. Reviewed By: DurhamG Differential Revision: D14028103 fbshipit-source-id: 4b6916792fc9da43795b0f9db57ec2a057c92283
2019-02-11 23:47:02 +03:00
$ $PYTHON $TESTDIR/truncate.py --size 20 $CACHEDIR/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
remotefilelog: handle corrupt loose files more gracefully Summary: If the history portion of a loose file was corrupt, users would often get errors like "ValueError: substring not found" when trying to split on the null separators. Let's catch that and handle it where appropriate. For cache stores, we move it out of the way and return a KeyError, so the union store will move on to try to fetch it from the server again. For local stores, we return a ValueError and surface it to the user since their local data may be corrupt. Reviewed By: quark-zju Differential Revision: D13976499 fbshipit-source-id: 39e63f19fc752ca0179fbc5b0908a28d46de3a3a
2019-02-08 00:31:07 +03:00
$ hg log -f y --config remotefilelog.validatecache=off
detected corruption in '$TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca', moving it aside
changeset: 0:f3d0bb0d1e48
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: xy
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over 0.00s
Reference in New Issue Copy Permalink