facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-09 16:31:02 +03:00
Code Issues Projects Releases Wiki Activity
c93ed71dcd
sapling/tests/test-fb-hgext-remotefilelog-corrupt-cache.t

72 lines
2.8 KiB
Perl
Raw Normal View History

remotefilelog: adjust PYTHONPATH during tests When remotefilelog moved from its own repo, the tests needed to be updated to adjust the PYTHONPATH to ensure the in-repo remotefilelog was loaded instead of the system one. This meant any local runs of remotefilelog tests would've been using the system remotefilelog unless the user had manually set the PYTHONPATH themselves.
2017-02-10 05:02:52 +03:00
Improve error message for corrupt cache files Summary: We've gotten reports of corrupt cache files, and the error message is pretty obtuse (ValueError for converting a string to an int). This refactors the size check into a function and provides a better error message. Test Plan: Added a test Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy Reviewed By: rmcelroy Differential Revision: https://phabricator.fb.com/D1774721 Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
2015-01-09 20:11:06 +03:00
$ . "$TESTDIR/library.sh"
$ hginit master
$ cd master
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> server=True
> EOF
$ echo x > x
$ echo y > y
$ echo z > z
$ hg commit -qAm xy
$ cd ..
$ hgcloneshallow ssh://user@dummy/master shallow -q
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
$ cd shallow
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
Verify corrupt cache handling repairs by default
$ hg up -q null
ioutil: move helper functions to ioutil Future patches will refactor the storage into more abstract APIs. Let's move these utility functions out to be on their own.
2016-04-05 02:26:12 +03:00
$ chmod u+w $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
$ echo x > $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
$ hg up tip
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
Improve error message for corrupt cache files Summary: We've gotten reports of corrupt cache files, and the error message is pretty obtuse (ValueError for converting a string to an int). This refactors the size check into a function and provides a better error message. Test Plan: Added a test Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy Reviewed By: rmcelroy Differential Revision: https://phabricator.fb.com/D1774721 Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
2015-01-09 20:11:06 +03:00
Verify corrupt cache error message
$ hg up -q null
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> validatecache=off
> EOF
ioutil: move helper functions to ioutil Future patches will refactor the storage into more abstract APIs. Let's move these utility functions out to be on their own.
2016-04-05 02:26:12 +03:00
$ chmod u+w $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
Improve error message for corrupt cache files Summary: We've gotten reports of corrupt cache files, and the error message is pretty obtuse (ValueError for converting a string to an int). This refactors the size check into a function and provides a better error message. Test Plan: Added a test Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy Reviewed By: rmcelroy Differential Revision: https://phabricator.fb.com/D1774721 Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
2015-01-09 20:11:06 +03:00
$ echo x > $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
remotefilelog: add revlog flags information to the protocol Summary: Make the unpacked file format to include the revlog flag information, and make the getfile(s) protocol support it. Note: The `getpackv1` protocol and packfile format is not changed yet. Test Plan: Run existing tests. Stronger tests and lfs integration test will be added when this area is more complete. Reviewers: #mercurial, rmcelroy, durham Reviewed By: durham Subscribers: rmcelroy, mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4903772 Signature: t1:4903772:1493152451:ab393b0208f0eee199ffc4c8fcfdfd5dd6d0f3ac
2017-04-26 23:08:13 +03:00
$ hg up tip 2>&1 | egrep "^RuntimeError"
RuntimeError: unexpected remotefilelog header: illegal format
Add optional cache validation Summary: There are reports of the local cache becoming invalid when stored on disk. This adds an option that will do some basic validation and remediation for those entries, and log some data to disk. This is optional, since it incurs some performance overhead. We just want to use it long enough to track down the issue. Test Plan: Added a test Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir Reviewed By: mitrandir Differential Revision: https://phabricator.fb.com/D1774724 Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
2015-01-09 05:59:04 +03:00
Verify detection and remediation when remotefilelog.validatecachelog is set
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> validatecachelog=$PWD/.hg/remotefilelog_cache.log
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
> validatecache=strict
Add optional cache validation Summary: There are reports of the local cache becoming invalid when stored on disk. This adds an option that will do some basic validation and remediation for those entries, and log some data to disk. This is optional, since it incurs some performance overhead. We just want to use it long enough to track down the issue. Test Plan: Added a test Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir Reviewed By: mitrandir Differential Revision: https://phabricator.fb.com/D1774724 Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
2015-01-09 05:59:04 +03:00
> EOF
ioutil: move helper functions to ioutil Future patches will refactor the storage into more abstract APIs. Let's move these utility functions out to be on their own.
2016-04-05 02:26:12 +03:00
$ chmod u+w $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
remotefilelog: add separate option to validate localcache files Summary: We've recently had to dig into two different issues that resulted in broken files landing in the localcache; one was due to a problem with the data source for our cacheprocess becoming corrupt and the other was due to a failed write (ENOSPC) causing a truncated file to be left in the local cache. It is desirable to perform some lightweight consistency checks before we return data up to the caller of localcache, but prior to this diff the validation functionality was coupled to configuring a log file. Due to the shared nature of the localcache it's not always clear cut where we want to log localcache consistency issues, so it feels more flexible to decouple logging from enabling checks. This diff introduces `remotefilelog.validatecache` as a separate option that can have three values: * `off` - no checks are performed * `on` - checks are performed during read and write * `strict` - checks are performed during __contains__, read and write The default is now `on`. Test Plan: `./run-tests.py --with-hg=../../hg-crew/hg` Reviewers: #sourcecontrol, ttung Differential Revision: https://phabricator.fb.com/D2941067 Tasks: 10044183, 9987694
2016-02-18 19:34:33 +03:00
$ echo x > $CACHEDIR/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
Add optional cache validation Summary: There are reports of the local cache becoming invalid when stored on disk. This adds an option that will do some basic validation and remediation for those entries, and log some data to disk. This is optional, since it incurs some performance overhead. We just want to use it long enough to track down the issue. Test Plan: Added a test Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir Reviewed By: mitrandir Differential Revision: https://phabricator.fb.com/D1774724 Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
2015-01-09 05:59:04 +03:00
$ hg up tip
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ cat .hg/remotefilelog_cache.log
corrupt $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0 during contains
remotefilelog: handle truncated server cache files Summary: If remotefilelog server cache files are truncated, when the server returns these files to the client, the client will crash trying to decompress them. Detect truncated files and treat them as cache misses. Test Plan: Add unit test for truncated server cache files. Reviewers: #fbhgext, simonfar Reviewed By: #fbhgext, simonfar Differential Revision: https://phab.mercurial-scm.org/D170
2017-07-21 18:07:09 +03:00
Verify handling of corrupt server cache
$ rm -f ../master/.hg/remotefilelogcache/y/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
$ touch ../master/.hg/remotefilelogcache/y/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
$ clearcache
$ hg prefetch -r .
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
$ test -s ../master/.hg/remotefilelogcache/y/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
$ hg debugremotefilelog $CACHEDIR/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
size: 2 bytes
path: $TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
key: 076f5e2225b3
node => p1 p2 linknode copyfrom
076f5e2225b3 => 000000000000 000000000000 f3d0bb0d1e48
Reference in New Issue Copy Permalink