mirror of
https://github.com/facebook/sapling.git
synced 2024-10-10 16:57:49 +03:00
3a97764d70
Summary: When using LFS, it's possible that a pointer may be present in the local LfsStore, but the blob would only be in the shared one. Such scenario can happen after an upload, when the blob is moved to the shared store for instance. In this case, during a `get` call, the local LFS store won't be able to find the blob and thus would return Ok(None), the shared LFS store woud not be able to find the pointer itself and would thus return Ok(None) too. If the server is not aware of the file node itself, the `ContentStore::get` would also return Ok(None), even though all the information is present locally. The main reason why this is happening is due to the `get` call operating primarily on file node based keys, and for content-based stores (like LFS), this means that the translation layer needs to be present in the same store, which in some case may not be the case. By allowing stores to return a `StoreKey` when progress was made in finding the key we can effectively solve the problem described above, the local store would translate the file node key onto a content key, and the shared store would read the blob properly. Reviewed By: DurhamG Differential Revision: D22565607 fbshipit-source-id: 94dd74a462526778f7a7e232a97b21211f95239f
152 lines
5.7 KiB
Perl
152 lines
5.7 KiB
Perl
#chg-compatible
|
|
|
|
$ disable treemanifest
|
|
|
|
$ . "$TESTDIR/library.sh"
|
|
|
|
$ hginit master
|
|
$ cd master
|
|
$ cat >> .hg/hgrc <<EOF
|
|
> [remotefilelog]
|
|
> server=True
|
|
> serverexpiration=-1
|
|
> EOF
|
|
$ echo x > x
|
|
$ echo y > y
|
|
$ hg commit -qAm xy
|
|
$ echo x >> x
|
|
$ echo y >> y
|
|
$ hg commit -qAm xy2
|
|
$ cd ..
|
|
|
|
$ hgcloneshallow ssh://user@dummy/master shallow -q
|
|
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
|
|
|
|
Set the prefetchdays config to zero so that all commits are prefetched
|
|
no matter what their creation date is.
|
|
$ cd shallow
|
|
$ cat >> .hg/hgrc <<EOF
|
|
> [remotefilelog]
|
|
> localdatarepack=True
|
|
> prefetchdays=0
|
|
> EOF
|
|
$ cd ..
|
|
|
|
$ cd shallow
|
|
$ findfilessorted $CACHEDIR/master/packs
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histidx
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histpack
|
|
$TESTTMP/hgcache/master/packs/9ae82bbfd147f28ff04367cf066c5ed3ef429be4.dataidx
|
|
$TESTTMP/hgcache/master/packs/9ae82bbfd147f28ff04367cf066c5ed3ef429be4.datapack
|
|
|
|
$ hg repack
|
|
|
|
$ findfilessorted $CACHEDIR/master/packs
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histidx
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histpack
|
|
$TESTTMP/hgcache/master/packs/9ae82bbfd147f28ff04367cf066c5ed3ef429be4.dataidx
|
|
$TESTTMP/hgcache/master/packs/9ae82bbfd147f28ff04367cf066c5ed3ef429be4.datapack
|
|
$TESTTMP/hgcache/master/packs/repacklock
|
|
|
|
Create some new data to pack into it
|
|
|
|
$ cd ../master
|
|
$ echo a > a
|
|
$ echo b > b
|
|
$ hg commit -qAm ab
|
|
$ echo a >> a
|
|
$ echo b >> b
|
|
$ hg commit -qAm ab2
|
|
$ cd ../shallow
|
|
$ hg pull -q
|
|
$ hg up -q tip
|
|
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob) (?)
|
|
$ findfilessorted $CACHEDIR/master/packs
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histidx
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histpack
|
|
$TESTTMP/hgcache/master/packs/767371d087d35f549346611a68c10805fa2e5083.dataidx
|
|
$TESTTMP/hgcache/master/packs/767371d087d35f549346611a68c10805fa2e5083.datapack
|
|
$TESTTMP/hgcache/master/packs/8054aa368dd33acb92013013096b517dfdfaf184.histidx
|
|
$TESTTMP/hgcache/master/packs/8054aa368dd33acb92013013096b517dfdfaf184.histpack
|
|
$TESTTMP/hgcache/master/packs/9ae82bbfd147f28ff04367cf066c5ed3ef429be4.dataidx
|
|
$TESTTMP/hgcache/master/packs/9ae82bbfd147f28ff04367cf066c5ed3ef429be4.datapack
|
|
$TESTTMP/hgcache/master/packs/repacklock
|
|
|
|
Truncate the historypack file in the middle of the filename length for "y"
|
|
$ chmod +w $TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histpack
|
|
$ python $TESTDIR/truncate.py --size 173 $TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histpack
|
|
|
|
Repack
|
|
$ hg repack
|
|
$ findfilessorted $CACHEDIR/master/packs
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histidx
|
|
$TESTTMP/hgcache/master/packs/37db2caec222ca26824a52d6bdc778344e0d1440.histpack
|
|
$TESTTMP/hgcache/master/packs/60374dd52300114836388912c35fe94f80e74889.dataidx
|
|
$TESTTMP/hgcache/master/packs/60374dd52300114836388912c35fe94f80e74889.datapack
|
|
$TESTTMP/hgcache/master/packs/8335018efaefd765d52fad1c07f44addc0371202.histidx
|
|
$TESTTMP/hgcache/master/packs/8335018efaefd765d52fad1c07f44addc0371202.histpack
|
|
$TESTTMP/hgcache/master/packs/repacklock
|
|
|
|
The history for y has to be refetched from the server.
|
|
$ hg log -f y -T '{desc}\n'
|
|
xy2
|
|
xy
|
|
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over 0.00s (?)
|
|
|
|
Next, do the same for local data. This time there is data loss, as there are no more copies
|
|
of the data available.
|
|
|
|
Create some local commits and pack them into a pack file
|
|
|
|
$ echo m > m
|
|
$ echo n > n
|
|
$ echo o > o
|
|
$ hg commit -qAm mno
|
|
$ echo m >> m
|
|
$ echo n >> n
|
|
$ echo o >> o
|
|
$ hg commit -qAm mno2
|
|
$ hg repack
|
|
|
|
Truncate the history in the middle of the filename length for "n"
|
|
$ chmod +w .hg/store/packs/822f755410ca9f664d7c706957b8391248327318.histpack
|
|
$ python $TESTDIR/truncate.py --size 173 .hg/store/packs/822f755410ca9f664d7c706957b8391248327318.histpack
|
|
|
|
Truncate the data in the middle of the filename length for "o"
|
|
$ chmod +w .hg/store/packs/f0a7036b83e36fd41dc1ea89cc67e6a01487f2cf.datapack
|
|
$ python $TESTDIR/truncate.py --size 130 .hg/store/packs/f0a7036b83e36fd41dc1ea89cc67e6a01487f2cf.datapack
|
|
|
|
Repack
|
|
$ hg repack 2>&1 | grep 'Repack failure'
|
|
*RustError: Repack failure: [("$TESTTMP/shallow/.hg/store/packs/f0a7036b83e36fd41dc1ea89cc67e6a01487f2cf", failed to fill whole buffer)] (glob)
|
|
$ findfilessorted .hg/store/packs
|
|
.hg/store/packs/074daadf58a47b23b71741ac7c922b7c02343598.dataidx
|
|
.hg/store/packs/074daadf58a47b23b71741ac7c922b7c02343598.datapack
|
|
.hg/store/packs/822f755410ca9f664d7c706957b8391248327318.histidx
|
|
.hg/store/packs/822f755410ca9f664d7c706957b8391248327318.histpack
|
|
.hg/store/packs/f0a7036b83e36fd41dc1ea89cc67e6a01487f2cf.dataidx
|
|
.hg/store/packs/f0a7036b83e36fd41dc1ea89cc67e6a01487f2cf.datapack
|
|
|
|
The local data and history for m is still available
|
|
$ hg cat m
|
|
m
|
|
m
|
|
$ hg log -f m -T '{desc}\n'
|
|
mno2
|
|
mno
|
|
|
|
The local data for n is still available
|
|
$ hg cat n
|
|
n
|
|
n
|
|
|
|
The history for n is lost
|
|
$ hg log -qf n 2>&1 | grep 'KeyError'
|
|
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("n"), hgid: HgId("c972a0820002b32c6fec4b7ca47d3aecdad8e1c5") })'
|
|
|
|
The local data and history for o is lost
|
|
$ hg cat -q o 2>&1 | grep 'KeyError'
|
|
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("o"), hgid: HgId("fd94f81d01bf8c9d960bb57abdd4e8375309ae43") })'
|
|
$ hg log -qf o 2>&1 | grep 'KeyError'
|
|
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("o"), hgid: HgId("fd94f81d01bf8c9d960bb57abdd4e8375309ae43") })'
|