lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-29 01:41:59 +03:00
|
|
|
# Initial setup
|
2017-03-25 05:00:30 +03:00
|
|
|
|
lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-29 01:41:59 +03:00
|
|
|
$ cat >> $HGRCPATH << EOF
|
2017-03-25 05:00:30 +03:00
|
|
|
> [extensions]
|
2017-03-25 05:01:42 +03:00
|
|
|
> lfs=$TESTDIR/../hgext3rd/lfs/
|
2017-03-25 05:00:30 +03:00
|
|
|
> [lfs]
|
|
|
|
> threshold=1000B
|
|
|
|
> chunksize=1000B
|
|
|
|
> blobstore=cache/localblobstore
|
|
|
|
> remotestore=dummy
|
lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-29 01:41:59 +03:00
|
|
|
> remotepath=$TESTTMP/dummy-remote/
|
2017-03-25 05:00:30 +03:00
|
|
|
> EOF
|
|
|
|
|
lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-29 01:41:59 +03:00
|
|
|
$ LONG=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
|
|
|
|
|
|
|
|
# Prepare server and enable extension
|
|
|
|
$ hg init server
|
|
|
|
$ hg clone -q server client
|
|
|
|
$ cd client
|
|
|
|
|
2017-03-25 05:00:30 +03:00
|
|
|
# Commit small file
|
|
|
|
$ echo s > smallfile
|
|
|
|
$ hg commit -Aqm "add small file"
|
|
|
|
|
|
|
|
# Commit large file
|
lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-29 01:41:59 +03:00
|
|
|
$ echo $LONG > largefile
|
2017-03-25 05:00:30 +03:00
|
|
|
$ hg commit --traceback -Aqm "add large file"
|
|
|
|
|
|
|
|
# Ensure metadata is stored
|
|
|
|
$ hg debugdata largefile 0
|
|
|
|
version https://git-lfs.github.com/spec/chunking
|
|
|
|
chunks d7dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18:1000,ed0f071aa4ff28ab9863b6cfc5f407e915612d70502422e4ab9b09f3dfec4a74:501
|
|
|
|
hashalgo sha256
|
|
|
|
size 1501
|
|
|
|
|
|
|
|
# Check the blobstore is populated
|
|
|
|
$ find .hg/cache/localblobstore | sort
|
|
|
|
.hg/cache/localblobstore
|
|
|
|
.hg/cache/localblobstore/d7
|
|
|
|
.hg/cache/localblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
|
|
|
|
.hg/cache/localblobstore/ed
|
|
|
|
.hg/cache/localblobstore/ed/0f071aa4ff28ab9863b6cfc5f407e915612d70502422e4ab9b09f3dfec4a74
|
|
|
|
|
|
|
|
# Check the blob stored contains the actual contents of the file
|
|
|
|
$ cat .hg/cache/localblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
|
|
|
|
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB (no-eol)
|
|
|
|
$ cat .hg/cache/localblobstore/ed/0f071aa4ff28ab9863b6cfc5f407e915612d70502422e4ab9b09f3dfec4a74
|
|
|
|
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
|
|
|
|
|
|
|
|
# Push changes to the server
|
2017-04-06 01:58:56 +03:00
|
|
|
$ hg push -v | egrep -v '^(uncompressed| )'
|
2017-03-25 05:00:30 +03:00
|
|
|
pushing to $TESTTMP/server (glob)
|
|
|
|
searching for changes
|
|
|
|
lfs: computing set of blobs to upload
|
2017-04-06 01:58:56 +03:00
|
|
|
lfs: uploading blobs to the remote (2 chunk(s), 1.47 KB)
|
|
|
|
2 changesets found
|
2017-03-25 05:00:30 +03:00
|
|
|
adding changesets
|
|
|
|
adding manifests
|
|
|
|
adding file changes
|
|
|
|
added 2 changesets with 2 changes to 2 files
|
|
|
|
|
2017-04-06 01:48:10 +03:00
|
|
|
# Unknown remotestore
|
|
|
|
|
|
|
|
$ hg push --config lfs.remotestore=404
|
|
|
|
abort: lfs: unknown remotestore: 404
|
|
|
|
[255]
|
|
|
|
|
2017-03-25 05:00:30 +03:00
|
|
|
$ cd ../
|
lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-29 01:41:59 +03:00
|
|
|
|
|
|
|
# Initialize new client (not cloning) and setup extension
|
|
|
|
$ hg init client2
|
2017-03-25 05:00:30 +03:00
|
|
|
$ cd client2
|
|
|
|
$ cat >> .hg/hgrc <<EOF
|
|
|
|
> [paths]
|
|
|
|
> default = $TESTTMP/server
|
|
|
|
> EOF
|
|
|
|
|
|
|
|
# Pull from server
|
|
|
|
$ hg pull default
|
|
|
|
pulling from $TESTTMP/server (glob)
|
|
|
|
requesting all changes
|
|
|
|
adding changesets
|
|
|
|
adding manifests
|
|
|
|
adding file changes
|
|
|
|
added 2 changesets with 2 changes to 2 files
|
|
|
|
(run 'hg update' to get a working copy)
|
|
|
|
|
|
|
|
# Check the blobstore is not yet populated
|
2017-03-25 05:01:42 +03:00
|
|
|
$ [ -f .hg/cache/localblobstore ]
|
2017-03-25 05:00:30 +03:00
|
|
|
[1]
|
|
|
|
|
|
|
|
# Update to the last revision containing the large file
|
|
|
|
$ hg update
|
|
|
|
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
|
|
|
|
|
|
|
# Check the blobstore has been populated on update
|
|
|
|
$ find .hg/cache/localblobstore | sort
|
|
|
|
.hg/cache/localblobstore
|
|
|
|
.hg/cache/localblobstore/d7
|
|
|
|
.hg/cache/localblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
|
|
|
|
.hg/cache/localblobstore/ed
|
|
|
|
.hg/cache/localblobstore/ed/0f071aa4ff28ab9863b6cfc5f407e915612d70502422e4ab9b09f3dfec4a74
|
|
|
|
|
|
|
|
# Check the contents of the file are fetched from blobstore when requested
|
|
|
|
$ hg cat -r . largefile
|
|
|
|
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
|
|
|
|
|
|
|
|
# Check the file has been copied in the working copy
|
|
|
|
$ cat largefile
|
|
|
|
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
|
lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-29 01:41:59 +03:00
|
|
|
|
|
|
|
$ cd ..
|
|
|
|
|
|
|
|
# Check blobstore could be an absolute path outside repo vfs:
|
|
|
|
|
|
|
|
$ mkdir -p $TESTTMP/global-store
|
|
|
|
$ cat >> $HGRCPATH <<EOF
|
|
|
|
> [lfs]
|
|
|
|
> blobstore=$TESTTMP/global-store
|
|
|
|
> EOF
|
|
|
|
|
|
|
|
$ hg init repo2
|
|
|
|
$ cd repo2
|
|
|
|
|
|
|
|
$ echo $LONG > largefile
|
|
|
|
$ hg add largefile
|
|
|
|
$ hg commit -m initcommit
|
|
|
|
$ [ -f $TESTTMP/global-store/ed/0f071aa4ff28ab9863b6cfc5f407e915612d70502422e4ab9b09f3dfec4a74 ]
|
|
|
|
$ [ -f $TESTTMP/global-store/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18 ]
|
2017-03-31 02:41:46 +03:00
|
|
|
|
|
|
|
$ cd ..
|
|
|
|
|
|
|
|
# Check rename, and switch between large and small files
|
|
|
|
|
|
|
|
$ hg init repo3
|
|
|
|
$ cd repo3
|
|
|
|
$ cat >> .hg/hgrc << EOF
|
|
|
|
> [lfs]
|
|
|
|
> threshold=10B
|
|
|
|
> chunksize=10B
|
|
|
|
> EOF
|
|
|
|
|
|
|
|
$ echo LONGER-THAN-TEN-BYTES-WILL-TRIGGER-LFS > large
|
|
|
|
$ echo SHORTER > small
|
|
|
|
$ hg add . -q
|
|
|
|
$ hg commit -m 'commit with lfs content'
|
|
|
|
|
|
|
|
$ hg mv large l
|
|
|
|
$ hg mv small s
|
|
|
|
$ hg commit -m 'renames'
|
|
|
|
|
|
|
|
$ echo SHORT > l
|
|
|
|
$ echo BECOME-LARGER-FROM-SHORTER > s
|
|
|
|
$ hg commit -m 'large to small, small to large'
|
|
|
|
|
|
|
|
$ echo 1 >> l
|
|
|
|
$ echo 2 >> s
|
|
|
|
$ hg commit -m 'random modifications'
|
|
|
|
|
|
|
|
$ echo RESTORE-TO-BE-LARGE > l
|
|
|
|
$ echo SHORTER > s
|
|
|
|
$ hg commit -m 'switch large and small again'
|
|
|
|
|
|
|
|
# Push and pull the above repo
|
|
|
|
|
|
|
|
$ hg --cwd .. init repo4
|
|
|
|
$ hg push ../repo4
|
|
|
|
pushing to ../repo4
|
|
|
|
searching for changes
|
|
|
|
adding changesets
|
|
|
|
adding manifests
|
|
|
|
adding file changes
|
|
|
|
added 5 changesets with 10 changes to 4 files
|
|
|
|
|
|
|
|
$ hg --cwd .. init repo5
|
|
|
|
$ hg --cwd ../repo5 pull ../repo3
|
|
|
|
pulling from ../repo3
|
|
|
|
requesting all changes
|
|
|
|
adding changesets
|
|
|
|
adding manifests
|
|
|
|
adding file changes
|
|
|
|
added 5 changesets with 10 changes to 4 files
|
|
|
|
(run 'hg update' to get a working copy)
|
|
|
|
|
|
|
|
$ cd ..
|
|
|
|
|
2017-04-05 02:08:36 +03:00
|
|
|
# Test clone
|
|
|
|
|
|
|
|
$ hg init repo6
|
|
|
|
$ cd repo6
|
|
|
|
$ cat >> .hg/hgrc << EOF
|
|
|
|
> [lfs]
|
|
|
|
> threshold=30B
|
|
|
|
> chunksize=20B
|
|
|
|
> EOF
|
|
|
|
|
|
|
|
$ echo LARGE-BECAUSE-IT-IS-MORE-THAN-30-BYTES > large
|
|
|
|
$ echo SMALL > small
|
|
|
|
$ hg commit -Aqm 'create a lfs file' large small
|
|
|
|
|
|
|
|
$ cd ..
|
|
|
|
|
|
|
|
$ hg clone repo6 repo7
|
|
|
|
updating to branch default
|
|
|
|
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
|
|
|
$ cd repo7
|
|
|
|
$ cat large
|
|
|
|
LARGE-BECAUSE-IT-IS-MORE-THAN-30-BYTES
|
|
|
|
$ cat small
|
|
|
|
SMALL
|
|
|
|
|
|
|
|
# Test bypass
|
|
|
|
|
|
|
|
$ hg cat -r tip large --config lfs.remotepath=$TESTTMP/404 --config lfs.blobstore=cache/404
|
|
|
|
abort: No such file or directory: $TESTTMP/404/2e81056070ae365867a5e7f804abe39c12e07bdf11c098994d1bc0ab9981910a
|
|
|
|
[255]
|
|
|
|
|
|
|
|
$ hg cat -r tip large --config lfs.remotepath=$TESTTMP/404 --config lfs.blobstore=cache/404 --config lfs.bypass=1
|
|
|
|
version https://git-lfs.github.com/spec/chunking
|
|
|
|
chunks 2e81056070ae365867a5e7f804abe39c12e07bdf11c098994d1bc0ab9981910a:20,903f301311488de7befd35be124433f73d3b7b6bce1d75440474907ff2ae7591:19
|
|
|
|
hashalgo sha256
|
|
|
|
size 39
|
|
|
|
|
|
|
|
$ cd ..
|
|
|
|
|
2017-03-31 02:41:46 +03:00
|
|
|
# Verify the repos
|
|
|
|
|
|
|
|
$ cat > $TESTTMP/dumpflog.py << EOF
|
|
|
|
> # print raw revision sizes, flags, and hashes for certain files
|
|
|
|
> import hashlib
|
|
|
|
> from mercurial import revlog
|
|
|
|
> from mercurial.node import short
|
|
|
|
> def hash(rawtext):
|
|
|
|
> h = hashlib.sha512()
|
|
|
|
> h.update(rawtext)
|
|
|
|
> return h.hexdigest()[:4]
|
|
|
|
> def reposetup(ui, repo):
|
|
|
|
> # these 2 files are interesting
|
|
|
|
> for name in ['l', 's']:
|
|
|
|
> fl = repo.file(name)
|
|
|
|
> if len(fl) == 0:
|
|
|
|
> continue
|
|
|
|
> sizes = [revlog.revlog.rawsize(fl, i) for i in fl]
|
|
|
|
> texts = [fl.revision(i, raw=True) for i in fl]
|
|
|
|
> flags = [fl.flags(i) for i in fl]
|
|
|
|
> hashes = [hash(t) for t in texts]
|
|
|
|
> print(' %s: rawsizes=%r flags=%r hashes=%r'
|
|
|
|
> % (name, sizes, flags, hashes))
|
|
|
|
> EOF
|
|
|
|
|
2017-04-05 02:08:36 +03:00
|
|
|
$ for i in client client2 server repo3 repo4 repo5 repo6 repo7; do
|
2017-03-31 02:41:46 +03:00
|
|
|
> echo 'repo:' $i
|
|
|
|
> hg --cwd $i verify --config extensions.dumpflog=$TESTTMP/dumpflog.py -q
|
|
|
|
> done
|
|
|
|
repo: client
|
|
|
|
repo: client2
|
|
|
|
repo: server
|
|
|
|
repo: repo3
|
|
|
|
l: rawsizes=[828, 6, 8, 216] flags=[8192, 0, 0, 8192] hashes=['2791', '948c', 'cc88', '5f1a']
|
|
|
|
s: rawsizes=[623, 283, 283, 8] flags=[8192, 8192, 8192, 0] hashes=['db08', 'ff24', '2217', '826b']
|
|
|
|
repo: repo4
|
|
|
|
l: rawsizes=[828, 6, 8, 216] flags=[8192, 0, 0, 8192] hashes=['2791', '948c', 'cc88', '5f1a']
|
|
|
|
s: rawsizes=[623, 283, 283, 8] flags=[8192, 8192, 8192, 0] hashes=['db08', 'ff24', '2217', '826b']
|
|
|
|
repo: repo5
|
|
|
|
l: rawsizes=[828, 6, 8, 216] flags=[8192, 0, 0, 8192] hashes=['2791', '948c', 'cc88', '5f1a']
|
|
|
|
s: rawsizes=[623, 283, 283, 8] flags=[8192, 8192, 8192, 0] hashes=['db08', 'ff24', '2217', '826b']
|
2017-04-05 02:08:36 +03:00
|
|
|
repo: repo6
|
|
|
|
repo: repo7
|