sapling/eden/scm/edenscmnative
Xavier Deguillard ed4021b8e3 revisionstore: disallow reading LFS pointers from packfiles
Summary:
For repositories that have the old-style LFS extension enabled, the pointers
are stored in packfiles/indexedlog alongside with a flag that signify to the
upper layers that the blob is externally stored. With the new way of doing LFS,
pointers are stored separately.

When both are enabled, we are observing some interesting behavior where
different get and get_meta calls may return different blobs/metadata for the
same filenode. This may happen if a filenode is stored in both a packfile as an
LFS pointers, and in the LFS store. Guaranteeing that the revisionstore code is
deterministic in this situation is unfortunately way too costly (a get_meta
call would for instance have to fully validate the sha256 of the blob, and this
wouldn't guarantee that it wouldn't become corrupted on disk before calling
get).

The solution take here is to simply ignore all the lfs pointers from
packfiles/indexedlog when remotefilelog.lfs is enabled. This way, there is no
risk of reading the metadata from the packfiles, and the blob from the
LFSStore. This brings however another complication for the user created blobs:
these are stored in packfiles and would thus become unreadable, the solution is
to simply perform a one-time full repack of the local store to make sure that
all the pointers are moved from the packfiles to to LFSStore.

In the code, the Python bindings are using ExtStoredPolicy::Ignore directly as
these are only used in the treemanifest code where no LFS pointers should be
present, the repack code uses ExtStoredPolicy::Use to be able to read the
pointers, it wouldn't be able to otherwise.

Reviewed By: DurhamG

Differential Revision: D22951598

fbshipit-source-id: 0e929708ba5a3bb2a02c0891fd62dae1ccf18204
2020-09-09 18:27:42 -07:00
..
bindings revisionstore: disallow reading LFS pointers from packfiles 2020-09-09 18:27:42 -07:00
__init__.py remediation of S205607 2020-07-17 17:16:13 -07:00
clindex.pyi add type annotations for a few more edenscmnative modules 2020-03-20 10:41:32 -07:00
clindex.pyx repoview: further remove repoview references 2020-07-06 14:04:27 -07:00
linelog.pyx py3/cython: set the language_level to 3str 2020-01-30 12:52:06 -08:00
patchrmdir.pyi add type annotations for a few more edenscmnative modules 2020-03-20 10:41:32 -07:00
patchrmdir.pyx py3: fix extdiff 2020-08-27 19:15:22 -07:00
traceprof.pyx py3/cython: set the language_level to 3str 2020-01-30 12:52:06 -08:00