Summary:
Setting `lfs.usercache` means every LFS object is available globally
for the test. Some tests would like to test pushing LFS objects from
one repo to another. This config hides real issues.
Reviewed By: DurhamG, zhh95
Differential Revision: D8986784
fbshipit-source-id: 3f8d925b28acfe1c6b8ebcb02bd3815642747629
Summary:
This is similar to what we have done for changegroups, but for non-changegroup
(addrawrevision) case. This is needed to make sure the delta application code
path can assume deltas are always against vanilla (non-LFS) rawtext so the next
fix becomes possible.
Reviewed By: DurhamG
Differential Revision: D6906202
fbshipit-source-id: a7d62dfed4206d45b42299f1dabf013620ae52b3
Summary:
There is no way to distinguish whether a delta base is LFS or non-LFS.
If the delta is against LFS rawtext, and the client trying to apply it has the
base revision stored as fulltext, the delta (aka. bundle) will fail to apply.
This patch forbids using delta on LFS revisions.
Note: this does not solve the problem entirely. Since the problem could also be
a client with base file revision being LFS tries to apply a non-LFS delta
(bundle).
Reviewed By: DurhamG
Differential Revision: D6878326
fbshipit-source-id: 9c3951e4673b8de61aae73a51e1bfff422f38d0f
Summary:
This is similar to `debugdata`, but instead of taking a file revision (or
file node in remotefilelog's case), it takes a revset.
This is more useful practically, since the user would know commit hashes
easily but file nodes are hidden from the UI.
This is intended to make it easier to investigate LFS contents.
Reviewed By: DurhamG, ryanmce
Differential Revision: D6891770
fbshipit-source-id: 415da9b773c30830a48c09eda9f1854c416e3222
Summary:
Previously there is no way to disable the global cache, since an empty
`lfs.usercache` means a default location.
This patch changes the meaning of an empty `lfs.usercache` to "disable".
This is useful for cases when the user cannot write to `~/.cache` and is
using the default configuration.
Reviewed By: ryanmce
Differential Revision: D6890047
fbshipit-source-id: 352dea5142178d5331e37a48957d9d2b8af26ab5
02c30db0443d (lfs: add a repo requirement for this extension once an lfs
file is committed) introduced a regression that prevents committing file
deletion. This patch fixes that.
Differential Revision: https://phab.mercurial-scm.org/D1717
# skip-blame because this was mechanically rewritten the following script. I
ran it on both *.t and *.py, but none of the *.py changes were proper. All *.t
ones appear to be, and they run without addition failures on both Windows and
Linux.
import argparse
import os
import re
ap = argparse.ArgumentParser()
ap.add_argument('path', nargs='+')
opts = ap.parse_args()
globre = re.compile(r'^(.*) \(glob\)(.*)$')
for p in opts.path:
tmp = p + '.tmp'
with open(p, 'rb') as src, open(tmp, 'wb') as dst:
for line in src:
m = globre.match(line)
if not m or '$LOCALIP' in line or '*' in line:
dst.write(line)
continue
if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
dst.write(line)
continue
dst.write(m.group(1) + m.group(2) + '\n')
os.unlink(p)
os.rename(tmp, p)
The shell construct here appears to be unevenly supported: it works in /bin/sh
on FreeBSD, but it doesn't seem to work when /bin/sh is dash. Using a Python
inline directive works fine, so let's just do that instead.
Differential Revision: https://phab.mercurial-scm.org/D1636
This is the same mechanism in place for largefiles, and solves several problems
working with multiple local repositories. The existing largefiles method is
reused in place, because I suspect that there are other functions that can be
shared. If we wait a bit to identify more before `hg cp lfutil.py ...`, the
history will be easier to trace.
The push between repo14 and repo15 in test-lfs.t arguably shouldn't be uploading
any files with a local push. Maybe we can revisit that when `hg push` without
'lfs.url' can upload files to the push destination. Then it would be consistent
for blobs in a local push to be linked to the local destination's cache.
The cache property is added to run-tests.py, the same as the largefiles
property, so that test generated files don't pollute the real location. Having
files available locally broke a couple existing lfs-test-server tests, so the
cache is cleared in a few places to force file download.
We do the same thing on clone for the largefiles extension, as a convenience.
Similar to largefiles, it's probably safer to only enable this extension on a
per repo basis because it is trivial to add an lfs file. And that gives the
repository some centralized VCS characteristics.
This covers both the vanilla repo -> lfs repo and largefiles -> lfs conversions.
The largefiles extension adds the requirement directly, because it has a
dedicated command to convert. Using the convert extension is better, because it
supports more features.
I'd like ideas about how to ensure that converting away from lfs works on all
files. (See comments in test-lfs.t)
Largefiles does the same thing (also delayed until the first largefile commit),
to prevent access to the repo without the extension. In the case of this
extension, not having the extension loaded while accessing an lfs file results
in cryptic errors about "missing processor for flag '0x2000'". If enabled
locally but not remotely, the cryptic error message is about no common
changegroup version. (It wants '03', which is currently experimental.)
The largefiles extension looks for any tracked file that starts with '.hglf/'.
Unfortunately, that doesn't work here. I didn't see any way to get the files
that were just committed, without doing a full status. But since there's no
secondary check on adding an lfs file once the extension is loaded and a
threshold set, the best practice is to only enable this locally on a repo that
needs it. That should minimize the unnecessary overhead for repos without an
lfs file.
The purpose of this is the same as the built-in largefiles extension- to handle
huge files outside of the normal storage system, generally to keep the amount of
data cloned to a lower amount. There are several benefits of implementing the
git-lfs protocol, instead of using the largefiles extension:
- Bitbucket and Github support (and probably wider support in 3rd party
hosting sites in general). [1][2]
- The number of hg internals monkey patched are several orders of magnitude
lower, so it will be easier to reason about and maintain. Future commands
will likely just work, without requiring various wrappers.
- The "standin" files are only written to the filelog, not the disk. That
should avoid weird edge cases where the largefile and standin files get out
of sync. [3] It also avoids the occasional printing of the "hidden" standin
file in various messages.
- Filesets like size() will work, even if the file isn't present. (It always
says 41 bytes for largefiles, whether present or not.)
The only place that I see where largefiles comes out on top is that it works
with `hg serve` for simple sharing, without external infrastructure. Getting
lfs-test-server working was a hassle, and took awhile to figure out. Maybe we
can do something to make it work in the future.
Long term, I expect that this will be highly preferred over largefiles. But if
we are to recommend this to largefile users, there are some UI issues to
bikeshed. Until they are resolved, I've marked this experimental, and am not
putting a pointer to this in the largefiles help. The (non exhaustive) list of
issues I've seen so far are:
- It isn't sufficient to just enable the largefiles extension- you have to
explicitly add a file with --large before it will pay attention to the
configured sizes and patterns on future adds. The justification being that
once you use it, you're stuck with it. I've seen people confused by this,
and haven't liked it myself. But it's also saved me a few times. Should we
do something like have a specific enabling config setting that must be set
in the local repo config, so that enabling this extension in the user or
system hgrc doesn't silently start storing lfs files?
- The largefiles extension adds a repo requirement when the first largefile is
committed, so that the extension must always be enabled in the future. This
extension is not doing that, and since I only enabled it locally to avoid
infecting other repos, I got a cryptic error about missing flag processors
when I cloned. Is there no repo requirement due to shallow/narrow clone
considerations (or other future advanced things)?
- In the (small amount of) reading I've done about the git implementation, it
seems that the files and sizes are stored in a tracked .gitattributes file.
I think a tracked file for this would be extremely useful for consistency
across developers, but this kind of touches on the tracked hgrc file
proposal a few months back.
- The git client can specify file patterns, not just sizes.
- The largefiles extension has a cache directory in the local repo, but also a
system wide one. We should probably implement a system wide cache too, so
that multiple clones don't have to refetch the files from the server.
- Jun mentioned other missing features, like SSH authentication, gc, etc.
The code corresponds to c0492b73c7ef in hg-experimental. [4] The only tweaks
are to load the extension in the tests with 'lfs=' instead of
'lfs=$TESTDIR/../hgext3rd/lfs', change the import in the *.py test to hgext
(from hgext3rd), add the 'testedwith' declaration, and mark it experimental for
now. The infinite-push, p4fastimport, and remotefilelog tests were left behind.
The devel-warnings for unregistered config options are not corrected yet, nor
are the import check warnings.
[1] https://www.mercurial-scm.org/pipermail/mercurial/2017-November/050699.html
[2] https://bitbucket.org/site/master/issues/3843/largefiles-support-bb-3903
[3] https://bz.mercurial-scm.org/show_bug.cgi?id=5738
[4] https://bitbucket.org/facebook/hg-experimental
