The purpose of this is the same as the built-in largefiles extension- to handle
huge files outside of the normal storage system, generally to keep the amount of
data cloned to a lower amount. There are several benefits of implementing the
git-lfs protocol, instead of using the largefiles extension:
- Bitbucket and Github support (and probably wider support in 3rd party
hosting sites in general). [1][2]
- The number of hg internals monkey patched are several orders of magnitude
lower, so it will be easier to reason about and maintain. Future commands
will likely just work, without requiring various wrappers.
- The "standin" files are only written to the filelog, not the disk. That
should avoid weird edge cases where the largefile and standin files get out
of sync. [3] It also avoids the occasional printing of the "hidden" standin
file in various messages.
- Filesets like size() will work, even if the file isn't present. (It always
says 41 bytes for largefiles, whether present or not.)
The only place that I see where largefiles comes out on top is that it works
with `hg serve` for simple sharing, without external infrastructure. Getting
lfs-test-server working was a hassle, and took awhile to figure out. Maybe we
can do something to make it work in the future.
Long term, I expect that this will be highly preferred over largefiles. But if
we are to recommend this to largefile users, there are some UI issues to
bikeshed. Until they are resolved, I've marked this experimental, and am not
putting a pointer to this in the largefiles help. The (non exhaustive) list of
issues I've seen so far are:
- It isn't sufficient to just enable the largefiles extension- you have to
explicitly add a file with --large before it will pay attention to the
configured sizes and patterns on future adds. The justification being that
once you use it, you're stuck with it. I've seen people confused by this,
and haven't liked it myself. But it's also saved me a few times. Should we
do something like have a specific enabling config setting that must be set
in the local repo config, so that enabling this extension in the user or
system hgrc doesn't silently start storing lfs files?
- The largefiles extension adds a repo requirement when the first largefile is
committed, so that the extension must always be enabled in the future. This
extension is not doing that, and since I only enabled it locally to avoid
infecting other repos, I got a cryptic error about missing flag processors
when I cloned. Is there no repo requirement due to shallow/narrow clone
considerations (or other future advanced things)?
- In the (small amount of) reading I've done about the git implementation, it
seems that the files and sizes are stored in a tracked .gitattributes file.
I think a tracked file for this would be extremely useful for consistency
across developers, but this kind of touches on the tracked hgrc file
proposal a few months back.
- The git client can specify file patterns, not just sizes.
- The largefiles extension has a cache directory in the local repo, but also a
system wide one. We should probably implement a system wide cache too, so
that multiple clones don't have to refetch the files from the server.
- Jun mentioned other missing features, like SSH authentication, gc, etc.
The code corresponds to c0492b73c7ef in hg-experimental. [4] The only tweaks
are to load the extension in the tests with 'lfs=' instead of
'lfs=$TESTDIR/../hgext3rd/lfs', change the import in the *.py test to hgext
(from hgext3rd), add the 'testedwith' declaration, and mark it experimental for
now. The infinite-push, p4fastimport, and remotefilelog tests were left behind.
The devel-warnings for unregistered config options are not corrected yet, nor
are the import check warnings.
[1] https://www.mercurial-scm.org/pipermail/mercurial/2017-November/050699.html
[2] https://bitbucket.org/site/master/issues/3843/largefiles-support-bb-3903
[3] https://bz.mercurial-scm.org/show_bug.cgi?id=5738
[4] https://bitbucket.org/facebook/hg-experimental
Following 30a570fd0165, running run-tests.py from outside tests/ would lead to
the creation of .testtimes and test-*.t.err in $PWD instead of $TESTDIR. This
patch fixes that and updates the relevant test.
In paper, coal, gitweb and monoblue a new "tag" (or multiple, if there are many
instabilities) is added to the same line that has phase, branch, etc of a
changeset; in gitweb and monoblue this element has a light red background, in
paper and coal the element is black and underlined. In spartan theme
instabilities are shown on a separate line.
While test-obsolete.t uses first(phasedivergent()) revset to pick a changeset
to test, that particular changeset is also an orphan, so two different
instability tags are displayed.
This anchor text is problematic in two ways: first, the "back to" part
assumes that you got to the page showing it via the filelog. This is
not necessarily true, as there are other ways to get to that view
besides the filelog view, such as for example following the history of
lines from a file. Second, it uses "filelog" jargon, which refers to
how each file has its own revlog. This is internal jargon that has no
business being exposed to the end user.
I just reworded this template to improve understanding.
As with phases, spartan theme shows a simple "obsolete: yes" on its own line
(this allows replacing "yes" with something more useful in future, like output
of obsfate* template functions). Everywhere else a new "tag" is added to the
same line that has phase, branch, etc of a changeset; in gitweb and monoblue
the element has gray background, in paper and coal the element is gray with a
dashed underline.
We add a pattern matching the infamous timestamp in http log. Now, we should be
able to have change appears in https log without having to re-glob the whole
thing over and over.
The bundle capabilites are sent with every getbundle ssh connection. Every time
the protocol is updated, that string is altered. We get the part about bundle2
string replaced by $USUAL_BUNDLE2_CAPS$ so that we only have to change the
substitution whenever this happens.
The bundle capabilites sent with every getbundle commands. Every time the
protocol is updated, that string is altered. We get that string replace by
$USUAL_BUNDLE_CAPS$ so that we only have to change the substitution whenever
this happens.
The compression list as to be matched with a glob because zstd might not be part
of the option. By using a substitution for these, we won't have to re-glob them
over and over.
We add a 'common-pattern.py' file that allow to define extra pattern. This seems
a cleaner approach than editing the 'run-test.py' file over and over. In
addition allowing arbitrary pattern registration will also help extension.
The format used is a python file is picked out of convenience defining a list of
tuple in 'substitutions' variable. This is picked out of convenience since it is
dead simple to implement.
The end goal is to register more pattern for Mercurial test. There are multiple
common patterns that change over time. That impact is annoying. Using pattern
emplacement for them would be handy.
The next patches will define all the needed patterns and the last patch will
mass-update the tests outputs as it was easier to do in a single pass.
Commit 30a570fd0165 made TESTDIR be the location of the arguments that were
passed to run-tests.py instead of just PWD. It assumed that these tests were
specified using relative paths, so if pwd was /tmp/foo, and the first argument
was /tmp/baz, it would set TESTDIR to /tmp/foo//tmp/baz.
Differential Revision: https://phab.mercurial-scm.org/D1433
In spartan theme phase is shown on its own table row, because there's no single
line of "tags". Everywhere else phase is prepended to the list of "tags" of a
changeset. Its element has a purple-ish color in gitweb and monoblue, and a
dotted line under it and no color in paper and coal (as these themes are frugal
with colors).
This patch intentionally doesn't touch graph, because it needs a rewrite. I'll
get to it pretty soon and in the process will add phase and everything that's
still coming (e.g. obsolescence and instabilities).
.. feature::
hgweb now displays phases of non-public changesets
This patch puts all these changeset "tags" into one template shared everywhere
in paper and coal themes. But it should be noted that some of the templates had
different sets of tags, in some cases it was intended, in others - most likely
not.
First, what's up with all these different ways to get changeset's branch. There
are actually 3 ways to do it in hgweb, they can all be seen in this patch;
"branches", "inbranch" and "branch". They are all lists that consist of 1 or 0
items:
- "branches" has ctx.branch() if current changeset is the tip of that branch
- "inbranch" has ctx.branch() if current changeset is _not_ the tip of that
branch and the branch is not "default"
- "branch" aka "changesetbranch" has ctx.branch() if the branch is not
"default"
The majority of cases (7 vs 2 + /graph) in paper theme used only option 3,
which meant that "default" was never displayed. But other parts of the theme
disagreed with this and used option 1 and option 2 together. For example, the
default view (log) displays "default" on the branch tip (can be seen right
about now on m-s.o/repo/hg), but it disappears when you click on the commit.
Also, using option 3 alone meant that there was no way to tell if a changeset
is the tip of its branch or not (it was always assumed that it's not, see how
some css classes change from "branchname" to the correct "branchhead" in tests)
-- so the two different css styles that exist in paper just for this were
underused.
I think this patch improves the situation, even though it changes the old (even
if inconsistent) behavior. The new behavior matches that of gitweb and
monoblue.
.. feature::
Let 'hg rebase' avoid content-divergence by skipping obsolete
changesets (and their descendants) when they are present in the rebase
set along with one of their successors but none of their successors is
in destination.
In the following example, when trying to rebase 3:: onto 2, the rebase
will abort with "this rebase will cause divergence from: 4":
o 7 f
|
| o 6 e
| |
| o 5 d'
| |
x | 4 d (rewritten as 5)
|/
o 3 c
|
| o 2 x
| |
o | 1 b
|/
o 0 a
By excluding obsolete changesets without a successor in destination (4
in the example above) and their descendants, we make rebase work in this
case, thus giving:
o 11 e
|
o 10 d'
|
o 9 c
|
o 8 b
|
| o 7 f
| |
| | x 6 e (rewritten using rebase as 11)
| | |
| | x 5 d' (rewritten using rebase as 10)
| | |
| x | 4 d
| |/
| x 3 c (rewritten using rebase as 9)
| |
o | 2 x
| |
| x 1 b (rewritten using rebase as 8)
|/
o 0 a
where branch 4:: is left behind while branch 5:: is rebased as expected.
The rationale is that users may not be interested in rebasing orphan
changesets when specifying a rebase set that include them but would
still want "stable" ones to be rebased. Currently, the user is suggested
to allow divergence (but probably does not want it) or they must specify
a rebase set excluding problematic changesets (which might be a bit
cumbersome). The approach proposed here corresponds to "Option 2" in
https://www.mercurial-scm.org/wiki/CEDRebase.
We extend _computeobsoletenotrebased() so that it also return a set of
obsolete changesets in rebase set without a successor in destination but
with at least one successor in rebase set. This
'obsoletewithoutsuccessorindestination' is then stored as an attribute
of rebaseruntime and used in _performrebasesubset() to:
* filter out descendants of these changesets from the revisions to
rebase;
* issue a message about these revisions being skipped.
This only occurs if 'evolution.allowdivergence' option is off and
'rebaseskipobsolete' is on.
This is a short topic to explain how command-line flags can be specified.
Some users have been confused by hg offerring different flag syntax than some
other libraries, so it'd be nice to point them to this rather than explaining
it every time.
Differential Revision: https://phab.mercurial-scm.org/D1270
These new colors and styles are already used in the tooltip that gets shown
when user hovers over line numbers on annotate page. The old styles, replaced
in this patch, look completely unrelated to gitweb or any other hgweb theme.
When origbackuppath is set, when looking to see if a file we are backing up
conflicts with a directory in the origbackuppath, we incorrectly match on
symlinks to directories. This means we try to call vfs.rmtree on the
symlink, which fails.
Differential Revision: https://phab.mercurial-scm.org/D1311
If origbackups are in use, a symlink to a valid directory is backed up, and an
update is made that attempts to backup a file or link over that symlink, we
abort with a bad error message instead of successfully updating.
Differential Revision: https://phab.mercurial-scm.org/D1310
We change subrepos.allowed from a list of allowed subrepo types to
a combination of a master switch and per-type boolean flag.
If the master switch is set, subrepos can be disabled wholesale.
If subrepos are globally enabled, then per-type options are
consulted. Mercurial repos are enabled by default. Everything else
is disabled by default.
We have a security issue with git subrepos. I'm not sure if svn subrepo is
vulnerable, but it seems not 100% safe to allow writing arbitrary data into
a metadata directory. So for now, only hg subrepo is enabled by default.
Maybe we should improve the help to describe why git/svn subrepos are
disabled.
This allows us to minimize the behavior change introduced by the next patch.
I have no idea which config style is preferred in UX POV, but I decided to
get things done.
a) list: 'allowed = hg, git, svn'
b) sub option: 'allowed.hg = True' or 'allowed:hg = True'
c) per-type action: 'hg = allow', 'git = abort'
This is an alternative workaround for the issue5730.
Perhaps this is the simplest way of disabling subrepo operations. It does
nothing clever, but just aborts if Mercurial starts accessing to a subrepo.
I think Greg's patch is more useful since it allows us to at least check
out the parent repository. However, that would be confusing if the default
is flipped to checkout=False and subrepos are silently ignored.
I don't like the config name 'allowed', but I couldn't get any better name.
It wasn't easy to extend the pathauditor to check symlink traversal across
subrepos because pathauditor._checkfs() rejects a directory having ".hg"
directory. That's why I added the explicit islink() check.
No idea if this patch is necessary after we've fixed the issue5730 by
splitting submerge() into planning and execution phases.
When cwd is removed and `hg` is executed, some shells may run `getcwd`
before forking and executing, some may not do it, some may print a
different error message.
The test should be shell-independent so let's just avoid checking the error
message.
Differential Revision: https://phab.mercurial-scm.org/D1282
The original condition was a bit harsh for extension authors since third-party
extensions need to preserve compatibility with older Mercurial versions, where
no defaults would be loaded from the configtable. So let's silence the warning
if the given default value matches, which should be harmless.
When a dirstate backup is restored, it is possible that no actual changes to
the dirstate have been made. In this case, the backup is still a hardlink
to the original dirstate.
Unfortunately, `os.rename` silently fails (nothing happens, and no error
occurs) when `src` and `dst` are hardlinks to the same file. As a result,
the backup is left lying around. Over time, these files accumulate.
When restoring dirstate backups, check if the backup and the dirstate are
the same file, and if so, just delete the backup.
Differential Revision: https://phab.mercurial-scm.org/D1201
I'm not sure why these weren't working on Windows. The failures were generally
in the style of:
- remote: phase-move: cb9a9f314b8b07ba71012fcdbc544b5a4d82ff5b: 1 -> 0
+ remote: "phase-move: $HG_NODE: $HG_OLDPHASE -> $HG_PHASE"
and
- abort: pretxnclose-bookmark.force-forward hook exited with status 1
- [255]
+ abort: pretxnclose-bookmark.force-public hook exited with status 255
+ [255]
These failures originated in 5625d0ddc285::6e3e88681b23.
Previously, the second last test (context.arbitraryfilectx(..)) returned True on
Windows. I changed the repo setup sequence to import a patch, so that way the
repo would have a proper symlink. That made the last test fail, since it is
comparing files in wdir(), one of which is not the expected symlink.
Apparently the (feature !) line matching doesn't work well with (no-eol), so I
had to conditionalize the test instead of the output.
This seems a bit hacky, but works well. There should be no reason that
static-http repo had to load dirstate.
Initially I tried to proxy os.stat() call through vfs so that statichttpvfs
could hook it, but there wasn't a good error value which the statichttpvfs
could return to get around the util.filestat issue.
