'unexpected putlfile response: None' when an http error occurs is not very
helpful.
Instead, leave the handling of urllib2.HTTPError exceptions to other layers.
If the store somehow got corrupted, users could end up in weird situations that
were very hard to recover from or lead to propagation of the corruption.
Instead, spend the extra time checking the hash when copying to the working
directory. If it doesn't match, emit a warning, and don't put wrong content in
the working directory.
Commit of corresponding normal/largefiles pairs would only commit the standin.
That is usually fine, except if either the normal file or the standin is a
remove while the other is an add. In that case it would either give duplicate
colliding entries or lose the file.
Instead, commit both filenames if one of them is a remove.
Before, when merging revisions with missing largefiles, the missing largefiles
would be fetched as a part of the merge. If that failed (for example because
the main repository temporarily was unavailable), the largefile would be left
missing. However, the next commit would abort and (seemed to) fail when
markcommitted tried to mark the standin file as normal and thus had to hash the
largefile that didn't exist. (Actually, the commit would succeed but the
largefile update that follows right after the commit transaction would abort -
quite confusing.)
To fix that, make sure that synclfdirstate only marks files as normal if they
actually exist.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
This regressed in f07f4c45a8f2, when trying to conditionally disable archiving
of largefiles.
I'm not sure if wrapfunction() is the right way to do this, but it seems to
work. The mysterious issue with lfstatus getting out of sync in the proxy and
the unfiltered view crops up again here. See the referenced cset for more info.
Previously, simply having the largefiles extension loaded without any largefiles
added would crash when amending with -I. The problem was with no files in the
matcher, the pattern list of files joined with 'standindir' was empty, and
scmutil.match() would match everything. In lfutil.composestandinmatcher(), the
match function is used to test if the file is a standin, and after getting a
false positive, proceeds to call lfutil.splitstandin(). This returns None
because it isn't a standin, which blows up when passed to rmatcher.matchfn().
Manually overriding _always in getstandinmatcher() probably isn't necessary
anymore, but we leave well enough alone on stable. This regressed in
78632d61a993.
There are currently no users of this, but it is a necessary step before
converting extdiff to use archive. It may be useful to add an argument to
extdiff in the future and allow largefiles to be diffed, but archiving
largefiles can have significant overhead and may not be very diffable, so
archiving them by default seems wrong.
It is a mystery to me why the lfstatus attribute needs to be set on the
unfiltered repo. However if it is set on the filtered repo instead (and the
filtered repo is passed to the original command), the lfstatus attribute is
False in the overrides for archival.archive() and hgsubrepo.archive() when
invoking the archive command. This smells like the buggy status behavior (see
9fc565fa1621, which was reverted in e1574a4a2cad). Neither the status nor
summary commands have this weird behavior in their respective overrides.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Python 2.6 introduced a new octal syntax: "0oXXX", replacing "0XXX". The
old syntax is not recognized in Python 3 and will result in a parse
error.
Mass rewrite all instances of the old octal syntax to the new syntax.
This patch was generated by `2to3 -f numliterals -w -n .` and the diff
was selectively recorded to exclude changes to "<N>l" syntax conversion,
which will be handled separately.
This is a step toward replacing the extdiff internals with archive, in order to
support 'extdiff -S'. Only Mercurial subrepos are supported for now.
If a file is missing from the filesystem, it is silently skipped. Perhaps it
should warn, but it cannot abort when working with extdiff because deleting a
file is a legitimate diff.
Previously, if there were any hidden changesets, the --lfa argument would cause
the command to abort with a hint about using --hidden when it tripped over a
hidden changeset.
The monkey patching in cat() can't be fixed, because it still delegates to the
original bad(). Overriding commands.cat() should go away in favor overriding
cmdutil.cat() anyway, and that matcher can be wrapped with matchmod.badmatch().
No known issues with the previous code since it restored the original method,
but this is cleaner.
The monkey patching in cat is harmless, because it is created locally, and
doesn't pass it anywhere (subrepo cat isn't supported with largefiles).
The logic in the convert extension is more advanced, supporting extra features
like converting revision IDs in 'extras' (e.g. 'amend_source'), supports
updating hashes in commit messages, and outputs an SHA map file. Rather than
try to duplicate all of that, just use the existing code.
Even though the convert extension supports user supplied options like filemap,
etc, those features aren't available on the lfconvert interface. Therefore, it
is safe to use the filemap mechanism (in memory) to handle the standin -> file
rename. The convert extension handles the destination locking for this path.
There was a comment in test-lfconvert.t about the hash on rev 5 being different
because it was doing a better job than "hg remove" + "hg merge" + "hg commit".
It isn't clear to me what was happening or why, but now the hashes match the
original repo exactly after a roundtrip, which seems like a good idea. If there
really was something beneficial about the previous behavior, perhaps merge can
be changed so that everyone benefits.
Converting to a largefiles repo still uses the original (limited) lfconvert
logic.
The choice between the "always" case and the other case is done in
getstandinmatcher() and the next patch will change how it's determined
based on the matcher, so let's prepare by passing in the matcher, not
just the matcher's files.
Extension authors (notably at companies using hg) have been
cargo-culting the `testedwith = 'internal'` bit from hg's own
extensions, which then defeats our "file bugs over here" logic in
dispatch. Let's be more aggressive about trying to give extension
authors a hint about what testedwith should say.
This is related to df069d92be27, but not required for that to function correctly
because 'prefix' always ends with '/', so os.path.join(prefix, _path) simply
concatenates the strings. Take the opportunity to drop the os.path usage, and
avoid confusion by using the same pattern as the overridden subrepo method.
The fileset-generated.t test previously failed with this:
+ hg: parse error: unknown identifier: .hglf/modified
+ (did you mean 'modified'?)
+ [255]
Filesets will find the standins on their own, without any help. While that's
useful for some things like modified(), clean(), etc., it is wrong for things
like size(). Proper fileset support for largefiles is not trivial, but this was
failing with just the extension enabled on a normal repo.
The immediate crash was when checking for requirements immediately after this,
but lfcommands.downloadlfiles() will also crash if --all-largefiles is
specified. That has been in place since atleast e01343f7da6f (2.3-rc) without
anyone noticing.
I can't tell from the peer classes if there's a way to make the custom largefile
functionality work in this case, but atleast it doesn't crash.
mergeupdate already set the flag to update all. This will thus only change
overriderevert and scmutilmarktouched ... where the flag effectually also were
true. The test coverage thus shows no change.
As the flag always is set, it is removed.
This is mainly a change for keeping the code simple and consistent and correct,
but it should also make it faster in many cases.
Before, a --clean update with largefiles would use the "optimization" that it
didn't read hashes from standin files before and after the update. Instead of
trusting the content of the standin files, it would rehash all the actual
largefiles that lfdirstate reported clean and update the standins that didn't
have the expected content. It could thus in some "impossible" situations
automatically recover from some "largefile got out sync with its standin"
issues (even there apparently still were weird corner cases where it could
fail). This extra checking is similar to what core --clean intentionally do
not do, and it made update --clean unbearable slow.
Usually in core Mercurial, --clean will rely on the dirstate to find the files
it should update. (It is thus intentionally possible (when trying to trick the
system or if there should be bugs) to end up in situations where --clean not
will restore the working directory content correctly.) Checking every file when
we "know" it is ok is however not an option - that would be too slow.
Instead, trust the content of the standin files. Use the same logic for --clean
as for linear updates and trust the dirstate and that our "logic" will keep
them in sync. It is much cheaper to just rehash the largefiles reported dirty
by a status walk and read all standins than to hash largefiles.
Most of the changes are just a change of indentation now when the different
kinds of updates no longer are handled that differently. Standins for added
files are however only written when doing a normal update, while deleted and
removed files only will be updated for --clean updates.
This allows passing a matcher down the pathcopies() stack to _forwardcopies().
This will let us add logic in a later patch to avoid tracing copies when not
necessary (like when doing hg diff -r 1 -r 2 foo.txt).
This makes treemanifest.walk() not visit submanifests that are known not to
have any matching files. It does this by calling match.visitdir() on
submanifests as it walks.
This change also updates largefiles to be able to work with this new behavior
in treemanifests. It overrides match.visitdir(), the function that dictates
how walk() and matches() skip over directories.
The greatest speed improvements are seen with narrower scopes. For example,
this commit speeds up the following command on the Mozilla repo from 1.14s
to 1.02s:
hg files -r . dom/apps/
Whereas with a wider scope, dom/, the speed only improves from 1.21s to 1.13s.
As with similar a similar optimization to treemanifest.matches(), this change
will bring out even bigger performance improvements once treemanifests are
loaded lazily. Once that happens, we won't just skip over looking at
submanifests, but we'll skip even loading them.
The benefit of retargeting the local store to the share source is that all
shares will always have access to the largefiles any one of them commit, even if
the user cache is deleted (which is documented to be OK to do). Further, any
push into the source (and now any shares), will likewise make the largefile(s)
visible to all related repositories.
In order to maintain compatibility with existing repos, where the largefiles
would be cached only in the local share, fallback to searching the local share
if it isn't found at the share source.
The unshare command should probably be taught to copy the source store into the
store for the repo being unshared to complete the loop.
This patch changes the test like this:
@@ -159,6 +159,5 @@
$ hg share -q src share_dst --config extensions.share=
$ hg -R share_dst update -r0
getting changed largefiles
- large: largefile $HASH not available from file:///$TESTTMP\share_dst
- 0 largefiles updated, 0 removed
+ 1 largefiles updated, 0 removed
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
The issue writeup mentions pushing a largefile from a remote repo to the main
local repo, and the largefile is then not available in any shares. Since the
push doesn't cache the largefile in $USERCACHE, the trashed $USERCACHE in this
test is equivalent.
