The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
Commit 359818e9961c (largefiles: don't create chain of __contains__
calls, 2015-03-11) introduced a typo with __class instead of
__class__. Let's fix it.
There are (obviously) no tests covering this code path, and I could
not figure out a way to trigger it, so it remains untested.
Spotted by Drew Gottlieb.
Matt Harbison pointed out that my recent cdf4a99e1657 might cause
__contains__ to continously get replaced by another version that calls
itself, since the manifest instance returned by the super method is
always the same instance due to @propertycache. He also suggested
replacing the class instead, as is done with the context class in the
surrounding code. Do so.
We're soon going to add an alternative manifest class
(treemanifest). Rather than extending both those classes by
largesfiles versions, let's replace the method on the manifest
instances.
Even if largefiles extension is enabled in a repository, "repo"
object, which isn't "largefiles.reposetup()"-ed, is passed to
overridden functions in the cases below unexpectedly, because
extensions are enabled for each repositories strictly.
(1) clone without -U:
(2) pull with -U:
(3) pull with --rebase:
combination of "enabled@src", "disabled@dst" and
"not-required@src" cause this situation.
largefiles requirement
@src @dst @src result
-------- -------- --------------- --------------------
enabled disabled not-required aborted unexpectedly
required requirement error (intentional)
-------- -------- --------------- --------------------
enabled enabled * success
-------- -------- --------------- --------------------
disabled enabled * success (only for "pull")
-------- -------- --------------- --------------------
disabled disabled not-required success
required requirement error (intentional)
-------- -------- --------------- --------------------
(4) update/revert with a subrepo disabling largefiles
In these cases, overridden functions cause accessing to largefiles
specific fields of not "largefiles.reposetup()"-ed "repo" object, and
execution is aborted.
- (1), (2), (4) cause accessing to "_lfstatuswriters" in
"getstatuswriter()" invoked via "updatelfiles()"
- (3) causes accessing to "_lfcommithooks" in "overriderebase()"
For safe accessing to these fields, this patch examines whether passed
"repo" object is "largefiles.reposetup()"-ed or not before accessing
to them.
This patch chooses examining existence of newly introduced
"_largefilesenabled" instead of "_lfcommithooks" and
"_lfstatuswriters" directly, because the former is better name for the
generic "largefiles is enabled in this repo" mark than the latter.
In the future, all other overridden functions should avoid largefiles
specific processing for efficiency, and "_largefilesenabled" is better
also for such purpose.
BTW, "lfstatus" can't be used for such purpose, because some code
paths set it forcibly regardless of existence of it in specified
"repo" object.
This effectively reverts 9fc565fa1621, which caused some normal file copies to
not be displayed as copies. Other normal file copies could be displayed- the
exact reason isn't clear. This also adds two tests that were failing prior to
this backout, so that this can be sorted out next cycle.
The difference between copy cases that worked and those that didn't seemed to be
in copies.pathcopies(). When largefiles isn't enabled for the changed test, or
lfstatus is not set in the commands.status() override, 'y.ancestor(x) == x'.
That wasn't true otherwise, which fell through to the _chain() method. In this
case, the copy is removed in the criss cross loop.
'y.ancestor(x)' returns a context.changectx type, while 'x' is a lfilesctx type
in the failing case. I tried adding the ancestor method to the lfilesctx class
to change the type of the ancestor context, however the context when printed as
a string then gains a '+'. This points to it being a context.committablectx,
which clearly isn't correct for an ancestor. Possibly the problem is the
lfilesctx needs to subclass context.committablectx in some cases, but
context.changectx in others, within the same invocation? I'm not sure how to
pull that off, and backing out this change is safer during the freeze.
As to the status changing when a path is specified, I haven't looked into it
yet.
When a directory is named in the commit file list, the previous behavior was to
walk the list, and if no normal files in the directory were also named, add the
corresponding standin for each largefile in that directory. The directory is
then dropped from the list, so that committing a directory with no normal file
changes works. It then added the corresponding standin directory for the first
largefile seen, by prefixing it with '.hglf/'.
The latter is unnecessary since each affected largefile is explicitly referenced
by its standin in the list. It also caused an abort if there were no changed
largefiles in the directory, because none of its standins changed:
abort: .hglf/foo/bar: no match under directory!
This list of files is used to tweak a matcher in lfutil.updatestandinsbymatch(),
which is what is passed to commit().
The status() call that is ultimately done in the commit code with this matcher
seems to have some OS specific differences. It is not necessary to append '.'
for Windows to run the largefiles tests cleanly. But if '.' is not added to the
list, the match function isn't called on Linux, so status() would miss any
normal files that were also in a named directory. The commit then proceeds
without those normal files, or says "nothing changed" if there were no changed
largefiles in the directory. This is not filesystem specific, as VFAT on Linux
had the same behavior as when run on ext4. It is also not an issue with
lfilesrepo.status(), since that only calls the overridden implementation when
paths are passed to commit. I dont have access to an OS X machine ATM to test
there.
Maybe there's a better way to do this. But since the standin directory for the
first largefile was previously being added, and that caused the same walk in
status(), there's no preformance change to this. There is no danger of
erroneously committing files in '.', because the original match function is
called, and if it fails, the lfutil.updatestandinsbymatch() tweaked matcher only
indicates a match if the file is in the list of standins- and '.' never is. The
added tests confirm this.
The comment about status being buggy with a repo proxy seems to be that status
wasn't being redirected to testing the largefiles themselves- lfstatus was
always False, and so the original status method was being called instead of
doing the largefiles processing. This is because when the various largefile
command overrides call 'repo.lfstatus = True', __setattr__() in repoview is in
turn setting the value on the unfiltered repo, and the value in the filtered
repo remains False.
Explicitly looking at the attribute on the unfiltered repo keeps all views in
sync.
Changeset 5b64e22ecd8e introduced the examination of exec bit of
largefiles in "hg status --rev REV" case, but it doesn't avoid it on
the platform being unaware of exec-bit (e.g. on NTFS of Windows).
Before this patch, while "hg convert", largefiles avoids copying
largefiles in the working directory into the store area by combination
of setting "repo._isconverting" in "mercurialsink{before|after}" and
checking it in "copytostoreabsolute".
This avoiding is needed while "hg convert", because converting doesn't
update largefiles in the working directory.
But this implementation is not efficient, because:
- invocation in "markcommitted" can easily ensure updating
largefiles in the working directory
"markcommitted" is invoked only when new revision is committed via
"commit" of "localrepository" (= with files in the working
directory). On the other hand, "commitctx" may be invoked directly
for in-memory committing.
- committing without updating the working directory (e.g. "import
--bypass") also needs this kind of avoiding
For efficiency of this kind of avoiding, this patch does:
- move "copyalltostore" invocation into "markcommitted"
- remove meaningless procedures below:
- hooking "mercurialsink{before|after}" to (un)set "repo._isconverting"
- checking "repo._isconverting" in "copytostoreabsolute"
This patch invokes "copyalltostore" also in "_commitcontext", because
"_commitcontext" expects that largefiles in the working directory are
copied into store area after "commitctx". In this case, the working
directory is used as a kind of temporary area to write largefiles out,
even though converted revisions are committed via "commitctx" (without
updating normal files).
"lfutil.getstatuswriter" is the utility to get appropriate function to
write largefiles specific status out from "repo._lfstatuswriters".
This patch uses "stack" with an element instead of flag like
"_isXXXXing" or so, because:
- the former works correctly even when customizations are nested, and
- ensuring at least one element can ignore empty check
This changes allows to customize pre-committing procedures according
to conditions.
This patch uses "stack" with an element instead of flag like
"_isXXXXing" or so, because:
- the former works correctly even when customizations are nested, and
- ensuring at least one element can ignore empty check
This patch factors out procedures to update standins for
pre-committing. This is one of preparations to avoid execution of such
procedures according to invocation context.
For example, resuming automated committing (e.g. "hg rebase
--continue") should update standins at the 1st commit, because
largefiles in the working directory may be modified manually. But on
the other hand, it should avoid updating standins at subsequent
committings for efficiency reason.
For simplicity, this patch just moves procedures mechanically only
with replacing below.
- "self" => "repo"
- "lfutil." => (none)
- "orig" invocation => returning "match"
Using "fstandin" instead "standin" as the name of local variable for
the loop below is the only special care, because the latter shadows
the same name function in "lfutil.py".
[before]
for standin in standins:
lfile = lfutil.splitstandin(standin)
if lfdirstate[lfile] != 'r':
lfutil.updatestandin(self, standin)
[after]
for fstandin in standins:
lfile = splitstandin(fstandin)
if lfdirstate[lfile] != 'r':
updatestandin(repo, fstandin)
Before this patch, procedures to update lfdirstate for post-committing
are scattered in "lfilesrepo.commit". In the case of "hg commit" with
patterns for target files ("Case 2"), lfdirstate is updated BEFORE
real committing.
This patch factors out procedures to update lfdirstate for
post-committing into "lfutil.markcommitted", and makes it callable via
"markcommitted" of the context passed to "lfilesrepo.commitctx".
"markcommitted" of the context is called, only when it is committed
successfully.
Passing original "markcommitted" of the context is meaningless in this
patch, but required in subsequent one to prepare something before
invocation of it.
Instead of checking for a partial merge by checking that the matches
has no files and no patterns, check that it's not an
always-matcher. Except for being shorter, it also catches the rare
case of an exact-matcher with no files.
We currently shortcircuit the checking for large file standins if only
patterns of type 'path' are given on the command line. That makes e.g.
"hg st 'glob:foo/**'" unnecessarily slow when the only large files are
in a sibling directory.
Relax the check to be that it is not an always-matcher and that no
large files match the patterns given on the command line.
Note that before this change, only the latter of the following two
would show the status of files in .hglf (since the -I makes
match.anypats() true). After this change, they both display the
status. This behavior doesn't seem correct, but it would be a separate
change to explicitly filter out .hglf even in the shortcircuit case.
hg st .hglf/$file
hg st .hglf/$file -I .
Before this patch, "hg status --rev REV" doesn't list largefiles up
with "M" mark, even if exec bit of them is changed, because
"lfilesrepo.status" doesn't examine exec bit in such case.
Before this patch, "hg status --rev REV" listed largefiles removed in
the working directory up with "R" mark, even if they aren't managed in
the REV. Normal files aren't listed up in such case.
When "lfilesrepo.status" is invoked for "hg status --rev REV", it
treats files on conditions below as "removed" (to avoid manifest full
scan in "ctx.status" ?):
- marked as "R" in lfdirstate, or
- files managed in the target revision but unknown in the manifest
of the working context (= not including "R" files)
But the former can include files not managed in the target context.
To ignore removal status of files not managed in the target context,
this patch drops files unknown in the target revision from "removed"
list.
The status tuple returned from dirstate.status() has an additional
field compared to the other status tuples: lookup/unsure. This field
is just an optimization and not something most callers care about
(they want the resolved value of 'modified' or 'clean'). To prepare
for a single future status type, let's separate out the 'lookup' field
from the rest by having dirstate.status() return a pair: (lookup,
status).
Instead of iterating over all files in the context and ignoring those
that are not standins, pass a standin-matcher to the context and
iterate over only the files matching.
Apart from making the intent clearer, this implementation will also
benefit from any future optimizations done to the manifest walking
code.
The variable 'lfiles' is first used for a set of the names of all the
large files. It is then overwritten with a tuple like the ones
returned from status(). To reduce confusion, let's create a separate
variable for the second use.
At the end of lfilesrepo.status(), we clear the lists of unknown,
ignored and clean files, depending on the values of 'listunknown'
etc. The lists originate from other calls to status(), and it is only
'clean' that may get updated after the calls. Let's remove the need to
clear any of the lists by explicitly only adding to 'clean' when
'listclean' is true.
After previous patches, largefiles in the working directory are
ensured to be updated before "repo.commit" invocation for automated
committing below:
- by "overrides.mergeupdate" via "merge.update" for rebase
- by "overrides.scmutilmarktouched" via "patch.patch" for transplant
This patch removes redundant "lfcommands.updatelfiles" invocation in
"Case 0" code path of "lfilesrepo.commit" for automated committing,
and revises detailed comment.
Before this patch, after successful "hg rebase" of the revision
removing largefiles, "hg status" may still show ""R" for such
largefiles unexpectedly.
"lfilesrepo.commit" executes the special code path for automated
committing while rebase/transplant, and lfdirstate entries for removed
files aren't updated in this code path, even after successful
committing.
Then, "R" entries still existing in lfdirstate cause unexpected "hg
status" output.
This patch synchronizes lfdirstate with dirstate after automated
committing.
This patch passes False as "normallookup" to "synclfdirstate", because
modified files in "files()" of the recent (= just committed) context
should be "normal"-ed.
This is a temporary way to fix with less changes. For fundamental
resolution of this kind of problems in the future, lfdirstate should
be updated with dirstate simultaneously. Hooking "markcommitted" of
ctx in "localrepository.commitctx" may achieve this.
This problem occurs, only when (1) the parent of the working directory
is rebased and (2) it removes largefiles, because:
- if the parent of the working directory isn't rebased, returning to
the initial revision (= update) after rebase hides this problem
- files added on "other" branch (= rebase target) are treated not as
"added" but as "modified" (= "normal" status and "unset"
timestamp) at merging
This patch tests also the status of added largefile, but it is only
for avoiding regression.
In addition to conditions above, "hg status" must not take existing
files to reproduce this problem, because existing files make
"match._files" not empty in "lfilesrepo.status" code path below:
def sfindirstate(f):
sf = lfutil.standin(f)
dirstate = self.dirstate
return sf in dirstate or sf in dirstate.dirs()
match._files = [f for f in match._files
if sfindirstate(f)]
Not empty "match._files" prevents "status" on lfdirstate from
returning the result containing problematic "R" files.
This is reason why "large1" (removed) and "largeX" (added) are checked
separately in this patch.
Problematic code path in "lfilesrepo.commit" is used also by "hg
transplant", but this problem doesn't occur at "hg transplant",
because invocation of "updatelfiles" after transplant-ing in
"overridetransplant" causes cleaning lfdirstate up.
This patch tests also "hg transplant" as same as "hg rebase", but it
is only for avoiding regression.
Before this patch, "hg push" invokes "findcommonoutgoing()" not only
in "exchange.push()" but also in "lfilesrepo.push()", when largefiles
is enabled. The latter is redundant.
This patch registers own "prepushoutgoinghook" function into
"prepushoutgoinghooks" of "localrepository" to reuse
"findcommonoutgoing()" result.
"prepushoutgoinghook" omits "changelog.nodesbetween()" invocation,
because "findcommonoutgoing()" invocation in "exchange.push()" takes
"onlyheads" argument and it considers "nodesbetween()".
Before this patch, "overrides.getoutgoinglfiles()" (called by
"overrideoutgoing()" and "overridesummary()") and "lfilesrepo.push()"
implement similar logic to get outgoing largefiles separately.
This patch centralizes the logic to get outgoing largefiles in
"lfutil.getlfilestoupload()".
"lfutil.getlfilestoupload()" takes "addfunc" argument, because each
callers need different information (and it is useful for enhancement
in the future).
- "overrides.getoutgoinglfiles()" needs only filenames
- "lfilesrepo.push()" needs only hashes of largefiles
This should correct an earlier couple of bad merges (5433856b2558 and
596960a4ad0d, now pruned) that accidentally brought in a change that had
been marked obsolete (244ac996a821).
Since changeset a8955c4d9ef5, "reposetup()" of each extensions is
invoked only on repositories enabling corresponded extensions.
This causes that largefiles specific interactions between the
repository enabling largefiles locally and remote (wire) peer fail,
because there is no way to know whether largefiles is enabled on the
remote repository behind the wire peer, and largefiles specific
"wireproto functions" are not given to any wire peers.
To avoid this problem, largefiles should be enabled in wider scope
than each repositories (e.g. user-wide "${HOME}/.hgrc").
This patch introduces "wirepeersetupfuncs" to setup wire peer by
extensions already enabled. Functions registered into
"wirepeersetupfuncs" are invoked for all wire peers.
This patch uses plain list instead of "util.hooks" for
"wirepeersetupfuncs", because the former allows to control order of
function invocation by order of extension enabling: it may be useful
for workaround of problems with combination of enabled extensions
Some extensions set configuration settings that showed up in 'hg showconfig
--debug' with 'none' as source. That was confusing.
Instead, they will now tell which extension they come from.
This change tries to be consistent and specify a source everywhere - also where
it perhaps is less relevant.
Since the localrepositoyry.push() method in mercurial/localrepo.py is defined
this way:
def push(self, remote, force=False, revs=None, newbranch=False):
it is better for largefiles to call push() on the super class with proper
kwargs to respect the API.
This will avoid breaking other extensions overriding the push method this way:
def push(self, remote, force=False, **kwargs):
Before this patch, if largefiles extension is enabled once in any of
target repositories, commands handling multiple repositories at a time
like below misunderstand that "largefiles" feature is supported also
in all other local repositories:
- clone/pull from or push to localhost
- recursive execution in subrepo tree
This patch registers "featuresetup()" into "featuresetupfuncs" of
"localrepository" to support "largefiles" features only in
repositories enabling largefiles extension, instead of adding
"largefiles" feature to class variable "_basesupported" of
"localrepository".
This patch also adds checking below to the largefiles specific class
derived from "localrepository":
- push to localhost: whether features supported in the local(= dst)
repository satisfies ones required in the remote(= src)
This can prevent useless looking up in the remote repository, when
supported and required features are mismatched: "push()" of
"localrepository" also checks it, but it is executed after looking up
in the remote.
The refactoring of all the context objects allows us to simply pass a basectx
to the __new__ constructor and have it return the same object without
allocating new memory.
This also removes the need to import the context module.
