The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
This patch uses "wvfs of the parent repository" ('pwvfs') instead of
'wvfs' of own repository, because 'self._path' is the path to this
subrepository as seen from the parent repository.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Python 2.6 introduced a new octal syntax: "0oXXX", replacing "0XXX". The
old syntax is not recognized in Python 3 and will result in a parse
error.
Mass rewrite all instances of the old octal syntax to the new syntax.
This patch was generated by `2to3 -f numliterals -w -n .` and the diff
was selectively recorded to exclude changes to "<N>l" syntax conversion,
which will be handled separately.
This is a step toward replacing the extdiff internals with archive, in order to
support 'extdiff -S'. Only Mercurial subrepos are supported for now.
If a file is missing from the filesystem, it is silently skipped. Perhaps it
should warn, but it cannot abort when working with extdiff because deleting a
file is a legitimate diff.
Some code cannot handle a subrepo based on the working directory (e.g.
sub.dirty()), so the caller must opt in. This will be useful for archive, and
perhaps some other commands. The git and svn methods where this is used may
need to be fixed up on a case by case basis.
While hopefully atypical, there are reasons that a subrepository revision can be
lost that aren't covered by corruption of the .hgsubstate revlog. Such things
can happen when a subrepo is amended, stripped or simply isn't pulled from
upstream because the parent repo revision wasn't updated yet. There's no way to
know if it is an error, but this will find potential problems sooner than when
some random revision is updated.
Until recently, convert made no attempt at rewriting the .hgsubstate file. The
impetuous for this is to verify the conversion of some repositories, and this is
orders of magnitude faster than a bash script from 0..tip that does an
'hg update -C $rev'. But it is equally useful to determine if everything has
been pulled down before taking a thumb drive on the go.
It feels somewhat wrong to leave this out of verifymod (mostly because the file
is already read in there, and the final summary is printed before the subrepos
are checked). But verifymod looks very low level, so importing subrepo stuff
there seems more wrong.
This code is already used in a couple of places, and will need to be used in
others where wdir() support is needed. Trying to get the wdir() revision stored
in self._substate[1] when creating the object in subrepo.subrepo() resulted in
breaking various status and diff tests. This is a far less invasive change.
This will be used in an upcoming patch. It's a one-off use, but seems better to
be contained in the subrepo module, than for the next patch to overwrite the
_ctx and _state fields in another module. '' is used as the default revision in
subrepo.state() if it can't be found, so it seems like a safe choice.
'git diff-index' by default does not recurse into subdirectories
when changes are found. As a result, the directory is shown as changed,
rather than the files in this directory.
Adding '-r' results in recursing until the blobs themselves are checked.
The previous behavior when archiving a subrepo 's' on Windows was to internally
name the file under it 's\file', due to the use of vfs.reljoin(). When printing
the file list from the archive on Windows or Linux, the file was named
's\\file'. The archive extracted OK on Windows, but if the archive was brought
to a Linux system, it created a file named 's\file' instead of a directory 's'
containing 'file'.
*.zip format achives seemed not to have the problem, but this was definitely an
issue with *.tgz archives.
Largefiles actually got this right, but a test is added to keep this from
regressing. The subrepo-deep-nested-change.t test was repurposed to archive to
a file, since there are several subsequent tests that archive to a directory.
The output change is losing the filesystem prefix '../archive_lf' and not
listing the directories 'sub1' and 'sub1/sub2'.
With many commands accepting a '-S' or an explicit path to trigger recursing
into subrepos, it seems that --hidden needs to be propagated too.
Unfortunately, many of the subrepo layer methods discard the options map, so
passing the option along explicitly isn't currently an option. It also isn't
clear if other filtered views need to be propagated, so changing all of those
commands may be insufficient anyway.
The specific jam I got into was amending an ancestor of qbase in a subrepo, and
then evolving. The patch ended up being hidden, and outgoing said it would only
push one unrelated commit. But push aborted with an 'unknown revision' that I
traced back to the patch. (Odd it didn't say 'filtered revision'.) A push with
--hidden worked from the subrepo, but that wasn't possible from the parent repo
before this.
Since the underlying problem doesn't actually require a subrepo, there's
probably more to investigate here in the discovery area. Yes, evolve + mq is
not exactly sane, but I don't know what is seeing the hidden revision.
In lieu of creating a test for the above situation (evolving mq should probably
be blocked), the test here is a marginally useful case where --hidden is needed
in a subrepo: cat'ing a file in a hidden revision. Without this change, cat
aborts with:
$ hg --hidden cat subrepo/a
skipping missing subrepository: subrepo
[1]
When passing a --rev, 'hg incoming -S' previously suffered from the same output
truncation or abort that was fixed for 'hg outgoing -S' in the previous patch,
for the same reasons.
Unlike push, subrepos are currently only pulled when the outer repo is updated,
not when the outer repo is pulled. That makes matching 'hg pull' behavior
impossible. Listing all incoming csets in the subrepo seems like the most
useful behavior, and is consistent with 'hg outgoing -S'.
The previous behavior didn't reflect what would actually be pushed- push will
ignore --rev and --branch in the subrepo and push everything. Therefore,
'push -r {rev}' would not list everything, unless {rev} was also the revision of
the subrepo's tip. Worse, if a hash was passed in, the command would abort
because that hash would either not be in the outer repo or not in the subrepo.
The '' that is used to represent the state of a not-yet-committed
subrepo cannot be written to the file, because the code that parses
the file splits on ' ' and expects two parts.
Given that the .hgsubstate file is automatically rewritten on commit, it seems
a little strange that the file is written out during a merge.
Prior to ef4d6b79dd3a, the subrelpath() (now _relpath) for hgsubrepo was
calculated by removing the root path of the outermost repo from the root path of
the subrepo. Since the root paths use platform specific separators, and the
relative path is printed by various commands, the output of these commands
require a glob (and check-code.py enforces this).
In an effort to be generic to all subrepos, ef4d6b79dd3a started calculating
this path based on the parent repo, and then joining the subrepo path in .hgsub.
One of the tests in test-subrepo.t creates a subrepo inside a directory, so the
path being joined contained '/' instead of '\'. This made the test fail with a
'~' status, because the glob is unnecessary[1]. Removing them made the test
work, but then check-code complains. We can't just drop the check-code rule,
because sub-subrepos are still joined with '\'. Presumably the other subrepo
types have this issue as well, but there likely isn't a test with git or svn
repos inside a subdirectory.
This simply restores the exact _relpath value (and output) for hgsubrepos prior
to ef4d6b79dd3a.
[1] http://www.selenic.com/pipermail/mercurial-devel/2015-April/068720.html
"dirpath" in the tuple yielded by "vfs.walk()" is relative one from
the root of specified vfs, and absolute path in the warning message is
composed by "vfs.join()".
On the other hand, target file "f" exists in "dirpath", and
"reljoin()" is needed to unlink "f" by "vfs.unlink()".
As a preparation for vfs migration of "_sanitize()", this patch passes
"wvfs" to "_sanitize()" and use "wvfs.base" instead of absolute path
to a subrepository.
This patch adds propertycache-ed "_relpath" field to
"abstractsubrepo", to centralize "subrelpath" logic into it.
Now, "subrelpath()" can always return "_relpath" field of the
specified subrepo object, because it is ensured that subrepo object
has it. To reduce changes in this patch, "subrelpath()" itself is
still kept, even though it seems to be redundant.
This is also a part of eliminating "os.path.*" API invocations for
"Windows UTF-8 Plan".
This patch doesn't create vfs object in "abstractsubrepo.__init__()"
but adds propertycache-ed "wvfs" field, because the latter can:
- delay vfs instantiation until it is actually needed
- allow to use "hgsubrepo._repo.wvfs" as "wvfs"
"hgsubrepo._repo" is initialized after
"abstractsubrepo.__init__()" invocation, and passing
"hgsubrepo._repo.wvfs" to "abstractsubrepo.__init__()" is
difficult.
This patch passes "ctx" and "path" instead of "ui" to
"abstractsubrepo.__init__()" and stores them as "_ctx" and "_path" to
use them in subsequent patches.
This also removes redundant field initializations in the constructor
of classes derived from "abstractsubrepo".
It isn't obvious which file is the problem with deep subrepos, so provide the
path. Since the parsing is done with a ctx and not a subrepo object, it isn't
possible to display a path from the root subrepo. Therefore, the path shown is
relative to cwd.
There's no test coverage for the first abort, and I couldn't figure out how to
trigger it, but it is changed for consistency.
This patch newly adds "dirtyreason()" to centralize composing dirty
reason message like "uncommitted changes in subrepository 'xxxx'".
There are 3 similar messages below, and this patch is a part of
preparations for unifying them into (1), too.
1. uncommitted changes in subrepository 'XXXX'
2. uncommitted changes in subrepository XXXX
3. uncommitted changes in subrepo XXXX
This patch chooses adding new method "dirtyreason()" instead of making
"dirty()" return "reason string", because:
- some of existing "dirty()" implementation is too complicated to do
so simply, and
- ill-mannered 3rd party subrepo classes, of which "dirty()" doesn't
return "reason string", cause meaningless message (even though it
is rare case)
Paths into the subrepo are not yet supported.
The need to use the workingctx in the subrepo will likely be used more in the
future, with the proposed working directory revset symbol. It is also needed
with archive, if that code is to be reused to support 'extdiff -S'.
Unfortunately, it doesn't seem possible to put the smarts in subrepo.subrepo(),
as it breaks various status and diff tests.
I opted not to pass the desired revision into the subrepo method explicitly,
because the only ones that do pass an explicit revision are methods like status
and diff, which actually operate on two contexts- the subrepo state and the
explicitly passed revision.
This brings parity with gitsubrepo and svnsubrepo (which already reference their
parent), and will be used in an upcoming patch.
I'm a bit concerned that the parent context could get stale (consider what
happens when the parent repo is reverted for example). I tried adding the
parent context to the substate tuple so that the parent is available everywhere
a state change is possible, but that made submerge() unhappy. Even with
removing the parent context inside submerge(), I wasn't able to get all of the
test diffs fixed.
But since the other subrepos reference their parent too, if there is a problem,
it is a preexisting one (that nobody seems to be running into). It can be fixed
if/when it pops up.
No behavior changes here since gitsubrepo.status() doesn't currently populate
clean, and ignores whether unknown files were actually requested. But this is
in line with other calls to status, and should avoid future surprises.
This follows normal Mercurial rules, and the message is lifted from
workingctx.add(). The file is printed with abs() to be consistent with how it
is printed in workingctx, even though that is inconsistent with how added files
are printed in verbose mode. Further, the 'already tracked' notifications come
after all of the files that are added are printed, like in Mercurial.
As a side effect, we now have the reject list to return to the caller, so that
'hg add' exits with the proper code. It looks like an abort occurs if git fails
to add the file. Prior to touching 'snake.python' in the test, this was the
result of attempting to add the file after a 'git rm':
fatal: pathspec 'snake.python' did not match any files
abort: git add error 128 in s (in subrepo s)
I'm not sure what happens when git is a deep subrepo, but the 'in s' and
'in subrepo s' from @annotatesubrepoerror are redundant here. Maybe we should
stat the files before invoking git to catch this case and print out the prettier
hg message? The other thing missing from workingctx.add() is the call to
scmutil.checkportable(), but that would need to borrow the parent's ui object.
The previous test gave a false success because only an hg-ignored pattern was
specified. Therefore match.files() was empty, and it fell back to the files
unknown to git. The simplest fix is to always consider what is unknown to git,
as well as anything specified explicitly. Files that are ignored by git can
only be introduced by an explicit mention in match.files().
This is in line with the other subrepo classes (i.e. only the filenames are
returned). Archive iterates over the manifest keys, and since subrepo archive()
uses this method, it does too now. This will allow largefiles to override its
manifest keys to present the largefiles instead of the standins. Once in place,
we shouldn't need the copy/paste overrides of archive and subrepo archive in
largefiles, that don't quite behave like the core methods they are overriding.
There are currently no tests for change/remove conflicts of subrepos,
and it's pretty broken. Add some tests demonstrating some of the
breakages and fix the most obvious one (a KeyError when trying to look
up a subrepo in the wrong context).
This passed an empty list to filerevert() if '--all' was specified, otherwise
the set of modified files. But then filerevert() immediately switched this and
reinitialized 'pats' to an empty list if '--all' was *not* specified.
Since the point of --dry-run is to show what will happen, the output with and
without it should agree. And since revert wasn't being called on subrepos with
--dry-run before, revert in the subrepo had to be defanged in this case.
Previously, git subrepo diffs did not have a newline at the end.
This caused multiple subrepo diffs to be joined on the same line.
Additionally, the command prompt after the diff still contained
a part of the diff.
The largefiles extensions needs to be able to pass --large, --normal and
--lfsize to subrepos via cmdutil.add() and hgsubrepo.add(). Rather than add
additional special purpose arguments, stop extracting the existing args from the
**opts passed to commands.add() and just pass them along.
This change touches every module in which repository.opener was being used, and
changes it for the equivalent repository.vfs. This is meant to make it easier
to split the repository.vfs into several separate vfs.
It should now be possible to remove localrepo.opener.
V2: use 'self._ctx.node()' instead of 'rev' in makefileobj.
As Matt Harbison mentioned, using 'rev' does not make sense,
since we'd be passing a git revision to the top-level
Mercurial repository.
Previously, revert was only possible if the '--no-backup'
switch was specified.
Now, to support backups, we explicitly go over all modified
files in the subrepo.
Up until now we compared the "path" and "sharedpath" repository properties to
check if a repository is shared. This was relying an implementation detail of
shared repositories. In order to make it possible to change the way shared
repositories are implemented, we encapsulate this check into its own localrepo
method, called shared.
This new method returns None if the repository is shared, and something else
(for now a string describing the short of share) otherwise.
The reason why I did not call this method "isshared" and made it return a
boolean is that I plan to introduce a new type of shared repository soon.
# NOTES:
This is the first patch in a series whose purpose is to add support for
creating "full repository shares", which are repositories that share everything
with the repository source except their current revision, branch and bookmark.
This series is RFC because I am not very sure of some of the solutions I took.
Comments are welcome!
This no longer needs to be explicitly passed because the subrepo object tracks
the 'ui' reference since d4e8aa61370d. See the change to 'archive' for details
about the differences between the output level in the root repo and subrepo 'ui'
object.
The only use for 'ui' in revert is to emit status and warning messages, and to
check the verbose flag prior to printing the action to be performed on a file.
The local repo's ui was already being used to print a warning message in
wctx.forget() and for 'ui.slash' when walking dirstate in the repo.status()
call. Unlike other methods where the matcher is passed along and narrowed, a
new matcher is created in each repo, and therefore the bad() method already used
the local repo's ui.
This no longer needs to be explicitly passed because the subrepo object tracks
the 'ui' reference since d4e8aa61370d. See the change to 'archive' for details
about the differences between the output level in the root repo and subrepo 'ui'
object.
The only use for 'ui' in remove is to emit status and warning messages, and to
check the verbose flag prior to printing files to be removed. The bad() method
on the matcher still uses the root repo's ui, because narrowing the matcher
doesn't change the ui object.
The local repo's ui was already being used to print a warning message in
wctx.forget() and for 'ui.slash' when walking dirstate in the repo.status()
call.
This no longer needs to be explicitly passed because the subrepo object tracks
the 'ui' reference since d4e8aa61370d. See the change to 'archive' for details
about the differences between the output level in the root repo and subrepo 'ui'
object.
The only use for 'ui' in forget is to emit status and warning messages, and to
check the verbose flag prior to printing files to be forgotten. The bad()
method on the matcher still uses the root repo's ui, because narrowing the
matcher doesn't change the ui object.
The local repo's ui was already being used to print a warning message in
wctx.forget() and for 'ui.slash' when walking dirstate in the repo.status()
call.
This no longer needs to be explicitly passed because the subrepo object tracks
a 'ui' reference since d4e8aa61370d. See the change to 'archive' for details
about the differences between the output level in the root repo and subrepo 'ui'
object.
The only use for 'ui' in cat is to emit a status message when a subrepo is
missing. The bad() method on the matcher still uses the root repo's ui, because
narrowing the matcher doesn't change the ui object.
The current state of subrepo methods is to pass a 'ui' object to some methods,
which has the effect of overriding the subrepo configuration since it is the
root repo's 'ui' that is passed along as deep as there are subrepos. Other
subrepo method are *not* passed the root 'ui', and instead delegate to their
repo object's 'ui'. Even in the former case where the root 'ui' is available,
some methods are inconsistent in their use of both the root 'ui' and the local
repo's 'ui'. (Consider hg._incoming() uses the root 'ui' for path expansion
and some status messages, but also calls bundlerepo.getremotechanges(), which
eventually calls discovery.findcommonincoming(), which calls
setdiscovery.findcommonheads(), which calls status() on the local repo 'ui'.)
This inconsistency with respect to the configured output level is probably
always hidden, because --verbose, --debug and --quiet, along with their 'ui.xxx'
equivalents in the global and user level hgrc files are propagated from the
parent repo to the subrepo via 'baseui'. The 'ui.xxx' settings in the parent
repo hgrc file are not propagated, but that seems like an unusual thing to set
on a per repo config file. Any 'ui.xxx' options changed by --config are also
not propagated, because they are set on repo.ui by dispatch.py, not repo.baseui.
The goal here is to cleanup the subrepo methods by dropping the 'ui' parameter,
which in turn prevents mixing subtly different 'ui' instances on a given subrepo
level. Some methods use more than just the output level settings in 'ui' (add
for example ends up calling scmutil.checkportabilityalert() with both the root
and local repo's 'ui' at different points). This series just goes for the low
hanging fruit and switches methods that only use the output level.
If we really care about not letting a subrepo config override the root repo's
output level, we can propagate the verbose, debug and quiet settings to the
subrepo in the same way 'ui.commitsubrepos' is in hgsubrepo.__init__.
Archive only uses the 'ui' object to call its progress() method, and gitsubrepo
calls status().
Creation of the subrepo's '_repo' object creates a new 'ui' by combining the
parent repo's 'baseui' and reading in the subrepo's hgrc file. This simply
avoids 'self.ui' and 'self._repo.ui' pointing to different objects, which seems
like a potential source of bugs.
Git and Svn subrepos are unchanged, because they don't have their own ui, and
have always used their parent's for their configuration.
The localrepository class has a 'ui' member, so keeping the names the same will
allow for duck typing with subrepo instances when accessing 'ui'. Changing this
is easier than finding all of the localrepository instance uses and renaming
that to '_ui'.
Previously, git subrepos did not support reverting.
This change adds basic support for reverting
when '--no-backup' is specified.
A warning is given (and the current state is kept)
when a revert is done without the '--no-backup' flag.
Since addremove on the top of a directory tree will recursively handle sub
directories, it should be the same with deep subrepos, once the user has
explicitly asked to process a subrepo. This really only has an effect when a
path that is a subrepo (or is in a subrepo) is given, since -S causes all
subrepos to be processed already. An addremove without a path that crosses into
a subrepo, will still not enter any subrepos, per backward compatibility rules.
The recursive addremove operation occurs completely before the first subrepo is
committed. Only hg subrepos support the addremove operation at the moment- svn
and git subrepos will warn and abort the commit.
This will be used in the next patch to print a warning from the base class. It
seems better than having to explicitly pass it to a new method, since a lot of
existing methods also require it.
So far, git subrepositories were silently ignored for diffs.
This patch adds support for git subrepositories,
with the remark that --include and --exclude are not supported.
If --include or --exclude are used, the subrepo is ignored.
This patch also replaces "self._getstorehashcachepath" (building
absolute path up) by "self._getstorehashcachename" (building relative
path up), because "vfs.writelines" requires relative path.
This patch also replaces "self._getstorehashcachepath" (building
absolute path up) by "self._getstorehashcachename" (building relative
path up), because "vfs.tryreadlines" requires relative path.
This patch makes "_readstorehashcache()" return "[]" (returned by
"vfs.tryreadlines()"), when cache file doesn't exist, even though
"_readstorehashcache()" returned '' (empty string) in such case before
this patch.
"_readstorehashcache()" is invoked only by the code path below in
"_storeclean()":
for filehash in self._readstorehashcache(path):
if filehash != itercache.next():
clean = False
break
In this case, "[]" and '' don't differ from each other, because both
of them cause avoiding iteration of "for loop".
This "vfs" object will be used by subsequent patches to handle cache
store hash files without direct file APIs.
This patch decorates "_cachestorehashvfs" with "@propertycache" to
delay vfs creation, because it is used only for cooperation with other
repositories.
In this patch, "/" is used as the path separator, even though
"self._repo.join" uses platform specific path separator (e.g. "\\" on
Windows). But it is reasonable enough, because "store" and other
management file handling already include such implementation, and they
work well.
"_calcfilehash" can be completely replaced by simple "vfs.tryread"
invocation.
def _calcfilehash(filename):
data = ''
if os.path.exists(filename):
fd = open(filename, 'rb')
data = fd.read()
fd.close()
return util.sha1(data).hexdigest()
Building absolute path "absname" up by "self._repo.join" for files in
"filelist" is avoided, because "vfs.tryread" does so internally.
Existance of specified "path" should be examined by "exists" via wvfs
of the parent repository, because the working directory of the parent
repository may be in UTF-8 mode. Wide API should be used via wvfs in
such case.
In this patch, "/" is used as the path separator, even though "path"
uses platform specific path separator (e.g. "\\" on Windows). But it
is reasonable enough, because "store" and other management file
handling already include such implementation, and they work well.
"util.makedirs" for the (sub-)repository root of "hgsubrepo" is also
executed in the constructor of "localrepository", if "create" is True
and ".hg" of it doesn't exist.
This patch avoids redundant "util.makedirs" invocation in the
constructor of "hgsubrepo".
Like 'forget', git and svn subrepos are currently not supported. Unfortunately
the name 'remove' is already used in the subrepo classes, so we break the
convention of naming the subrepo function after the command.
We do modify the lists that make up the status in several places, so
it seems risky to use the same instance of a list for several
different status types. Use a separate empty list for each type
instead.