Before this change, you could get in a start where the checker would either
complain about importing local module before stdlib one or complain about the
local one being wrongly lexically sorted with the stdlib one.
We detect the boundary and avoid complaining about lexical sort across it.
Before this patch, fromlocalfunc() assumes that "module" attribute of
ast.ImportFrom is None for "from . import a", and Python 2.7.x
satisfies this assumption.
On the other hand, with Python 2.6.x, "module" attribute of
ast.ImportFrom is an empty string for "from . import a", and this
causes failure of test-check-module-imports.t.
It doesn't make sense that (a) is allowed whereas (b) is disallowed.
a) from mercurial import hg
from mercurial.i18n import _
b) from . import hg
from .i18n import _
So far fromlocal recognizes relative imports of the form:
from . import D
from .. import E
It wasn't prepared for recognizing relative imports like:
from ..F import G
The bug was not found so far because all relative imports starting
from the parent was in the list of allowsymbolicimports like:
from ..i18n import
from ..node import
We don't really need to report SyntaxErrors, since in theory
docchecker or a test will catch them, but they happen, and
we can't just have the code crash, so for now, we're reporting
them.
On pypy datetime and cProfile are modules written in Python, not in C.
For the purpose of this test, just list them explicitely as builtins,
which silences warnings about them being imported before stdlib modules.
The custom porting fixers are removed. A comment related to 2to3
has been removed from the import checker.
After this patch, no references to 2to3 remain.
Not having this caused warnings on Windows:
mercurial/pure/osutil.py:12: stdlib import follows local import: os
mercurial/pure/osutil.py:13: stdlib import follows local import: socket
mercurial/pure/osutil.py:14: stdlib import follows local import: stat
mercurial/pure/osutil.py:15: stdlib import follows local import: sys
Before this patch, import-checker.py didn't know if a name in ImportFrom
statement are module or not. Therefore, it complained the following example
did "direct symbol import from mercurial".
# hgext/foo.py
from mercurial import hg
This patch reuses the dict of local modules to filter out sub-module names.
I got the following error by rewriting hgweb/webcommands.py to use
absolute_import. It is false-positive because the import line appears in
"help" function:
hgweb/webcommands.py:1297: higher-level import should come first: mercurial
This patch makes the import checker aware of the function scope and apply
rules recursively.
I got the following error by rewriting hgweb/__init__.py to use
absolute_import, which is obviously wrong:
Import cycle: mercurial.hgweb.__init__ -> mercurial.hgweb.__init__
"from foo import bar" should not make a cycle if "foo" is a package and
if "bar" is a module or a package. On the other hand, it should be detected
as a cycle if "bar" is a non-module name. Both cases are doc-tested already,
so this patch does not add new doctest.
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
$ ls '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6'/BaseHTTPServer.py*
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/BaseHTTPServer.pyc
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/BaseHTTPServer.pyo
On my linux machines multiprocessing appears to defeat the logic in
import-checker to detect stdlib modules. Since we now only use
versions of Python which ship with multiprocessing, let's just
whitelist the module.
If mercurial was installed into a directory other than the site-packages,
test-module-imports.t failed as 'mercurial.node' was listed in stdlib_modules:
testpackage/latesymbolimport.py relative import of stdlib module
Instead, we should exclude our packages explicitly.
We can't assume that the site-packages is the only directory that has Python
files but is not handled as a package. For example, we have dist-packages
directory on Debian.
Before this patch, `import-checker.py` exits with non-0 code, if no
error is detected. This is unusual as Unix command.
This change may be a one of preparations for issue4677, because this
can avoid extra explanation about unusual exit code of
`import-checker.py` for third party tool developers.
We introduce a new convention for declaring imports and enforce it via
the import checker script.
The new convention is only active when absolute imports are used, which is
currently nowhere. Keying off "from __future__ import absolute_import" to
engage the new import convention seems like the easiest solution. It is
also beneficial for Mercurial to use this mode because it means less work
and ambiguity for the importer and potentially better performance due to
fewer stat() system calls because the importer won't look for modules in
relative paths unless explicitly asked.
Once all files are converted to use absolute import, we can refactor
this code to again only have a single import convention and we can
require use of absolute import in the style checker.
The rules for the new convention are documented in the docstring of the
added function. Tests have been added to test-module-imports.t. Some
tests are sensitive to newlines and source column position, which makes
docstring testing difficult and/or impossible.
A future patch will formalize the modern import convention. In
preparation for that, introduce a new wrapper function that will invoke
the proper function.
"from . import X" will produce an ImportFrom ast node with .module =
None. This resulted in a run-time error from attempting to concatenate
None with a str.
Another problem with relative imports is that the prefix may be dynamic
based on the "level" attribute of the import. e.g. "from ." has level 1
and "from .." has level 2.
We teach the "fromlocal" function how to cope with relative imports.
Where appropriate, the consumer passes in the level so relative module
names may be resolved properly.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
The previous patch ensures all module names are recorded in `imports`
as absolute names, so we no longer need to treat modules as ones
imported relatively from the target source if they appear to not be
from the stdlib.
This patch makes `imported_modules()` always yield absolute
`dotted_name_of_path()`-ed name by strict detection with
`fromlocal()`.
This change improves circular detection in some points:
- locally defined modules, of which name collides against one of
standard library, can be examined correctly
For example, circular import related to `commands` is overlooked
before this patch.
- names not useful for circular detection are ignored
Names below are also yielded before this patch:
- module names of standard library (= not locally defined one)
- non-module names (e.g. `node.nullid` of `from node import nullid`)
These redundant names decrease performance of circular detection.
For example, with files at 13dc86d189c9, average loops per file in
`checkmod()` is reduced from 165 to 109.
- `__init__` can be handled correctly in `checkmod()`
For example, current implementation has problems below:
- `from xxx import yyy` doesn't recognize `xxx.__init__` as imported
- `xxx.__init__` imported via `import xxx` is treated as `xxx`,
and circular detection is aborted, because `key` of such
module name is not `xxx` but `xxx.__init__`
- it is easy to enhance for `from . import xxx` style or so (in the
future)
Module name detection in `imported_modules()` can use information
in `ast.ImportFrom` fully.
It is assumed that all locally defined modules are correctly specified
to `import-checker.py` at once.
Strictly speaking, when `from foo.bar.baz import module1` imports
`foo.bar.baz.module1` module, current `imported_modules()` yields only
`foo.bar.baz.__init__`, even though also `foo.__init__` and
`foo.bar.__init__` should be yielded to detect circular import
exactly.
But this limitation is reasonable one for improvement in this patch,
because current `__init__` files in Mercurial seems to be implemented
carefully.
`fromlocalfunc()` uses:
- `modulename` (of the target source) to compose absolute module
name imported relatively from it
It is assumed that `modulename` is an `dotted_name_of_path()`-ed
source file, which may have `.__init__` at the end of it.
This assumption makes composing `prefix` of relative name easy.
- `localmods` to examine whether there is a locally defined (=
Mercurial specific) module matching against the specified name
It is assumed that module names not existing in `localmods` are
ones of Python standard library.
This is a preparation for subsequent patches, which expect that all
locally defined (= mercurial specific) modules are already known
before examinations.
Looping twice for specified modules is a little redundant, but
reasonable cost for improvement in subsequent patches.
Before this patch, "import-check.py" is invoked via "xargs" in
"test-module-imports.t", but it doesn't ensure that
"import-checker.py" is certainly invoked with all mercurial specific
files at once.
"xargs" may invoke specified command multiple times with part of
arguments given from stdin: according to "xargs(1)" man page, this
dividing arguments is system-dependent.
This patch adds "xargs" like mode to "import-checker.py".
This can ensure that "import-checker.py" is certainly invoked with all
mercurial specific files at once in "test-module-imports.t". This is
assumed by subsequent patches.
These are Windows dlls, and eliminate the following import check diffs that are
not on Unix:
mercurial/changegroup.py mixed imports
stdlib: os, struct, tempfile, zlib
relative: bz2
mercurial/encoding.py mixed imports
stdlib: locale, os
relative: unicodedata
