As the fromlist gives the names of sub-modules, they should be searched in
the parent directory of the package's __init__.py, which is level=1.
I got the following error by rewriting hgweb to use absolute_import, where
the "mercurial" package is referenced as ".." (level=2):
ValueError: Attempted relative import beyond toplevel package
I know little about the import mechanism, but this change seems correct.
Before this patch, the following code did import the os module with no error:
from mercurial import demandimport
demandimport.enable()
from mercurial import os
print os.name
_demandmod instances may be referenced by multiple importing modules.
Before this patch, the _demandmod instance only maintained a reference
to its first consumer when using the "from X import Y" syntax. This is
because we only created a single _demandmod instance (attached to the
parent X module). If multiple modules A and B performed
"from X import Y", we'd produce a single _demandmod instance
"demandmod" with the following references:
X.Y = <demandmod>
A.Y = <demandmod>
B.Y = <demandmod>
The locals from the first consumer (A) would be stored in <demandmod1>.
When <demandmod1> was loaded, we'd look at the locals for the first
consumer and replace the symbol, if necessary. This resulted in state:
X.Y = <module>
A.Y = <module>
B.Y = <demandmod>
B's reference to Y wasn't updated and was still using the proxy object
because we just didn't record that B had a reference to <demandmod> that
needed updating!
With this patch, we add support for tracking which modules in addition
to the initial importer have a reference to the _demandmod instance and
we replace those references at module load time.
In the case of posix.py, this fixes an issue where the "encoding" module
was being proxied, resulting in hundreds of thousands of
__getattribute__ lookups on the _demandmod instance during dirstate
operations on mozilla-central, speeding up execution by many
milliseconds. There are likely several other operation that benefit from
this change as well.
The new mechanism isn't perfect: references in locals (not globals) may
likely linger. So, if there is an import inside a function and a symbol
from that module is used in a hot loop, we could have unwanted overhead
from proxying through _demandmod. Non-global imports are discouraged
anyway. So hopefully this isn't a big deal in practice. We could
potentially deploy a code checker that bans use of attribute lookups of
function-level-imported modules inside loops.
This deficiency in theory could be avoided by storing the set of globals
and locals dicts to update in the _demandmod instance. However, I tried
this and it didn't work. One reason is that some globals are _demandmod
instances. We could work around this, but it's a bit more work. There
also might be other module import foo at play. The solution as
implemented is better than what we had and IMO is good enough for the
time being.
It's worth noting that this sub-optimal behavior was made worse by the
introduction of absolute_import and its recommended "from . import X"
syntax for importing modules from the "mercurial" package. If we ever
wrote performance tests, measuring the amount of module imports and
__getattribute__ proxy calls through _demandmod instances would be
something I'd have it check.
Before, we didn't support lazy loading if absolute_import was in
effect and a fromlist was used. This meant that "from . import X"
wasn't lazy and performance could suffer as a result.
With this patch, we now support lazy loading for this scenario.
As part of developing this, I discovered issues when module names
are defined. Since the enforced import style only allows
"from X import Y" or "from .X import Y" in very few scenarios
when absolute_import is enabled - scenarios where Y is not a
module and thus there is nothing to lazy load - I decided to drop
support for this case instead of chasing down the errors. I don't
think much harm will come from this. But I'd like to take another
look once all modules are using absolute_import and I can see the
full extent of what is using names in absolute_import mode.
demandimport doesn't currently support absolute imports (level >= 0).
In preparation for this, we add some documentation and a code branch
to handle the absolute_import case.
The removed code was to support an __import__ function that doesn't
support the "level" argument. This argument was added in Python 2.5,
which we no longer support.
This module depends on _winreg, which is windows-only. Recent versions
of setuptools load distutils.msvc9compiler and expect it to
ImportError immediately when on non-Windows platforms, so we need to
let them do that. This breaks in an especially mystifying way, because
setuptools uses vars() on the imported module. We then throw an
exception, which vars doesn't pick up on well. For example:
In [3]: class wat(object):
...: @property
...: def __dict__(self):
...: assert False
...:
In [4]: vars(wat())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-4-2781ada5ffe6> in <module>()
----> 1 vars(wat())
TypeError: vars() argument must have __dict__ attribute
Which is similar to the problem we run into.
demandimport was failing in Python 3 with a ValueError because
__import__'s level=-1 has gone away (-1 means to try both relative
and absolute imports and relative imports don't exist in Python 3).
With this patch, demandimport still doesn't work in Python 3 (it
fails when importing a non-package module).
This fixes an issue introduced in 818c8992811a where, when disabling
demandimport while running hooks, it's inadvertently re-enabled even when
it was never enabled in the first place.
This doesn't affect normal command line usage of Mercurial; it only matters
when Mercurial is run with demandimport intentionally disabled.
Before this patch, python modules of each extensions can't import
another one in own extension by absolute name, because root modules of
each extensions are loaded with "hgext_" prefix.
For example, "import extroot.bar" in "extroot/foo.py" of "extroot"
extension fails, even though "import bar" in it succeeds.
Installing extensions into site-packages of python library path can
avoid this problem, but this solution is not reasonable in some cases:
using binary package of Mercurial on Windows, for example.
This patch retries to import with "hgext_" prefix after ImportError,
if the module in the extension may try to import another one in own
extension.
This patch doesn't change some "_import()"/"_origimport()" invocations
below, because ordinary extensions shouldn't cause such invocations.
- invocation of "_import()" when root module imports sub-module by
absolute path without "fromlist"
for example, "import a.b" in "a.__init__.py".
extensions are loaded with "hgext_" prefix, and this causes
execution of another (= fixed by this patch) code path.
- invocation of "_origimport()" when "level != -1" with "fromlist"
for example, importing after "from __future__ import
absolute_import" (level == 0), or "from . import b" or "from .a
import b" (0 < level),
for portability between python versions and environments,
extensions shouldn't cause "level != -1".
Before this patch, demandimport of Mercurial may fail to load external
libraries using "from __future__ import absolute_import": for example,
importing "foo" in "bar.baz" module will load "bar.foo" if it exists,
even though "absolute_import" is enabled in "bar.baz" module.
So, extensions for Mercurial can't use such external libraries.
This patch saves "level" of import request for on-demand module
loading in the future: default value of level is -1, and level is 0
when "absolute_import" is enabled.
"level" value is passed to built-in import function in
"_demandmod._load()" and it should load target module correctly.
This patch changes only one "_demandmod" construction case other than
cases below:
- construction in "_demandmod._load()"
this code path should be used only in relative sub-module
loading case
- constructions other than patched one in"_demandimport()"
these code paths shouldn't be used in "level != -1" case
We don't use util.safehasattr() here to avoid adding new dependencies
for demandimport. This change may expose previously-silenced
deprecation warnings to appear, as hasattr silently hides warnings
that occur during module import when using demandimport.
The Python default for this function is -1, indicating both relative
and absolute imports should be used.[1] Previously, we relied on the
Python VM not passing level when such semantics were
requisted. This is not the case for PyPy, however, where a level of -1
is always passed to __import__.
[1] <http://docs.python.org/library/functions.html#__import__>
Python's __import__() function has 'level' as the fourth argument, not the
third. The code path in question probably never worked.
(This was seen trying to run Mercurial in PyPy. Fixing this made it
die somewhere else...)
The 'level' argument to __import__ was added in Python 2.6, and is
specified for either relative or absolute imports. The fix introduced
in 5b0fda8ff209 allowed such imports to proceed without failure, but
effectively disabled demandimport for them. This is particularly
unfortunate in Python 3.x, where *all* imports are either relative or
absolute.
The solution introduced here is to store the level argument on the
demandimport instance, and propagate it to _origimport() when its
value isn't None.
Please note that this patch hasn't been tested in Python 3.x, and thus
may not be complete. I'm worried about how sub-imports are handled; I
don't know what they are, or whether the level argument should be
modified for them. I've added 'TODO' notes to these cases; hopefully,
someone more knowledgable of these issues will deal with them.
Demandimport breaks gtk. You get a meaningless error about
'failed loading gobject\_gobject.pyd'. Mercurial does not use gtk,
but this trips up many extension writers.
I wanted to check if mercurial.demandimport could speed up the loading of
PyObjC, and ran into this: the level argument for __import__, available in
Python 2.5 and later, is silently dropped when doing an 'import *'. I have no
idea what these arguments mean, but this minor change made it work.
(Oh, and because of that 'from ... import *', PyObjC still took about 2s...)
Mercurial does not work on python2.6 because __import__ takes an
additional argument called level. This patch merely calls the
built-in __import__ when level is passed.
