Attempting to build Mercurial from source using MinGW from
msys2 on Windows produces a hg.exe that attempts to load e.g.
python27.dll. MinGW prefixes its library name with "lib" and
adds a period between the major and minor versions. e.g.
"libpython2.7.dll."
Before this patch, hg.exe files in a MinGW environment would
either fail to find a Python DLL or would attempt to load a
non-MinGW DLL, which would summarily explode. Either way,
hg.exe wouldn't work.
This patch improves the code that determines the Python DLL
filename to actually use the loaded Python DLL instead of
inferring it. Basically we take the handle of the loaded DLL
from sys.dllhandle and call a Windows API to try to resolve
that handle to a filename.
Before this patch, fsmonitor was not installed along with other extensions. It
did correctly build the C files needed but forgot to copy over the python
files. This patch fixes it by adding fsmonitor and fsmonitor.pywatchman to the
correct install variable.
Mercurial extensions are not meant to be normal python package/module. Yet the
lack of an official location to install them means that a lot of them actually
install as root level python package, polluting the global Python package
namespace and risking collision with more legit packages. As we recently
discovered, core python actually support namespace package. A way for multiples
distinct "distribution" to share a common top level package without fear of
installation headache. (Namespace package allow submodule installed in different
location (of the 'sys.path') to be imported properly. So we are fine as long as
extension includes a proper 'hgext3rd.__init__.py' to declare the namespace
package.)
Therefore we introduce a 'hgext3rd' namespace packages and search for extension
in it. We'll then recommend third extensions to install themselves in it.
Strictly speaking we could just get third party extensions to install in 'hgext'
as it is also a namespace package. However, this would make the integration of
formerly third party extensions in the main distribution more complicated as the third
party install would overwrite the file from the main install. Moreover, having an
explicit split between third party and core extensions seems like a good idea.
The name 'hgext3rd' have been picked because it is short and seems explicit enough.
Other alternative I could think of where:
- hgextcontrib
- hgextother
- hgextunofficial
In preparation for the filesystem monitor extension, include the pywatchman
library. The fbmonitor extension relies on this library to communicate with
the Watchman service. The library is BSD licensed and is taken from
https://github.com/facebook/watchman/tree/master/python.
This package has not been updated to mercurial code standards.
This is not technically needed, since mercurial.__version__
does not exist as a native module, but, without this style wrappings,
if something else had a native flavor, the module loader would get
upset.
In principle, the `env` object is trying to set HGMODULEPOLICY for
children, so, conceptually we should set it for this in-process
child.
Instead of rewriting __init__ to define the modulepolicy,
write out a __modulepolicy__.py file like __version__.py
This should work for both system-wide installation and in-place build. Therefore
we can avoid relying on two separate modulepolicy rules, '@MODULELOADPOLICY@'
and 'mercurial/modulepolicy'.
Before this patch, "setup.py --pure" fails on Windows, because
hgbuildscripts.run() tries to copy "hg.exe", which doesn't generated
at "setup.py --pure".
At that time, run_command('build_hgexe') invoked in
hgbuildscripts.run() does nothing and returns successfully. Therefore,
subsequent procedure assuming existence of "hg.exe" fails.
This patch avoids procedure related to "hg.exe" (= all of
hgbuildscripts.run() except for build_scripts.run() invocation) at
"setup.py --pure".
The b() helper was needed because Python < 2.6 didn't support bytes
literals (b''). Now that we don't support Python < 2.6, we no longer
need this helper.
This is necessary to produce wheels that install properly. More
details are captured in an in-line comment.
After this patch, produced wheels can be installed via `pip install`
and appear to "just work," including on Windows.
Currently, packaging Mercurial on Windows will produce a
Scripts\hg Python script and a Scripts\hg.bat batch script. The
py2exe distribution contains a hg.exe which loads a Python
interpretter and invokes the "hg" Python script. Running a
exe directly has benefits over batch scripts because batch
scripts do things like muck around with command arguments.
This patch implements a custom "build_scripts" command which
attempts to build hg.exe on Windows. If hg.exe is built, it is
marked as a "script" file and installed into the Scripts\
directory on Windows. Since hg.exe is redundant and better than
hg.bat, if hg.exe is built, hg.bat is not installed.
Since some environments don't support compiling C programs,
we treat hg.exe as optional and catch failures building it. This
is not ideal. However, I reckon most Windows users will not be
installing Mercurial from source: they will get it from the MSI
installer or via `pip install Mercurial`, which will download a
wheel that has hg.exe in it. So, I don't think this is a big deal.
Previously, .py files under mercurial/pure/ were copied to mercurial/*
during installation if we were performing a pure Python installation.
Now that the new import hooks and module load policy are in place, this
hackery from the past is no longer necessary.
With this patch, we stop copying modules from mercurial/pure/* to
mercurial/*. Instead, we preserve the files at their original
hierarchy, mirroring the source repository structure.
In addition, we always install the pure modules. Before, we would only
include the pure modules in the distribution/installation if the
install-time settings requested a pure Python installation. The upside
of this change is that CPython and PyPy can run from the same Mercurial
installation, making packaging and distribution of Mercurial simpler.
The inclusion of pure Python modules in the installation sounds
risky, as it could lead to inadvertent loading of non-C modules.
This shouldn't be a problem. The default module load policy is "C
only" (or at least will be shortly) and the only way to load pure
modules from an installation is if a) pure installation was requested
b) the HGMODULELOADPOLICY overrides the requirement for C modules.
The default module load policy as defined in source is a special string
whose default value from the checkout is equivalent to the "C only"
policy (again, not exactly the state right now). For pure
installations, this default policy is not appropriate and will not
work. This patch adds support for rewriting __init__.py during
installation to reflect the module load policy that should be in
place accoding to the installation settings. For default CPython
installs, the value in the source file will change but there will
be no functional change. For pure installations, the default policy
will be set to "py," allowing them to work without having to set
environment variables.
There are a handful of modules that have both pure Python and C
extension implementations. Currently, setup.py copies files from
mercurial/pure/*.py to mercurial/ during the install process if C
extensions are not available. This way, "import mercurial.X" will
work whether C extensions are available or not.
This approach has a few drawbacks. First, there aren't run-time checks
verifying the C extensions are loaded when they should be. This could
lead to accidental use of the slower pure Python modules. Second, the
C extensions aren't compatible with PyPy and running Mercurial with
PyPy requires installing Mercurial - you can't run ./hg from a source
checkout. This makes developing while running PyPy somewhat difficult.
This patch implements a PEP-302 import hook for finding and loading the
modules with both C and Python implementations. When a module with dual
implementations is requested for import, its import is handled by our
import hook.
The importer has a mechanism that controls what types of modules we
allow to load. We call this loading behavior the "module load policy."
There are 3 settings:
* Only load C extensions
* Only load pure Python
* Try to load C and fall back to Python
An environment variable allows overriding this policy at run time. This
is mainly useful for developers and for performing actions against the
source checkout (such as installing), which require overriding the
default (strict) policy about requiring C extensions.
The default mode for now is to allow both. This isn't proper and is
technically backwards incompatible. However, it is necessary to
implement a sane patch series that doesn't break the world during
future bisections. The behavior will be corrected in future patch.
We choose the main mercurial/__init__.py module for this code out of
necessity: in a future world, if the custom module importer isn't
registered, we'll fail to find/import certain modules when running
from a pure installation. Without the magical import-time side-effects,
*any* importer of mercurial.* modules would be required to call a
function to register our importer. I'm not a fan of import time side
effects and I initially attempted to do this. However, I was foiled by
our own test harness, which has numerous `python` invoked scripts that
"import mercurial" and fail because the importer isn't registered.
Realizing this problem is probably present in random Python scripts
that have been written over the years, I decided that sacrificing
purity for backwards compatibility is necessary. Plus, if you are
programming Python, "import" should probably "just work."
It's worth noting that now that we have a custom module loader, it
would be possible to hook up demand module proxies at this level
instead of replacing __import__. We leave this work for another time,
if it's even desired.
This patch breaks importing in environments where Mercurial modules
are loaded from a zip file (such as py2exe distributions). This will
be addressed in a subsequent patch.
This should allow easier experimentation with using setuptools in mercurial's
build automation, without breaking anything that currently depends on distutils
behavior
This makes the root install folder (on Windows) nice and tidy. The
only files left in the root folder are:
hg.exe
python27.dll
COPYING.rtf
ReadMe.html
the last of which was probably out-of-date 7 years ago
This will raise a syntax error for people who attempt to use Py2.4,
but that's already going to fail and we have no way to keep other
2.6isms from creeping in since we've removed the check-code rules and
the buildbot.
The last blocker for dropping Python 2.4 was Centos 5. We now provide our own
Mercurial package for Centos 5 with a bundled Python2.7.
I'm therefore happy to officially drop compatibility with Python 2.4 (and
Python 2.5 that nobody really cares about). This open the season for code
cleanup.
It is war's prize to take all vantage.
This lets us iterate manifests in order, but do a _lot_ less work in
the common case when we only care about a few manifest entries.
Many thanks to Mike Edgar for reviewing this in advance of it going
out to the list, which caught many things I missed.
This version of the patch includes C89 fixes from Sean Farley and
many correctness/efficiency cleanups from Martin von
Zweigbergk. Thanks to both!
Some Python installations like the ones available from the optware
project
for the Synology DiskStation NASes don't have that package, which means
running the setup script will crash and exit right away. Instead, we now
just use an empty/fake class for the HackedMingw32CCompiler, which we
likely won't use anyway.
For a Mercurial built on the merge from stable into default right after 3.2.2
was released -- fa2ec68252fb -- the version number produced was "3.2.2+4". This
is potentially misleading, since in reality the built Mercurial includes many
more changes compared to 3.2.2.
Change the versioning scheme so that we take into consideration all the changes
present in the current revision that aren't present in the latest tag. For
fa2ec68252fb the new versioning scheme results in a version number of
"3.2.2+256". This gives users a much better idea of how many changes have
actually happened since the latest release.
Since changessincelatesttag is always greater than or equal to the
latesttagdistance, this will produce version numbers that are always greater
than or equal to the old scheme. Thus there's minimal compatibility risk.
changessincelatesttag gives one a better idea of how much the code has changed
since. Since changessincelatesttag is always greater than or equal to the
latesttagdistance (see previous patch for why), this will always produce
version numbers greater than or equal to the previous scheme.
This helps providing a more consistent user experience on all platforms and
with all packaging.
The exact location of default.d depends on how Mercurial is installed and
whether it is 'frozen'. The exact location should never be relevant to users
and is intentionally not explained in details in the documentation. It will
however always be next to the help and templates files.
Note that setting HGRCPATH also disables these defaults. I don't know if that
should be considered a bug or a feature.
This will give PKI-secure behaviour out of the box, without any configuration.
Setting web.cacerts to any value or empty will disable this trick.
This dummy cert trick only works on OS X 10.6+, but 10.5 had Python 2.5 which
didn't have certificate validation at all.
OS X would offer to expand the zip so the (multi file) installer inside it
could be run ... but that would leave the expanded zip folder around.
Instead, use a .dmg file that automatically will be mounted - that seems more
common on OS X.
Still, there is two levels of levels of clicking before actually launching the
installer. Having a single file installer would be better ... but seems to be
hard. A more feasible improvement would be some fancy layout inside the .dmg .
"make clean" already removed __index__.py[cdo], but not the __index__.py
(automatically generated by "python setup.py build_hgextindex").
"setup.py build_hgextindex" did not generate a new index if file
__index__.py[cdo] already existed, because if __index__.py was removed,
the compiled file containing the old information was imported and used.
Generate an empty file (with a new timestamp to generate a new .py[cdo])
instead and make mercurial.extensions ignore the unset docs attribute.
One of the problems was a failed test-help.t, to reproduce:
$ rm hgext/__index__.py*
$ echo 'docs = {"mq": "dummy"}' > hgext/__index__.py
$ make test-help.t
With this a "make clean" or "python setup.py build_hgextindex" helps.
This extension has always had correctness issues and has been
unmaintained for years. It is now removed in favor of the third-party
hgwatchman which is maintained and appears to be correct.
Users with inotify enabled in their config files will fall back to
standard status performance.
This is over twice as fast as the Python dirs code. Upcoming changes
will nearly double its speed again.
perfdirs results for a working dir with 170,000 files:
Python 638 msec
C 244
We give up using CPython's PythonXX.lib import libraries (and Python.h), and
now "manually" call the LoadLibrary() / GetProcAddress() Windows API's instead.
If there is a "hg-python" subdirectory (the canonical directory name for
HackableMercurial's private Python copy) next to the hg.exe, we load the
pythonXX.dll from there (feeding an absolute path to LoadLibrary) and we set
Py_SetPythonHome() to that directory, so that the Python libraries are used
from there as well.
If there is no "hg-python" subdir found next to the hg.exe, we do not feed an
absolute path to LoadLibrary. This continues to allow to find a globally
installed Python DLL, as before this change - that is, without having to edit,
delete, rename, or configure anything.
Note that the hg.exe built is still bound to a *specific* major version of the
pythonXX.dll (e.g. python27.dll). What version it is, is inferred from the
version of the python interpreter that was used when calling setup.py. For
example
C:\python27_x86\python.exe setup.py build_hgexe -i --compiler=mingw32
builds a hg.exe (using the mingw32 tool chain) bound to (x86) Python 2.7. And
C:\python27_x86\python.exe setup.py build_hgexe -i
builds the same using the Microsoft C compiler/linker. (Note that the Microsoft
toolchain combined with x64 CPython can be used to build an x64 hg.exe.)
setup.py is changed to write the name of the pythonlib into the generated header
file "mercurial/hgpythonlib.h", which is #included by exewrapper.c. For a Python
2.7 build, it for example contains:
#define HGPYTHONLIB "python27"
exewrapper.c then uses HGPYTHONLIB for the name of the Python dll to load.
We don't want to track mercurial/hgpythonlib.h, so we add it to .hgignore.
The old calculation code failed to properly identify revs that
weren't tagged, leaving us with a version of "unknown" most of the
time during development.
Not yet used (will be enabled in a later patch).
This patch is a stripped down version of patches originally created by
Bryan O'Sullivan <bryano@fb.com>
This gcc option has been deprecated since at least 2009 (gcc 4.4),
and it causes compilations to fail entirely with gcc 4.6.x.
Upstream distutils bug: http://bugs.python.org/issue12641
This patch contains support for Plan 9 from Bell Labs. A README is
provided in contrib/plan9 which describes the port in greater detail.
A new extension is also provided named factotum which permits the
factotum(4) authentication agent to provide credentials for HTTP
repositories. This extension is also applicable to other POSIX
platforms which make use of Plan 9 from User Space (aka plan9ports).
Apparently, it prints nothing at all if the user installed only the
command-line tools. In that case, don't try to parse the empty output
-- just assume they have Xcode >= 4.
Merge the code from contrib/setup3.py in setup.
The argument for executing is marked as experimental.
Reason: The file in contrib was outdated (packages, cmdclass, ...)
If Python interpreter was built under Linux 3.x kernel, it reports
sys.platform to be 'linux3' (it is fixed for Python 3, but not for 2.x).
This cancels building inotify extension, which was built only for 'linux2'
platform. Improved test checks if sys.platform begins with 'linux', and together
with test for kernel version to be greater than 2.6 it seems to cover all known
cases.
It generates prebuilt index of all extensions, which will be used by
frozen exe when running 'hg help extensions'.
Now py2exe invokes this command automatically.
Fink's python is either i386 or amd64, but not universal. Setting ARCHFLAGS to the
empty string produces a successful build against both OS X python and fink python.
The modules will no longer be universal -- if that is an issue, we can change the
test to extract ARCHFLAGS from distutils.sysconfig and remove ppc if necessary.
This is revision a4229f13c374 of
http://py-nonblocking-http.googlecode.com/ with a no-check-code
comment added to the end of each file using `for fi in $(hg manifest |
grep mercurial/httpclient/) ; echo '# no-check-code' >> $fi`.
The previous test assumed that 'os.name' was "mac" on Mac OS X. This
is not the case; 'mac' was classic Mac OS, whereas Mac OS X has 'os.name'
be 'posix'.
Please note that this change will break Mercurial on hypothetical
non-Mac OS X deployments of Darwin.
Credit to Brodie Rao for thinking of CGSessionCopyCurrentDictionary()
and Kevin Bullock for testing.
Sometimes xcodebuild prints warnings to stderr, but runcmd() assumes anything
printed to stderr implies failure. Since runcmd() was originally only
intended to run hg, this was fine until it was pressed into service for
running xcodebuild. Thus: split runcmd() into two parts: runcmd(), which does
the minimal amount of work to run a subprocess, and runhg(), which calls
runcmd().
Add missing calls to close() to many places where files are
opened. Relying on reference counting to catch them soon-ish is not
portable and fails in environments with a proper GC, such as PyPy.
This provides two new features:
- Mercurial may be installed into a non-standard location without
having to set PYTHONPATH.
- Multiple installations can use Mercurial from different locations.
This patch adds access to util.h for the inotify C module by adding the
"mercurial" directory as an include dir, enabling access to the macros defined
in util.h.
In py3k, subprocess.Popen.communicate's output are bytes objects. String
literals are Unicode objects. Thus, when a bytes object startswith method is
called, with string literals, it fails. What this patch does is:
* Convert the string (unicode in py3k) literals to bytes objects;
* As "bytes" is not a builtin in python < 2.6, it defines a "b" helper
function that merely returns its argument, as suggested by Antoine Pitrou.
Windows python 2.4 os.stat() reports times including DST offset, while osutil.c
reports the correct value, which makes status() systematically compare files
content. This bug is fixed in python 2.5. Using osutil.py instead of osutil.c
is 4x times slower on large repositories but current code is completely
unusable. Given few people are likely to use python 2.4 on Windows this
solution was considered a good trade-off compared to more invasive solutions
trying to address the offset issue.
Fixes
warning: py2exe: Version Info will not be included:
could not parse version number ...
which was seen when doing nightly builds. hg.exe files
of nightly builds did not have any version info resoure,
which may cause problems with installers.
Also setting a copyright string for the version resource
(was missing).
Though we only support Python 2.4 or greater, we should keep setup.py
compatible with earlier versions on the syntactic level. Otherwise
people will simply get a SyntaxError when trying to install Mercurial
with an old version of Python. With this change, the setup.py file can
be imported with Python 2.3 and we then issue a friendly error message
when we detect that Python is too old.
Python will issue an ImportWarning when seeing 'import locale' if
there is a locale/ directory present without a __init__.py file.
The warning is silent by default, but it somehow shows up anyway on
Windows when setup.py executed hg. The warning causes runcmd to panic
since it sees output on stderr.
This patch ignores warnings on stderr about not importing a package.
Here is an array summarizing the mercurial version string:
[A] [B] [C] [D]
[1] clone tag clean => tag
[2] clone hash clean => latesttag+latesttagdistance-hash
[3] clone tag dirty => tag+date
[4] clone hash dirty => latesttag+latesttagdistance-hash+date
[5] archive tag clean => tag
[6] archive hash clean => latesttag+latesttagdistance-hash
Column [A]: Mercurial built from an hg *archive* or hg *clone* working directory
Column [B]: revision built has a *tag* or else default to the SHA1 *hash*
Column [C]: working tree *clean* or *dirty*
Column [D]: Mercurial version string
Over the previous version:
- row [5] did return just the node hash, now it returns the tag
- prepend the latest tag and the distance to it to rows [2][4][6]
- append also the date to row [3]; previously, it was just the tag
- the version string is with an empty string to avoid possible TypeError
exceptions during string manipulations
- factorize the function to run hg commands; remove the error message as it is
no more specific to the function.
This scheme enables to have first part of the version strings that can be
compared, whether it has been built from a tagged or untagged revision.
The second part of the version adds a hash for untagged revisions and today's
date if the working tree has local modifications.
As the version string does not contain spaces or special characters, it should
not break script parsing the 'hg version' command and should be usable for use
in file names.
The new code also ensure that the version string has exactly the same version
string, whether it has been built from an archive or from a clone.
The selection is somewhat arbitrary. In the case of the Zsh completion
file, it will not conflict with the builtin Zsh completions: they
are in a file named `_mercurial', not `_hg'.
Remove the `install_package_data' subclass of `install_data' and use
the `package_data' functionality provided by distutils instead. As
package data must be located within the package directory, the data
files are now generated in the build directory.
To simplify the functionality of this change, the top-level `doc' and
`templates' directories have been moved into the `mercurial' package
directory.
Previously, setup.py was enhanced to identify the Mercurial version
from either .hg/ or mercurial/__version__.py. When archives are
created using 'hg archive' or via hgweb, neither of those options are
available. However, there is a .hg_archival.txt file in the root of
the archive that has the information. This patch enhances setup.py to
identify the Mercurial version from the .hg_archival.txt file when
there is no .hg/ or mercurial/__version__.py available.
There is a problem with setup.py where it will not identify the Mercurial
version properly when not being ran in within a repository even if
mercurial/__version__.py exists.
To fix, use mercurial.__version__.version when available before defaulting
to "unknown". (Using mercurial.util.version() is not an option due to a
dependency issue where osutil can be referenced before it is built.)
- don't do version detection if there's no .hg directory
- shrink try: clause
- don't write __version__.py if version is unknown
(we might overwrite the real version)
This flag will make the build_py step install the pure Python modules
in mercurial/pure/ into mercurial/ and furthermore prevent building
the C extensions.
The target update-pot extracts strings using pygettext and updates the
i18n/hg.pot file. The translators can then use msgmerge to merge the
new strings in hg.pot with their xx.po file when they want to.
The setup.py file now includes files under both templates/ and i18n/
as data files.
- simplify version detection code
- move detection code into setup.py
- move version reading function into util.py
- drop version.py code
This makes hg more closely follow its own recommendation of how to deal with
versioning your builds: use hg id in your build script.
Use information provided by FindFile... Win32 calls
to generate stat information without lstat call per file.
rwx bits in st_mode are ignored as they are not stored in Win32 fs
and Mercurial does not use them
Unicode path / path names over _MAX_PATH are intentionally not supported.
With Python 2.5, the email package is not fully loaded by py2exe, due to
dynamic imports which are not found by modulefinder. This breaks the patchbomb
extension. This patch forces the whole email package to be included so that
the dynamic imports work as expected.
Debian Etch doesn't include a sys/inotify.h header, which makes it
impossible to compile _inotify.c, making hg uninstallable.
The cc.has_function() method is implemented by trying to compile a
simple C program. Since there's no redirection involved all error
messages are sent to the terminal. This is not particularly pretty
but at least it allows the installation to complete.
Further the installation of packagescan over demandload is moved to the
packagescan module.
I added as well few more comments in the packagescan module to avoid
the wrong use of package scan in the future.
Reason:
mercurial.packagescan acts as fake mercurial.demandload during a py2exe
run. Unfortunatly the import of mercurial.version in setup.py is done
before mercurial.packagescan is installed. This results in few imports
without mercurial.packagescan in charge and therefore not all dependend
modules are detected when running mercurial.packagescan.getmodules
later e.g. winerror is missed.