This patch also adds new check-code.py pattern to detect invalid usage
of "mercurial@selenic.com".
Change for test-convert-tla.t is tested, but similar change for almost
same test-convert-baz.t isn't yet tested actually, because I couldn't
find out the way to get "GNU Arch baz client".
AFAIK, buildbot skips test-convert-baz.t, too. Does anybody have
appropriate environment for testing?
The next release from upstream adds another named argument to this
function. Specify arguments by name so there is no ambiguity about
which argument is being passed.
Downstream packagers will inevitably want to disable building the
vendored python-zstandard Python package. Rather than force them
to patch setup.py, let's give them a knob to use.
distutils Command classes support defining custom options. It requires
setting certain class attributes (yes, class attributes: instance
attributes don't work because the class type is consulted before it
is instantiated).
We already have a custom child class of build_ext, so we set these
class attributes, implement some scaffolding, and override
build_extensions to filter the Extension instance for the zstd
extension if the `--no-zstd` argument is specified.
Example usage:
$ python setup.py build_ext --no-zstd
Now that zstd and python-zstandard are vendored, we can start compiling
them as part of the install.
python-zstandard provides a self-contained Python function that returns
a distutils.extension.Extension, so it is really easy to add zstd
to our setup.py without having to worry about defining source files,
include paths, etc. The function even allows specifying the module
name the extension should be compiled as. This conveniently allows us
to compile the module into the "mercurial" package so "our" version
won't collide with a version installed under the canonical "zstd"
module name.
We are going to use setproctitle (provided by FreeBSD) if it's available in
the next patch. Therefore provide a macro to give some clues to the C
pre-processor so it could choose code path wisely.
This patch moves all setup*cffi stuff to mercurial/cffi to make the root
directory cleaner. The idea was from mpm [1]:
> It seems like we could have a fair amount of cffi definitions, and
> cluttering the root directory (or mercurial/) with them is probably not
> a great long-term solution. We could probably add a cffi/ directory
> under mercurial/ to parallel pure/.
[1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-July/086442.html
Attempting to build Mercurial from source using MinGW from
msys2 on Windows produces a hg.exe that attempts to load e.g.
python27.dll. MinGW prefixes its library name with "lib" and
adds a period between the major and minor versions. e.g.
"libpython2.7.dll."
Before this patch, hg.exe files in a MinGW environment would
either fail to find a Python DLL or would attempt to load a
non-MinGW DLL, which would summarily explode. Either way,
hg.exe wouldn't work.
This patch improves the code that determines the Python DLL
filename to actually use the loaded Python DLL instead of
inferring it. Basically we take the handle of the loaded DLL
from sys.dllhandle and call a Windows API to try to resolve
that handle to a filename.
Before this patch, fsmonitor was not installed along with other extensions. It
did correctly build the C files needed but forgot to copy over the python
files. This patch fixes it by adding fsmonitor and fsmonitor.pywatchman to the
correct install variable.
Mercurial extensions are not meant to be normal python package/module. Yet the
lack of an official location to install them means that a lot of them actually
install as root level python package, polluting the global Python package
namespace and risking collision with more legit packages. As we recently
discovered, core python actually support namespace package. A way for multiples
distinct "distribution" to share a common top level package without fear of
installation headache. (Namespace package allow submodule installed in different
location (of the 'sys.path') to be imported properly. So we are fine as long as
extension includes a proper 'hgext3rd.__init__.py' to declare the namespace
package.)
Therefore we introduce a 'hgext3rd' namespace packages and search for extension
in it. We'll then recommend third extensions to install themselves in it.
Strictly speaking we could just get third party extensions to install in 'hgext'
as it is also a namespace package. However, this would make the integration of
formerly third party extensions in the main distribution more complicated as the third
party install would overwrite the file from the main install. Moreover, having an
explicit split between third party and core extensions seems like a good idea.
The name 'hgext3rd' have been picked because it is short and seems explicit enough.
Other alternative I could think of where:
- hgextcontrib
- hgextother
- hgextunofficial
In preparation for the filesystem monitor extension, include the pywatchman
library. The fbmonitor extension relies on this library to communicate with
the Watchman service. The library is BSD licensed and is taken from
https://github.com/facebook/watchman/tree/master/python.
This package has not been updated to mercurial code standards.
This is not technically needed, since mercurial.__version__
does not exist as a native module, but, without this style wrappings,
if something else had a native flavor, the module loader would get
upset.
In principle, the `env` object is trying to set HGMODULEPOLICY for
children, so, conceptually we should set it for this in-process
child.
Instead of rewriting __init__ to define the modulepolicy,
write out a __modulepolicy__.py file like __version__.py
This should work for both system-wide installation and in-place build. Therefore
we can avoid relying on two separate modulepolicy rules, '@MODULELOADPOLICY@'
and 'mercurial/modulepolicy'.
Before this patch, "setup.py --pure" fails on Windows, because
hgbuildscripts.run() tries to copy "hg.exe", which doesn't generated
at "setup.py --pure".
At that time, run_command('build_hgexe') invoked in
hgbuildscripts.run() does nothing and returns successfully. Therefore,
subsequent procedure assuming existence of "hg.exe" fails.
This patch avoids procedure related to "hg.exe" (= all of
hgbuildscripts.run() except for build_scripts.run() invocation) at
"setup.py --pure".
The b() helper was needed because Python < 2.6 didn't support bytes
literals (b''). Now that we don't support Python < 2.6, we no longer
need this helper.
This is necessary to produce wheels that install properly. More
details are captured in an in-line comment.
After this patch, produced wheels can be installed via `pip install`
and appear to "just work," including on Windows.
Currently, packaging Mercurial on Windows will produce a
Scripts\hg Python script and a Scripts\hg.bat batch script. The
py2exe distribution contains a hg.exe which loads a Python
interpretter and invokes the "hg" Python script. Running a
exe directly has benefits over batch scripts because batch
scripts do things like muck around with command arguments.
This patch implements a custom "build_scripts" command which
attempts to build hg.exe on Windows. If hg.exe is built, it is
marked as a "script" file and installed into the Scripts\
directory on Windows. Since hg.exe is redundant and better than
hg.bat, if hg.exe is built, hg.bat is not installed.
Since some environments don't support compiling C programs,
we treat hg.exe as optional and catch failures building it. This
is not ideal. However, I reckon most Windows users will not be
installing Mercurial from source: they will get it from the MSI
installer or via `pip install Mercurial`, which will download a
wheel that has hg.exe in it. So, I don't think this is a big deal.
Previously, .py files under mercurial/pure/ were copied to mercurial/*
during installation if we were performing a pure Python installation.
Now that the new import hooks and module load policy are in place, this
hackery from the past is no longer necessary.
With this patch, we stop copying modules from mercurial/pure/* to
mercurial/*. Instead, we preserve the files at their original
hierarchy, mirroring the source repository structure.
In addition, we always install the pure modules. Before, we would only
include the pure modules in the distribution/installation if the
install-time settings requested a pure Python installation. The upside
of this change is that CPython and PyPy can run from the same Mercurial
installation, making packaging and distribution of Mercurial simpler.
The inclusion of pure Python modules in the installation sounds
risky, as it could lead to inadvertent loading of non-C modules.
This shouldn't be a problem. The default module load policy is "C
only" (or at least will be shortly) and the only way to load pure
modules from an installation is if a) pure installation was requested
b) the HGMODULELOADPOLICY overrides the requirement for C modules.
The default module load policy as defined in source is a special string
whose default value from the checkout is equivalent to the "C only"
policy (again, not exactly the state right now). For pure
installations, this default policy is not appropriate and will not
work. This patch adds support for rewriting __init__.py during
installation to reflect the module load policy that should be in
place accoding to the installation settings. For default CPython
installs, the value in the source file will change but there will
be no functional change. For pure installations, the default policy
will be set to "py," allowing them to work without having to set
environment variables.
There are a handful of modules that have both pure Python and C
extension implementations. Currently, setup.py copies files from
mercurial/pure/*.py to mercurial/ during the install process if C
extensions are not available. This way, "import mercurial.X" will
work whether C extensions are available or not.
This approach has a few drawbacks. First, there aren't run-time checks
verifying the C extensions are loaded when they should be. This could
lead to accidental use of the slower pure Python modules. Second, the
C extensions aren't compatible with PyPy and running Mercurial with
PyPy requires installing Mercurial - you can't run ./hg from a source
checkout. This makes developing while running PyPy somewhat difficult.
This patch implements a PEP-302 import hook for finding and loading the
modules with both C and Python implementations. When a module with dual
implementations is requested for import, its import is handled by our
import hook.
The importer has a mechanism that controls what types of modules we
allow to load. We call this loading behavior the "module load policy."
There are 3 settings:
* Only load C extensions
* Only load pure Python
* Try to load C and fall back to Python
An environment variable allows overriding this policy at run time. This
is mainly useful for developers and for performing actions against the
source checkout (such as installing), which require overriding the
default (strict) policy about requiring C extensions.
The default mode for now is to allow both. This isn't proper and is
technically backwards incompatible. However, it is necessary to
implement a sane patch series that doesn't break the world during
future bisections. The behavior will be corrected in future patch.
We choose the main mercurial/__init__.py module for this code out of
necessity: in a future world, if the custom module importer isn't
registered, we'll fail to find/import certain modules when running
from a pure installation. Without the magical import-time side-effects,
*any* importer of mercurial.* modules would be required to call a
function to register our importer. I'm not a fan of import time side
effects and I initially attempted to do this. However, I was foiled by
our own test harness, which has numerous `python` invoked scripts that
"import mercurial" and fail because the importer isn't registered.
Realizing this problem is probably present in random Python scripts
that have been written over the years, I decided that sacrificing
purity for backwards compatibility is necessary. Plus, if you are
programming Python, "import" should probably "just work."
It's worth noting that now that we have a custom module loader, it
would be possible to hook up demand module proxies at this level
instead of replacing __import__. We leave this work for another time,
if it's even desired.
This patch breaks importing in environments where Mercurial modules
are loaded from a zip file (such as py2exe distributions). This will
be addressed in a subsequent patch.
This should allow easier experimentation with using setuptools in mercurial's
build automation, without breaking anything that currently depends on distutils
behavior
This makes the root install folder (on Windows) nice and tidy. The
only files left in the root folder are:
hg.exe
python27.dll
COPYING.rtf
ReadMe.html
the last of which was probably out-of-date 7 years ago
This will raise a syntax error for people who attempt to use Py2.4,
but that's already going to fail and we have no way to keep other
2.6isms from creeping in since we've removed the check-code rules and
the buildbot.
The last blocker for dropping Python 2.4 was Centos 5. We now provide our own
Mercurial package for Centos 5 with a bundled Python2.7.
I'm therefore happy to officially drop compatibility with Python 2.4 (and
Python 2.5 that nobody really cares about). This open the season for code
cleanup.
It is war's prize to take all vantage.
This lets us iterate manifests in order, but do a _lot_ less work in
the common case when we only care about a few manifest entries.
Many thanks to Mike Edgar for reviewing this in advance of it going
out to the list, which caught many things I missed.
This version of the patch includes C89 fixes from Sean Farley and
many correctness/efficiency cleanups from Martin von
Zweigbergk. Thanks to both!
Some Python installations like the ones available from the optware
project
for the Synology DiskStation NASes don't have that package, which means
running the setup script will crash and exit right away. Instead, we now
just use an empty/fake class for the HackedMingw32CCompiler, which we
likely won't use anyway.
For a Mercurial built on the merge from stable into default right after 3.2.2
was released -- fa2ec68252fb -- the version number produced was "3.2.2+4". This
is potentially misleading, since in reality the built Mercurial includes many
more changes compared to 3.2.2.
Change the versioning scheme so that we take into consideration all the changes
present in the current revision that aren't present in the latest tag. For
fa2ec68252fb the new versioning scheme results in a version number of
"3.2.2+256". This gives users a much better idea of how many changes have
actually happened since the latest release.
Since changessincelatesttag is always greater than or equal to the
latesttagdistance, this will produce version numbers that are always greater
than or equal to the old scheme. Thus there's minimal compatibility risk.
changessincelatesttag gives one a better idea of how much the code has changed
since. Since changessincelatesttag is always greater than or equal to the
latesttagdistance (see previous patch for why), this will always produce
version numbers greater than or equal to the previous scheme.
This helps providing a more consistent user experience on all platforms and
with all packaging.
The exact location of default.d depends on how Mercurial is installed and
whether it is 'frozen'. The exact location should never be relevant to users
and is intentionally not explained in details in the documentation. It will
however always be next to the help and templates files.
Note that setting HGRCPATH also disables these defaults. I don't know if that
should be considered a bug or a feature.
This will give PKI-secure behaviour out of the box, without any configuration.
Setting web.cacerts to any value or empty will disable this trick.
This dummy cert trick only works on OS X 10.6+, but 10.5 had Python 2.5 which
didn't have certificate validation at all.
OS X would offer to expand the zip so the (multi file) installer inside it
could be run ... but that would leave the expanded zip folder around.
Instead, use a .dmg file that automatically will be mounted - that seems more
common on OS X.
Still, there is two levels of levels of clicking before actually launching the
installer. Having a single file installer would be better ... but seems to be
hard. A more feasible improvement would be some fancy layout inside the .dmg .
"make clean" already removed __index__.py[cdo], but not the __index__.py
(automatically generated by "python setup.py build_hgextindex").
"setup.py build_hgextindex" did not generate a new index if file
__index__.py[cdo] already existed, because if __index__.py was removed,
the compiled file containing the old information was imported and used.
Generate an empty file (with a new timestamp to generate a new .py[cdo])
instead and make mercurial.extensions ignore the unset docs attribute.
One of the problems was a failed test-help.t, to reproduce:
$ rm hgext/__index__.py*
$ echo 'docs = {"mq": "dummy"}' > hgext/__index__.py
$ make test-help.t
With this a "make clean" or "python setup.py build_hgextindex" helps.
This extension has always had correctness issues and has been
unmaintained for years. It is now removed in favor of the third-party
hgwatchman which is maintained and appears to be correct.
Users with inotify enabled in their config files will fall back to
standard status performance.
This is over twice as fast as the Python dirs code. Upcoming changes
will nearly double its speed again.
perfdirs results for a working dir with 170,000 files:
Python 638 msec
C 244
We give up using CPython's PythonXX.lib import libraries (and Python.h), and
now "manually" call the LoadLibrary() / GetProcAddress() Windows API's instead.
If there is a "hg-python" subdirectory (the canonical directory name for
HackableMercurial's private Python copy) next to the hg.exe, we load the
pythonXX.dll from there (feeding an absolute path to LoadLibrary) and we set
Py_SetPythonHome() to that directory, so that the Python libraries are used
from there as well.
If there is no "hg-python" subdir found next to the hg.exe, we do not feed an
absolute path to LoadLibrary. This continues to allow to find a globally
installed Python DLL, as before this change - that is, without having to edit,
delete, rename, or configure anything.
Note that the hg.exe built is still bound to a *specific* major version of the
pythonXX.dll (e.g. python27.dll). What version it is, is inferred from the
version of the python interpreter that was used when calling setup.py. For
example
C:\python27_x86\python.exe setup.py build_hgexe -i --compiler=mingw32
builds a hg.exe (using the mingw32 tool chain) bound to (x86) Python 2.7. And
C:\python27_x86\python.exe setup.py build_hgexe -i
builds the same using the Microsoft C compiler/linker. (Note that the Microsoft
toolchain combined with x64 CPython can be used to build an x64 hg.exe.)
setup.py is changed to write the name of the pythonlib into the generated header
file "mercurial/hgpythonlib.h", which is #included by exewrapper.c. For a Python
2.7 build, it for example contains:
#define HGPYTHONLIB "python27"
exewrapper.c then uses HGPYTHONLIB for the name of the Python dll to load.
We don't want to track mercurial/hgpythonlib.h, so we add it to .hgignore.
The old calculation code failed to properly identify revs that
weren't tagged, leaving us with a version of "unknown" most of the
time during development.
Not yet used (will be enabled in a later patch).
This patch is a stripped down version of patches originally created by
Bryan O'Sullivan <bryano@fb.com>
This gcc option has been deprecated since at least 2009 (gcc 4.4),
and it causes compilations to fail entirely with gcc 4.6.x.
Upstream distutils bug: http://bugs.python.org/issue12641
This patch contains support for Plan 9 from Bell Labs. A README is
provided in contrib/plan9 which describes the port in greater detail.
A new extension is also provided named factotum which permits the
factotum(4) authentication agent to provide credentials for HTTP
repositories. This extension is also applicable to other POSIX
platforms which make use of Plan 9 from User Space (aka plan9ports).
Apparently, it prints nothing at all if the user installed only the
command-line tools. In that case, don't try to parse the empty output
-- just assume they have Xcode >= 4.
Merge the code from contrib/setup3.py in setup.
The argument for executing is marked as experimental.
Reason: The file in contrib was outdated (packages, cmdclass, ...)
If Python interpreter was built under Linux 3.x kernel, it reports
sys.platform to be 'linux3' (it is fixed for Python 3, but not for 2.x).
This cancels building inotify extension, which was built only for 'linux2'
platform. Improved test checks if sys.platform begins with 'linux', and together
with test for kernel version to be greater than 2.6 it seems to cover all known
cases.
It generates prebuilt index of all extensions, which will be used by
frozen exe when running 'hg help extensions'.
Now py2exe invokes this command automatically.
