For reasons I can't explain (but likely have something to do with a
combination of __import__ inferring default values for arguments and
the demand importer mechanism further assuming defaults), the demand
importer isn't playing well with IPython. Without this patch, we get
a failure "ValueError: Attempted relative import in non-package" when
attempting to import "IPython." The stack has numerous demandimport
calls on it and adding "IPython" to the exclude list in demandimport
isn't enough to make the problem go away, which means the issue is
likely somewhere in the bowells of IPython. It's easier to just disable
the demand importer when importing the debugger.
The goal of commit summary keywords is to help us sort, categorize,
and filter our voluminous commits for our release notes in a way
that's helpful and meaningful to end users. Lately, there have been a
huge number of "keywords" that are neither words nor particularly key.
This patch tries to discourage that by narrowing the allowed
characters to alphanumeric. In particular, it doesn't allow "."
(method, function names, and file extensions) and "/" (filenames). It
also gives a short reminder of what a keyword ought to be.
This addition to the inno installer script means that the windows uninstaller
registry key “DisplayVersion" is set to the application version number and
will show in Add/Remove Programs.
This makes the changes in 68b7b759ebff and 71a3703364df available on Windows.
I'm not setup to make the installer, so someone with experience in this area
should probably give it a look. In looking around to try to figure out how to
build the installer, it looks like the Makefile may need an update to $DOCFILES.
My version of docker (1.8.3) have a different formating for 'docker version'
that broke the build script. We make the version matching more generic in to
work with both version.
A subsequent patch will refactor _chunks() and the calculation of the
offset will no longer occur in that function. Prepare by returning the
offset from _chunkraw().
There are make targets for building mercurial packages for various
distributions using docker. One of the preparation steps before building is to
create inside the docker image a user with the same uid/gid as the current user
on the host system, so that the resulting files have appropriate
ownership/permissions.
It's possible to run `make docker-<distro>` as a user with uid or gid that is
already present in a vanilla docker container of that distibution. For example,
issue4657 is about failing to build fedora packages as a user with uid=999 and
gid=999 because these ids are already used in fedora, and groupadd fails.
useradd would fail too, if the flow ever got to it (and there was a user with
such uid already).
A straightforward (maybe too much) way to fix this is to allow non-unique uid
and gid for the new user and group that get created inside the image. I'm not
sure of the implications of this, but marmoute encouraged me to try and send
this patch for stable.
Previously the -rc in our rc tags got dropped, meaning that those
packages looked newer to the packaging system than the later release
build. This rectifies the issue, though some damage may already have
been done on 3.6-rc builds.
I'm mostly cargo-culting the RPM version format - there don't appear
to be rules for RPM about how to handle this. Hopefully an RPM
enthusiast can fix up what I've done as a followup.
We want to support editors with parameters, eg EDITOR="vim -O" or whatever.
So remove the quotes from around $ED and assume that the editor variable is
properly escaped already.
Now, 'dirstate.write(tr)' delays writing in-memory changes out, if a
transaction is running.
This may cause treating this revision as "the first bad one" at
bisecting in some cases using external hook process inside transaction
scope, because some external hooks and editor process are still
invoked without HG_PENDING and pending changes aren't visible to them.
'dirstate.write()' callers below in localrepo.py explicitly use 'None'
as 'tr', because they can assume that no transaction is running:
- just before starting transaction
- at closing transaction, or
- at unlocking wlock
This fix adds a caret to the start of the regex looking for merge markers. This
avoids the issue arises when you've real merge conflicts in a file that tests
for the existance of merge markers in test output. Editmerge will not open on
the fake/tested merge markers because they'll be indented in.
This adds a test extension to check that the non-normal set contains the
expected entries. It wraps several methods of the dirstate to check that
the non-normal set has the correct values before and after the call. The
extension lives in contrib so that paranoid developers can easily
enable it to make sure that the non-normal set is consistent across more
complex operations than the included tests.
Whenever check-code finds something wrong, the diffs it
generated were fairly hard to read.
The problem is that check-code before this change
would list files that were white listed using
no- check- code but without a glob marker.
Whereas, the test-check-code.t expected output has
no-che?k-code (glob) in order to avoid having itself
flagged as a file to skip.
Thus, in addition to any lines relating to things you
did wrong, all of the white-listed files are listed as
changed.
There is no reason for things to be this painful.
This change makes the output from check-code.py match
the expected output in test-check-code.t
This came up before, but the tests in check-code.py don't find -U (only -u)
and they don't work when the diff is inside a shell function. This fixes
the offending tests and beefs up check-code.py.
Not having this caused warnings on Windows:
mercurial/pure/osutil.py:12: stdlib import follows local import: os
mercurial/pure/osutil.py:13: stdlib import follows local import: socket
mercurial/pure/osutil.py:14: stdlib import follows local import: stat
mercurial/pure/osutil.py:15: stdlib import follows local import: sys
We have a convention of using -c|-m|FILE elsewhere for reading from
revlogs. Use it for `hg perfrevlog`.
While I was here, I also added a docstring to document what this
command does, as "perfrevlog" is ambiguous.
As part of investigating performance improvements to revlog reading,
I needed a mechanism to measure every part of revlog reading so I knew
where time was spent and how effective optimizations were.
This patch implements a perf command for benchmarking the various
stages of reading a single revlog revision.
When executed against a manifest revision at the end of a 30,000+
long delta chain in mozilla-central, the command demonstrates that
~80% of time is spent in zlib decompression.
The old code only partially cleared the caches. Now that we have a
comprehensive method for wiping all caches, let's call it.
This appears to introduce a marginal regression in `hg perfmanifest`
on mozilla-central. This is good because the new result is more
accurate since caches aren't being used.
The /dev/null redirect was causing the following error:
The system cannot find the path specified.
Adjusting HGRCPATH as part of the command line causes the system to try to
execute 'HGRCPATH'.
Once we get a matcher down into manifestmerge, we can make narrowhg
work more easily and potentially let manifest.match().diff() do less
work in manifestmerge.
Python 3 is inevitable. There have been incremental movements towards
converting the code base to be Python 3 compatible. Unfortunately, we
don't have any tests that look for Python 3 compatibility. This patch
changes that.
We introduce a check-py3-compat.py script whose role is to verify
Python 3 compatibility of the files passed in. We add a test that
calls this script with all .py files from the source checkout.
The script currently only verifies that absolute_import and
print_function are used. These are the low hanging fruits for Python
compatbility. Over time, we can include more checks, including
verifying we're able to load each Python file with Python 3. You
have to start somewhere.
Accepting this patch means that all new .py files must have
absolute_import and print_function (if "print" is used) to avoid
a new warning about Python 3 incompatibility. We've already
converted several files to use absolute_import and print_function
is in the same boat, so I don't think this is such a radical
proposition.
Before this patch, import-checker.py didn't know if a name in ImportFrom
statement are module or not. Therefore, it complained the following example
did "direct symbol import from mercurial".
# hgext/foo.py
from mercurial import hg
This patch reuses the dict of local modules to filter out sub-module names.
Because OpenSSL is compiled without SSLv3 support on Debian sid, Python 2.6.9
can't be built without this hack. Python 2.7 is patched appropriately, but
2.6 isn't.
http://bugs.python.org/issue22935
This allows builddeb to handle distributions that are not Debian.
Distributor ID is reported by lsb_release --id, and in case of builddeb it's
usually Debian or Ubuntu.
Debian and Ubuntu releases have both codenames and traditional version numbers.
An entire "branch" of releases is referred to by its codename, and version
numbers (e.g. 8.2, 14.04.3) are used to address individual releases.
Since we use codenames for building .deb packages, let's call the option and
the variable appropriately.
The pull url header can easily grow over 80 chars. The check-commit script was
confusing this with a too long summary line. We update the regular expression to
not match other header.
Many revset consumers construct changectx instances for each returned
result. Add support for benchmarking this to our revset benchmark
script.
In the future, we might want to have some kind of special syntax in
the parsed revset files to engage this mode automatically. This would
enable us to load changectxs for revsets that do that in the code and
would more accurately benchmark what's actually happening. For now,
running all revsets with or without changectxs is sufficient.
Previously, perfrevset called repo.revs(), which only returns integer
revisions. Many revset consumers call repo.set(), which returns
changectx instances. Or they obtain a context manually later.
Since obtaining changectx instances when evaluating revsets is common,
this patch adds support for benchmarking this use case.
While we added an if conditional for every benchmark loop, it
doesn't appear to matter since revset evaluation dwarfs the cost
of a single if.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
default value are common to all call. Using mutable value is a classical source
of bug in Python. We forbid it.
The regexp (Courtesy of Matt Mackall) is only catching such value on the first
line of a definition, but that will be good enough for now.
Leaving the hgk binary in /usr/bin causes some lintian warnings, and
downstream packages poke it in /usr/share/mercurial, so we'll just
stash it in there. Rather than patch hgk.py as part of the Mercurial
install, just drop a config file in /etc/mercurial/hgrc.d that points
to the installed hgk.
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
$ ls '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6'/BaseHTTPServer.py*
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/BaseHTTPServer.pyc
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/BaseHTTPServer.pyo
This is a much larger commit than I'd like, but I honestly don't see a
good way to break it up and leave things working. Summary:
We now use debian/rules with debhelper to build our debs. This is much
more standard, and means we use dh_python2 to do things like handle
leaving .pyc files out of the built debs.
The resulting package is split into mercurial and mercurial-common,
with the former being the hg stub and all the native .sos, and the
latter being basically everything else.
builddeb and dockerdeb are updated to use the new system. The old way
(using dpkg by hand) breaks with the above changes because
debian/control no longer contains a version string (that's now guessed
from the phony changelog.)
Tests are updated to assert that the right files end up in the right
debs.
Debian now rolls their own official Vagrant base boxes, so use that. At
the same time, we're updating from Debian 7.4 (wheezy) to 8.1 (jessie),
and switching from 32-bit to 64-bit (Debian does not provide 32-bit base
boxes).
Previously:
$ /c/Program\ Files/Mercurial/hg help -k merge-tools
abort: No such file or directory: c:\Program Files\Mercurial\help\scripting.txt
The Inno installer seems OK, but the TortoiseHg required the same fix. That got
queued with an additional change of 'helpFolder.guid' in guids.wxi (probably by
Steve). I'm not sure if that is necessary here too.
This allows getting a Python stack trace at any time on Unix by
hitting Ctrl-\ (or Ctrl-T on BSDs). Useful for debugging mysterious
hangs on the fly. Sample output:
$ hg log -k nosuchmessage
^\
File "/home/mpm/hg/mercurial/revset.py", line 3089, in _iterfilter
if cond(x):
File "/home/mpm/hg/mercurial/util.py", line 415, in f
cache[arg] = func(arg)
File "/home/mpm/hg/mercurial/revset.py", line 1215, in matches
for t in c.files() + [c.user(), c.description()])
File "/home/mpm/hg/mercurial/context.py", line 525, in files
return self._changeset[3]
File "/home/mpm/hg/mercurial/util.py", line 531, in __get__
result = self.func(obj)
File "/home/mpm/hg/mercurial/context.py", line 498, in _changeset
return self._repo.changelog.read(self.rev())
File "/home/mpm/hg/mercurial/changelog.py", line 338, in read
text = self.revision(node)
File "/home/mpm/hg/mercurial/revlog.py", line 1092, in revision
bins = self._chunks(chain)
File "/home/mpm/hg/mercurial/revlog.py", line 1013, in _chunks
ladd(decompress(buffer(data, chunkstart - offset, chunklength)))
File "/home/mpm/hg/mercurial/revlog.py", line 91, in decompress
return _decompress(bin)
----
As of this change, we no longer produce broken debs, but I've already
got followups written that will produce much more standard-looking
packages and test the resulting packages.
On my linux machines multiprocessing appears to defeat the logic in
import-checker to detect stdlib modules. Since we now only use
versions of Python which ship with multiprocessing, let's just
whitelist the module.
I got the following error by rewriting hgweb/webcommands.py to use
absolute_import. It is false-positive because the import line appears in
"help" function:
hgweb/webcommands.py:1297: higher-level import should come first: mercurial
This patch makes the import checker aware of the function scope and apply
rules recursively.
I got the following error by rewriting hgweb/__init__.py to use
absolute_import, which is obviously wrong:
Import cycle: mercurial.hgweb.__init__ -> mercurial.hgweb.__init__
"from foo import bar" should not make a cycle if "foo" is a package and
if "bar" is a module or a package. On the other hand, it should be detected
as a cycle if "bar" is a non-module name. Both cases are doc-tested already,
so this patch does not add new doctest.
This script scans files for lines that look like either ui.config
usage or config variable documentation. It then ensures:
- ui.config calls for each option agree on types and defaults
- every option appears to be mentioned in documentation
It doesn't complain about devel/experimental options and allows
marking options that are not intended to be public.
Since we haven't been able to come up with a good scheme for
documenting config options at point of use, this will help close the
loop of making sure all options that should be documented are.
If mercurial was installed into a directory other than the site-packages,
test-module-imports.t failed as 'mercurial.node' was listed in stdlib_modules:
testpackage/latesymbolimport.py relative import of stdlib module
Instead, we should exclude our packages explicitly.
We can't assume that the site-packages is the only directory that has Python
files but is not handled as a package. For example, we have dist-packages
directory on Debian.
Before this patch, `import-checker.py` exits with non-0 code, if no
error is detected. This is unusual as Unix command.
This change may be a one of preparations for issue4677, because this
can avoid extra explanation about unusual exit code of
`import-checker.py` for third party tool developers.
We introduce a new convention for declaring imports and enforce it via
the import checker script.
The new convention is only active when absolute imports are used, which is
currently nowhere. Keying off "from __future__ import absolute_import" to
engage the new import convention seems like the easiest solution. It is
also beneficial for Mercurial to use this mode because it means less work
and ambiguity for the importer and potentially better performance due to
fewer stat() system calls because the importer won't look for modules in
relative paths unless explicitly asked.
Once all files are converted to use absolute import, we can refactor
this code to again only have a single import convention and we can
require use of absolute import in the style checker.
The rules for the new convention are documented in the docstring of the
added function. Tests have been added to test-module-imports.t. Some
tests are sensitive to newlines and source column position, which makes
docstring testing difficult and/or impossible.
A future patch will formalize the modern import convention. In
preparation for that, introduce a new wrapper function that will invoke
the proper function.
"from . import X" will produce an ImportFrom ast node with .module =
None. This resulted in a run-time error from attempting to concatenate
None with a str.
Another problem with relative imports is that the prefix may be dynamic
based on the "level" attribute of the import. e.g. "from ." has level 1
and "from .." has level 2.
We teach the "fromlocal" function how to cope with relative imports.
Where appropriate, the consumer passes in the level so relative module
names may be resolved properly.
We just rewrote all files to use modern exception syntax. Ban the old
form.
This will detect the "except type, instance" and
"except (type1, type2), instance" forms.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
The canonical way of doing 'roots(X)' is 'X - children(X)'. This is what the
implementation used to be. However, computing children is expensive because it
is unbounded. Any changesets in the repository may be a children of '0' so you
have to look at all changesets in the repository to compute children(0).
Moreover the current revsets implementation for children is not lazy, leading to
bad performance when fetching the first result.
There is a more restricted algorithm to compute roots:
roots(X) = [r for r in X if not parents(r) & X]
This achieve the same result while only looking for parent/children relation in
the X set itself, making the algorithm 'O(len(X))' membership operation.
Another advantages is that it turns the check into a simple filter, preserving
all laziness property of the underlying revsets.
The speed is very significant and some laziness is restored.
-) revset without 'roots(...)' to compare to base line
0) before this change
1) after this change
revset #0: roots((tip~100::) - (tip~100::tip))
plain min last
-) 0.001082 0.000993 0.000790
0) 0.001366 0.001385 0.001339
1) 0.001257 92% 0.001028 74% 0.000821 61%
revset #1: roots((0::) - (0::tip))
plain min last
-) 0.134551 0.144682 0.068453
0) 0.161822 0.171786 0.157683
1) 0.137583 85% 0.146204 85% 0.070012 44%
revset #2: roots(tip~100:)
plain min first last
-) 0.000219 0.000225 0.000231 0.000229
0) 0.000513 0.000529 0.000507 0.000539
1) 0.000463 90% 0.000269 50% 0.000267 52% 0.000463 85%
revset #3: roots(:42)
plain min first last
-) 0.000119 0.000146 0.000146 0.000146
0) 0.000231 0.000254 0.000253 0.000260
1) 0.000216 93% 0.000186 73% 0.000184 72% 0.000244 93%
revset #4: roots(not public())
plain min first
-) 0.000478 0.000502 0.000504
0) 0.000611 0.000639 0.000634
1) 0.000604 0.000560 87% 0.000558
revset #5: roots((0:tip)::)
plain min max first last
-) 0.057795 0.004905 0.058260 0.004908 0.038812
0) 0.132845 0.118931 0.130306 0.114280 0.127742
1) 0.111659 84% 0.005023 4% 0.111658 85% 0.005022 4% 0.092490 72%
revset #6: roots(0::tip)
plain min max first last
-) 0.032971 0.033947 0.033460 0.032350 0.033125
0) 0.083671 0.081953 0.084074 0.080364 0.086069
1) 0.074720 89% 0.035547 43% 0.077025 91% 0.033729 41% 0.083197
revset #7: 42:68 and roots(42:tip)
plain min max first last
-) 0.006827 0.000251 0.006830 0.000254 0.006771
0) 0.000337 0.000353 0.000366 0.000350 0.000366
1) 0.000318 94% 0.000297 84% 0.000353 0.000293 83% 0.000351
revset #8: roots(0:tip)
plain min max first last
-) 0.002119 0.000145 0.000147 0.000147 0.000147
0) 0.047441 0.040660 0.045662 0.040284 0.043435
1) 0.038057 80% 0.000187 0% 0.034919 76% 0.000186 0% 0.035097 80%
revset #0: roots(:42 + tip~42:)
plain min max first last sort
-) 0.000321 0.000317 0.000319 0.000308 0.000369 0.000343
0) 0.000772 0.000751 0.000811 0.000750 0.000802 0.000783
1) 0.000632 81% 0.000369 49% 0.000617 76% 0.000358 47% 0.000601 74% 0.000642 81%
If the computation of a set for each phase (done in C) is available,
we use it directly instead of applying a simple filter. This give a
massive speed-up in the vast majority of cases.
On my mercurial repo with about 15000 out of 40000 draft changesets:
revset: draft()
plain min first last
0) 0.011201 0.019950 0.009844 0.000074
1) 0.000284 2% 0.000312 1% 0.000314 3% 0.000315 x4.3
Bad performance for "last" come from the handling of the 15000 elements set
(memory allocation, filtering hidden changesets (99% of it) etc. compared to
applying the filter only on a handfuld of revisions (the first draft changesets
being close of tip).
This is not seen as an issue since:
* Timing is still pretty good and in line with all the other one,
* Current user of Vanilla Mercurial will not have 1/3 of their repo draft,
This bad effect disappears when phase's set is smaller. (about 200 secrets):
revset: secret()
plain min first last
0) 0.011181 0.022228 0.010851 0.000452
1) 0.000058 0% 0.000084 0% 0.000087 0% 0.000087 19%
Using 'repo[X]' is much slower because it creates a 'changectx' object and goes
though multiple layers of code to do so. It is also error prone if there is
tags, bookmarks, branch or other names that could map to a node hash and take
precedence (user are wicked).
This provides a significant performance boost on repository with a lot of
heads. Benchmark result for a repo with 1181 heads.
revset: head()
plain min last reverse
0) 0.014853 0.014371 0.014350 0.015161
1) 0.001402 9% 0.000975 6% 0.000874 6% 0.001415 9%
revset: head() - public()
plain min last reverse
0) 0.015121 0.014420 0.014560 0.015028
1) 0.001674 11% 0.001109 7% 0.000980 6% 0.001693 11%
revset: draft() and head()
plain min last reverse
0) 0.015976 0.014490 0.014214 0.015892
1) 0.002335 14% 0.001018 7% 0.000887 6% 0.002340 14%
The speed up is visible even when other more costly revset are in use
revset: head() and author("mpm")
plain min last reverse
0) 0.105419 0.090046 0.017169 0.108180
1) 0.090721 86% 0.077602 86% 0.003556 20% 0.093324 86%
This file should gather all revsets ever thought interesting by
anyone. That way one can check the impact of a change when touching
something revset-ish. See inline comments for details.
This file have been refilled with all the entry I could automatically
find from changeset descriptions. I assume we missed some not using
'revsetbenchmarks.py' output.
We rename the file and document its purpose. We'll be introducing another file
gathering revsets useful for benchmark of the predicate themsleves in a coming
changesets.
We remove revset making use of min and max as this is covered by the variants.
We could use variant for roots too, but it is not in the default so keep it
here.
We need more advanced variants in some cases. For example, "The last
rev of the sorted version".
We introduce a syntax for this: `reverse+last` means `last(reverse(REVSET))`.
We now use an 8 char display for timing (from 10), we add some logic to drop
precision if the number grows too large (as we do not care about sub-0 digit
in this case). This allow to pack more variants in a single screen.
The current benchmarks were only testing the whole iteration. This is suboptimal
because some changes are meaningful for things like first result, minimum or
sorting.
We introduce a "variants" feature that let you systematically add some variants
to all revsets tested.
A typical variants value would be 'plain,min,last,sort'. When testing 'all()' it
will also provide testing for:
- all()
- min(all())
- last(all())
- sort(sort)
and output:
plain min last sort
0) 0.034568 0.037857 0.000074 0.034238
1) 0.011358 32% 0.020181 53% 0.000080 108% 0.011405 33%
Using revsets (who hit the API) instead of the internal API add some overhead,
but the overhead should be the same everywhere so it still allow comparison.
This is is more simple to implement and allows comparison with older versions
who do not have the same API.
If the time difference is more than 5% from the previous run, we'll display
relative information. This makes it much simpler to spot performance changes in
a sea of benchmarks.
We mostly only care about total time. Dropping this output give us some room to
display more useful information (like percentage different) in future
changesets.
The file doc was saying something, the code was doing something else, the
argument validation was doing a third thing.
Doc and behavior now comply with the argument defined in the code.
We cannot just ask perfrevset to provide debug output because we usually want
to compare output from old version of Mercurial that do not support it. So, we
are using a regular expression.
(/we now have \d problems/).
