This is a cherry pick of an upstream fix. The free() of uninitialed
memory could likely only occur if a malloc() inside zstd fails.
The patched functions aren't currently used by Mercurial. But I don't
like leaving footguns sitting around.
Previously, when runcommand raises, chg aborts with, and does not wait for
pager. The call stack is like:
hgc_runcommand -> handleresponse -> readchannel -> debugmsg("failed to
read channel") -> exit(255)
That means, chg returns to the shell, then both the pager and the shell will
read from the terminal at the same time, causing problems.
This patch fixes that by using "atexit" to register the pager cleanup
function so chg will always wait for pager even if runcommand raises.
Commit 63c68d6f5fc8de4afd9bde81b13b537beb4e47e8 from
https://github.com/indygreg/python-zstandard is imported without
modifications (other than removing unwanted files).
This includes minor performance and feature improvements. It also
changes the vendored zstd library from 1.1.1 to 1.1.2.
# no-check-commit
pycompat.getenv returns os.getenvb on py3 which is not available on Windows.
This patch replaces them with encoding.environ.get and checks to ensure no
new instances of os.getenv or os.setenv are introduced.
Module-level @cachefunc usage is risky because it can easily create a
memory "leak". Let's reject it completely for now. If a valid usage
comes up in the future, we can always improve the check or reconsider.
Now that the revlog has a reference to a compressor, it is
possible to swap in other compression engines. So, teach
`hg perfrevlogchunks` to do that.
The default behavior of `hg perfrevlogchunks` is now to measure the
compression performance of all compression engines implementing the
revlog compressor API. This effectively adds the no-op "none"
compressor and zstd (when available) into the default set.
While we can't yet plug alternate compressors into revlogs, this
command gives us a preview of the performance. On the mozilla-unified
repository:
$ hg perfrevlogchunks -c
! compress w/ none
! wall 0.115159 comb 0.110000 user 0.110000 sys 0.000000 (best of 86)
! compress w/ zlib
! wall 5.681406 comb 5.680000 user 5.680000 sys 0.000000 (best of 3)
! compress w/ zstd
! wall 2.624781 comb 2.620000 user 2.620000 sys 0.000000 (best of 4)
$ hg perfrevlogchunks -m
! compress w/ none
! wall 0.124486 comb 0.120000 user 0.120000 sys 0.000000 (best of 79)
! compress w/ zlib
! wall 10.144701 comb 10.150000 user 10.150000 sys 0.000000 (best of 3)
! compress w/ zstd
! wall 4.383118 comb 4.390000 user 4.390000 sys 0.000000 (best of 3)
Those numbers for zstd look promising. But they aren't the full story.
For that, we'll need to look at decompression times and storage sizes.
Stay tuned...
Upcoming patches will convert revlogs to use the compression engine
APIs to perform all things compression. The yet-to-be-introduced
APIs support a persistent "compressor" object so the same object
can be reused for multiple compression operations, leading to
better performance. In addition, compression engines like zstd
may wish to tweak compression engine state based on the revlog
(e.g. per-revlog compression dictionaries).
A global and shared decompress() function will shortly no longer
make much sense. So, we move decompress() to be a method of the
revlog class. It joins compress() there.
On the mozilla-unified repo, we can measure the impact of this change
on reading performance:
$ hg perfrevlogchunks -c
! chunk
! wall 1.932573 comb 1.930000 user 1.900000 sys 0.030000 (best of 6)
! wall 1.955183 comb 1.960000 user 1.930000 sys 0.030000 (best of 6)
! chunk batch
! wall 1.787879 comb 1.780000 user 1.770000 sys 0.010000 (best of 6
! wall 1.774444 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)
"chunk" appeared to become slower but "chunk batch" got faster. Upon
further examination by running both sets multiple times, the numbers
appear to converge across all runs. This tells me that there is no
perceived performance impact to this refactor.
Selecting single and multiple revisions is closely related, so let's
put it in one place, so users can easily find it. We actually did not
even point to "hg help revsets" from "hg help revisions", but now that
they're on a single page, that won't be necessary.
This patch uses the newly introduced "setprocname" interface to update the
process title server-side, to make it easier to tell what a worker is actually
doing.
The new title is "chg[worker/$PID]", where PID is the process ID of the
connected client. It can be directly observed using "ps -AF" under Linux, or
"ps -A" under FreeBSD.
We have enough bits to switch to the new chg pager code path in runcommand.
So just remove the legacy getpager support.
This is a red-only patch, and will break chg's pager support temporarily.
This patch implements the simple S-channel pager handling at chg
client-side.
Note: It does not deal with environ and cwd currently for simplicity, which
will be fixed later.
Previously S channel is only used to send system commands. It will also be
used to send pager commands. So add a type parameter.
This breaks older chg clients. But chg and hg should always come from a
single commit and be packed into a single package. Supporting running
inconsistent versions of chg and hg seems to be unnecessarily complicated
with little benefit. So just make the change and assume people won't use
inconsistent chg with hg.
After dropping the garbage collector hack, `hg perfstartup` started yelling
about not being able to import the evolve extension, which I have in my user
config. Launching `env` shows that an empty HGRCPATH isn't exported to the
environment. Since `env` doesn't quote, I have no idea if the variable is
trimmed, but Mercurial doesn't complain when processing it.
We previously weren't looking for this config helper. And, surprise,
profiling.py references config options without docs.
If I tried hard enough, I could have combined the regexps using a
positive lookbehind assertion or something. But I didn't want to make
my brain explode.
At some point, we should probably do this linting at the tokenizer or
ast layer. I'm not willing to open that can of worms right now.
Instead of mentioning 127.0.0.1, we should use $LOCALIP. Anytime
$LOCALIP appears in output, we should make sure we use (glob) on that
line of output so that weird environments that do remapping jiggery
pokery (such as our FreeBSD buildbot that's in a jail) don't get
spurious test failures.
Commit 81e1f5bbf1fc54808649562d3ed829730765c540 from
https://github.com/indygreg/python-zstandard is imported without
modifications (other than removing unwanted files).
Updates relevant to Mercurial include:
* Support for multi-threaded compression (we can use this for
bundle and wire protocol compression).
* APIs for batch compression and decompression operations using
multiple threads and optimal memory allocation mechanism. (Can
be useful for revlog perf improvements.)
* A ``BufferWithSegments`` type that models a single memory buffer
containing N discrete items of known lengths. This type can be
used for very efficient 0-copy data operations.
# no-check-commit
I softly formalized the concept of a "bundle specification" a while
ago when I was working on clone bundles and stream clone bundles and
wanted a more robust way to define what exactly is in a bundle file.
The concept has existed for a while. Since it is part of the clone
bundles feature and exposed to the user via the "-t" argument to
`hg bundle`, it is something we need to support for the long haul.
After the 4.1 release, I heard a few people comment that they didn't
realize you could generate zstd bundles with `hg bundle`. I'm
partially to blame for not documenting it in bundle's docstring.
Additionally, I added a hacky, experimental feature for controlling
the compression level of bundles in 054e64c4d837. As the commit
message says, I went with a quick and dirty solution out of time
constraints. Furthermore, I wanted to eventually store this
configuration in the "bundlespec" so it could be made more flexible.
Given:
a) bundlespecs are here to stay
b) we don't have great documentation over what they are, despite being
a user-facing feature
c) the list of available compression engines and their behavior isn't
exposed
d) we need an extensible place to modify behavior of compression
engines
I want to move forward with formalizing bundlespecs as a user-facing
feature. This commit does that by introducing a "bundlespec" help
page. Leaning on the just-added compression engine documentation
and API, the topic also conveniently lists available compression
engines and details about them. This makes features like zstd
bundle compression more discoverable. e.g. you can now
`hg help -k zstd` and it lists the "bundlespec" topic.
Currently, Mercurial has a number of commands to show information. And,
there are features coming down the pipe that will introduce more
commands for showing information.
Currently, when introducing a new class of data or a view that we
wish to expose to the user, the strategy is to introduce a new command
or overload an existing command, sometimes both. For example, there is
a desire to formalize the wip/smartlog/underway/mine functionality that
many have devised. There is also a desire to introduce a "topics"
concept. Others would like views of "the current stack." In the
current model, we'd need a new command for wip/smartlog/etc (that
behaves a lot like a pre-defined alias of `hg log`). For topics,
we'd likely overload `hg topic[s]` to both display and manipulate
topics.
Adding new commands for every pre-defined query doesn't scale well
and pollutes `hg help`. Overloading commands to perform read-only and
write operations is arguably an UX anti-pattern: while having all
functionality for a given concept in one command is nice, having a
single command doing multiple discrete operations is not. Furthermore,
a user may be surprised that a command they thought was read-only
actually changes something.
We discussed this at the Mercurial 4.0 Sprint in Paris and decided that
having a single command where we could hang pre-defined views of
various data would be a good idea. Having such a command would:
* Help prevent an explosion of new query-related commands
* Create a clear separation between read and write operations
(mitigates footguns)
* Avoids overloading the meaning of commands that manipulate data
(bookmark, tag, branch, etc) (while we can't take away the
existing behavior for BC reasons, we now won't introduce this
behavior on new commands)
* Allows users to discover informational views more easily by
aggregating them in a single location
* Lowers the barrier to creating the new views (since the barrier
to creating a top-level command is relatively high)
So, this commit introduces the `hg show` command via the "show"
extension. This command accepts a positional argument of the
"view" to show. New views can be registered with a decorator. To
prove it works, we implement the "bookmarks" view, which shows a
table of bookmarks and their associated nodes.
We introduce a new style to hold everything used by `hg show`.
For our initial bookmarks view, the output varies from `hg bookmarks`:
* Padding is performed in the template itself as opposed to Python
* Revision integers are not shown
* shortest() is used to display a 5 character node by default (as
opposed to static 12 characters)
I chose to implement the "bookmarks" view first because it is simple
and shouldn't invite too much bikeshedding that detracts from the
evaluation of `hg show` itself. But there is an important point
to consider: we now have 2 ways to show a list of bookmarks. I'm not
a fan of introducing multiple ways to do very similar things. So it
might be worth discussing how we wish to tackle this issue for
bookmarks, tags, branches, MQ series, etc.
I also made the choice of explicitly declaring the default show
template not part of the standard BC guarantees. History has shown
that we make mistakes and poor choices with output formatting but
can't fix these mistakes later because random tools are parsing
output and we don't want to break these tools. Optimizing for human
consumption is one of my goals for `hg show`. So, by not covering
the formatting as part of BC, the barrier to future change is much
lower and humans benefit.
There are some improvements that can be made to formatting. For
example, we don't yet use label() in the templates. We obviously
want this for color. But I'm not sure if we should reuse the existing
log.* labels or invent new ones. I figure we can punt that to a
follow-up.
At the aforementioned Sprint, we discussed and discarded various
alternatives to `hg show`.
We considered making `hg log <view>` perform this behavior. The main
reason we can't do this is because a positional argument to `hg log`
can be a file path and if there is a conflict between a path name and
a view name, behavior is ambiguous. We could have introduced
`hg log --view` or similar, but we felt that required too much typing
(we don't want to require a command flag to show a view) and wasn't
very discoverable. Furthermore, `hg log` is optimized for showing
changelog data and there are things that `hg display` could display
that aren't changelog centric.
There were concerns about using "show" as the command name.
Some users already have a "show" alias that is similar to `hg export`.
There were also concerns that Git users adapted to `git show` would
be confused by `hg show`'s different behavior. The main difference
here is `git show` prints an `hg export` like view of the current
commit by default and `hg show` requires an argument. `git show`
can also display any Git object. `git show` does not support
displaying more complex views: just single objects. If we
implemented `hg show <hash>` or `hg show <identifier>`, `hg show`
would be a superset of `git show`. Although, I'm hesitant to do that
at this time because I view `hg show` as a higher-level querying
command and there are namespace collisions between valid identifiers
and registered views.
There is also a prefix collision with `hg showconfig`, which is an
alias of `hg config`.
We also considered `hg view`, but that is already used by the "hgk"
extension.
`hg display` was also proposed at one point. It has a prefix collision
with `hg diff`. General consensus was "show" or "view" are the best
verbs. And since "view" was taken, "show" was chosen.
There are a number of inline TODOs in this patch. Some of these
represent decisions yet to be made. Others represent features
requiring non-trivial complexity. Rather than bloat the patch or
invite additional bikeshedding, I figured I'd document future
enhancements via TODO so we can get a minimal implmentation landed.
Something is better than nothing.
In filerevision view (/file/<rev>/<fname>) we add some event listeners on
mouse clicks of <span> elements in the <pre class="sourcelines"> block.
Those listeners will capture a range of lines selected between two mouse
clicks and a box inviting to follow the history of selected lines will then
show up. Selected lines (i.e. the block of lines) get a CSS class which make
them highlighted. Selection can be cancelled (and restarted) by either
clicking on the cancel ("x") button in the invite box or clicking on any other
source line. Also clicking twice on the same line will abort the selection and
reset event listeners to restart the process.
As a first step, this action is only advertised by the "cursor: cell" CSS rule
on source lines elements as any other mechanisms would make the code
significantly more complicated. This might be improved later.
All JavaScript code lives in a new "linerangelog.js" file, sourced in
filerevision template (only in "paper" style for now).
revlog.revision takes either node or rev, but taking a rev is more
efficient, because converting rev to node is just a seek and read.
That's cheaper than converting node to rev, which may require O(n) walk in
revlog index for the first times, and then triggering building the radix
tree index. Even with the radix tree built, rev -> node is still faster than
node -> rev because the radix tree requires more jumps in memory.
So r.revision(r.node(rev)) should be changed to r.revision(rev). This patch
adds a check-code rule to detect that.
This is similar to what 9704c8e70d2d does. Since 1363aaf74791 has changed
"127.0.0.1" to "$LOCALIP". The glob pattern needs update accordingly. It is
expected to fix tests running in some BSD jails.
This will ensure new extensions are consistent and `hg help -e` has a
consistent output.
I have to add a new group since the normal "pypats" will be filtered by
"pyfilters", which will remove comments and docstrings.
Previously, chg.c maintains the pagerpid. Let's move it to procutil.c.
Note: chg.c still have a pagerpid to decide whether to call attachio or not.
In the future, attachio may be moved from hgc_open to hgc_runcommand, and
hgc_runcommand handles both pager and attachio so we don't need to run
attachio twice. And chg.c will be free of pagerpid.
In the future hgclient will deal with pager directly inside runcommand, so
related signal handling stuff needs to be decoupled from chg.c.
The signal handling and pager logic are coupled because we need to forward
SIGPIPE when pager exits. So they are moved together, otherwise a global
variable (pagerpid) is inevitable.
This patch moves related functions from chg.c to procutil.c, which was
marked as copied to maintain annotate history.
The move is done without code modification for easy review, therefore
`#include "procutil.c"` was introduced temporarily.
$XDG_RUNTIME_DIR [1] is a better place for user daemons. Let's use it and
fallback to $TMPDIR.
After this patch, chg will try socket paths in the following order:
1. $CHGSOCKNAME
2. $XDG_RUNTIME_DIR/chg/server
3. ${TMPDIR:-tmp}/chg$UID/server
[1]: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
"sizeof(sun_path)" is too small. Use the chdir trick to support long socket
path, like "mercurial.util.bindunixsocket".
It's useful for cases where TMPDIR is long. Modern OS X rewrites TMPDIR to a
long value. And we probably want to use XDG_RUNTIME_DIR [2] for Linux.
The approach is a bit different from the previous plan, where we will have
hgc_openat and pass cmdserveropts.sockdirfd to it. That's because the
current change is easier: chg has to pass a full path to "hg" as the
"--address" parameter. There is no "--address-basename" or "--address-dirfd"
flags. The next patch will remove "sockdirfd".
Note: It'd be nice if we can use a native "connectat" implementation.
However, that's not available everywhere. Some platform (namely FreeBSD)
does support it, but the implementation has bugs so it cannot be used [2].
[1]: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
[2]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-April/082892.html
We have our own bytes versions of things like, getopt.getopt, os.sep, os.name,
sys.executable, os.environ and few more for python 3 portability. Its better
to come up with warnings if someone breaks the things which we have fixed.
After this patch, check-code will warn us to use our bytes version.
These checks run on mercurial/ and hgext/ and pycompat.py is excluded.
See the previous two patches for the reason. The advantage is a simplified
code base and better throughput when starting multiple servers with multiple
confighashes. The disadvantage is starting multiple servers in parallel with
a single confighash will waste some CPU time, which is probably fine in
common use-cases.
This makes it easier to switch to relative paths to support long unix domain
socket paths.
See the previous patch for motivation. Previously, the server is started at
a globally shared address. This patch appends pid to the address so it
becomes unique.
Note: with Linux pid namespace, the address may be non-unique, but it does
not affect correctness of chg - chg client will receive an redirection and
that's it.
Before this change, you could get in a start where the checker would either
complain about importing local module before stdlib one or complain about the
local one being wrongly lexically sorted with the stdlib one.
We detect the boundary and avoid complaining about lexical sort across it.
ui.load() has been available since d83ca854 and at the time of writing isn't
available on stable branch breaking benchmarking newer stable revisions.
Add historical portability policy note on contrib/benchmarks
In the next patch, we will be creating a bytes version of getopt.getopt() and
doing that will leave getopt as unused import in fancyopts. So before removing
that there are instances in codebase where instead of importing getopt, we
have used fancyopts.getopt. This patch will switch all those cases so that
the next patch can remove the import of getopt from fancyopts without breaking
things.
This allows us to write doctests depending on a ui object, but not on global
configs.
ui.load() is a class method so we can do wsgiui.load(). All ui() calls but
for doctests are replaced with ui.load(). Some of them could be changed to
not load configs later.
See the commit message of the previous patch for the reason. In short,
according to the current POSIX standard, "-r" is "removed", and "-R" is the
current standard way to do "copy file hierarchies".
It was an extension just because there were several dependency cycles I
needed to address.
I don't add 'chgserver' to extensions._builtin since chgserver is considered
an internal extension so nobody should enable it by their config.
Upcoming commits will teach revlogs to leverage the new compression
engine API so that new compression formats can more easily be
leveraged in revlogs. We want to be sure this refactoring doesn't
regress performance. So this commit introduces "perfrevchunks" to
explicitly test performance of reading, decompressing, and
recompressing revlog chunks.
Here is output when run on the mozilla-unified repo:
$ hg perfrevlogchunks -c
! read
! wall 0.346603 comb 0.350000 user 0.340000 sys 0.010000 (best of 28)
! read w/ reused fd
! wall 0.337707 comb 0.340000 user 0.320000 sys 0.020000 (best of 30)
! read batch
! wall 0.013206 comb 0.020000 user 0.000000 sys 0.020000 (best of 221)
! read batch w/ reused fd
! wall 0.013259 comb 0.030000 user 0.010000 sys 0.020000 (best of 222)
! chunk
! wall 1.909939 comb 1.910000 user 1.900000 sys 0.010000 (best of 6)
! chunk batch
! wall 1.750677 comb 1.760000 user 1.740000 sys 0.020000 (best of 6)
! compress
! wall 5.668004 comb 5.670000 user 5.670000 sys 0.000000 (best of 3)
$ hg perfrevlogchunks -m
! read
! wall 0.365834 comb 0.370000 user 0.350000 sys 0.020000 (best of 26)
! read w/ reused fd
! wall 0.350160 comb 0.350000 user 0.320000 sys 0.030000 (best of 28)
! read batch
! wall 0.024777 comb 0.020000 user 0.000000 sys 0.020000 (best of 119)
! read batch w/ reused fd
! wall 0.024895 comb 0.030000 user 0.000000 sys 0.030000 (best of 118)
! chunk
! wall 2.514061 comb 2.520000 user 2.480000 sys 0.040000 (best of 4)
! chunk batch
! wall 2.380788 comb 2.380000 user 2.360000 sys 0.020000 (best of 5)
! compress
! wall 9.815297 comb 9.820000 user 9.820000 sys 0.000000 (best of 3)
We already see some interesting data, such as how much slower
non-batched chunk reading is and that zlib compression appears to be
>2x slower than decompression.
I didn't have the data when I wrote this commit message, but I ran this
on Mozilla's NFS-based Mercurial server and the time for reading with a
reused file descriptor was faster. So I think it is worth testing both
with and without file descriptor reuse so we can make informed
decisions about recycling file descriptors.
Currently, no explicit garbage collection is performed when running
the microbenchmarks in `hg perf`. I think this is wrong because
garbage collection can have a significant impact on execution times.
And, if gc is triggered via the default heuristics, it will
fire effectively randomly during subsequent benchmark iterations
due to variable amount of garbage left over from previous runs.
Running a gc before invoking the measured function will help ensure
state is more consistent across all iterations.
Running 'Start-Process -Wait "vim" "+10" "filename"' from PowerShell
actually spawns a separate cmd window to run vim in. This looks ugly
and in most cases not what user wants. During my initial testing I was
using the Cmder app, which made me not notice this (it captures new
windows as new tabs).
This change is needed for cases when user does not put editmergeps.bat
directly into PATH, but rather uses 'merge-tools.editmergeps.executable'
config option to provide a full path to editmergeps.bat. In such cases,
editmergeps.ps1 cannot be run simply by name, it needs a full path. In
BATCH file %~dp0 stands for the directory in which the original file
is located.
We didn't have explicit microbenchmark coverage for loading revlog
indexes. That seems like a useful thing to have, so let's add it.
We currently measure the low-level nodemap APIs. There is room to
hook in at the actual revlog layer. This could be done as a follow-up.
The hackiest thing about this patch is specifying revlog paths.
Other commands have arguments that allow resolution of changelog,
manifest, and filelog. I needed to hook in at a lower level of
the revlog API than what the existing helper functions to resolve
revlogs allowed. I was too lazy to write some new APIs. This could
be done as a follow-up easily enough.
Example output for `hg perfrevlogindex 00changelog.i` on my
Firefox repo (404418 revisions):
! revlog constructor
! wall 0.003106 comb 0.000000 user 0.000000 sys 0.000000 (best of 912)
! read
! wall 0.003077 comb 0.000000 user 0.000000 sys 0.000000 (best of 924)
! create index object
! wall 0.000000 comb 0.000000 user 0.000000 sys 0.000000 (best of 1803994)
! retrieve index entry for rev 0
! wall 0.000193 comb 0.000000 user 0.000000 sys 0.000000 (best of 14037)
! look up missing node
! wall 0.003313 comb 0.000000 user 0.000000 sys 0.000000 (best of 865)
! look up node at rev 0
! wall 0.003295 comb 0.010000 user 0.010000 sys 0.000000 (best of 858)
! look up node at 1/4 len
! wall 0.002598 comb 0.010000 user 0.010000 sys 0.000000 (best of 1103)
! look up node at 1/2 len
! wall 0.001909 comb 0.000000 user 0.000000 sys 0.000000 (best of 1507)
! look up node at 3/4 len
! wall 0.001213 comb 0.000000 user 0.000000 sys 0.000000 (best of 2275)
! look up node at tip
! wall 0.000453 comb 0.000000 user 0.000000 sys 0.000000 (best of 5697)
! look up all nodes (forward)
! wall 0.094615 comb 0.100000 user 0.100000 sys 0.000000 (best of 100)
! look up all nodes (reverse)
! wall 0.045889 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
! retrieve all index entries (forward)
! wall 0.078398 comb 0.080000 user 0.060000 sys 0.020000 (best of 100)
! retrieve all index entries (reverse)
! wall 0.079376 comb 0.080000 user 0.070000 sys 0.010000 (best of 100)
We have a couple of commands beginning with "perfrevlog." The
actual "perfrevlog" command actually measures resolving the fulltext
of multiple revisions. So let's rename it to reflect what it actually
does.
Since extension modules aren't included in the list of source files, they
need to be populated somehow. Otherwise the import from cext/cffi would be
treated as a global one.
Loading the obsstore can become a large part of the time necessary to compute
the important volatile set. We add a flag purging all known obsstore related
data.
For example, computing the 'bumped' set currently requires reading the full
obsstore, so timing greatly differ with or without that flag:
Without:
! bumped
! wall 0.005047 comb 0.000000 user 0.000000 sys 0.000000 (best of 446)
With:
! bumped
! wall 0.512367 comb 0.510000 user 0.480000 sys 0.030000 (best of 15)
The goal is to get rid of the debugcommands -> commands dependency.
Since globalopts is the property of the commands, it's kept in the commands
module.
Using Start-Process -Wait makes it wait until the process finishes,
which is necesssary for Windows GUI applications. My short testing
also demonstrated that it does not hurt with command line vim.
cmdutil.command wasn't a member of the registrar framework only for a
historical reason. Let's make that happen. This patch keeps cmdutil.command
as an alias for extension compatibility.
This just adds a translation of existing contrib/editmerge to powershell.
It allows users on Windows to iteratively resolve conflicts in their
editor of choice.
# no-check-commit
I removed this in ff97c183abcf thinking it wasn't necessary. In fact,
we need to always pass a node so the code is compatible with revisions
before f4a6c9197dbd.
The new code uses a variable to avoid check-style complaining
about "r.revision(r.node(" patterns.
Per discussion on the mailing list and elsewhere, we've decided that
Python 2.6 is too old to continue supporting. We keep accumulating
hacks/fixes/workarounds for 2.6 and this is taking time away from
more important work.
So with this patch, we officially drop support for Python 2.6 and
require Python 2.7 to run Mercurial.
We don't need to measure the time it takes to open the revlog or
calculate its length.
This is more consistent with what other perf* functions do.
While I was here, I also renamed the revlog variable from "r" to
"rl" - again in the name of consistency.
cmdutil.openrevlog() may return a cached revlog instance. This /may/
be a recent "regression" due to refactoring of the manifest API. I'm
not sure.
Either way, this perf command was broken for at least manifests because
subsequent invocations of the perf function would get cache hits from
previous invocations, invalidating results. In the extreme case,
testing the last revision in the revlog resulted in near-instantanous
execution of subsequent runs (since the fulltext is cached). A time
of ~1us would be reported in this case.
This completes our rename of internal revlog methods to
distinguish between low-level raw revlog data "segments" and
higher-level, per-revision "chunks."
perf.py has been updated to consult both names so it will work
against older Mercurial versions.
To prepare for renaming revlog._chunkraw, we stuff a reference to this
metho in a local variable. This does 2 things. First, it moves the
attribute lookup outside of a loop, which more accurately measures
the time of the code being invoked. Second, it allows us to alias
to different methods depending on their presence (perf.py needs to
support running against old Mercurial versions).
Removing an attribute lookup from a tigh loop appears to shift the
numbers slightly with mozilla-central:
$ hg perfrevlogchunks -c
! read
! wall 0.354789 comb 0.340000 user 0.330000 sys 0.010000 (best of 28)
! wall 0.335932 comb 0.330000 user 0.290000 sys 0.040000 (best of 30)
! read w/ reused fd
! wall 0.342326 comb 0.340000 user 0.320000 sys 0.020000 (best of 29)
! wall 0.332857 comb 0.340000 user 0.290000 sys 0.050000 (best of 30)
! read batch
! wall 0.023623 comb 0.020000 user 0.000000 sys 0.020000 (best of 124)
! wall 0.023666 comb 0.020000 user 0.000000 sys 0.020000 (best of 125)
! read batch w/ reused fd
! wall 0.023828 comb 0.020000 user 0.000000 sys 0.020000 (best of 124)
! wall 0.023556 comb 0.020000 user 0.000000 sys 0.020000 (best of 126)
Previously, the "startrev" argument would be ignored due to
"startrev = 0" in the benchmark function. This meant that
`hg perfrevlog` always started at revision 0.
Rename the local variable to "beginrev" so the variable does the
right thing.
As the commit message for the previous changeset says, we wish
for zstd to be a 1st class citizen in Mercurial. To make that
happen, we need to enable Python to talk to the zstd C API. And
that requires bindings.
This commit vendors a copy of existing Python bindings. Why do we
need to vendor? As the commit message of the previous commit says,
relying on systems in the wild to have the bindings or zstd present
is a losing proposition. By distributing the zstd and bindings with
Mercurial, we significantly increase our chances that zstd will
work. Since zstd will deliver a better end-user experience by
achieving better performance, this benefits our users. Another
reason is that the Python bindings still aren't stable and the
API is somewhat fluid. While Mercurial could be coded to target
multiple versions of the Python bindings, it is safer to bundle
an explicit, known working version.
The added Python bindings are mostly a fully-featured interface
to the zstd C API. They allow one-shot operations, streaming,
reading and writing from objects implements the file object
protocol, dictionary compression, control over low-level compression
parameters, and more. The Python bindings work on Python 2.6,
2.7, and 3.3+ and have been tested on Linux and Windows. There are
CFFI bindings, but they are lacking compared to the C extension.
Upstream work will be needed before we can support zstd with PyPy.
But it will be possible.
The files added in this commit come from Git commit
e637c1b214d5f869cf8116c550dcae23ec13b677 from
https://github.com/indygreg/python-zstandard and are added without
modifications. Some files from the upstream repository have been
omitted, namely files related to continuous integration.
In the spirit of full disclosure, I'm the maintainer of the
"python-zstandard" project and have authored 100% of the code
added in this commit. Unfortunately, the Python bindings have
not been formally code reviewed by anyone. While I've tested
much of the code thoroughly (I even have tests that fuzz APIs),
there's a good chance there are bugs, memory leaks, not well
thought out APIs, etc. If someone wants to review the code and
send feedback to the GitHub project, it would be greatly
appreciated.
Despite my involvement with both projects, my opinions of code
style differ from Mercurial's. The code in this commit introduces
numerous code style violations in Mercurial's linters. So, the code
is excluded from most lints. However, some violations I agree with.
These have been added to the known violations ignore list for now.
zstd is a new compression format and it is awesome, yielding
higher compression ratios and significantly faster compression
and decompression operations compared to zlib (our current
compression engine of choice) across the board.
We want zstd to be a 1st class citizen in Mercurial and to eventually
be the preferred compression format for various operations.
This patch starts the formal process of supporting zstd by vendoring
a copy of zstd. Why do we need to vendor zstd? Good question.
First, zstd is relatively new and not widely available yet. If we
didn't vendor zstd or distribute it with Mercurial, most users likely
wouldn't have zstd installed or even available to install. What good
is a feature if you can't use it? Vendoring and distributing the zstd
sources gives us the highest liklihood that zstd will be available to
Mercurial installs.
Second, the Python bindings to zstd (which will be vendored in a
separate changeset) make use of zstd APIs that are only available
via static linking. One reason they are only available via static
linking is that they are unstable and could change at any time.
While it might be possible for the Python bindings to attempt to
talk to different versions of the zstd C library, the safest thing to
do is link against a specific, known-working version of zstd. This
is why the Python zstd bindings themselves vendor zstd and why we
must as well. This also explains why the added files are in a
"python-zstandard" directory.
The added files are from the 1.1.1 release of zstd (Git commit
4c0b44f8ced84c4c8edfa07b564d31e4fa3e8885 from
https://github.com/facebook/zstd) and are added without modifications.
Not all files from the zstd "distribution" have been added. Notably
missing are files to support interacting with "legacy," pre-1.0
versions of zstd. The decision of which files to include is made by
the upstream python-zstandard project (which I'm the author of). The
files in this commit are a snapshot of the files from the 0.5.0
release of that project, Git commit
e637c1b214d5f869cf8116c550dcae23ec13b677 from
https://github.com/indygreg/python-zstandard.
This broke in c7236da49964 due to a refactored manifest API.
The fix is a bit hacky - perfbdiff doesn't yet support tree manifests
for example. But it gets the job done.
A test has been added for --alldata so this doesn't happen again.
Airspeed velocity (ASV) is a python framework for benchmarking Python packages
over their lifetime. The results are displayed in an interactive web frontend.
Add ASV benchmarks for mercurial that use contrib/perf.py extension that could
be run against multiple reference repositories.
The benchmark suite now includes revsets from contrib/base-revsets.txt with
variants, perftags, perfstatus, perfmanifest and perfheads.
Installation requires asv>=0.2, python-hglib and virtualenv
This is part of PerformanceTrackingSuitePlan
https://www.mercurial-scm.org/wiki/PerformanceTrackingSuitePlan
Now that all the functionality has been moved to manifestlog/manifestrevlog/etc,
we can finally change all the uses of repo.manifest to use the new versions. A
future diff will then delete repo.manifest.
One additional change in this commit is to change repo.manifestlog to be a
@storecache property instead of @property. This is required by some uses of
repo.manifest require that it be settable (contrib/perf.py and the static http
server). We can't do this in a prior change because we can't use @storecache on
this until repo.manifest is no longer used anywhere.
The --all argument changes the behavior of `perfbdiff` to pull
in fulltext revision pairs for all changes related to a changeset.
The p1 and p2 manifests will be bdiffed against current. Every file
that changed between p1 and current will have its file revisions
loaded and bdiffed.
This mode of operation effectively measured the bdiff time required
for `hg commit`.
Before, we only supported benchmarking a single pair of texts
with bdiff. We want to enable feeding larger corpora into this
benchmark. So rewrite the code to support that.
Now that the 'vfs' classes moved in their own module, lets use the new module
directly. We update code iteratively to help with possible bisect needs in the
future.
SIGUSR1 and SIGUSR2 are reserved for user-defined behaviors. They may be
redefined by an hg extension [1], but cannot be easily redefined for chg.
Since the default behavior (kill) is not that useful for chg, let's forward
them to hg, hoping it got redefined there and could be more useful.
[1] https://bitbucket.org/facebook/hg-experimental/commits/e7c883a465
Now that the feature no longer lives in the extension, we document it in the
help of the core config. This include the new 'ui.color' option introduced in
the previous changesets.
As a result the color extensions can now be deprecated.
This is a documentation patch only; color is still disabled by default.
I'm adding some performance logging to ui.write - this benchmark lets us
confirm that the cost of that logging is acceptably low.
At this point, the microbenchmark on Linux over SSH shows:
! wall 3.213560 comb 0.410000 user 0.350000 sys 0.060000 (best of 4)
while on the Mac locally, it shows:
! wall 0.342325 comb 0.180000 user 0.110000 sys 0.070000 (best of 20)
Commit 3054ae3a66112970a091d3939fee32c2d0c1a23e from
https://github.com/indygreg/python-zstandard is imported without
modifications (other than removing unwanted files).
The vendored zstd library within has been upgraded from 1.1.2 to
1.1.3. This version introduced new APIs for threads, thread
pools, multi-threaded compression, and a new dictionary
builder (COVER). These features are not yet used by
python-zstandard (or Mercurial for that matter). However,
that will likely change in the next python-zstandard release
(and I think there are opportunities for Mercurial to take
advantage of the multi-threaded APIs).
Relevant to Mercurial, the CFFI bindings are now fully
implemented. This means zstd should "just work" with PyPy
(although I haven't tried). The python-zstandard test suite also
runs all tests against both the C extension and CFFI bindings to
ensure feature parity.
There is also a "decompress_content_dict_chain()" API. This was
derived from discussions with Yann Collet on list about alternate
ways of encoding delta chains.
The change most relevant to Mercurial is a performance enhancement in
the simple decompression API to reuse a data structure across
operations. This makes decompression of multiple inputs significantly
faster. (This scenario occurs when reading revlog delta chains, for
example.)
Using python-zstandard's bench.py to measure the performance
difference...
On changelog chunks in the mozilla-unified repo:
decompress discrete decompress() reuse zctx
1.262243 wall; 1.260000 CPU; 1.260000 user; 0.000000 sys 170.43 MB/s (best of 3)
0.949106 wall; 0.950000 CPU; 0.950000 user; 0.000000 sys 226.66 MB/s (best of 4)
decompress discrete dict decompress() reuse zctx
0.692170 wall; 0.690000 CPU; 0.690000 user; 0.000000 sys 310.80 MB/s (best of 5)
0.437088 wall; 0.440000 CPU; 0.440000 user; 0.000000 sys 492.17 MB/s (best of 7)
On manifest chunks in the mozilla-unified repo:
decompress discrete decompress() reuse zctx
1.367284 wall; 1.370000 CPU; 1.370000 user; 0.000000 sys 274.01 MB/s (best of 3)
1.086831 wall; 1.080000 CPU; 1.080000 user; 0.000000 sys 344.72 MB/s (best of 3)
decompress discrete dict decompress() reuse zctx
0.993272 wall; 0.990000 CPU; 0.990000 user; 0.000000 sys 377.19 MB/s (best of 3)
0.678651 wall; 0.680000 CPU; 0.680000 user; 0.000000 sys 552.06 MB/s (best of 5)
That should make reads on zstd revlogs a bit faster ;)
# no-check-commit
According to the specification [1], $XDG_RUNTIME_DIR should be ignored
unless:
The directory MUST be owned by the user, and he MUST be the only one
having read and write access to it. Its Unix access mode MUST be 0700.
This patch adds a check and ignores it if it does not meet part of the
criteria.
[1]: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
Previously, the code was similar to what revlog._chunks() was doing,
which took a raw data segment and delta chain, obtained buffers for
the raw revlog chunks within, and decompressed them.
This commit splits the "get raw chunks" action from "decompress." The
goal of this change is to more accurately measurely decompression
performance.
On a ~50k deltachain for a manifest in mozilla-central:
! full
! wall 0.430548 comb 0.440000 user 0.410000 sys 0.030000 (best of 24)
! deltachain
! wall 0.016053 comb 0.010000 user 0.010000 sys 0.000000 (best of 181)
! read
! wall 0.008078 comb 0.010000 user 0.000000 sys 0.010000 (best of 362)
! rawchunks
! wall 0.033785 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
! decompress
! wall 0.327126 comb 0.320000 user 0.320000 sys 0.000000 (best of 31)
! patch
! wall 0.032391 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)
! hash
! wall 0.012587 comb 0.010000 user 0.010000 sys 0.000000 (best of 233)
Now, all URL in Mercurial source tree should refer mercurial-scm.org
domain instead of selenic.com.
*.po files are ignored in this patch, because they might contain
msgid/msgstr coming from old source files.
This ignorance seems safe enough, because such msgstr should be
ignored at runtime, because:
- msgid corresponded to it should be invalid, or
- msgstr itself should be marked as fuzzy at synchronized to recent hg.pot
If any additional examination for *.po files is needed in the future,
let i18n/check-translation.py achieve such examination.
BTW, some binary files (e.g. *.png) are meaningless for checking
reference to old domain in this patch, but aren't ignored like as *.po
files, because excluding multiple suffixes is difficult for regexp
matching.
Before this patch, check-code.py applies filtering on the file
content, to which filtering of previous check is already applied.
This might hide issues, which should be detected by a subsequent check
in "checks" list.
Fortunately, this problem hasn't appeared, because there is no
overlapping of filename matching (examined in the order below).
1. *.py or *.cgi
2. test-* (not *.t suffix)
3. *.c or *.h
4. *.t
5. *.txt
6. *.tmpl
For example, adding a test, which wants to examine raw comment text in
*.py files, at the end of current "checks" list doesn't work as
expected, because a filter for *.py files normalizes comment text in
them.
Putting such test at the beginning of "checks" list also resolves this
problem, but such dependence on the order decreases maintainability of
check-code.py itself.
This patch discards filtering result of previous check at the
beginning of each checks, for independence of each checks.
The old @property on manifestlog was broken. It meant that we would always
recreate the manifestlog instance, which meant the cache was never hit. Since
we'll eventually remove repo.manifest and make manifestlog the only property,
let's go ahead and make manifestlog the @storecache property, have manifestlog
own the manifest instance, and have repo.manifest refer to it via manifestlog.
This means all accesses go through repo.manifestlog, which is now invalidated
correctly.
On systems with large repositories and slow disks,
the calls to 'hg status' make autocomplete annoyingly slow.
This fix makes it possible to avoid the slowdown.
Before this patch, "hg perftags" command doesn't measure performance
of "repo.tags()" correctly, because it doesn't clear tags cache
correctly.
a043ed82a5cd replaced repo._tags with repo._tagscache, but didn't
change the code path to clear tags cache in perftags() at that time.
BTW, full history of "tags cache" is:
- b8d757d45f24 (or 0.6) introduced repo.tagscache as the first "tags cache"
- 4cbf51c74e8c (or 1.4) replaced repo.tagscache with repo._tags
- a043ed82a5cd (or 2.0) replaced repo._tags with repo._tagscache
- 04c204f1ed65 (or 2.5) made repo._tagscache filteredpropertycache
To make perftags clear tags cache correctly, and to increase
"historical portability" of perftags, this patch examines existence of
attributes in repo object, and guess appropriate procedure to clear
tags cache.
To avoid examining existence of attributes at each repetition, this
patch makes repocleartagscachefunc() return the function, which
actually clears tags cache.
mozilla-central repo (85 tags on 308365 revs) with each Mercurial
version between before and after this patch.
==== ========= =========
ver before after
==== ========= =========
1.9 0.476062 0.466464
------- *1 -------
2.0 0.346309 0.458327
2.1 0.343106 0.454489
------- *2 -------
2.2 0.069790 0.071263
2.3 0.067829 0.069340
2.4 0.068075 0.069573
------- *3 -------
2.5 0.021896 0.022406
2.6 0.021900 0.022374
2.7 0.021883 0.022379
2.8 0.021949 0.022327
2.9 0.021877 0.022330
3.0 0.021860 0.022314
3.1 0.021869 0.022669
3.2 0.021831 0.022668
3.3 0.021809 0.022691
3.4 0.021861 0.022916
3.5 0.019335 0.020749
3.6 0.019319 0.020866
3.7 0.018781 0.020251
------- *4 -------
3.8 0.068262 0.072558
3.9 0.069682 0.073773
==== ========= =========
(*1) repo._tags was replaced with repo._tagscache at this point
"repo._tags = None" in perftags "before" this patch doesn't clear
tags cache for Mercurial 2.0 or later. This causes significant
gap of "before" between 1.9 and 2.0 .
(*2) I'm not sure about significant gap at this point, but release
note of 2.2 described "a number of significant performance
improvements for large repositories"
(*3) filtered changelog was cached in repoview as repoview.changelog
at this point (by 131b01a4654d)
This avoids calculation of filtered changelog at each repetition
of t().
(*4) calculation of filtered changelog was included into wall time at
this point (by adf01efe43a5), again
See below for detail about this significant gap:
https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-April/083410.html
Before this patch, using ui.configint() prevents perf.py from
measuring performance with Mercurial earlier than 1.9 (or
12e7e9fbf243), because ui.configint() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
This patch replaces ui.configint() invocations by newly introduced
getint().
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of ui.configint() in perf.py.
BTW, this patch doesn't choose adding configint() method at runtime by
replacing ui.__class__ like below, even though this is the recommended
way to modern Mercurial extensions.
def uisetup(ui):
if not util.safehasattr(ui, 'configint'):
class uiwrap(ui.__class__):
def configint(self, section, name, ....):
....
ui.__class__ = uiwrap
Because changes to ui.__class__ by uisetup() of loaded extension have
been propagated since 1.6.1 (or 07a6e7bd0cc1), the recommended way
above doesn't work as expected with Mercurial earlier than it.
Before this patch, referring ui.ferr prevents perf.py from measuring
performance with Mercurial earlier than 1.9 (or bac01d164cc1), because
ui.ferr isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 1.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
Before this patch, using ui.formatter() prevents perf.py from
measuring performance with Mercurial earlier than 2.2 (or
045c8375c770), because ui.formatter() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 2.2 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
This patch defines formatter class locally, and use it instead of the
value returned by ui.formatter(), if perf.py is used with Mercurial
earlier than 2.2.
In this case, we don't need to think about -T/--template option for
formatter, because previous patch made -T/--template disabled for
perf.py with Mercurial earlier than 3.2 (or 44a82ed65df7).
Before this patch, using svfs prevents perf.py from measuring
performance of Mercurial earlier than 2.3 (or 12df7401e8cd), because
svfs isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 2.3 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
To get appropriate vfs-like object to access files under .hg/store,
this patch adds getsvfs() (and also getvfs(), for future use).
To avoid examining existence of attribute at each repetition while
measuring performance, getsvfs() is invoked outside the function to be
called repeatedly.
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of repo.(vfs|svfs|opener|sopener) in perf.py.
Mercurial 2.5 (or 0eb7dcc721cb) introduced "perfbranchmap" command,
and tried to avoid actual writing branch cache out by replacing
write() of branchcache class in branchmap.py with no-op function
(probably, for elimination of noisy and heavy file I/O factor).
But its implementation isn't correct, because 0eb7dcc721cb replaced
not branchmap.branchcache.write() but branchmap.write(). The latter
doesn't exist, even at that change.
To avoid actual writing branch cache out correctly, this patch
replaces branchmap.branchcache.write() with no-op function.
To detect mistake of replacement or change of API in the future
quickly, this patch uses safeattrsetter() instead of direct attribute
assignment. For similarity between replacements, this patch also
changes replacement of branchmap.read().
In this patch, replacement of read()/write() can run safely outside
"try" block, because two safeattrsetter() invocations ensure that
replacement doesn't cause exception.
FYI, the table below compares "base" filter wall time of perfbranchmap
on recent mozilla-central repo with each Mercurial version between
before and after this patch.
==== ========= =========
ver before after
==== ========= =========
2.5 18.492334 18.232455
2.6 18.733858 18.156702
2.7 18.245598 18.349210
2.8 18.289070 18.528422
2.9 17.572742 16.989655
3.0 17.406953 17.615012
3.1 17.228419 17.689805
3.2 17.862961 17.718367
3.3 2.632110 2.707960
3.4 3.285683 3.272060
3.5 3.370141 3.352176
3.6 3.366939 3.242455
3.7 3.300778 3.367328
3.8 3.300132 3.267298
3.9 3.418996 3.370265
==== ========= =========
IMHO, there is no serious overlooking performance regression.
Before this patch, using branchmap.subsettable prevents perfbranchmap
from measuring performance of Mercurial earlier than 2.9 (or
aad678a92970), because aad678a92970 moved subsettable from repoview.py
to branchmap.py, even though there are some code paths for Mercurial
earlier than 2.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
To get subsettable from appropriate module, this patch examines
existence of subsettable in branchmap and repoview.
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of subsettable attribute in perf.py.
Referring not-existing attribute immediately causes failure, but
assigning a value to such attribute doesn't.
For example, perf.py has code paths below, which assign a value to
not-existing attribute. This causes incorrect performance measurement,
but these code paths are executed successfully.
- "repo._tags = None" in perftags()
recent Mercurial has tags cache information in repo._tagscache
- "branchmap.write = lambda repo: None" in perfbranchmap()
branchmap cache is written out by branchcache.write() in branchmap.py
"util.safehasattr() before assignment" can avoid this issue, but might
increase mistake at "copy & paste" attribute name or so.
To centralize (1) examining existence of, (2) assigning a value to,
and (3) restoring an old value to the attribute, this patch introduces
safeattrsetter(). This is used to replace direct attribute assignment
in subsequent patches.
Encapsulation of restoring is needed to completely remove direct
attribute assignment from perf.py, even though restoring isn't needed
so often.
Otherwise no code transformation would be applied to the modules which are
imported only by imp.load_module().
This change means modules are imported from PYTHONPATH, not from the paths
given by command arguments. This isn't always correct, but seems acceptable.
Here are the relevant symbol descriptions from [0]
optspec:...
The colon indicates handling for one or more arguments to the option; if it
is not present, the option is assumed to take no arguments.
-optname+
The first argument may appear immediately after optname in the same word,
or may appear as a separate word after the option. For example, ‘-foo+:...’
specifies that the completed option and argument will look like either
‘-fooarg’ or ‘-foo arg’.
-optname=
The argument may appear as the next word, or in same word as the option
name provided that it is separated from it by an equals sign, for example
‘-foo=arg’ or ‘-foo arg’.
There are 3 types of changes in this patch:
- addition of '=': means that '--repository ~/foo' can now also be spelled as
'--reporitory=~/foo'
- addition of '+': means '-r 0' can now also be spelled as '-r0'
- removal of '+', '=' and ':': means that '-u|--untrusted' doesn't take any
arguments, so '-uq' is definitely '-u -q' and not '--untrusted=q'
Occasionally, ':' had to be added together with '=' and '+'.
This patch is mostly just making zsh_completion file look more like zsh's own
hg completion file so that we can more easily take improvements from them or
send patches to them. Some context for this patch from zsh project: [2] [3].
[0]: http://zsh.sourceforge.net/Doc/Release/Completion-System.html
[1]: https://sourceforge.net/p/zsh/code/ci/master/tree/Completion/Unix/Command/_hg
[2]: 92584634d3/
[3]: http://www.zsh.org/mla/workers/2015/msg02551.html
This command can be used for testing the performance of producing the
changelog portion of a changegroup.
We could use additional perf* commands for testing other parts of
changegroup. Those can be written another time, when they are needed.
(And those may want to refactor the changegroup generation API so code
can be reused.) Speaking of code reuse, yes, this command does reinvent
a small wheel. I didn't want to scope bloat to change the changegroup
API because that will invite bikeshedding.
The Mercurial wire protocol is under-documented. This includes a lack
of source docstrings and comments as well as pages on the official
wiki.
This patch adds the beginnings of "internals" documentation on the
wire protocol.
The documentation should have nearly complete coverage on the
lower-level parts of the protocol, such as the different transport
mechanims, how commands and arguments are sent, capabilities, and,
of course, the commands themselves.
As part of writing this documentation, I discovered a number of
deficiencies in the protocol and bugs in the implementation. I've
started sending patches for some of the issues. I hope to send a lot
more.
This patch starts with the scaffolding for a new internals page.
I've caught multiple extensions in the wild lying about being
'internal', so it's time to move the goalposts on people. Goalpost
moving will continue until third party extensions stop trying to
defeat the system.
These signals are meant to send to a process group, instead of a single
process: SIGINT is usually emitted by the terminal and sent to the process
group. SIGHUP usually happens to a process group if termination of a process
causes that process group to become orphaned.
Before this patch, chg will only forward these signals to the single server
process. This patch changes it to the server process group.
This will allow us to properly kill processes started by the forked server
process, like a ssh process. The behavior difference can be observed by
setting SSH_ASKPASS to a dummy script doing "sleep 100" and then run
"chg push ssh://dest-need-password-auth". Before this patch, the first Ctrl+C
will kill the hg process while ssh-askpass and ssh will remain alive. This
patch will make sure they are killed properly.
We recently discovered a case in production that chg uses 100% CPU and is
trying to read data forever:
recvfrom(4, "", 1814012019, 0, NULL, NULL) = 0
Using gdb, apparently readchannel() got wrong data. It was reading in an
infinite loop because rsize == 0 does not exit the loop, while the server
process had ended.
(gdb) bt
#0 ... in recv () at /lib64/libc.so.6
#1 ... in readchannel (...) at /usr/include/bits/socket2.h:45
#2 ... in readchannel (hgc=...) at hgclient.c:129
#3 ... in handleresponse (hgc=...) at hgclient.c:255
#4 ... in hgc_runcommand (hgc=..., args=<optimized>, argsize=<optimized>)
#5 ... in main (argc=...486922636, argv=..., envp=...) at chg.c:661
(gdb) frame 2
(gdb) p *hgc
$1 = {sockfd = 4, pid = 381152, ctx = {ch = 108 'l',
data = 0x7fb05164f010 "st):\nTraceback (most recent call last):\n"
"Traceback (most recent call last):\ne", maxdatasize = 1814065152,"
" datasize = 1814064225}, capflags = 16131}
This patch addresses the infinite loop issue by detecting continuously empty
responses and abort in that case.
Note that datasize can be translated to ['l', ' ', 'l', 'a']. Concatenate
datasize and data, it forms part of "Traceback (most recent call last):".
This may indicate a server-side channeledoutput issue. If it is a race
condition, we may want to use flock to protect the channels.
This patch makes an extra check pattern to be prepared by
"_preparepats()" as similarly as existing patterns, if it is added to
"checks" array before invocation of "main()" in check-code.py.
This is a part of preparation for adding check-code.py extra checks by
another python script in subsequent patch.
This is also useful for SkeletonExtensionPlan.
https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan
demandimport of early Mercurial loads an imported module immediately,
if a module is imported absolutely by "from a import b" style. Recent
perf.py satisfies this condition, because it does:
- have "from __future__ import absolute_import" line
- use "from a import b" style for modules in "mercurial" package
Before this patch, importing modules below prevents perf.py from being
loaded by earlier Mercurial, because these aren't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9.
- branchmap 2.5 (or e3354ba12eef)
- repoview 2.5 (or 7d207cb7e38a)
- obsolete 2.3 (or b50a017cd9ac)
- scmutil 1.9 (or 065064cdde5f)
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with Mercurial earlier than 1.8 (or
1299f0c14572).
After this patch, "mercurial.error" is the only blocker in "from
mercurial import" statement for loading perf.py with Mercurial earlier
than 1.2. This patch ignores it, because just importing it separately
isn't enough.
The BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer has been merged into
http.server in python 3. All of them has been merged as util.httpserver to use
in both python 2 and 3. This patch adds a regex to check-code to warn against
the use of BaseHTTPServer. Moreover this patch also includes updates to lower
part of test-check-py3-compat.t which used to remain unchanged.
The most painful part of ensuring Python code runs on both Python 2
and 3 is string encoding. Making this difficult is that string
literals in Python 2 are bytes and string literals in Python 3 are
unicode. So, to ensure consistent types are used, you have to
use "from __future__ import unicode_literals" and/or prefix literals
with their type (e.g. b'foo' or u'foo').
Nearly every string in Mercurial is bytes. So, to use the same source
code on both Python 2 and 3 would require prefixing nearly every
string literal with "b" to make it a byte literal. This is ugly and
not something mpm is willing to do at this point in time.
This patch implements a custom module loader on Python 3 that performs
source transformation to convert string literals (unicode in Python 3)
to byte literals. In effect, it changes Python 3's string literals to
behave like Python 2's.
In addition, the module loader recognizes well-known built-in
functions (getattr, setattr, hasattr) and methods (encode and decode)
that barf when bytes are used and prevents these from being rewritten.
This prevents excessive source changes to accommodate this change
(we would have to rewrite every occurrence of these functions passing
string literals otherwise).
The module loader is only used on Python packages belonging to
Mercurial.
The loader works by tokenizing the loaded source and replacing
"string" tokens if necessary. The modified token stream is
untokenized back to source and loaded like normal. This does add some
overhead. However, this all occurs before caching: .pyc files will
cache the transformed version. This means the transformation penalty
is only paid on first load.
As the extensive inline comments explain, the presence of a custom
source transformer invalidates assumptions made by Python's built-in
bytecode caching mechanism. So, we have to wrap bytecode loading and
writing and add an additional header to bytecode files to facilitate
additional cache validation when the source transformations
change in the future.
There are still a few things this code doesn't handle well, namely
support for zip files as module sources and for extensions. Since
Mercurial doesn't officially support Python 3 yet, I'm inclined to
leave these as to-do items: getting a basic module loading mechanism
in place to unblock further Python 3 porting effort is more important
than comprehensive module importing support.
check-py3-compat.py has been updated to ignore frames. This is
necessary because CPython has built-in code to strip frames from the
built-in importer. When our custom code is present, this doesn't work
and the frames get all messed up. The new code is not perfect. It
works for now. But once you start chasing import failures you find
some edge cases where the files aren't being printed properly. This
only burdens people doing future Python 3 porting work so I'm inclined
to punt on the issue: the most important thing is for the source
transforming module loader to land.
There was a bit of churn in test-check-py3-compat.t because we now
trip up on str/unicode/bytes failures as a result of source
transformation. This is unfortunate but what are you going to do.
It's worth noting that other approaches were investigated.
We considered using a custom file encoding whose decode() would
apply source transformations. This was rejected because it would
require each source file to declare its custom Mercurial encoding.
Furthermore, when changing the source transformation we'd need to
version bump the encoding name otherwise the module caching layer
wouldn't know the .pyc file was invalidated. This would mean mass
updating every file when the source transformation changes. Yuck.
We also considered transforming at the AST layer. However, Python's
ast module is quite gnarly and doing AST transforms is quite
complicated, even for trivial rewrites. There are whole Python packages
that exist to make AST transformations usable. AST transforms would
still require import machinery, so the choice was basically to
perform source-level, token-level, or ast-level transforms.
Token-level rewriting delivers the metadata we need to rewrite
intelligently while being relatively easy to understand. So it won.
General consensus seems to be that this approach is the best available
to avoid bulk rewriting of '' to b''. However, we aren't confident
that this approach will never be a future maintenance burden. This
approach does unblock serious Python 3 porting efforts. So we can
re-evaulate once more work is done to support Python 3.
Prior to this, check-code wouldn't notice things like (glob)
annotations or similar in a test if they were after a # anywhere in
the line. This resolves a defect in a future change, and also exposed
a couple of small spots that needed some attention.
Before this patch, using cmdutil.command() for "@command" annotation
prevents perf.py from being loaded by Mercurial earlier than 1.9 (or
d4096ee63f8e), because cmdutil.command() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
In addition to it, "norepo" option of command annotation has been
available since 3.1 (or 9cbb59f03d57), and this is another blocker for
loading perf.py with earlier Mercurial.
============ ============ ======
command of
hg version cmdutil norepo
============ ============ ======
3.1 or later o o
1.9 or later o x
earlier x x
============ ============ ======
This patch defines "command()" for annotation locally as below:
- define wrapper of existing cmdutil.command(), if cmdutil.command()
doesn't support "norepo"
(for Mercurial earlier than 3.1)
- define full command() locally with minimum function, if
cmdutil.command() isn't available at runtime
(for Mercurial earlier than 1.9)
This patch also defines parsealiases() locally without examining
whether it is available or not, because it is small enough to define
locally.
Before this patch, referring commands.formatteropts prevents perf.py
from being loaded by Mercurial earlier than 3.2 (or 44a82ed65df7),
because it isn't available in such Mercurial, even though formatting
itself has been available since 2.2 (or 045c8375c770).
In addition to it, there are some code paths for Mercurial earlier
than 3.2. For example, setting "_prereadsize" attribute in perfindex()
and perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
This patch uses empty option list as formatteropts, if it isn't
available in commands module at runtime.
Disabling -T/--template option for earlier Mercurial should be
reasonable, because:
- since 0c03b2a1206a, -T/--template for formatter has been available
- since 44a82ed65df7, commands.formatteropts has been available
- the latter revision is direct child of the former
Before this patch, referring commands.debugrevlogopts prevents perf.py
from being loaded by Mercurial earlier than 3.7 (or 1a89336e03aa),
because it isn't available in such Mercurial, even though
cmdutil.openrevlog(), a user of these options, has been available
since 1.9 (or f32fd7cab084).
In addition to it, there are some code paths for Mercurial earlier
than 3.7. For example, setting "_prereadsize" attribute in perfindex()
and perfnodelookup() is effective only with hg earlier than 1.8 (or
1299f0c14572).
But just "using locally defined revlog option list" might cause
unexpected behavior at runtime. If --dir option is specified to
cmdutil.openrevlog() of Mercurial earlier than 3.5 (or e3ab0b30c05e),
it is silently ignored without any warning or so.
============ ============ ===== ===============
debugrevlogopts
hg version openrevlog() --dir of commands
============ ============ ===== ===============
3.7 or later o o o
3.5 or later o o x
1.9 or later o x x
earlier x x x
============ ============ ===== ===============
Therefore, this patch does:
- use locally defined option list, if commands.debugrevlogopts isn't
available (for Mercurial earlier than 3.7)
- wrap cmdutil.openrevlog(), if it is ambiguous whether
cmdutil.openrevlog() can recognize --dir option correctly
(for Mercurial earlier than 3.5)
This wrapper function aborts execution, if:
- --dir option is specified, and
- localrepository doesn't have "dirlog" attribute, which indicates
that localrepository has a function for '--dir'
BTW, extensions.wrapfunction() has been available since 1.1 (or
56ba0b824b91), and this seems old enough for "historical
portability" of perf.py, which has been available since 1.1 (or
bca5e7427e89).
Before this patch, using util.safehasattr() prevents perf.py from
being loaded by Mercurial earlier than 1.9.3 (or 36f152380d7d),
because util.safehasattr() isn't available in such Mercurial, even
though there are some code paths for Mercurial earlier than 1.9.3.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with Mercurial earlier than 1.8 (or
1299f0c14572).
This patch is a preparation for using util.safehasattr() safely in
subsequent patches.
This patch defines util.safehasattr() forcibly without examining
whether it is available or not, because:
- examining existence of "safehasattr" safely itself needs similar logic
- safehasattr() is small enough to define locally
The httplib library is renamed to http.client in python 3. So the
import is conditionalized and a test is added in check-code to warn
to use util.httplib
If the user press 'q' to leave the 'less' pager, it is expected to end the
hg process immediately. We currently rely on SIGPIPE for this behavior. But
SIGPIPE won't arrive if we don't write anything (like doing heavy
computation, reading from network etc). If that happens, the user will feel
that the hg process just hangs.
The patch address the issue by adding a SIGCHLD signal handler and sends
SIGPIPE to the server as soon as the pager exits.
This is also an issue with hg's pager implementation.
Rebuilding translation table (256 size) at each repquote() invocations
is redundant.
For example, this patch decreases user time of command invocation
below from 18.297s to 13.445s (about -27%) on a Linux box. This
command is main part of test-check-code.t.
hg locate | xargs python contrib/check-code.py --warnings --per-file=0
This patch adds "_repquote" prefix to functions and variables factored
out from repquote() to avoid conflict of name in the future.
Before this patch, "missing _() in ui message" rule overlooks
translatable message, which starts with other than alphabet.
To detect "missing _() in ui message" more exactly, this patch
improves the regexp with assumptions below.
- sequence consisting of below might precede "translatable message"
in same string token
- formatting string, which starts with '%'
- escaped character, which starts with 'b' (as replacement of '\\'), or
- characters other than '%', 'b' and 'x' (as replacement of alphabet)
- any string tokens might precede a string token, which contains
"translatable message"
This patch builds an input file, which is used to examine "missing _()
in ui message" detection, before '"$check_code" stringjoin.py' in
test-contrib-check-code.t, because this reduces amount of change churn
in subsequent patch.
This patch also applies "()" instead of "_()" on messages below to
hide false-positives:
- messages for ui.debug() or debug commands/tools
- contrib/debugshell.py
- hgext/win32mbcs.py (ui.write() is used, though)
- mercurial/commands.py
- _debugchangegroup
- debugindex
- debuglocks
- debugrevlog
- debugrevspec
- debugtemplate
- untranslatable messages
- doc/gendoc.py (ReST specific text)
- hgext/hgk.py (permission string)
- hgext/keyword.py (text written into configuration file)
- mercurial/cmdutil.py (formatting strings for JSON)
When auto-completing hg commands, aliases are listed, but not the available
switches for an alias, because `HGPLAIN=1` filters these out. Add a
`HGPLAINEXCEPT=alias` exception to resolve this.
We make heavy use of aliases that drive hg log with custom revsets, sorting and
the -G switch, but want our users to be able to auto-complete any additional
command-line switches.
Before this patch, fromlocalfunc() assumes that "module" attribute of
ast.ImportFrom is None for "from . import a", and Python 2.7.x
satisfies this assumption.
On the other hand, with Python 2.6.x, "module" attribute of
ast.ImportFrom is an empty string for "from . import a", and this
causes failure of test-check-module-imports.t.
Our signal handlers forward signals to the server process, but it will
disappear soon after hgc_close(). So we should unregister handlers before
hgc_close(). Otherwise chg would abort due to kill(perrpid, sig) failure.
The problem is spotted by SIGWINCH while waiting pager termination.
Before this patch, chg will give up when it cannot connect to the new server
within 10 seconds. If the host has high load during that time, 10 seconds
is not enough.
This patch makes it adjustable using the CHGTIMEOUT environment variable.
Before this patch, chg uses the old pager behavior (pre 55f6f7fb60d2), which
executes pager in the main process. The user will see the exit code of the
pager, instead of the hg command.
Like 55f6f7fb60d2, this patch fixes the behavior by executing the pager in
the child process, and wait for it at the end of the main process.
This patch makes repquote() distinguish more characters below, as a
preparation for exact detection in subsequent patch.
- "%" as "%"
- "\\" as "b"(ackslash)
- "*" as "A"(sterisk)
- "+" as "P"(lus)
- "-" as "M"(inus)
Characters other than "%" don't use itself as replacement, because
they are treated as special ones in regexp.
d19c9c93ad10 tried to detect '.. note::' more exactly. But
implementation of it seems not correct, because:
- fromc.find(c) returns -1 for other than "." and ":"
- tochr[-1] returns "q" for such characters, but
- expected result for them is "o"
This patch uses dict to manage replacement instead of replacing
str.find() by str.index(), for improvement/refactoring in subsequent
patches. Examination by fixedmap is placed just after examination for
' ' and '\n', because subsequent patch will integrate the latter into
the former.
This patch also changes regexp for 'string join across lines with no
space' rule, and adds detailed test for it, because d19c9c93ad10 did:
- make repquote() distinguish "." (as "p") and ":" (as "q") from
others (as "o"), but
- not change this regexp without any reason (in commit log, at
least), even though this regexp depends on what "o" means
This patch doesn't focuses on deciding whether "." and/or ":" should
be followed by whitespace or not in translatable messages.
It doesn't make sense that (a) is allowed whereas (b) is disallowed.
a) from mercurial import hg
from mercurial.i18n import _
b) from . import hg
from .i18n import _
Since (b) is banned, we should do the same for (a) for consistency.
a) from mercurial import hg
from mercurial.i18n import _
b) from . import hg
from .i18n import _
This is needed so that launchpad and friends have a unique version number for
each distroseries (trusty, wily, xenial, etc). It was discovered when trying to
upload 3.8 to launchpad.
Debian maintainers already have this and lintian warns us about not
listing 'wish' as a dependency or suggestion so this patch does indeed
just that. The issue, by the way, is that we are shipping hgk (which is
written in tcl/tk) so we should be good citizens and list wish (a meta
package for tcl/tk) as a dependency.
Instead of using bdist_mpkg, we use the modern Apple-provided tools to
build an OS X Installer package directly. This has several advantages:
* Avoids bdist_mpkg which seems to be barely maintained and is hard to
use.
* Creates a single unified .pkg instead of a .mpkg.
* The package we produce is in the modern, single-file format instead of
a directory bundle that we have to zip up for download.
In addition, this way of building the package now correctly:
* Installs the manpages, bringing the `make osx`-generated package in
line with the official Mac packages we publish on the website.
* Installs files with the correct permissions instead of encoding the
UID of the user who happened to build the package.
Thanks to Augie for updating the test expectations.
As we don't use sockdirfd yet, this is the simplest workaround to compile chg
on old Unices where AT_FDCWD does not exist. Foozy pointed out Mac OS X 10.10
is required for AT_FDCWD as well as xxxat() functions.
This bug is hidden by the current bias towards matches at the
beginning of the file. When this bias is tweaked later to address
recursion balancing, the normalization code could cause the next block
to shrink to a negative length, thus creating invalid delta chunks. We
add checks here to disallow that.
This bug requires test cases that are an awkwardly large size for the test
suite, but is very rapidly picked up by the included torture tester.
So far fromlocal recognizes relative imports of the form:
from . import D
from .. import E
It wasn't prepared for recognizing relative imports like:
from ..F import G
The bug was not found so far because all relative imports starting
from the parent was in the list of allowsymbolicimports like:
from ..i18n import
from ..node import
