When doing streaming compression with zlib, zlib appears to emit chunks
with data after ~20-30kb on average is available. In other words, most
calls to compress() return an empty string. On the mozilla-unified repo,
only 48,433 of 921,167 (5.26%) of calls to compress() returned data.
In other words, we were sending hundreds of thousands of empty chunks
via a generator where they touched who knows how many frames (my guess
is millions). Filtering out the empty chunks from the generator
cuts down on overhead.
In addition, we were previously feeding 8kb chunks into zlib
compression. Since this function tends to emit *compressed* data after
20-30kb is available, it would take several calls before data was
produced. We increase the amount of data fed in at a time to 32kb.
This reduces the number of calls to compress() from 921,167 to
115,146. It also reduces the number of output chunks from 48,433 to
31,377. This does increase the average output chunk size by a little.
But I don't think this will matter in most scenarios.
The combination of these 2 changes appears to shave ~6s CPU time
or ~3% from a server serving the mozilla-unified repo.
Currently, running `hg serve --profile` doesn't yield anything useful:
when the process is terminated the profiling output displays results
from the main thread, which typically spends most of its time in
select.select(). Furthermore, it has no meaningful results from
mercurial.* modules because the threads serving HTTP requests don't
actually get profiled.
This patch teaches the hgweb wsgi applications to profile individual
requests. If profiling is enabled, the profiler kicks in after
HTTP/WSGI environment processing but before Mercurial's main request
processing.
The profile results are printed to the configured profiling output.
If running `hg serve` from a shell, they will be printed to stderr,
just before the HTTP request line is logged. If profiling to a file,
we only write a single profile to the file because the file is not
opened in append mode. We could add support for appending to files
in a future patch if someone wants it.
Per request profiling doesn't work with the statprof profiler because
internally that profiler collects samples from the thread that
*initially* requested profiling be enabled. I have plans to address
this by vendoring Facebook's customized statprof and then improving
it.
Before this patch, the HTTP transport protocol would always zlib
compress certain responses (notably "getbundle" wire protocol commands)
at zlib compression level 6.
zlib can be a massive CPU resource sink for servers. Some server
operators may wish to reduce server-side CPU requirements while
requiring more bandwidth. This is common on corporate intranets, for
example. Others may wish to use more CPU but reduce bandwidth.
This patch introduces a config option to allow server operators
to control the zlib compression level.
On the "mozilla-unified" generaldelta repository, setting this
value to "0" (disable compression) results in server-side CPU
utilization for a `hg clone` going from ~180s to ~124s CPU time on
my i7-6700K. A level of "1" (which increases the transfer size from
~1,074 MB at level 6 to ~1,222 MB) utilizes ~132s CPU time.
The BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer has been merged into
http.server in python 3. All of them has been merged as util.httpserver to use
in both python 2 and 3. This patch adds a regex to check-code to warn against
the use of BaseHTTPServer. Moreover this patch also includes updates to lower
part of test-check-py3-compat.t which used to remain unchanged.
This patch transitions the built-in HTTPS server to use sslutil for
creating the server socket.
As part of this transition, we implement developer-only config options
to control CA loading and whether to require client certificates. This
eliminates the need for the custom extension in test-https.t to define
these.
There is a slight change in behavior with regards to protocol
selection. Before, we would always use the TLS 1.0 constant to define
the protocol version. This would *only* use TLS 1.0. sslutil defaults
to TLS 1.0+. So this patch improves the security of `hg serve` out of
the box by allowing it to use TLS 1.1 and 1.2 (if available).
Doing this will allow access to the lines in arbitrary order (because the
result of enumerate() is an iterator), and that will help calculating rowspan
for annotate blocks.
The link is embedded into a div with class="annotate-info" that only shows up
upon hover of the annotate column. To avoid duplicate hover-overs (this new
one and the one coming from link's title), drop "title" attribute from a
element and put it in the annotate-info element.
Previously, ETag headers from hgweb weren't correctly formed, because rfc2616
(section 14, header definitions) requires double quotes around the content of
the header. str(web.mtime) didn't do that.
Additionally, strong ETags signify that the resource representations are
byte-for-byte identical. That is, they can be reconstructed from byte ranges if
client so wishes. Considering ETags for all hgweb pages is just mtime of
00changelog.i and doesn't consider of e.g. .hg/hgrc with description, contact
and other fields, it's clearly shouldn't be strong. The W/ prefix marks it as
weak, which still allows caching the whole served file/page, but doesn't allow
byte-range requests.
hgweb currently offers limited functionality for "classifying"
repositories. This patch aims to change that.
The web.labels config option list is introduced. Its values
are exposed to the "index" and "summary" templates. Custom
templates can use template features like ifcontains() to e.g.
look for the presence of a specific label and engage specific
behavior. For example, a site operator may wish to assign a
"defunct" label to a repository so the repository is prominently
marked as dead in repository indexes.
I.e. when a revision blames a block of source lines, only display the
revision link on the first line of the block (this is identified by the
"blockhead" key in annotate context).
This addresses item "Visual grouping of changesets" of the blame improvements
plan (https://www.mercurial-scm.org/wiki/BlamePlan) which states: "Typically
there are block of lines all attributed to the same revision. Instead of
rendering the revision/changeset for every line, we could only render it once
per block."
Change summary webcommand to yield each element of the shortlog instead of the
entire list.
This makes generated json more readable since each entry can be formatted
separately, instead of returning all the shortlog content in a single string.
New frommapfile() function will make it clear when template aliases will be
loaded. They should be applied to command arguments and templates in hgrc,
but not to map files. Otherwise, our stock styles and web templates
(i.e map-file templates) could be modified unintentionally.
Future patches will add "aliases" argument to __init__(), but not to
frommapfile().
Changes, branches and tags are already in revlog order on /summary, /branches
and /tags, let's now make bookmarks be sorted by the same principle. It's more
helpful to show more "recent" bookmarks on top. This will affect /bookmarks
page in all styles, including atom, rss and raw, and also /summary page.
Bookmarks are sorted using a (revision number, bookmark name) tuple.
Let's do the same thing that /tags page does. It gets sorted tags and then if
it needs the latest only, it just slices the first item from the list. Since
it's a slice and not a min(), it doesn't throw an exception if the list is
empty. This fixes HTTP 500 error from issue5022.
Entries prepared in webutil.changelistentry() skip showing parents in the
trivial case when there's only one parent and it's the previous revision. This
doesn't work well for the json-log template, which is supposed to just dump raw
data in an easy-to-parse format, so let's provide all parents as another
keyword: allparents.
Using a lambda function here means that the performance of templates that don't
use allparents won't be affected (see 88bd6697bfad).
narrowhg (for its narrow spec) and remotefilelog (for its large batch
requests) would like to be able to make requests with argument sets so
absurdly large that they blow out total request size limit on some
http servers. As a workaround, support stuffing args at the start
of the POST body.
We will probably want to leave this behavior off by default in servers
forever, because it makes the old "POSTs are only for writes"
assumption wrong, which might break some of the simpler authentication
configurations.
This is necessary to preserve filename encoding over JSON. Instead, this
patch inserts "|utf8" where non-ascii local-encoding texts can be passed
to "|json".
See also the commit that introduced "utf8" filter.
Because future patches will change "|json" filter to handle input bytes
transparently, i.e. use UTF-8b encoding, "{jsdata}" must keep data in UTF-8
bytes, whereas "{nodes}" are text.
This patch inserts encodestr() where localstr is likely to survive.
The new function is used to fill basic information about a ctx, such as
revision number and hash, author, commit message, etc. Before, every webcommand
used to get this basic information on its own using some boilerplate code, and
some things in some places just weren't available.
Before this patch, it was unclear why the httpservice object could read the
server options (e.g. --port) from 'ui'. It just worked because repo.ui is ui.
Since createservice() was moved to hgweb and hgweb imports both hgweb_mod and
hgwebdir_mod, we no longer have to force hgweb() function to select one of
them by the type of 'o' variable. Let's be explicit!
This patch does not change hgweb() function because it is the interface of
existing WSGI and CGI scripts.
A block of code above this one already says "if fctx is not None", and it's
also what this code actually intends to check, so let's be specific as PEP-8
recommends.
c0ebd60607e9 didn't remove it, let's do it now.
Placing the added lines into the already existing "if fctx is not None" block
also makes webcommands.comparison() look a bit more like
webcommands.filediff(), which eases possible future refactoring. And fctx is
not None only when path in ctx, so logically it's equivalent.
When comparing a file that was removed at the current revision, parents used to
show grandparents instead, due to how fctx was "shifted" from the current
revision to its p1. Let's not do that.
The fix is pretty much copied from webcommands.filediff().
The next patch will merge the cmdutil.service() calls of both commandserver
and hgweb. Before doing it, this patch wipes out the code specific to hgweb
from commands.serve().
_siblings is a helper that is used for displaying changeset parents and
children in hgweb. Before, when it was a simple generator, it couldn't tell its
length without being consumed, and that required a special case when preparing
data for changeset template (see 3468fd599ef4).
Let's make it into a class (similar to templatekw._hybrid) that allows len(...)
without side-effects.
Log pages, i.e. changelog, filelog and search results page computed children
and parents for each changeset shown, because spartan hgweb style shows this
info. Turns out, computing all this is heavy and also unnecessary for log pages
in all other hgweb styles.
Luckily, templates allow an easy way to do computations on demand: just pass
the heavy part of code as a callable and it will be only called when needed.
Here are some benchmarks on the mercurial repository (best of 3):
time wget http://127.0.0.1:8021/
before: 0m0.050s
after: 0m0.040s
time wget http://127.0.0.1:8021/?revcount=960
before: 0m1.164s
after: 0m0.389s
time wget http://127.0.0.1:8021/log/tip/mercurial/commands.py
before: 0m0.047s
after: 0m0.042s
time wget http://127.0.0.1:8021/log/tip/mercurial/commands.py?revcount=960
before: 0m0.830s
after: 0m0.434s
