Commit Graph

976 Commits

Author SHA1 Message Date
Martijn Pieters
de1f10a894 hgweb: fix the MRO in Python 3
object should appear at the end, otherwise it tries to pre-empt the other
new-style classes in the MRO, resulting in an unresolvable MRO in Py3. We still
need to include object because otherwise in 2.7 we end up with an old-style
class if threading is not supported, new-style if it is.
2016-10-08 19:11:19 +02:00
Jun Wu
a68511813b hgweb: make fctx.annotate a separated function so it could be wrapped
This patch moves "fctx.annotate" used by the "annotate" webcommand, along
with the diffopts to a separated function which takes a ui and a fctx.
So it could be replaced by other implementations which don't want to replace
the core "fctx.annotate" directly.
2016-10-08 16:10:34 +01:00
Gregory Szorc
36f039b85b wireproto: rename argument to groupchunks()
groupchunks() is a generic "turn a file object into a generator"
function. It isn't limited to changegroups. Rename the argument
and update the docstring to reflect this.
2016-09-25 12:20:31 -07:00
Durham Goode
52b8095f37 manifest: remove last uses of repo.manifest
Now that all the functionality has been moved to manifestlog/manifestrevlog/etc,
we can finally change all the uses of repo.manifest to use the new versions. A
future diff will then delete repo.manifest.

One additional change in this commit is to change repo.manifestlog to be a
@storecache property instead of @property. This is required by some uses of
repo.manifest require that it be settable (contrib/perf.py and the static http
server). We can't do this in a prior change because we can't use @storecache on
this until repo.manifest is no longer used anywhere.
2016-11-10 02:13:19 -08:00
Augie Fackler
5229cbbf19 protocol: drop unused import of zlib
Something weird is happening that breaks pyflakes installed via 'pip
install --user'. I haven't had a chance to finish debugging this, but
this at least fixes the build.
2016-11-10 15:14:05 -05:00
Gregory Szorc
f8bef20b48 hgweb: use compression engine API for zlib compression
More low-level compression code elimination because we now have nice
APIs.

This patch also demonstrates why we needed and implemented the
"level" option on the "compressstream" API.
2016-11-07 18:54:35 -08:00
Gregory Szorc
312f42b6e4 hgweb: tweak zlib chunking behavior
When doing streaming compression with zlib, zlib appears to emit chunks
with data after ~20-30kb on average is available. In other words, most
calls to compress() return an empty string. On the mozilla-unified repo,
only 48,433 of 921,167 (5.26%) of calls to compress() returned data.
In other words, we were sending hundreds of thousands of empty chunks
via a generator where they touched who knows how many frames (my guess
is millions). Filtering out the empty chunks from the generator
cuts down on overhead.

In addition, we were previously feeding 8kb chunks into zlib
compression. Since this function tends to emit *compressed* data after
20-30kb is available, it would take several calls before data was
produced. We increase the amount of data fed in at a time to 32kb.
This reduces the number of calls to compress() from 921,167 to
115,146. It also reduces the number of output chunks from 48,433 to
31,377. This does increase the average output chunk size by a little.
But I don't think this will matter in most scenarios.

The combination of these 2 changes appears to shave ~6s CPU time
or ~3% from a server serving the mozilla-unified repo.
2016-08-14 21:29:46 -07:00
Gregory Szorc
118980f02b hgweb: document why we don't allow untrusted settings to control zlib
Added comment per discussion on mercurial-devel.
2016-08-15 20:39:33 -07:00
Gregory Szorc
4cfd8623b8 hgweb: profile HTTP requests
Currently, running `hg serve --profile` doesn't yield anything useful:
when the process is terminated the profiling output displays results
from the main thread, which typically spends most of its time in
select.select(). Furthermore, it has no meaningful results from
mercurial.* modules because the threads serving HTTP requests don't
actually get profiled.

This patch teaches the hgweb wsgi applications to profile individual
requests. If profiling is enabled, the profiler kicks in after
HTTP/WSGI environment processing but before Mercurial's main request
processing.

The profile results are printed to the configured profiling output.
If running `hg serve` from a shell, they will be printed to stderr,
just before the HTTP request line is logged. If profiling to a file,
we only write a single profile to the file because the file is not
opened in append mode. We could add support for appending to files
in a future patch if someone wants it.

Per request profiling doesn't work with the statprof profiler because
internally that profiler collects samples from the thread that
*initially* requested profiling be enabled. I have plans to address
this by vendoring Facebook's customized statprof and then improving
it.
2016-08-14 18:37:24 -07:00
Gregory Szorc
2ed4e485bc hgweb: abstract call to hgwebdir wsgi function
The function names and behavior now matches hgweb. The reason for this
will be obvious in the next patch.
2016-08-14 16:03:30 -07:00
Gregory Szorc
8720bcb69f hgweb: config option to control zlib compression level
Before this patch, the HTTP transport protocol would always zlib
compress certain responses (notably "getbundle" wire protocol commands)
at zlib compression level 6.

zlib can be a massive CPU resource sink for servers. Some server
operators may wish to reduce server-side CPU requirements while
requiring more bandwidth. This is common on corporate intranets, for
example. Others may wish to use more CPU but reduce bandwidth.

This patch introduces a config option to allow server operators
to control the zlib compression level.

On the "mozilla-unified" generaldelta repository, setting this
value to "0" (disable compression) results in server-side CPU
utilization for a `hg clone` going from ~180s to ~124s CPU time on
my i7-6700K.  A level of "1" (which increases the transfer size from
~1,074 MB at level 6 to ~1,222 MB) utilizes ~132s CPU time.
2016-08-07 18:09:58 -07:00
Anton Shestakov
72f684f821 paper: make different blocks of annotated lines have different colors 2016-07-16 14:49:07 +08:00
Pulkit Goyal
6b3bc52b40 py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import
The BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer has been merged into
http.server in python 3. All of them has been merged as util.httpserver to use
in both python 2 and 3. This patch adds a regex to check-code to warn against
the use of BaseHTTPServer. Moreover this patch also includes updates to lower
part of test-check-py3-compat.t which used to remain unchanged.
2016-07-13 23:38:29 +05:30
Gregory Szorc
c023f774e6 hgweb: use sslutil.wrapserversocket()
This patch transitions the built-in HTTPS server to use sslutil for
creating the server socket.

As part of this transition, we implement developer-only config options
to control CA loading and whether to require client certificates. This
eliminates the need for the custom extension in test-https.t to define
these.

There is a slight change in behavior with regards to protocol
selection. Before, we would always use the TLS 1.0 constant to define
the protocol version. This would *only* use TLS 1.0. sslutil defaults
to TLS 1.0+. So this patch improves the security of `hg serve` out of
the box by allowing it to use TLS 1.1 and 1.2 (if available).
2016-07-12 23:12:03 -07:00
Gregory Szorc
6052136b07 hgweb: pass ui into preparehttpserver
Upcoming patches will need the built-in HTTPS server to be more
configurable.
2016-07-13 00:14:50 -07:00
Anton Shestakov
2f0cc33284 hgweb: enumerate lines in loop header, not before
Doing this will allow access to the lines in arbitrary order (because the
result of enumerate() is an iterator), and that will help calculating rowspan
for annotate blocks.
2016-07-14 12:33:44 +08:00
Denis Laxalde
07a35f6357 hgweb: add link to parents of annotated revision in annotate view
The link is embedded into a div with class="annotate-info" that only shows up
upon hover of the annotate column. To avoid duplicate hover-overs (this new
one and the one coming from link's title), drop "title" attribute from a
element and put it in the annotate-info element.
2016-06-28 11:42:42 +02:00
Anton Shestakov
662db21d2e hgweb: emit a valid, weak ETag
Previously, ETag headers from hgweb weren't correctly formed, because rfc2616
(section 14, header definitions) requires double quotes around the content of
the header. str(web.mtime) didn't do that.

Additionally, strong ETags signify that the resource representations are
byte-for-byte identical. That is, they can be reconstructed from byte ranges if
client so wishes. Considering ETags for all hgweb pages is just mtime of
00changelog.i and doesn't consider of e.g. .hg/hgrc with description, contact
and other fields, it's clearly shouldn't be strong. The W/ prefix marks it as
weak, which still allows caching the whole served file/page, but doesn't allow
byte-range requests.
2016-07-09 03:26:24 +08:00
Gregory Szorc
5255c3f24b hgweb: expose list of per-repo labels to templates
hgweb currently offers limited functionality for "classifying"
repositories. This patch aims to change that.

The web.labels config option list is introduced. Its values
are exposed to the "index" and "summary" templates. Custom
templates can use template features like ifcontains() to e.g.
look for the presence of a specific label and engage specific
behavior. For example, a site operator may wish to assign a
"defunct" label to a repository so the repository is prominently
marked as dead in repository indexes.
2016-06-30 18:59:53 -07:00
Pulkit Goyal
38a359ce5c py3: conditionalize SocketServer import
The SocketServer is renamed to socketserver in python 3
2016-06-27 16:48:54 +05:30
Denis Laxalde
2acd032eb6 hgweb: display blamed revision once per block in annotate view
I.e. when a revision blames a block of source lines, only display the
revision link on the first line of the block (this is identified by the
"blockhead" key in annotate context).

This addresses item "Visual grouping of changesets" of the blame improvements
plan (https://www.mercurial-scm.org/wiki/BlamePlan) which states: "Typically
there are block of lines all attributed to the same revision. Instead of
rendering the revision/changeset for every line, we could only render it once
per block."
2016-06-07 12:10:01 +02:00
Laura Médioni
d95b4f5e78 templates: add support for summary webcommand in json style
Change summary webcommand to yield each element of the shortlog instead of the
entire list.

This makes generated json more readable since each entry can be formatted
separately, instead of returning all the shortlog content in a single string.
2016-06-06 15:14:11 +02:00
Denis Laxalde
675f39ef38 hgweb: remove unused code in annotate web command 2016-06-01 15:16:38 +02:00
timeless
a1cb3173a2 py3: convert to next() function
next(..) was introduced in py2.6 and .next() is not available in py3

https://docs.python.org/2/library/functions.html#next
2016-05-16 21:30:53 +00:00
Yuya Nishihara
2d4abff2d9 hgweb: make log streams compatible with command server
Even though it would be useless to start a web server by a command server,
it should be doable in principle. Also, we can't use sys.stdout/err directly
on Python 3 because they are unicode streams.
2014-06-28 13:13:32 +09:00
Yuya Nishihara
3f981af86b templater: separate function to create templater from map file (API)
New frommapfile() function will make it clear when template aliases will be
loaded. They should be applied to command arguments and templates in hgrc,
but not to map files. Otherwise, our stock styles and web templates
(i.e map-file templates) could be modified unintentionally.

Future patches will add "aliases" argument to __init__(), but not to
frommapfile().
2016-04-03 23:26:48 +09:00
timeless
109fcbc79e pycompat: switch to util.urlreq/util.urlerr for py3 compat 2016-04-06 23:22:12 +00:00
timeless
f77cdcd3b1 pycompat: switch to util.stringio for py3 compat 2016-04-10 20:55:37 +00:00
Anton Shestakov
9f3a77930a hgweb: generate last change date for an empty atom-bookmarks feed (issue5022)
RFC 4287 states that atom feeds must have an <updated> element, so let's add
one even when repo doesn't have a single bookmark.
2016-03-31 15:37:21 +08:00
Anton Shestakov
d9e41fcffc hgweb: sort bookmarks in revlog order of their nodes
Changes, branches and tags are already in revlog order on /summary, /branches
and /tags, let's now make bookmarks be sorted by the same principle. It's more
helpful to show more "recent" bookmarks on top. This will affect /bookmarks
page in all styles, including atom, rss and raw, and also /summary page.

Bookmarks are sorted using a (revision number, bookmark name) tuple.
2016-03-31 15:22:06 +08:00
Anton Shestakov
0741a98a71 hgweb: sort bookmarks early
Let's do the same thing that /tags page does. It gets sorted tags and then if
it needs the latest only, it just slices the first item from the list. Since
it's a slice and not a min(), it doesn't throw an exception if the list is
empty. This fixes HTTP 500 error from issue5022.
2016-03-31 14:23:27 +08:00
Anton Shestakov
e81bf97ae9 hgweb: add parents to json-log (issue5074)
Entries prepared in webutil.changelistentry() skip showing parents in the
trivial case when there's only one parent and it's the previous revision. This
doesn't work well for the json-log template, which is supposed to just dump raw
data in an easy-to-parse format, so let's provide all parents as another
keyword: allparents.

Using a lambda function here means that the performance of templates that don't
use allparents won't be affected (see 88bd6697bfad).
2016-03-31 18:09:09 +08:00
Augie Fackler
b3f8347d29 http: support sending hgargs via POST body instead of in GET or headers
narrowhg (for its narrow spec) and remotefilelog (for its large batch
requests) would like to be able to make requests with argument sets so
absurdly large that they blow out total request size limit on some
http servers. As a workaround, support stuffing args at the start
of the POST body.

We will probably want to leave this behavior off by default in servers
forever, because it makes the old "POSTs are only for writes"
assumption wrong, which might break some of the simpler authentication
configurations.
2016-03-11 11:37:00 -05:00
Yuya Nishihara
2e33bbee20 templatefilters: make json filter be byte-transparent (BC) (issue4926)
This is necessary to preserve filename encoding over JSON. Instead, this
patch inserts "|utf8" where non-ascii local-encoding texts can be passed
to "|json".

See also the commit that introduced "utf8" filter.
2015-12-27 17:59:57 +09:00
Yuya Nishihara
bd35a9a988 hgweb: add option to convert encoding of graphdata()
Because future patches will change "|json" filter to handle input bytes
transparently, i.e. use UTF-8b encoding, "{jsdata}" must keep data in UTF-8
bytes, whereas "{nodes}" are text.

This patch inserts encodestr() where localstr is likely to survive.
2016-01-04 23:05:09 +09:00
Yuya Nishihara
0633dff1e2 hgweb: remove unused argument from graphdata() factory
As graphdata() is wrapped by lambda, there's no reason to pass unused
arguments to it.
2016-01-04 22:55:05 +09:00
Gregory Szorc
a3e59a30b6 hgweb: support rendering a sub-topic
If the requested topic contains a "." we assume a sub-topic is
requested and display it.
2015-12-30 17:15:10 -07:00
Gregory Szorc
a320e743a7 hgweb: support rendering sub-topic indexes
If the requested topic name is the name of a sub-topic, we now render
an index of topics within that sub-topic.
2015-12-30 17:34:51 -07:00
Anton Shestakov
2e206a44b7 hgweb: move entry-preparing code from webcommands to webutils.commonentry()
The new function is used to fill basic information about a ctx, such as
revision number and hash, author, commit message, etc. Before, every webcommand
used to get this basic information on its own using some boilerplate code, and
some things in some places just weren't available.
2015-11-13 14:35:36 +08:00
Yuya Nishihara
9ed442849f hgweb: load server settings from --web-conf (issue4699)
It copies the ui before loading the webconf and passes the copied ui only
to the service. This way, the hgwebdir app can reload configs cleanly.
2015-10-31 22:50:03 +09:00
Yuya Nishihara
5cfcdd71d7 hgweb: make sure command options are set to all ui objects
Before this patch, it was unclear why the httpservice object could read the
server options (e.g. --port) from 'ui'. It just worked because repo.ui is ui.
2015-11-21 13:28:12 +09:00
Yuya Nishihara
8d4fdbd827 hgweb: eliminate duck-typing to select hgweb or hgwebdir by command option
Since createservice() was moved to hgweb and hgweb imports both hgweb_mod and
hgwebdir_mod, we no longer have to force hgweb() function to select one of
them by the type of 'o' variable. Let's be explicit!

This patch does not change hgweb() function because it is the interface of
existing WSGI and CGI scripts.
2015-10-31 22:26:50 +09:00
Anton Shestakov
4f46ce6f31 webcommands: test that fctx is not None in filediff()
A block of code above this one already says "if fctx is not None", and it's
also what this code actually intends to check, so let's be specific as PEP-8
recommends.
2015-11-28 17:06:29 +08:00
Anton Shestakov
7359ceac6b webcommands: stop using ersatz if-else ternary operator for rename variable
c0ebd60607e9 didn't remove it, let's do it now.

Placing the added lines into the already existing "if fctx is not None" block
also makes webcommands.comparison() look a bit more like
webcommands.filediff(), which eases possible future refactoring. And fctx is
not None only when path in ctx, so logically it's equivalent.
2015-11-28 16:46:31 +08:00
Anton Shestakov
9f284e62ca webcommands: get correct parents when comparing a removed file (issue4962)
When comparing a file that was removed at the current revision, parents used to
show grandparents instead, due to how fctx was "shifted" from the current
revision to its p1. Let's not do that.

The fix is pretty much copied from webcommands.filediff().
2015-11-28 16:02:22 +08:00
Yuya Nishihara
c64bc024c8 hgweb: extract factory function of httpservice object
The next patch will merge the cmdutil.service() calls of both commandserver
and hgweb. Before doing it, this patch wipes out the code specific to hgweb
from commands.serve().
2015-10-31 22:15:16 +09:00
Yuya Nishihara
4fb8995ece hgweb: move httpservice object from commands module
This avoids the deep import of hgweb.server at the commands module.
2015-10-31 21:57:45 +09:00
Anton Shestakov
5c74a8a582 paper: show current revision on file log page
Most of the pages in paper (and coal) style show the current revision and its
branch, tags and bookmarks. Let's also show all this on file log page.
2015-11-13 18:31:58 +08:00
Yuya Nishihara
47690f822c hgweb: use absolute_import 2015-10-31 22:07:40 +09:00
Yuya Nishihara
6f91677786 hgweb: import wsgicgi at top level
There should be no practical reason to delay the import of wsgicgi.
2015-11-01 15:09:35 +09:00
Yuya Nishihara
3a2ca88d5a hgweb: import BaseHTTPServer as module at top level
This will avoid future warning spotted by the import checker.
2015-11-01 15:07:08 +09:00
Yuya Nishihara
e142543972 hgweb: do not import hgweb_mod.hgweb and .makebreadcrumb as symbol
It will be enforced by the import checker.
2015-11-01 14:23:23 +09:00
Anton Shestakov
53412195f4 webutil: make _siblings into an object with __iter__ and __len__
_siblings is a helper that is used for displaying changeset parents and
children in hgweb. Before, when it was a simple generator, it couldn't tell its
length without being consumed, and that required a special case when preparing
data for changeset template (see 3468fd599ef4).

Let's make it into a class (similar to templatekw._hybrid) that allows len(...)
without side-effects.
2015-11-10 17:22:40 +08:00
Yuya Nishihara
8110fbc21f hgweb: unify import style of error classes
It will be enforced by the import checker.
2015-11-01 14:43:25 +09:00
Yuya Nishihara
01e2649d34 hgweb: do not import templatefilters.revescape and websub as symbol
It will be enforced by the import checker.
2015-11-01 14:00:14 +09:00
Yuya Nishihara
ad55f13658 hgweb: alias ui module as uimod
It will be enforced by the import checker.
2015-11-01 13:58:32 +09:00
Anton Shestakov
90da27730e hgweb: compute changeset parents and children for log pages lazily
Log pages, i.e. changelog, filelog and search results page computed children
and parents for each changeset shown, because spartan hgweb style shows this
info. Turns out, computing all this is heavy and also unnecessary for log pages
in all other hgweb styles.

Luckily, templates allow an easy way to do computations on demand: just pass
the heavy part of code as a callable and it will be only called when needed.

Here are some benchmarks on the mercurial repository (best of 3):

time wget http://127.0.0.1:8021/

before: 0m0.050s
after:  0m0.040s

time wget http://127.0.0.1:8021/?revcount=960

before: 0m1.164s
after:  0m0.389s

time wget http://127.0.0.1:8021/log/tip/mercurial/commands.py

before: 0m0.047s
after:  0m0.042s

time wget http://127.0.0.1:8021/log/tip/mercurial/commands.py?revcount=960

before: 0m0.830s
after:  0m0.434s
2015-11-10 23:02:59 +08:00
Siddharth Agarwal
6b30464363 hgweb.server: drop support for Python 2.4 2015-10-24 16:23:42 -07:00
Anton Shestakov
9b60cbd85f hgweb: replace some str.split() calls by str.partition() or str.rpartition()
Since Python 2.5 str has new methods: partition and rpartition. They are more
specialized than the usual split and rsplit, and they sometimes convey the
intent of code better and also are a bit faster (faster than split/rsplit with
maxsplit specified). Let's use them in appropriate places for a small speedup.

Example performance (partition):

$ python -m timeit 'assert "apple|orange|banana".split("|")[0] == "apple"'
1000000 loops, best of 3: 0.376 usec per loop

$ python -m timeit 'assert "apple|orange|banana".split("|", 1)[0] == "apple"'
1000000 loops, best of 3: 0.327 usec per loop

$ python -m timeit 'assert "apple|orange|banana".partition("|")[0] == "apple"'
1000000 loops, best of 3: 0.214 usec per loop

Example performance (rpartition):

$ python -m timeit 'assert "apple|orange|banana".rsplit("|")[-1] == "banana"'
1000000 loops, best of 3: 0.372 usec per loop

$ python -m timeit 'assert "apple|orange|banana".rsplit("|", 1)[-1] == "banana"'
1000000 loops, best of 3: 0.332 usec per loop

$ python -m timeit 'assert "apple|orange|banana".rpartition("|")[-1] == "banana"'
1000000 loops, best of 3: 0.219 usec per loop
2015-11-02 23:37:49 +08:00
Mads Kiilerich
09567db49a spelling: trivial spell checking 2015-10-17 00:58:46 +02:00
Pierre-Yves David
30913031d4 error: get Abort from 'error' instead of 'util'
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.

For great justice.
2015-10-08 12:55:45 -07:00
Matt Mackall
5e1b4ad958 urls: bulk-change primary website URLs 2015-09-30 15:43:49 -05:00
Yuya Nishihara
d7b6a95763 hgweb: overwrite cwd to resolve file patterns relative to repo (issue4568)
It's useless to handle file patterns as relative to the cwd of the server
process. The only sensible way in hgweb is to resolve paths relative to the
repository root.

It seems dirstate.getcwd() isn't used to get a real file path, so this patch
won't cause problem.
2015-09-20 20:11:31 +09:00
Gregory Szorc
4e3c6d9b7a hgweb: consume generator inside context manager (issue4756)
If code inside a context manager returns a generator, the context
manager exits before the generator is iterated.

hgweb was using a context manager to control thread safe access to a
localrepository instance. But it was returning a generator, so there was
a race condition between a previous request streaming a response to the
client and a new request obtaining the released but in use repository.
By iterating the generator inside the context manager, we ensure we
don't release the repo instance until after the response has finished.

With this change, hgweb finally appears to have full localrepository
isolation between threads. I can no longer reproduce the 2 exceptions
reported in issue4756.

test-hgweb-non-interactive.t has been modified to consume the output
of calling into a WSGI application. Without this, execution of the WSGI
application stalls because of the added yield statement.
2015-09-12 12:16:39 -07:00
Gregory Szorc
216aba88f8 hg: always create new localrepository instance
cachedlocalrepo.copy() didn't actually create new localrepository
instances. This meant that the new thread isolation code in hgweb wasn't
actually using separate localrepository instances, even though it was
properly using separate cachedlocalrepo instances.

Because the behavior of the API changed, the single caller in hgweb had
to be refactored to always call _webifyrepo() or it may not have used
the proper filter.

I confirmed via print() debugging that id(repo) is in fact different on
each thread. This was not the case before.

For reasons I can't yet explain, this does not fix issue4756. I suspect
there is shared cache somewhere that isn't thread safe.
2015-09-12 11:31:56 -07:00
Matt Mackall
a8c75dcbec hgweb: drop unused import 2015-09-11 12:48:09 -07:00
Gregory Szorc
39247b40e9 hgweb: use separate repo instances per thread
Before this change, multiple threads/requests could share a
localrepository instance. This meant that all of localrepository needed
to be thread safe. Many bugs have been reported telling us that
localrepository isn't actually thread safe.

While making localrepository thread safe is a noble cause, it is a lot
of work. And there is little gain from doing so. Due to Python's GIL,
only 1 thread may be processing Python code at a time. The benefits
to multi-threaded servers are marginal.

Thread safety would be a lot of work for little gain. So, we're not
going to even attempt it.

This patch establishes a pool of repos in hgweb. When a request arrives,
we obtain the most recently used repository from the pool or create a
new one if none is available. When the request has finished, we put that
repo back in the pool.

We start with a pool size of 1. For servers using a single thread, the
pool will only ever be of size 1. For multi-threaded servers, the pool
size will grow to the max number of simultaneous requests the server
processes.

No logic for pruning the pool has been implemented. We assume server
operators either limit the number of threads to something they can
handle or restart the Mercurial process after a certain amount of
requests or time has passed.
2015-08-22 18:43:24 -07:00
Gregory Szorc
83818c1fca hg: establish a cache for localrepository instances
hgweb contained code for determining whether a cached localrepository
instance was up to date. This code was way too low-level to be in
hgweb.

This functionality has been moved to a new "cachedlocalrepo" class
in hg.py. The code has been changed slightly to facilitate use
inside a class. hgweb has been refactored to use the new API.

As part of this refactor, hgweb.repo no longer exists! We're very close
to using a distinct repo instance per thread.

The new cache records state when it is created. This intelligence
prevents an extra localrepository from being created on the first
hgweb request. This is why some redundant output from test-extension.t
has gone away.
2015-08-22 18:54:34 -07:00
Gregory Szorc
2aa478bfd2 hgweb: create function to perform actions on new repo
We perform some common tasks when a new repo instance is obtained. In
preparation for changing how we obtain repo instances, factor this
functionality into a standalone function.
2015-08-22 18:15:42 -07:00
Gregory Szorc
e03cfea13e hgweb: remove proxy to hgweb instance
We were temporarily routing attributes until all request-specific
attributes from hgweb were moved to requestcontext. We have finally
reached that juncture and we can remove the proxy.

At this point, only the repo instance is prone to race conditions
between threads. This will be dealt with shortly.
2015-08-22 17:50:28 -07:00
Gregory Szorc
fc3513ba2d hgweb: assign ctype to requestcontext
The very existence of ctype is a bit hacky. But we roll with it.

Before this patch, there was possibly a race condition between 2
threads handling file requests: 1 thread could set the ctype and
another serving a different file would read and use that potentially
wrong ctype.
2015-08-22 17:08:37 -07:00
Gregory Szorc
e74f9acd5d hgweb: add reponame to requestcontext
We have to use object.__setattr__ until the app proxy is gone.
2015-08-22 17:04:24 -07:00
Gregory Szorc
c3d67d78b9 hgweb: don't access self.repo during request processing
We want all repository accesses to go through requestcontext.repo
so the request is isolated from the application.
2015-08-22 16:54:52 -07:00
Gregory Szorc
29e7feecf4 hgweb: extract _getview to own function
While we're refactoring code, we might as well remove a method that
doesn't need to be a method.
2015-08-22 16:44:36 -07:00
Gregory Szorc
6e97351675 hgweb: regenerate web substitutions when repo is refreshed
Previously, changes to the configuration would not be picked up by a
running server. That feels like a bug. Regenerate the web substitutions
table when the repository changes.
2015-08-22 16:41:02 -07:00
Matt Mackall
9389191839 hgweb: drop unused import 2015-09-09 12:40:57 -07:00
timeless@mozdev.org
096175f16e hgweb.server: fix _httprequesthandlerssl help text 2015-09-08 15:32:20 -04:00
timeless@mozdev.org
9059b84547 hgweb: remove ErrorResponse.message
BaseException.message is deprecated:
https://www.python.org/dev/peps/pep-0352/#retracted-ideas
2015-09-08 14:56:29 -04:00
Gregory Szorc
123302288c hgweb: move templater instantiation to requestcontext
This code needs to access a lot of config options. All our config
lookups have moved to requestcontext. It makes sense to move this
function there.
2015-08-22 16:38:51 -07:00
Matt Mackall
78dbeac869 merge with stable 2015-09-03 18:32:17 -05:00
Matt Mackall
5a89fbd24f hgweb: use latest mtime for caching tag (issue4814)
Previously, we were using the last mtime we saw when reporting the
HTTP cache etag. When we added bookmarks to the end of the list of
files checked, unchanged or missing bookmarks would keep the client
cache from being invalidated.
2015-09-03 12:19:43 -05:00
Matt Mackall
12d69c8e7a hgweb: fix trust of templates path (BC)
Long ago we disabled trust of the templates path with a comment
describing the (insecure) behavior before the change. At some later
refactor, the code was apparently changed back to match the comment,
unaware that the intent of the comment was to describe the behavior to
avoid.

This change disables the trust and updates the comment to explicitly
say not only what the old problem was, but also that it was in fact a
problem and the action taken to prevent it.

Impact: prior to this change, if you had a UNIX-based hgweb server
where users can write hgrc files, those users could potentially read
any file readable by the web server.

This is marked as a backwards compatibility issue because people may
have configured templates without proper trust settings. Issue spotted
by Greg Szorc.
2015-09-01 16:08:07 -05:00
Gregory Szorc
8201e17909 hgweb: create websubtable on requestcontext 2015-08-22 16:39:29 -07:00
Gregory Szorc
6d1194795f hgweb: move templatepath to requestcontext
This does change behavior in that the templatepath could change during
the lifetime of the server. But everything else can change, I don't see
why template paths can't.
2015-08-22 16:28:22 -07:00
Gregory Szorc
4b659c309d hgweb: extract web substitutions table generation to own function
It doesn't use any state in hgweb except for the repo instance.
Move it to a standalone function.
2015-08-22 15:40:33 -07:00
Gregory Szorc
fe3df2ac82 hgweb: remove hgweb.configbool
It is redundant with requestcontext.configbool.
2015-08-22 15:32:16 -07:00
Gregory Szorc
1cdb989214 hgweb: move additional state setting outside of refresh
We want refresh() to only be about refreshing repository
instances. This state doesn't belong in requestcontext
because it is shared across multiple threads.
2015-08-22 15:30:39 -07:00
Gregory Szorc
b629be8ff2 hgweb: initialize repostate to None
The initial value is irrelevant since refresh() compares it to
a tuple of tuples of file mtime and size. None != tuple and
None is a better default value than a tuple containing irrelevant
values.
2015-08-22 15:21:45 -07:00
Matt Mackall
58b892a1cb merge with stable 2015-09-01 17:09:00 -05:00
Gregory Szorc
34c68673a7 hgweb: move archive related attributes to requestcontext
As part of this, "archive_specs" was renamed to "archivespecs" to align
with naming conventions.

"archive_specs" didn't technically need to be moved from hgweb. But it
seemed to make sense to have all the archive code in the same class.

As part of this, hgweb.configlist is no longer used, so it was deleted.
2015-08-22 15:12:52 -07:00
Gregory Szorc
820065a592 hgweb: move some config options to requestcontext
Various config options from the repository were stored on the
hgweb instance. While unlikely, there could be race conditions between
a new request updating these values and an in-flight request seeing both
old and new values, leading to weird results.

We move some of options/attributes from hgweb to requestcontext.

As part of this, we establish config* helpers on requestcontext. As
part of the move, we changed int() casts to configint() calls. The
int() usage likely predates the existence of configint().

We also removed config option updating from once every refresh to every
request. I don't believe obtaining config options is expensive enough to
warrant only doing when the repository has changed.

The excessive use of object.__setattr__ is unfortunate. But it will
eventually disappear once the proxy is no longer necessary.
2015-08-22 15:02:41 -07:00
Gregory Szorc
bdb55cdf30 hgweb: establish class for holding per request context
Currently, hgweb applications have many instance variables holding
mutated state. This is somewhat problematic because multiple threads
may race accessing or changing this state.

This patch starts a series that will add more thread safety to
hgweb applications. It will do this by moving mutated state out
of hgweb and into per-request instances of the newly established
"requestcontext" class.

Our new class currently behaves like a proxy to hgweb instances. This
should change once all state is captured in it instead of hgweb. The
effectiveness of this proxy is demonstrated by passing instances of
it - not hgweb instances/self - to various functions.
2015-08-22 14:59:36 -07:00
Gregory Szorc
a8f65a043f hgweb: don't make request optional
The "request" argument has been optional in this code since
it was introduced in bb95879961db in 2009. There are no consumers that
don't pass this argument. So don't make it an optional argument.
2015-08-22 14:22:40 -07:00
Gregory Szorc
962145920a hgweb: add some documentation
It took longer than I wanted to grok how the various parts of hgweb
worked. So I added some class and method documentation to help whoever
hacks on this next.
2015-08-22 13:58:59 -07:00
Anton Shestakov
df9bda4834 hgweb: limit branches shown on summary page to 10
Tags and bookmarks on summary page are already limited to 10, let's limit
branches as well.

Each of the blocks (tags, bookmarks, branches) currently has a link to the full
list.
2015-09-01 23:35:06 +08:00
Anton Shestakov
7e5ab1949a hgweb: use webutil.branchentries for branches on summary page
This allows showing correct status for each branch, which was missing on
/summary. Usually that means that closed branches get the same css class
(resulting in e.g. different color/shade) as they do on /branches page.

The sorting of the branches on summary page also changes and is now the same as
on /branches page: closed branches are now at the end of the list.
2015-09-01 23:29:30 +08:00
Anton Shestakov
3fecbc5edf hgweb: move branchentries code from webcommands to webutil 2015-09-01 22:28:45 +08:00
Gregory Szorc
6e680ddd62 hgweb: make refresh interval configurable
hgwebdir refreshes the set of known repositories periodically. This
is necessary because refreshing on every request could add significant
request latency.

More than once I've found myself wanting to tweak this interval at
Mozilla. I've also wanted the ability to always refresh (often when
writing tests for our replication setup).

This patch makes the refresh interval configurable. Negative values
indicate to always refresh. The default is left unchanged.
2015-08-22 22:59:51 -07:00
Anton Shestakov
3c48ed2bcf style: adjust whitespaces in webutil.py
Turns out, all this came from the single 3ff83729b63f.
2015-08-11 13:19:42 +08:00
Anton Shestakov
939c576f07 templates: introduce revescape filter for escaping symbolic revisions
There needs to be a way to escape symbolic revisions containing forward
slashes, but urlescape filter doesn't escape slashes at all (in fact, it is
used in places where forward slashes must be preserved).

The filter considers @ to be safe just for bookmarks like @ and @default to
look good in urls.
2015-07-12 16:47:56 +08:00