If the computation of a set for each phase (done in C) is available,
we use it directly instead of applying a simple filter. This give a
massive speed-up in the vast majority of cases.
On my mercurial repo with about 15000 out of 40000 draft changesets:
revset: draft()
plain min first last
0) 0.011201 0.019950 0.009844 0.000074
1) 0.000284 2% 0.000312 1% 0.000314 3% 0.000315 x4.3
Bad performance for "last" come from the handling of the 15000 elements set
(memory allocation, filtering hidden changesets (99% of it) etc. compared to
applying the filter only on a handfuld of revisions (the first draft changesets
being close of tip).
This is not seen as an issue since:
* Timing is still pretty good and in line with all the other one,
* Current user of Vanilla Mercurial will not have 1/3 of their repo draft,
This bad effect disappears when phase's set is smaller. (about 200 secrets):
revset: secret()
plain min first last
0) 0.011181 0.022228 0.010851 0.000452
1) 0.000058 0% 0.000084 0% 0.000087 0% 0.000087 19%
Code for draft and secret are the same. We'll make it more complex to
take advantages of the set recomputed in C, so we first refactor the
code to only have one place to update (and make sure all behave
properly).
We do not refactor the 'public()' code because it does not have a natively
computed set.
Using 'repo[X]' is much slower because it creates a 'changectx' object and goes
though multiple layers of code to do so. It is also error prone if there is
tags, bookmarks, branch or other names that could map to a node hash and take
precedence (user are wicked).
This provides a significant performance boost on repository with a lot of
heads. Benchmark result for a repo with 1181 heads.
revset: head()
plain min last reverse
0) 0.014853 0.014371 0.014350 0.015161
1) 0.001402 9% 0.000975 6% 0.000874 6% 0.001415 9%
revset: head() - public()
plain min last reverse
0) 0.015121 0.014420 0.014560 0.015028
1) 0.001674 11% 0.001109 7% 0.000980 6% 0.001693 11%
revset: draft() and head()
plain min last reverse
0) 0.015976 0.014490 0.014214 0.015892
1) 0.002335 14% 0.001018 7% 0.000887 6% 0.002340 14%
The speed up is visible even when other more costly revset are in use
revset: head() and author("mpm")
plain min last reverse
0) 0.105419 0.090046 0.017169 0.108180
1) 0.090721 86% 0.077602 86% 0.003556 20% 0.093324 86%
The '_notpublic()' internal revset was "returning" a set. That was wrong. We now
return a 'baseset' as appropriate. This has no effect on performance in most case,
because we do the exact same operation than what the combination with a
'fullreposet' was doing. This as a small effect on some operation when combined
with other set, because we now apply the filtering in all cases. I think the
correctness is worth the impact on some corner cases. The optimizer should take
care of these corner cases anyway.
revset #0: not public()
plain min max first last reverse
0) 0.000465 0.000491 0.000495 0.000500 0.000494 0.000479
1) 0.000484 0.000503 0.000498 0.000505 0.000504 0.000491
revset #1: (tip~1000::) - public()
plain min max first last reverse
0) 0.002765 0.001742 0.002767 0.001730 0.002761 0.002782
1) 0.002847 0.001777 0.002776 0.001741 0.002764 0.002858
revset #2: not public() and branch("default")
plain min max first last reverse
0) 0.012104 0.011138 0.011189 0.011138 0.011166 0.011578
1) 0.011387 94% 0.011738 105% 0.014220 127% 0.011223 0.011184 0.012077
revset #3: (not public() - obsolete())
plain min max first last reverse
0) 0.000583 0.000556 0.000552 0.000555 0.000552 0.000610
1) 0.000613 105% 0.000559 0.000557 0.000573 0.000558 0.000613
revset #4: head() - public()
plain min max first last reverse
0) 0.010869 0.010800 0.011547 0.010843 0.010891 0.010891
1) 0.011031 0.011497 106% 0.011087 0.011100 0.011100 0.011085
Unlike other styles, paper and coal had only one link to current revision: in
the sidebar. Since those links now use symbolic revisions after 4b263b99440b,
it's nice to have a link that allows going from /rev/tip to /rev/<tip hash>,
for instance. Let's make the node hash in the page header that new link.
If we are the very first rev access (or if the phase cache just got
invalidated) the phasesets will be None even if we support the native
computation. So we explicitly trigger a computation if needed.
This was not an issue before because requesting any phase information
would have triggered such computation.
This function is:
- already loading the data in place,
- used once in the code.
So we drop the return value and change the name to make this obvious. We keep
the function public because we'll have to use it in revset.
Let's make paper (and coal, since it borrows so much from paper) templates use
symbolic revision in navigation links.
The majority of links (log, filelog, annotate, etc) still use node hashes.
Some pages don't have permanent links to current node hash (so it's not very
easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future
patches.
Let's make gitweb templates use symbolic revision in navigation links.
The majority of links (log, filelog, annotate, etc) still use node hashes.
Some pages don't have permanent links to current node hash (so it's not very
easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future
patches.
Let's make monoblue templates use symbolic revision in navigation links.
The majority of links (log, filelog, annotate, etc) still use node hashes.
Some pages don't have permanent links to current node hash (so it's not very
easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future
patches.
Let's make spartan templates use symbolic revision in navigation links.
The majority of links (log, filelog, annotate, etc) still use node hashes, and
many pages also have permanent link to current node hash (i.e. you can go from
/rev/tip to /rev/<tip hash> without manual url editing), so it's safe to
update navigation.
One of the features of hgweb is that current position in repo history is
remembered between separate requests. That is, links from /rev/<node_hash> lead
to /file/<node_hash> or /log/<node_hash>, so it's easy to dig deep into the
history. However, such links could only use node hashes and local revision
numbers, so while staying at one exact revision is easy, staying on top of the
changes is not, because hashes presumably can't change (local revision numbers
can, but probably not in a way you'd find useful for navigating).
So while you could use 'tip' or 'default' in a url, links on that page would be
permanent. This is not always desired (think /rev/tip or /graph/stable or
/log/@) and is sometimes just confusing (i.e. /log/<not the tip hash>, when
recent history is not displayed). And if user changed url deliberately to say
default instead of <some node hash>, the page ignores that fact and uses node
hash in its links, which means that navigation is, in a way, broken.
This new property, symrev, is used for storing current revision the way it was
specified, so then templates can use it in links and thus "not dereference" the
symbolic revision. It is an additional way to produce links, so not every link
needs to drop {node|short} in favor of {symrev}, many will still use node hash
(log and filelog entries, annotate lines, etc).
Some pages (e.g. summary, tags) always use the tip changeset for their context,
in such cases symrev is set to 'tip'. This is needed in case the pages want to
provide archive links.
highlight extension needs to be updated, since _filerevision now takes an
additional positional argument (signature "web, req, tmpl" is used by most of
webcommands.py functions).
More references to symbolic revisions and related gripes: issue2296, issue2826,
issue3594, issue3634.
This is a step toward replacing the extdiff internals with archive, in order to
support 'extdiff -S'. Only Mercurial subrepos are supported for now.
If a file is missing from the filesystem, it is silently skipped. Perhaps it
should warn, but it cannot abort when working with extdiff because deleting a
file is a legitimate diff.
Some code cannot handle a subrepo based on the working directory (e.g.
sub.dirty()), so the caller must opt in. This will be useful for archive, and
perhaps some other commands. The git and svn methods where this is used may
need to be fixed up on a case by case basis.
I've tried to unify gettemplate() with buildtemplate(), but it didn't go well
because gettemplate() have to bypass mapping dict.
For example, web templates have '{tags%changelogtag}' and 'changelogtag' is
defined in both mapping, the default, and context.cache, sourced from map file.
In general, mapping shadows context variables, but gettemplate() have to pick
it from context.cache.
The previous patch made 'string' is always interpreted as a template. So
this patch removes the special handling of r'rawstring' instead. Now r''
disables template processing at all.
This patch series is intended to unify the interpretation of string literals.
It is breaking change that boldly assumes
a. string literal "..." never contains template-like fragment or it is
intended to be a template
b. we tend to use raw string literal r"..." for regexp pattern in which "{"
should have different meaning
Currently, we don't have a comprehensible rule how string literals are
evaluated in template functions. For example, fill() takes "initialindent"
and "hangindent" as templates, but not for "text", whereas "text" is a
template in pad() function.
date(date, fmt)
diff(includepattern, excludepattern)
fill(text, width, initialident: T, hangindent: T)
get(dict, key)
if(expr, then: T, else: T)
ifcontains(search, thing, then: T, else: T)
ifeq(expr1, expr2, then: T, else: T)
indent(text, indentchars, firstline)
join(list, sep)
label(label: T, expr: T)
pad(text: T, width, fillchar, right)
revset(query, formatargs...])
rstdoc(text, style)
shortest(node, minlength)
startswith(pattern, text)
strip(text, chars)
sub(pattern, replacement, expression: T)
word(number, text, separator)
expr % template: T
T: interpret "string" or r"rawstring" as template
This patch series adjusts the rule as follows:
a. string literal, '' or "", starts template processing (BC)
b. raw string literal, r'' or r"", disables both \-escape and template
processing (BC, done by subsequent patches)
c. fragment not surrounded by {} is non-templated string
"ccc{'aaa'}{r'bbb'}"
------------------ *: template
--- c: string
--- a: template
--- b: rawstring
Because this can eliminate the compilation of template arguments from the
evaluation phase, "hg log -Tdefault" gets faster.
% cd mozilla-central
% LANG=C HGRCPATH=/dev/null hg log -Tdefault -r0:10000 --time > /dev/null
before: real 4.870 secs (user 4.860+0.000 sys 0.010+0.000)
after: real 3.480 secs (user 3.440+0.000 sys 0.030+0.000)
Also, this will allow us to parse nested templates at once for better error
indication.
The next patch will introduce buildtemplate function that should be defined
near runtemplate. But I don't want to insert it between buildmap and runmap.
While hopefully atypical, there are reasons that a subrepository revision can be
lost that aren't covered by corruption of the .hgsubstate revlog. Such things
can happen when a subrepo is amended, stripped or simply isn't pulled from
upstream because the parent repo revision wasn't updated yet. There's no way to
know if it is an error, but this will find potential problems sooner than when
some random revision is updated.
Until recently, convert made no attempt at rewriting the .hgsubstate file. The
impetuous for this is to verify the conversion of some repositories, and this is
orders of magnitude faster than a bash script from 0..tip that does an
'hg update -C $rev'. But it is equally useful to determine if everything has
been pulled down before taking a thumb drive on the go.
It feels somewhat wrong to leave this out of verifymod (mostly because the file
is already read in there, and the final summary is printed before the subrepos
are checked). But verifymod looks very low level, so importing subrepo stuff
there seems more wrong.
Since node is None for workingctx, it can't use the base class
implementation of 'hex(self.node())'.
It doesn't appear that there are any current callers of this, but there will be
when archive supports 'wdir()'. My first thought was to use "{p1node}+", but
that would cause headaches elsewhere [1].
We should probably fix up localrepository.__getitem__ to accept this hash for
consistency, as a followup. This works, if the full hash is specified:
@@ -480,7 +480,7 @@
return dirstate.dirstate(self.vfs, self.ui, self.root, validate)
def __getitem__(self, changeid):
- if changeid is None:
+ if changeid is None or changeid == 'ff' * 20:
return context.workingctx(self)
if isinstance(changeid, slice):
return [context.changectx(self, i)
That differs from null, where it will accept any number of 0s, as long as it
isn't ambiguous.
[1] https://www.selenic.com/pipermail/mercurial-devel/2015-June/071166.html
This uses a simple heuristic to avoid expensive resizes.
On a real-world repo with around 400,000 files, perfdirstate:
before: ! wall 0.155562 comb 0.160000 user 0.150000 sys 0.010000 (best of 64)
after: ! wall 0.132638 comb 0.130000 user 0.120000 sys 0.010000 (best of 75)
On another real-world repo with around 250,000 files:
before: ! wall 0.098459 comb 0.100000 user 0.090000 sys 0.010000 (best of 100)
after: ! wall 0.089084 comb 0.090000 user 0.080000 sys 0.010000 (best of 100)
Index lookups raise RevlogError when the lookup fails. The previous
implementation was caching a reference to the RevlogError type in a
static variable. This assumed that the "mercurial.error" module was
only loaded once and there was only a single copy of it floating
around in memory. Unfortunately, in some situations - including
certain mod_wsgi configurations - this was not the case: the
"mercurial.error" module could be reloaded. It was possible for a
"RevlogError" reference from the first interpreter to be used by
a second interpreter. While the underlying thing was a
"mercurial.error.RevlogError," the object IDs were different, so
the Python code in revlog.py was failing to catch the exception! This
error has existed since the C index lookup code was implemented in
changeset 1e47437a1ca7, which was first released in Mercurial 2.2 in
2012.
http://emptysqua.re/blog/python-c-extensions-and-mod-wsgi/#static-variables-are-shared
contains more details.
This patch removes the caching of the RevlogError type from the
function.
Since pretty much the entire function was refactored and the return
value of the function wasn't used, I changed the function signature
to not return anything.
For reasons unknown to me, we were calling PyErr_SetObject()
with the type of RevlogError and an instance of RevlogError. This
was equivalent to the Python code "raise RevlogError(RevlogError)".
This seemed wonky and completely unnecessary. The Python code only
cares about the type of the exception, not its contents. So I got
rid of this complexity.
This is my first Python C extension patch. Please give extra scrutiny
to it during review.
The problem was spotted at cd1b50e99ed8, that says "this patch invokes it
with "strtoken='rawstring'" in "_evalifliteral()", because "t" is the result
of "arg" evaluation and it should be "string-escape"-ed if "arg" is "string"
expression." This workaround is no longer valid since 72be08a15d8d introduced
strict parsing of '\{'.
Instead, we should interpret bare token as "string" or "rawstring" template.
This is what buildmap() does at parsing phase.
When moving crecord to core, I did a search an replace to remove all of the
variable with caps letter. Doing so, I unfortunately changed some commands
with capital letter shortcut to their lowercase counterpart. I fixed one
of these cases in bfcbb3995330. This patch fixes all of the remaining cases.
When moving crecord to core, I did a search an replace to remove all of the
variable with caps letter. Doing so, I unfortunately changed the 'F' command
to 'f' leading to two 'f' commands, one never used. This patch fixes this
mistake and re enables the behavior of 'F'.
Previously, if a subrepo parent had 'web.hidden=True' set, neither the parent
nor child had a repository entry. However, the directory entry for the parent
would be listed (it wouldn't have the fancy 'web.name' if configured), and that
link went to the repo's summary page, effectively making it not hidden.
This simply disables the directory processing if a valid repository is present.
Whether or not the subrepo should be hidden is debatable, but this leaves that
behavior unchanged (i.e. it stays hidden).
The previous scheme was:
1) lookup node for all pulled revision,
2) pull said node
3) lookup the node of the checkout target
4) update the repository there.
If the remote repo changes between (1) and (3), the resolved name will be
different and (3) crash. There is actually no need for a remote lookup during
(3), we could just set the value in (1). This prevent the race condition and
save a possible network roundtrip.
Previously, when 'web.name' was set on a subrepo parent and 'web.collapse=True',
the parent repo would show in the list with the configured 'web.name', and a
directory with the parent repo's filesystem name (with a trailing slash) would
also appear. The subrepo(s) would unexpectedly be excluded from the list of
repositories. Clicking the directory entry would go right to the repo page.
Now both the parent and the subrepos show up, without the additional directory
entry.
The configured hgweb paths used '**' for finding the repos in this scenario.
A couple of notes about the tests:
- The area where the subrepo was added has a comment that it tests subrepos,
though none previously existed there. One now does.
- The 'web.descend' option is required for collapse to work. I'm not sure what
the previous expectations were for the test. Nothing changed with it set,
prior to adding the code in this patch. It is however required for this test.
- The only output changes are for the hyperlinks, obviously because of the
'web.name' parameter.
- Without this code change, there would be an additional diff:
--- /usr/local/mercurial/tests/test-hgwebdir.t
+++ /usr/local/mercurial/tests/test-hgwebdir.t.err
@@ -951,7 +951,7 @@
/rcoll/notrepo/e/
/rcoll/notrepo/e/e2/
/rcoll/notrepo/f/
- /rcoll/notrepo/f/f2/
+ /rcoll/notrepo/f/
Test repositories inside intermediate directories
I'm not sure why the fancy name doesn't come out, but it is enough to
demonstrate that the parent is not listed redundantly, and the subrepo isn't
skipped.
Before this patch, template keywords `{file_mods}`, `{file_adds}` and
`{file_dels}` use values gotten by `repo.status(ctx.p1().node(),
ctx.node())`.
But this doesn't work as expected if `ctx` is `memctx` or
`workingcommitctx`. Typical case of templating with these contexts is
customization of the text shown in the commit message editor by
`[committemplate]` configuration.
In this case, `ctx.node()` returns None and it causes comparison
between `ctx.p1()` and `workingctx`. `workingctx` lists up all changed
files in the working directory even at selective committing.
BTW, `{files}` uses `ctx.files()` and it works as expected.
To compare target context and its parent exactly, this patch passes
`ctx.p1()` and `ctx` without `node()`-nize. This avoids unexpected
comparison with `workingctx`.
This patch uses a little redundant template configurations in
`test-commit.t`, but they are needed to avoid regression around
problems fixed by 17e2fda16f58 and 2b999bc2d89a: accessing on `ctx`
may break `ctx._status` field.
This makes _includeroots more like _fileroots and gives visitdir() a
nice symmetry in the two.
I'm hoping to later combine the two (_fileroots and _includeroots),
and having them treated similarly should make that step easier to
follow.
