I'm not sure if this is better. If we're planning to add a template keyword
that returns obsoleted nodes unavailable in the repo (i.e. they have no valid
revision numbers), we might want to use the current "node"-only format
everywhere.
context.py seems not a good place to host these functions.
% wc -l mercurial/context.py mercurial/dagop.py
2306 mercurial/context.py
424 mercurial/dagop.py
2730 total
This module hosts the following functions. They are somewhat similar (e.g.
scanning revisions using heap queue or stack) and seem non-trivial in
algorithmic point of view.
- _revancestors()
- _revdescendants()
- reachableroots()
- _toposort()
I was thinking of adding revset._fileancestors() generator for better follow()
implementation, but it would be called from context.py as well. So I decided
to create new module.
Naming is hard. I couldn't come up with any better module name, so it's called
"dag operation" now. I rejected the following candidates:
- ancestor.py - existing, revlog-level DAG algorithm
- ancestorset.py - doesn't always return a set
- dagalgorithm.py - hard to type
- dagutil.py - existing
- revancestor.py - I want to add fileancestors()
% wc -l mercurial/dagop.py mercurial/revset.py
339 mercurial/dagop.py
2020 mercurial/revset.py
2359 total
We move the feature to a proper configuration and document it. The config goes
in the 'server' section because it feels like something the server owner would
want to decide. We pick and open field because it seems likely that other
checking levels will emerge in the future. (eg: server like the mozilla-try
server will likely wants a "none" value)
The option name contains 'push' since this affects 'push' only. The option value
'check-related' is preferred over one explicitly containing 'allow' or 'deny'
because the client still have a strong decision power here. Here, the server is
just advising the client on the check mode to use.
Add a 'predecessors' template that returns the list of all closest known
predecessors for a changectx. The elements of the list are row changectx node id
formatted by default as short nodes.
The "closest predecessors" are the first locally known revisions encountered
while, walking predecessors markers. For example:
1) If a (A, (B)) markers exists and both A and B are locally known A is a
closest predecessors of B.
2) If a (A, (B)) and (B, (C)) markers exists and only A and C are known
locally, A will be the closest precursors of C.
This logic respect repository filtering. So hidden revision will be skipped by
this logic unless --hidden is specified. Since we only display the visible
predecessors, this template will not display anything in most case. It makes a
good candidate for inclusion in the default log output.
I added a new test-file for testing the precursors in various scenarios. This
test file will also be used for the successors template.
A new "obsutil" module has been added to start gathering utility function
outside of the large obsolete.py module.
Now spec.ref should be '' if spec.tmpl is specified. Since spec.ref is the
option to select the initial template to be rendered, it doesn't make sense
to store the given literal template as spec.ref.
Since a map file has another level to select a template (spec -> mapfile
-> topic), this isn't exactly the same as how a map file works. But I believe
most users would expect the new behavior.
A literal template is stored as an unnamed template so that it will never
conflict with the templates defined in [templates] section.
This provides a simpler API for callers which don't need full templating
stack. Instead of storing the given template as the name specified by topic,
use '' as the default template to be rendered.
We've been talking for years about a one-stop config knob to opt in to
better behavior. There have been a lot of ideas thrown around, but
they all seem to be too complicated to get anyone to actually do the
work.. As such, this patch is the stupidest thing that can possibly
work in the name of getting a good feature to users.
Right now it's just three config settings that I think are generally
uncontroversial, but I expect to add more soon. That will likely
include adding new config knobs for the express purpose of adding them
to tweakdefaults.
It seems like the reason for "content" is that the variable contains
the nodes that the changegroup "contains", see c2901cddb53f (revlog:
make addgroup returns a list of node contained in the added source,
2012-01-13), but "nodes" seems much clearer.
The comment seems to refer to code that was deleted in a46638c640d8
(revlog.addgroup(): always use _addrevision() to add new revlog
entries, 2010-10-08).
pycompat.py is unorganized and looks ugly. Next few patches will try to make it
look more cleaner so that adding more code is easy and reading code also.
This patch moves the multiline comments above functions to function docs. While
moving, I improved the comments and make them better suitable for func doc.
While I was here I drop a unrequired and misplaced comment.
statprof.display_hotpath() accepts a "limit" function to choose the
minimum threshold for samples to display. The default is 0.05, which
means you don't need individual items contributing less than 5%.
I had a need to adjust this threshold. We already have a config
option for it. So let's reuse it.
check-config.py doesn't like having multiple defaults for the
ui.configwith() calls. The behavior is obviously correct. I'm
not sure if it is worth teaching check-config.py how to ignore
this. So I've just accepted the new output.
This will allow us to change the initial template reference depending on how
the template is looked up. For example,
-Tdefault => (ref='changeset', tmpl=None, mapfile='map-cmdline.default')
-T'{rev}' => (ref='', tmpl='{rev}', mapfile=None)
A literal template given by -T option will be stored as an unnamed template,
which will free up the template namespace so that we can load named templates
from [templates] section of user config.
I'm going to add more options to the templatespec tuple.
cmdutil.logtemplatespec() is just an alias now, but it will be changed to
a factory function later.
changeset_templater has lots of arguments, but most callers only need to
specify a literal template 'tmpl'.
"hg debugtemplate" has no diff option, which means 'opts' were effectively {},
so dropped opts.
Since it's highly use-case dependent how template should be looked up,
gettemplater() function isn't useful. Keeping it would introduce another
bug I've made and fixed earlier in this series.
ui.graphnodetemplate config should be a literal template as ui.logtemplate is.
The use of formatter.gettemplater() is valid only for a template string
given by -T/--template option.
I believe it was a mistake to open file in text mode. Now '\r' is preserved
on Windows, but it should be okay to print '\r\n' as long as users live in
such platform.
Yuya Nishihara gave this suggestion on the mailing list after the
previous patch was queued, and honestly this seems much simpler and
looks more efficient.
This renames the spanset class to _spanset, and moves its __init__ to new
spanset() function. spanset() is now a factory function.
This allows us to construct a spanset without keeping a repo instance.
Inspired by the dirstate fix in 39954a8760cd, this should fix any race
conditions with the fsmonitor state changing from underneath.
Since we now grab the wlock for any non-invalidate writes, the only situation
this appears to happen in is with a concurrent invalidation. Test that.
Some extensions like fsmonitor need to run code after dirstate.status is
called, but while the wlock is held. The extensions could grab the wlock again,
but that has its own peculiar race issues. For example, fsmonitor would not
like its state to be written out if the dirstate has changed underneath (see
issue5581 for what can go wrong in that sort of case).
To protect against these sorts of issues, allow extensions to declare that they
would like to run some code to run at fixup time.
fsmonitor will switch to using this in the next patch in the series.
Context manager has a mechanism to control extension propagation. It is not
used by profiling right now, but making the code correct will help prevent bug
in the future.
There are no way for this to happen today, but better be safe than sorry, no
one know how the code will evolve. We now make sure the file pointer is closed
even is profiler is None.
This replaces 'if y in subset' with '& subset'. first(null) and last(wdir())
are fixed thanks to fullreposet.__and__.
This also revealed that first() and last() don't follow the order of the
input set. 'ls & subset' is valid only if the ordering requirement is 'define'
or 'any'.
No performance regression observed:
revset #0: limit(0:9999, 100, 9000)
0) 0.001164
1) 0.001135
revset #2: 9000 & limit(0:9999, 100, 9000)
0) 0.001224
1) 0.001181
revset #3: last(0:9999, 100)
0) 0.000237
1) 0.000199
last() is implemented using a reversed iterator, so the result should be
reversed again.
I've marked this as BC since it's quite old bug seen in 3.0. The first bad
revision is 1ef0875a62f8 "revset: changed last implementation to use lazy
classes."
Instead of walking all ancestors to compute the full set we now check membership
lazily. This massively speed.
On a million-ish revision repository, this remove 14 seconds from the push logic,
making the checkheads function disappear from profile.
The obsolete post-processing needs to know the extend of the pushed set. The
way it is implemented is... suboptimal. It build a full set of all nodes in the
pushset and it does so using changectx. We have much better API for this now.
The simplest is to use the existing lazy ancestors computation. That logic uses
revnum and not node (for good reason) so we start with updating the
post-processing code to handle a "allfuturecommon" set containing revision
numbers.
We are about to call 'torev' on node that might be locally missing. In this
case, 'nodemap.revs' will return None (something valid in our usecase) while
'changelog.rev' would raise an exception.
We make this change in a distinct changeset to show it does not impact the
tests.
We are about to make "allfuturecommon" a set of revs instead of a set of nodes.
The function updated in this patch do not needs to know about these details so
we just pass it a 'ispushed(node)' function. This will simplify the next
changeset.
It seems sufficiently simple to use "profile(enabled=X)" to not justify having
a dedicated context manager just to read the config.
(I do not have a too strong opinion about this).
We now process the "--profile" a second time after alias has been processed and
the command argument fully parsed. If appropriate we enable profiling at that
time.
In these situation, the --profile will cover less than if the full --profile
flag was passed on the command line. This is better than the previous behavior
(flag ignored) and still fullfil multiple valid usecases.
This is a step toward allowing context where the profiling in enabled
withing the context range.
This also open the way to kill the dedicated "maybeprofile" context manager
and keep only one of 'profile' and 'maybeprofile'.
The start method is doing all profiler setup and activation. It is currently
unconditionally called by '__init__' but this will be made more flexible in
later changesets.
So far we have been able to use a simple decorator for this. However using the
current context manager makes the scope of the profiling in dispatch
constrainted and the time frame to decide to enable profiling quite limited
(using "maybeprofile")
This is the first step toward the ability to enable the profiling from within
the profiling scope. eg::
with maybeprofiling(ui) as profiler:
...
bar.foo():
...
if options['profile']:
profiler.start()
...
fooz()
...
My target usecase is adding support for "--profile" to alias definitions with
effect. These are to be used with "profiling.output=blackbox" to gather data
about operation that get slow from time to time (eg: pull being minutes instead
of seconds from time to time).
Of course, in such case, the scope of the profiling would be smaller since
profiler would be started after running extensions 'reposetup' (and other
potentially costly logic), but these are not relevant for my target usecase
(multiple second commits, multiple tens of seconds pull).
Currently adding '--profile' to a command through alias requires to re-spin a
Mercurial binary (using "!$HG" in alias), which as a significant performance
impact, especially in context where startup performance is being worked on...
An alternative approach would be to stop using the context manager in dispatch
and move back to a try/finally setup.
This can happen if another process (even another hg process!) comes along and
removes the file at that time.
This partly resolves issue5584, but not completely -- a bogus dirstate update
can still happen. However, the full fix is too involved for stable.
The 'srvheads' list contains all server heads including the common ones. We
adjust 'ui.log' message to provide more useful information about server heads
locally unknown. The performance impact of turning the list to set is
negligible (about 1e-4s) compared to the rest of the discovery cost, so I'm
taking the easy path.
gitweb was missing the hint hover box. So that was added.
Also, the positioning of the form was absolute and it didn't
vertically align on all pages. The element has been moved inline
with the navigation links (which now are contained in a div) and
flexbox is used to obtain sane alignment of the navigation links
and search form. For those new to flexbox,
"justify-content: space-between" basically says to maximize space
elements. You can use it to easily get left and right justified
containers without having to worry about width, floating, etc.
"align-items: center" centers all items in a cross-axis. I've
literally wasted hours trying to figure out both these problems
before flexbox. Flexbox is amazing.
Flexbox has been supported by Chrome and Firefox for a few years.
But it is only supported by IE 11. I'm willing to wager that
people using this either won't be using IE or will be using IE 11.
So I'm willing to be a bit aggressive in adopting flexbox because
it makes CSS alignment so much easier.
Paper has it on all pages. Not sure why gitweb doesn't. I think it
should be everywhere because it is a useful feature.
Also, we weren't consistently adding the HTML in the same place. This
was OK since the element is absolutely positioned. But this bothered
me a bit, so I went ahead and fixed it.
AFAICT this was mostly a bunch of copy pasta. The only variation is
some pages defined a "value" attribute. The "query" variable will
just be empty on pages that don't accept it. So let's consolidate
the template and remove the redundancy.
Before this patch, workingctx.status() may cause writing outdated
dirstate out, if:
- .hg/dirstate is changed simultaneously after last loading it,
- there is any file, which should be dirstate.normal()-ed
Typical issue case is:
- the working directory is updated by "hg update"
- .hg/dirstate is updated in background (e.g. fsmonitor)
This patch compares identities of dirstate before and after
acquisition of wlock, and avoids writing outdated dirstate out, if
change of .hg/dirstate is detected.
This identity is used to examine whether dirstate is simultaneously
changed in storage after previous caching (see issue5584 for detail).
util.cachestat can't be used for this purpose, because it has no
valuable information on Windows.
On the other hand, util.filestat can detect changing dirstate in
storage certainly, regardless of platforms.
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
Strictly speaking, if underlying filesystem doesn't support
ctime/mtime, util.filestat can't detect simultaneous changing in
storage as expected. But simultaneous changing on such (very rare)
platform can't be detected regardless of this patch series.
Therefore, util.filestat should be reasonable identity for almost all
usecases.
In order to fix issue5418, 0d920bcb0fd1 made vfs.rename(checkambig=True)
omit advancing mtime of renamed file, if renamed file is owned by
another (EPERM is raised in this case).
But this omission causes rewinding mtime at restoration in such
situation, and makes avoiding file stat ambiguity difficult, because
ExactCacheValidationPlan assumes that mtime should be advanced, if a
file is changed in same ctime.
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
Ambiguity of file stat also requires issue5584 to be fixed with other
than file stat, but "hash of file", "generation ID" and so on were
already rejected ideas (please see original RFC linked from "Outline
of issue" in ExactCacheValidationPlan page).
This omission occurs:
- only for non append-only files (dirstate, bookmarks, and phaseroots), and
- only if previous transaction is rollbacked by another user
The latter means "sharing a repository clone via group permission".
This is reasonable usecase, but not ordinary for many users, IMHO.
"hg rollback" itself has been deprecated since Mercurial 2.7, too.
Therefore, increasing the cost at rollbacking previous transaction
executed by another a little seems reasonable, for avoidance of file
stat ambiguity.
This patch does:
- create copy of (already renamed) source file, if advancing mtime
fails for EPERM
- rename from copied file to destination file, and
- advance mtime of renamed file, which is now owned by current user
This patch also factors "self.join(src)" out to reduce redundancy.
This seems like a prudent thing to do. As the inline comment says,
we may want to make this abort once the functionality is stabilized
as part of `hg bundle`. Let's save that debate for another day.
Previously, a repo containing secret changesets would be served via
stream clone, transferring those secret changesets. While secret
changesets aren't meant to imply strong security (if you really
want to keep them secret, others shouldn't have read access to the
repo), we should at least make an effort to protect secret changesets
when possible.
After this commit, we no longer serve stream clones for repos
containing secret changesets by default. This is backwards
incompatible behavior. In case anyone is relying on the behavior,
we provide a config option to opt into the old behavior.
Note that this defense is only beneficial for remote repos
accessed via the wire protocol: if a client has access to the
files backing a repo, they can get to the raw data and see secret
revisions.
Since 1d07d9da84a0, pycompat.bytestr() wrapped by win32mbcs returns
unicode object, if an argument is not byte-str object. And this causes
unexpected failure at colorization.
pycompat.bytestr() is used to convert from color effect "int" value to
byte-str object in color module. Wrapped pycompat.bytestr() returns
unicode object for such "int" value, because it isn't byte-str.
If this returned unicode object is used to colorize non-ASCII byte-str
in cases below, UnicodeDecodeError is raised at an operation between
them.
- colorization uses "ansi" color mode, or
Even though this isn't default on Windows, user might use this
color mode for third party pager.
- ui.write() is buffered with labeled=True
Buffering causes "ansi" color mode internally, regardless of
actual color mode. With "win32" color mode, extra escape sequences
are omitted at writing data out.
For example, with "win32" color mode, "hg status" doesn't fail for
non-ASCII filenames, but "hg log" does for non-ASCII text, because
the latter implies buffered formatter.
There are many "color effect" value lines in color.py, and making them
byte-str objects isn't suitable for fixing on stable. In addition to
it, pycompat.bytestr will be used to get byte-str object from any
types other than int, too.
To resolve this issue, this patch does:
- replace pycompat.bytestr in checkwinfilename() with newly added
hook point util._filenamebytestr, and
- make win32mbcs reverse-wrap util._filenamebytestr
(this is a replacement of 1d07d9da84a0)
This patch does two things above at same time, because separately
applying the former change adds broken revision (from point of view of
win32mbcs) to stable branch.
"_" prefix is added to "filenamebytestr", because it is win32mbcs
specific hook point.
This commit essentially reverts 62ad9c1dbce9.
urllib2.URLError receives a "reason" argument. It isn't always a
tuple. Mozilla has experienced at least IndexError failures due
to the reason[1] access.
https://bugzilla.mozilla.org/show_bug.cgi?id=1364687
Before this patch, "hg CMD --pager on" on Windows shows output
unintentionally decorated with ANSI color escape sequences, if color
mode is "auto". This issue occurs in steps below.
1. dispatch() invokes ui.pager() at detection of "--pager on"
2. stdout of hg process is redirected into stdin of pager process
3. "ui.formatted" = True, because isatty(stdout) is so before (2)
4. color module is loaded for colorization
5. color.w32effects = None, because GetConsoleScreenBufferInfo()
fails on stdout redirected at (2)
6. "ansi" color mode is chosen, because of "not w32effects"
7. output is colorized in "ansi" mode because of "ui.formatted" = True
Even if "ansi" color mode is chosen, ordinarily redirected stdout
makes ui.formatted() return False, and colorization is avoided. But in
this issue case, "ui.formatted" = True at (3) forces output to be
colorized.
For correct console information on win32, it is needed to ensure that
color module is loaded before redirection of stdout for pagination.
BTW, if any of enabled extensions has "colortable" attribute, this
issue is avoided even before this patch, because color module is
imported as a part of loading such extension, and extension loading
occurs before setting up pager. For example, mq and keyword have
"colortable".
Adding markers to the repository might affect the set of obsolete changesets. So we
most remove the "volatile" set who rely in that data. We add two missing
invalidations after merging markers. This was caught by code change in the evolve
extensions tests.
This issues highlight that the current way to do things is a bit fragile,
however we keep things simple for stable.
Graft used findmissingrevs to find the candidates for graft duplicates in the
destination. That function operates with the constraint:
1. N is an ancestor of some node in 'heads'
2. N is not an ancestor of any node in 'common'
For our purpose, we do however have to work correctly in cases where the graft
set has multiple roots or where merges between graft ranges are skipped. The
only changesets we can be sure doesn't have ancestors that are grafts of any
changeset in the graftset, are the ones that are common ancestors of *all*
changesets in the graftset. We thus need:
2. N is not an ancestor of all nodes in 'common'
This change will graft more correctly, but it will also in some cases make
graft slower by making it search through a bigger and unnecessary large sets of
changes to find duplicates. In the general case of grafting individual or
linear sets, we do the same amount of work as before.
"merge-tools" help topic has described that the merge of the file
fails if no tool is found to merge binary or symlink, since
9da9bced2226 (or Mercurial 1.7), which based on (already removed)
MergeProgram wiki page.
But even at that revision, and of course now, merge of the file
doesn't fail automatically for binary/symlink. ":prompt" (or
equivalent logic) is used, if there is no appropriate tool
configuration for binary/symlink.
As an aside, I'm having trouble parsing the help text meaning for HG when it is
unset or empty. How can it be the frozen name or searched if it is empty?
This may be too subtle of a change to get the point across, but when I first
read the original text, I thought maybe the pager would only be invoked if
writing more than a screenful. The distinction between this and a pager that
simply exits after printing less than a screenful is important on Windows, given
the inability of `more` to color output.
Even though I figured this out a few weeks ago, I was initially puzzled where
the color went when I upgraded to 4.2 on a different Windows machine. Let's
point users reading the help into the right direction.
I wonder if we should be even more explicit about cmd.exe/MSYS/pager/color
interplay, but at least all of the breadcrumbs are here (I think).
This option was never released except for a release candidate. Dropping
compatibility with this option will free the 'pager.enable' config option for
other usage in the future.
This aligns with what we do for color (see cea7a760c58d). Pager is a central
enough notion that having the master config in the [ui] section makes senses. It
will helps with consistency, discoverability. It will also help having a simple
and clear example hgrc mentioning pager.
The previous form of the option had never been released in a non-rc version but
we keep it around for convenience. If both are set, 'ui.pager' take priority.
Previously, 'ui.color=yes' meant "always show color", While
"ui.color=auto" meant "use color automatically when it appears
sensible".
This feels problematic to some people because if an administrator has
disabled color with "ui.color=off", and a user turn it back on using
"color=on", it will get surprised (because it breaks their output when
redirected to a file.) This patch changes ui.color=true to only move the
default value of --color from "never" to "auto".
I'm not really in favor of this changes as I suspect the above case will
be pretty rare and I would rather keep the logic simpler. However, I'm
providing this patch to help the 4.2 release in the case were others
decide to make this changes.
Users that want to force colors without specifying --color on the
command line can use the 'ui.formatted' config knob, which had to be
enabled in a handful of tests for this patch.
Nice summary table (credit: Augie Fackler)
That is, before this patch:
+--------------------+--------------------+--------------------+
| | not a tty | a tty |
| | --color not set | --color not set |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color (not set) | no color | no color |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color = auto | no color | color |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color = yes | *color* | color |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color = no | no color | no color |
| | | |
+--------------------+--------------------+--------------------+
(if --color is specified, it always clobbers the setting in [ui])
and after this patch:
+--------------------+--------------------+--------------------+
| | not a tty | a tty |
| | --color not set | --color not set |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color (not set) | no color | no color |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color = auto | no color | color |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color = yes | *no color* | color |
| | | |
+--------------------+--------------------+--------------------+
| [ui] | | |
| color = no | no color | no color |
| | | |
+--------------------+--------------------+--------------------+
(if --color is specified, it always clobbers the setting in [ui])
"Churn" is not the useful example we have, but this is the one used in
'hg help config.extensions'. As we need something to replace the deprecated
'pager' extension in the example config, we are adding 'churn'.
Since we no longer set '_clean = False' during the initialization loop, we can
move the attribute assignment earlier in the function for clarity.
(no speed improvement expected or measured ;-) )
The bmstore '__setitem__' method is setting an extra flag that is not needed
during initialization. Skipping the method will allow further cleanup and yield
some speedup as a side effect.
Before:
! wall 0.009120 comb 0.010000 user 0.010000 sys 0.000000 (best of 312)
After:
! wall 0.007874 comb 0.010000 user 0.010000 sys 0.000000 (best of 360)
Since we already have an exception context open, for other thing, we can
simplify the code a bit and rely on exception handling for invalid lines.
Speed is not the main motivation for this changes. However as I'm in the middle
of benchmarking things we can see a small positive impact.
Before:
! wall 0.009358 comb 0.000000 user 0.000000 sys 0.000000 (best of 303)
After:
! wall 0.009173 comb 0.010000 user 0.010000 sys 0.000000 (best of 310)
We know the content of the file is supposed to be full hex. So we can do the
translation ourselves and directly check if the node is known.
As nice side effect we now have proper error handling for invalid node value.
Before:
! wall 0.021580 comb 0.020000 user 0.020000 sys 0.000000 (best of 134)
After:
! wall 0.009342 comb 0.010000 user 0.010000 sys 0.000000 (best of 302)
Skipping the attribute lookup up raise a significant speedup.
Example on a repository with about 4000 bookmarks.
Before:
! wall 0.026027 comb 0.020000 user 0.020000 sys 0.000000 (best of 112)
After:
! wall 0.021580 comb 0.020000 user 0.020000 sys 0.000000 (best of 134)
(This is also in its own changeset to clarify the perf win from another coming
changesets)
This is obviously an instance attribute, not a type attribute. The
modern Python style is to use __init__ for defining these.
This exposes statichttprepo as inheriting from localrepository
without calling its __init__. As a result, its __init__ defines
a lot of variables that methods on localrepository's methods need.
But factoring the common bits into a separate class is for another
day.
match.match already interprets "!bool(patterns)" as matching
everything (but includes and excludes still apply). We might as well
allow None, which lets us simplify some callers a bit.
I originally wrote this patch while trying to change
match.match(patterns=[]) to mean to match no patterns. This patch is
one step towards that goal. I'm not sure it'll be worth the effort to
go all the way there, but I think this patch still makes sense on its
own.
For performance reasons we have several repositories where the files in the working
directory of 1 repo are hardlinks to the files of the other repo
When an update in one repo results in a chmod of a such a file, the hardlink
has to be deleted and replaced by a regular file to make sure that the change
does not happen in the other repo
We log the discovery summary, the number of roundtrips and the elapsed time.
This is useful to understand where slow push might come from when lloking at
the blackbox.
Client has a mechanism for the server to check that nothing changed server side
since the client prepared a push. That check is wide and any head changed on
the server will lead to an aborted push. We introduce a way for the client to
send a less strict checking. That logic will check that no heads impacted by
the push have been affected. If other unrelated heads (including named branches
heads) have been affected, the push will proceed.
This is very helpful for repositories with high developers traffic on different
heads, a common setup.
That behavior is currently controlled by an experimental option. The config
should live in the "server" section but bike-shedding of the name will happen
in the next changesets. Servers advertise this capability through a new bundle2
capability 'checkeads', using the value 'related'.
The 'test-push-race.t' is updated to check that new capabilities on the
documented cases.
The goal is to have the function directly return something meaningful and
useful for the whole pull.
Note: we skip adding post-processing in '_oldheadssummary' because if a client
is too old for branchmap it will be too old for obsolescence too.
Our goal is to be able to perform the post processing directly into the
'_headssummary' function. However before this patch the '_headsummary' function
only had access to repo, remote, outgoing while the '_postprocessobsolete'
function takes a 'pushop' object. Experience shows that having the 'pushop'
object helps extensions so we update '_headssummary' to take a pushop
object as argument.
We extract this function from the loop and gather it with the rest of the
obsolescence specific code. That will help to clarify the move of the whole logic
inside the "heads summary" computation.
Negative offsets to the `~` operator now search for descendents. The search is
aborted when a node has more than one child as we do not have a definition for
'nth child'. Optionally we can introduce such a notion and take the nth child
ordered by rev number.
The current revset language does provides a short operator for ancestor lookup
but not for descendents. This gives user a simple revset to move to the previous
changeset, e.g. `hg up '.~1'` but not to the 'next' changeset. With this change
userse can now use `.~-1` as a shortcut to move to the next changeset.
This fits better into allowing users to specify revisions via revsets and
avoiding the need for special `hg next` and `hg prev` operations.
The alternative to negative offsets is adding a new operator. We do not have
many operators in ascii left that do not require bash escaping (',', '_', and
'/' come to mind). If we decide that we should add a more convenient short
operator such as ('/', e.g. './1') we can later add it and allow ascendents
lookup via negative numbers.
Currently when we have multiple heads on the same branch, update tells us that
there some more heads for the current branch but does not tells us the head to
which the repository has been updated to. It makes more sense showing the
head we updated to and then telling there are some more heads.
There are a number of improvements we want to make to revlogs
that will require a new version - version 2. It is unclear what the
full set of improvements will be or when we'll be done with them.
What I do know is that the process will likely take longer than a
single release, will require input from various stakeholders to
evaluate changes, and will have many contentious debates and
bikeshedding.
It is unrealistic to develop revlog version 2 up front: there
are just too many uncertainties that we won't know until things
are implemented and experiments are run. Some changes will also
be invasive and prone to bit rot, so sitting on dozens of patches
is not practical.
This commit introduces skeleton support for version 2 revlogs in
a way that is flexible and not bound by backwards compatibility
concerns.
An experimental repo requirement for denoting revlog v2 has been
added. The requirement string has a sub-version component to it.
This will allow us to declare multiple requirements in the course
of developing revlog v2. Whenever we change the in-development
revlog v2 format, we can tweak the string, creating a new
requirement and locking out old clients. This will allow us to
make as many backwards incompatible changes and experiments to
revlog v2 as we want. In other words, we can land code and make
meaningful progress towards revlog v2 while still maintaining
extreme format flexibility up until the point we freeze the
format and remove the experimental labels.
To enable the new repo requirement, you must supply an experimental
and undocumented config option. But not just any boolean flag
will do: you need to explicitly use a value that no sane person
should ever type. This is an additional guard against enabling
revlog v2 on an installation it shouldn't be enabled on. The
specific scenario I'm trying to prevent is say a user with a
4.4 client with a frozen format enabling the option but then
downgrading to 4.3 and accidentally creating repos with an
outdated and unsupported repo format. Requiring a "challenge"
string should prevent this.
Because the format is not yet finalized and I don't want to take
any chances, revlog v2's version is currently 0xDEAD. I figure
squatting on a value we're likely never to use as an actual revlog
version to mean "internal testing only" is acceptable. And
"dead" is easily recognized as something meaningful.
There is a bunch of cleanup that is needed before work on revlog
v2 begins in earnest. I plan on doing that work once this patch
is accepted and we're comfortable with the idea of starting down
this path.
Since 348863ccec7e "util: always force line buffered stdout when stdout is
a tty", we have two file objects attached to the same STDOUT_FILENO. If one
is closed, the underlying file descriptor is also closed, and writing to
the other file object would crash the Python interpreter in a hard way, at
least on Windows.
So, it seems safer to not close the standard streams. This also matches
the behavior of the default sys.stdout/stderr.close(), which never close
the FILE* streams in C layer.
https://hg.python.org/cpython/file/v2.7.13/Python/sysmodule.c#l1401
It's been there since 84af5a079c7d (2007-02-19), but seems wrong since any
I/O operations to a closed file would raise ValueError, not IOError. We should
keep the file object open even if the underlying file descriptor is half dead.
If a marker has no parent information, parents field is set to None, which
caused TypeError. I think this shouldn't normally happen, but somehow
I have such marker in my repo.
We point out at the help of the config option for user who wants to learn more
about the possible config value.
The suggested command returns some other extra (color related) results, but this
is bug to fix outside of the freeze.
The --color option is described to accept "boolean, always, auto,
never, or debug". Let's use a similar description for ui.color. Also
fix the formatting to use double quotes as we seem to use for about
half the values in config.txt (the other half uses double
backticks). Also use uppercase Boolean for consistency within the
file.
Acquiring lock by vfs.makelock() and getting lock holder (aka
"locker") information by vfs.readlock() aren't atomic action.
Therefore, failure of the former doesn't ensure success of the latter.
Before this patch, lock is unintentionally acquired, if ENOENT
causes failure of vfs.readlock() while 5 times retrying, because
lock._trylock() returns to caller silently after retrying, and
lock.lock() assumes that lock._trylock() returns successfully only if
lock is acquired.
In this case, lock symlink (or file) isn't created, even though lock
is treated as acquired in memory.
To avoid this issue, this patch makes lock._trylock() raise
LockHeld(EAGAIN) at the end of it, if lock isn't acquired while
retrying.
An empty "locker" meaning "busy for frequent lock/unlock by many
processes" might appear in an abortion message, if lock acquisition
fails. Therefore, this patch also does:
- use '%r' to increase visibility of "locker", even if it is empty
- show hint message to explain what empty "locker" means
Acquiring lock by vfs.makelock() and getting lock holder (aka
"locker") information by vfs.readlock() aren't atomic action.
Therefore, failure of the former doesn't ensure success of the latter.
Before this patch, lock is unintentionally acquired, if
self.parentlock is None (this is default value), and lock._readlock()
returns None for ENOENT at vfs.readlock(), because these None are
recognized as equal to each other.
In this case, lock symlink (or file) isn't created, even though lock
is treated as acquired in memory.
To avoid this issue, this patch retries lock acquisition immediately,
if lock._readlock() returns None "locker".
This issue will be covered by a test added in subsequent patch,
because simple emulation of ENOENT at vfs.readlock() easily causes
another issue. "raising ENOENT only at the first vfs.readlock()
invocation" is too complicated for unit test, IMHO.
Another raising PeerTransportError for "incomplete response" in
httppeer.py uses this (changed) hint message. This unification reduces
cost of translation.
There are some paragraphs, which aren't rendered in online help as
expected because of indentation and literal blocking issues.
- hgext/rebase.py
- paragraph before example code ends with ":", which treats
subsequent indented paragraphs as normal block
=> replace ":" with "::" to treat subsequent paragraphs as literal block
- help/pager.txt
- paragraph before a list of --pager option values ends with "::",
which treats subsequent indented paragraphs as literal block
=> replace "::" with ":" to treat subsequent paragraphs as normal block
- the second line of explanation for no/off --pager option value is
indented incorrectly (this also causes failure of "make" in doc)
=> indent correctly
- help/revisions.txt
- explanation following example code of "[revsetalias]" section
isn't suitable for literal block
=> un-indent explanation paragraph to treat it as normal block
- indentation of "For example" before example of tag() revset
predicate matching is meaningless
- descriptive text for tag() revset predicate matching isn't
suitable for literal block
=> un-indent concatenated two paragraphs to treat them as normal block
When on a filelog head, we are certain that there will be no descendant so the
target of the "descending" link will lead to an empty log result. Do not
display the link in this case.
We set parent._descendantrev = child.rev() when walking parents in
blockancestors() so that, when linkrev adjustment is perform for these, it
starts from a close descendant instead of possibly topmost introrev. (See
`self._adjustlinkrev(self._descendantrev)` in filectx._changeid().)
This is similar to changeset 8758896efb1c, which added a "f._changeid"
instruction in annotate() for the same purpose.
However, here, we set _descendantrev explicitly instead of relying on the
'_changeid' cached property being accessed (with effect to set _changeid
attribute) so that, in _parentfilectx() (called from parents()), we go through
`if '_changeid' in vars(self) [...]` branch in which instruction
`fctx._descendantrev = self.rev()` finally appears and does what we want.
With this, we can roughly get a 3x speedup (including in example of issue5538
from mozilla-central repository) on usage of followlines revset (and
equivalent hgweb request).
Update the hunk selector help message to use the operation name instead
of using "record" for all operations. Extract the help message in the same way
as other single and multiple message line.
Update tests to make sure that both "revert" and "discard" variants are tested.
Previously, calling blockancestors() with a fctx not touching file would
sometimes yield this filectx first, instead of the first "block ancestor",
because when compared to its parent it may have changes in specified line
range despite not touching the file at all.
Fixing this by starting the algorithm from the "base" filectx obtained using
fctx.introrev() (as done in annotate()).
In tests, add a changeset not touching file we want to follow lines of to
cover this case. Do this in test-annotate.t for followlines revset tests and
in test-hgweb-filelog.t for /log/<rev>/<file>?linerange=<from>:<to> tests.
Resolves test failures on FreeBSD, but I'm not happy about the fix.
A previous version of this also wrapped readline by putting the hack
in the _call method on doublepipe. That was confusing for readers and
wasn't necessary - just doing this on read() is sufficient to fix the
bugs I'm observing. We can always come back and do readline later if
needed.
`hg show work` is much more usable if output is colored. This patch
implements coloring via label() in a very hacky way.
In a default Mercurial install, you'll see yellow node labels for all
phases. Branches and bookmarks use the same formatting as the commit
message. So this change doesn't help much in a default install. But if
you have a custom colors defined for these things, output is much more
readable.
The implementation obviously needs some work. But for a minor change
on a feature that isn't convered by BC, this seems like a clear win
for the feature in 4.2.
Durham and I both like this better than "underway." We can add aliases
and bikeshed on the name during the 4.3 cycle, as this whole extension is
highly experimental.
See the inline comment how this could mitigate the issue.
I couldn't reproduce the exact problem on my Linux machine, but there are
at least two people who got EINTR in progress.py, and it seems file_write()
of Python 2 is fundamentally broken [1]. Let's make something in on 4.2.
[1]: https://hg.python.org/cpython/file/v2.7.13/Objects/fileobject.c#l1850
Some shared-ssh installations assume that 'hg serve --stdio' is a safe
command to run for minimally trusted users. Unfortunately, the messy
implementation of argument parsing here meant that trying to access a
repo named '--debugger' would give the user a pdb prompt, thereby
sidestepping any hoped-for sandboxing. Serving repositories over HTTP(S)
is unaffected.
We're not currently hardening any subcommands other than 'serve'. If
your service exposes other commands to users with arbitrary repository
names, it is imperative that you defend against repository names of
'--debugger' and anything starting with '--config'.
The read-only mode of hg-ssh stopped working because it provided its hook
configuration to "hg serve --stdio" via --config parameter. This is banned for
security reasons now. This patch switches it to directly call ui.setconfig().
If your custom hosting infrastructure relies on passing --config to
"hg serve --stdio", you'll need to find a different way to get that configuration
into Mercurial, either by using ui.setconfig() as hg-ssh does in this patch,
or by placing an hgrc file someplace where Mercurial will read it.
mitrandir@fb.com provided some extra fixes for the dispatch code and
for hg-ssh in places that I overlooked.
If we use the normal vfs, store encoding will be applied when we
.join() the path to be copied. This results in attempting to copy
a file that (likely) doesn't exist. Using the rawvfs operates on
the raw file path, which is returned by vfs.readdir().
Users at Mozilla are encountering this, as I've instructed them to
run `hg debugupgraderepo` to upgrade to generaldelta. While Mercurial
shouldn't deposit any files under .hg/store that require encoding, it
is possible for e.g. .DS_Store files to be created by the operating
system.
Previously, the "oldheads" variable was a list. On a repository at
Mozilla with 46,492 heads, profiling revealed that list membership
testing was dominating execution time of applying small changegroups.
This patch converts the list of old heads to a set. This makes
membership testing significantly faster. On the aforementioned
repository with 46,492 heads:
$ hg unbundle <file with 1 changeset>
before: 18.535s wall
after: 1.303s
Consumers of this variable only check for truthiness (`if oldheads`),
length (`len(oldheads)`), and (most importantly) item membership
(`h not in oldheads` - which occurs twice). So, the change to a set
should be safe and suitable for stable.
The practical effect of this change is that changegroup application
and related operations (like `hg push`) no longer exhibit an O(n^2)
CPU explosion as the number of heads grows.
Previously, `hg bundle zstd` on a non-generaldelta repo would
attempt to use a v1 bundle. This would fail because zstd is not
supported on v1 bundles.
This patch changes the behavior to automatically use a v2 bundle
when the user explicitly requests a bundlespec that is a compression
engine not supported on v1. If the bundlespec is <engine>-v1, it is
still explicitly rejected because that request cannot be fulfilled.
Version 1 bundles only support a fixed set of compression engines.
Before this change, we would accept any compression engine for v1
bundles, even those that may not work on v1. This could lead to
an error.
We define a fixed set of compression engines known to work with v1
bundles and we add checking to ensure a newer engine (like zstd)
won't work with v1 bundles.
I also took the liberty of adding test coverage for unknown compression
names because I noticed we didn't have coverage of it before.
For consistency with the other template options. Quotes are necessary if
you want to preserve leading/trailing whitespace, which would be stripped
by config parser.
I got the following error by running "hg log" and quitting the pager
immediately. Any output here may trigger another SIGPIPE, so only thing
we can do is to swallow the exception and exit with an error status.
Traceback (most recent call last):
File "./hg", line 45, in <module>
mercurial.dispatch.run()
File "mercurial/dispatch.py", line 83, in run
status = (dispatch(req) or 0) & 255
File "mercurial/dispatch.py", line 167, in dispatch
req.ui.warn(_("interrupted!\n"))
File "mercurial/ui.py", line 1224, in warn
self.write_err(*msg, **opts)
File "mercurial/ui.py", line 790, in write_err
self._write_err(*msgs, **opts)
File "mercurial/ui.py", line 798, in _write_err
self.ferr.write(a)
File "mercurial/ui.py", line 129, in _catchterm
raise error.SignalInterrupt
mercurial.error.SignalInterrupt
Perhaps this wasn't visible before ee4f321cd621 because the original stderr
handle was restored very late.
Before, it crashed because mapping['templ'] was missing. As it didn't support
the legacy list template from the beginning, we can simply use hybridlist().
See the previous commit for why.
splitlines() does not pass a mapping dict, which would probably mean the
legacy template didn't work from the beginning.
Now that all known format variants exists outside of the function, we can gather
them in a lists. This build a single entry point other code can use (current
target: extensions).
The repository upgrade code is updated to simply use entries from this list.
As a side effect this will also allow extensions to register their own format
variants, to do this "properly" we should introduce a "registrar" for this
category of object. However I prefer to keep this series simple, and that will
be adventure for future time.
Our goal here is to get top level definition for all the format variants. Having
them defined outside of the function enabled other users of that logic.
They are two keys components of a format variant:
1) the name and various descriptions of its effect,
2) the code that checks if the repo is using this variant and if the config
enables it.
That second items make us pick a class-based approach, since different variants
requires different code (even if in practice, many can reuse the same logic).
Each variants define its own class that is then used like a singleton. The
class-based approach also clarify the definitions part a bit since each are
simple assignment in an indented block.
The 'fromdefault' and 'fromconfig' are respectively replaced by a class
attribute and a method to be called at the one place where "fromconfig"
matters.
Overall, they are many viable approach for this, but this is the one I picked.
The 'deficiency' type has multiple specificities. We create a dedicated class to
host them. More logic will be added incrementally in future changesets.
I've not found anything related to color + windows on the bug tracker. So I'm
suggesting we get bolder and turn it on for windows too in the release
candidate. We can always backout this changeset if we find serious issue on
windows.
Color support is all in core for a couple of months. I've browsed the bug tracker
without finding any blocker bug. So I'm moving forward and enable color on by
default before '4.2-rc'. In the worse case, having it on in the release
candidate will help us to find blocker bug and we can turn it off for the final
release.
I remember people talking about issue with Windows during the freeze so I'm
keeping it off by default on that OS.
We could do various cleaning of the color used and the label issued. However
the label are probably already in our backward compatibility envelope since the
color extensions has been around since for ever and I do not think the color
choice themself should be considered BC. So I think we should rather gives color
to all user sooner than later.
A couple of test needs to be updated to avoid having color related control code
spoil the tested output.
Many have seen a "stream ended unexpectedly" error. This message is
raised from changegroup.readexactly() when a read(n) operation fails
to return exactly N bytes.
I believe most occurrences of this error in the wild stem from
the code changed in this patch. Before, if bundle2's part applicator
raised an Exception when processing/applying parts, the exception
handler would attempt to iterate the remaining parts. If I/O
during this iteration failed, it would likely raise the
"stream ended unexpectedly" exception.
The problem with this approach is that if we already encountered
an I/O error iterating the bundle2 data during application, then
any further I/O would almost certainly fail. If the original stream
were closed, changegroup.readexactly() would obtain an empty
string, triggering "stream ended unexpectedly" with "got 0." This is
the error message that users would see. What's worse is that the
original I/O related exception would be lost since a new exception
would be raised. This made debugging the actual I/O failure
effectively impossible.
This patch changes the exception handler for bundle2 application to
ignore errors when seeking the underlying stream. When the underlying
error is I/O related, the seek should fail fast and the original
exception will be re-raised. The output changes in
test-http-bad-server.t demonstrate this.
When the underlying error is not I/O related and the stream can be
seeked, the old behavior is preserved.
As part of writing test-http-bad-server.t, I noticed that some
requests include an empty Vary HTTP request header.
The Vary HTTP request header indicates which headers should be taken
into account when determining if a cached response can be used. It also
accepts the special value of "*".
The previous code unconditionally added a Vary header. This could lead
to an empty header value. While I don't believe this violates the HTTP
spec, this is weird and just wastes bytes. So this patch changes
behavior to only send a Vary header when it has a value.
Some low-level wire protocol byte reporting tests changed. In some
cases, the exact point of data termination changed. However, the
behavior being tested - that clients react when the connection is
closed in the middle of an HTTP request line or header - remains
unchanged.
The previous logic had many shortcoming (eg: looking at the head only, not
handling prune, etc...), the new logic use a more robust approach:
For each head, we check if after the push all changesets exclusive to this heads
will be obsolete. If they are, the branch considered be "replaced".
To check if a changeset will be obsolete, we simply checks:
* the changeset phase
* the existence of a marker relevant to the "pushed set" that affects the
changesets..
This fixes two major issues of the previous algorithm:
* branch partially rewritten (eg: head but not root) are no longer detected as
replaced,
* Prune are now properly handled.
(This implementation was introduction in the evolve extension, version 6.0.0.)
This new algorithm has an extended number of tests, a basic one is provided
in this patch. The others will be introduced in their own changeset for clarity.
In addition, we stop trying to process heads unknown locally, we do not have
enough data to take an informed decision so we should stop pretending we do.
This reflect a test that is now update.
I've been using `hg serve --web-conf ...` with a simple '/=projects/**' [paths]
configuration for awhile without issue. Let's ditch the need for the manual
configuration in this case, and limit the repos served to the actual subrepos.
This doesn't attempt to handle the case where a new subrepo appears while the
server is running. That could probably be handled with a hook if somebody wants
it. But it's such a rare case, it probably doesn't matter for the temporary
serves.
The main repo is served at '/', just like a repository without subrepos. I'm
not sure why the duplicate 'adding ...' lines appear on Linux. They don't
appear on Windows (see 3f4ff1bdf101), so they are optional.
Subrepositories that are configured with '../path' or absolute paths are not
cloneable from the server. (They aren't cloneable locally either, unless they
also exist at their configured source, perhaps via the share extension.) They
are still served, so that they can be browsed, or cloned individually. If we
care about that cloning someday, we can probably just add the extra entries to
the webconf dictionary. Even if the entries use '../' to escape the root, only
the related subrepositories would end up in the dictionary.
Now that the response instance itself is wrapped with error
handling, we no longer need this code. This code became dead
with the previous patch because the added code catches
HTTPException and re-raises as something else.
There were a handful of places in the code where HTTPResponse.read()
was called with no explicit error handling or with inconsistent
error handling. In order to eliminate this class of bug, we globally
swap out HTTPResponse.read() with a unified error handler.
I initially attempted to fix all call sites. However, after
going down that rabbit hole, I figured it was best to just change
read() to do what we want. This appears to be a worthwhile
change, as the tests demonstrate many of our uncaught exceptions
go away.
To better represent this class of failure, we introduce a new
error type. The main benefit over IOError is it can hold a hint.
I'm receptive to tweaking its name or inheritance.
An upcoming patch will add low-level testing of the bytes being sent
over the wire. As part of developing that test, I discovered that the
order of headers in HTTP requests wasn't deterministic. This patch
makes the order deterministic to make things easier to test.
We parse "descend" symbol as a Boolean using getboolean (prior extraction by
getargsdict already checked that it is a symbol).
In tests, check for error cases and vary Boolean values here and there.
The tag changes information we compute is now written to disk. This gives
hooks full access to that data.
The format picked for that file uses a 2 characters prefix for the action:
-R: tag removed
+A: tag added
-M: tag moved (old value)
+M: tag moved (new value)
This format allows hooks to easily select the line that matters to them without
having to post process the file too much. Here is a couple of examples:
* to select all newly tagged changeset, match "^+",
* to detect tag move, match "^.M",
* to detect tag deletion, match "-R".
Once again we rely on the fact the tag tests run through all possible
situations to test this change.
We now compute the proper actuall differences between tags before and after the
transaction. This catch a couple of false positives in the tests.
The compute the full difference since we are about to make this data available
to hooks in the next changeset.
This changeset introduces detection of tags changes during transaction. When
this happens a 'tag_moved=1' argument is set for hooks, similar to what we do
for bookmarks and phases.
This code is disabled by default as there are still various performance
concerns. Some require a smarter use of our existing tag caches and some other
require rework around the transaction logic to skip execution when unneeded.
These performance improvements have been delayed, I would like to be able to
experiment and stabilize the feature behavior first.
Later changesets will push the concept further and provide a way for hooks to
know what are the actual changes introduced by the transaction. Similar work
is needed for the other families of changes (bookmark, phase, obsolescence,
etc). Upgrade of the transaction logic will likely be performed at the same
time.
The current code can report some false positive when .hgtags file changes but
resulting tags are unchanged. This will be fixed in the next changeset.
For testing, we simply globally enable a hook in the tag test as all the
possible tag update cases should exist there. A couple of them show the false
positive mentioned above.
See in code documentation for more details.
This will get used to compare tags between two set of revisions during a
transaction (pre and post heads). The end goal is to be able to track tags
movement in transaction hooks.
If initial 'fctx' has changes in line range with respect to its parents, we
yield it first. This makes 'followlines(..., descend=True)' consistent with
'descendants()' revset which yields the starting revision.
We reuse one iteration of blockancestors() which does exactly what we want.
In test-annotate.t, adjust 'startrev' in one case to cover the situation where
the starting revision does not touch specified line range.
