Before this patch, "hg pull --update" doesn't advance current active
bookmark correctly, if pulling itself doesn't advance it, even though
"hg pull" + "hg update" does so.
Existing test for "pull --update works the same as pull && update" in
test-bookmarks.t doesn't examine this case, because pulling itself
advance current active bookmark before actual updating the working
directory in that test case.
To advance current active bookmark at "hg pull --update" correctly,
this patch examines 'movemarkfrom' instead of 'not checkout'.
Even if 'not checkout' at the invocation of postincoming(), 'checkout'
is overwritten by "the revision to update to" value returned by
destutil.destupdate() in such case. Therefore, 'not checkout'
condition means "update destination is revision #0", and isn't
suitable for examining whether active bookmark should be advanced.
Even though examination around "movemarkfrom == repo['.'].node()" may
seem a little redundant just for this issue, this makes it easier to
compare (and unify in the future, maybe) with the same logic to update
bookmark at "hg update" below.
if not ret and movemarkfrom:
if movemarkfrom == repo['.'].node():
pass # no-op update
elif bookmarks.update(repo, [movemarkfrom], repo['.'].node()):
ui.status(_("updating bookmark %s\n") % repo._activebookmark)
else:
# this can happen with a non-linear update
ui.status(_("(leaving bookmark %s)\n") %
repo._activebookmark)
bookmarks.deactivate(repo)
There were two mistakes: one was accidental reuse of the fclnode
variable from the loop gathering file nodes, and the other (masked by
that bug) was not correctly handling deleted directories. Both cases
are now fixed and the test passes.
On repos with lots of heads, the filelog() code could spend several
minutes decompressing manifests. This change instead tries to
efficiently scan the changelog for candidates and decompress as few
manifests as possible. This is a regression introduced in 3.3 by the
linkrev adjustment code. Prior to that, filelog was nearly instant.
For the repo in the bug report, this improves time of a simple log
command from ~3 minutes to ~.5 seconds, a 360x speedup.
For the main Mercurial repo, a log of commands.py slows down from
1.14s to 1.45s, a 27% slowdown. This is still faster than the file()
revset, which takes 2.1 seconds.
The largefiles extension needs to set lfstatus for this status call. Otherwise,
if a missing largefile is explicitly named, a confusing message is issued that
says the largefile wasn't found, followed by another that says nothing changed.
In a4119550f1e1 (context: clarify why we don't compare file contents
when nodeid differs, 2016-01-12), I also changed "node2 != _newnode"
into "self.rev() is not None". I don't remember why. They are similar,
but the former also catches the case where the file is clean in the
dirstate (so node2 is not _newnode), but different from the "other"
context. This resulted in unnecessary file content comparison a few
lines further down in the code. Let's just back out the code change.
Thanks to Durham Goode for spotting this.
Instead of rewriting __init__ to define the modulepolicy,
write out a __modulepolicy__.py file like __version__.py
This should work for both system-wide installation and in-place build. Therefore
we can avoid relying on two separate modulepolicy rules, '@MODULELOADPOLICY@'
and 'mercurial/modulepolicy'.
A filteredset is heavily used, but it cannot provide a printable information
how given set is filtered because a condition is an arbitrary callable object.
This patch adds an optional "condrepr" object that is used only by repr(). To
minimize the maintaining/runtime overhead of "condrepr", its type is overloaded
as follows:
type example
-------- ---------------------------------
tuple ('<not %r>', other)
str '<branch closed>'
callable lambda: '<branch %r>' % sorted(b)
object other
If your mercurial/templates/ directory is dirty, then the template system would
otherwise import duplicate templates from the .orig files and potentially try to
parse .rej files.
Since editing/reverting these templates isn't an unexpected action, and since
they're in .hgignore, it's best that the template system know to skip them."
To make all built-in predicates be known to hggettext, loading
built-in predicates by loadpredicate() should be placed before fixing
i18nfunctions but after all of predicate decorating.
This patch consists of changes below (these can't be applied
separately).
- replace revset.extpredicate by registrar.revsetpredicate in
extensions
- remove setup() on an instance named as revsetpredicate in
uisetup()/extsetup() of each extensions
registrar.revsetpredicate doesn't have setup() API.
- put new entry for revsetpredicate into extraloaders in dispatch
This causes implicit loading predicate functions at loading
extension.
This loading mechanism requires that an extension has an instance
named as revsetpredicate, and this is reason why
largefiles/__init__.py is also changed in this patch.
Before this patch, test-revset.t tests that all decorated revset
predicates are loaded by explicit setup() at once ("all or nothing").
Now, test-revset.t tests that any revset predicate isn't loaded at
failure of loading extension, because loading itself is executed by
dispatch and it can't be controlled on extension side.
revsetpredicate is used to replace revset.predicate and
revset.extpredicate in subsequent patches.
This patch also adds loadpredicate() to revset, because this
combination helps to figure out how the name of safe predicate is put
into safesymbols.
This patch still uses safesymbols set to examine whether the predicate
corresponded to the 'name' is safe from DoS attack or not, because
just setting func._safe property needs changes below for such
examination.
before:
name in revset.safesymbols
after:
getattr(revset.symbols.get(name, None), '_safe', False)
"automatic registration" described in help doc of revsetpredicate
class will be achieved by the subsequent patch, which lists
loadpredicate() up in dispatch.extraloaders.
_funcregistrarbase differs from funcregistrar in points below:
- every code paths should use same class derived from
_funcregistrarbase to register functions in a same category
funcregistrar expects (3rd party) extensions to use (a class
derived from) delayregistrar.
- actual extra setup should be executed in another function
For example, marking revset predicate as "safe" is executed in a
class derived from _funcregistrarbase, but putting name of "safe"
predicate into safesymbols is executed in another function for it.
funcregistrar expects derived classes to do so.
New class is named as module private one, because code paths, which
register functions, should use not it directly but one derived from
it.
This patch makes loading extra information from extension module at
dispatching extensible. Factoring 'loadcmdtable()' into commands.py is
a part of generalization of loading extra information.
This extensibility assumes registration of new function like below,
for example:
- revset predicate
- fileset predicate
- template keyword
- template filter
- template function
- internal merge tool
- web command
This patch requires not loader function itself but container module
and the name of it, because listing loader function directly up
implies actual loading module of it, even if it isn't used at runtime
(for example, extensions don't always define revset predicate)
The old native head revs logic would iterate over every node, starting from 0,
and check if every node was filtered (by testing it against the filteredrevs
python set). On large repos with hundreds of thousands of commits, this could
take 150ms.
This new logic iterates over the nodes in reverse order, and skips the filtered
check if we've seen an unfiltered child of the node. This saves approximately a
bagillion filteredrevs set checks, which shaves the time down from 150ms to
20ms during every branch cache write.
Before this patch, destupdate() returns the tipmost (descendant)
branch head regardless of closed or not. But updating to closed branch
head isn't reasonable for ordinary workflow, because:
- "hg heads" doesn't show closed heads (= updated parent itself) by
default
- subsequent committing on it re-opens closed branch
even if inactivation of closed head is needed, update destination
isn't it, because it should be merged into to another branch in
such case.
This patch chooses non-closed descendant branch head as default update
destination at first. If all descendant branch heads are closed,
destupdate() returns the tipmost closed branch head.
For simplicity, this patch chooses adding _destupdatebranchfallback()
instead largely changing _destupdatebranch().
This patch changes not only normal lookup code path, but also the "no
default branch" code path, for consistency.
To describe the bug this fix is addressing, one can do
``$ hg status -T "{label('red', path)}\n" --color=debug``
and observe that the label is not applied before my fix and applied with it.
The walker knows when an edge leads to a direct parent, a grandparent (skipping
revisions not part of the revset) and parents that are missing altogether
(neither it nor a grandparent is in the revset). Add this information to the
parents sequence yielded.
In this case, a template is parsed recursively with no thunk for lazy
evaluation. This patch prevents recursion by putting a dummy of the same name
into a cache that will be referenced while parsing if there's a recursion.
changeset = {files % changeset}\n
~~~~~~~~~
= [(_runrecursivesymbol, 'changeset')]
It would be nice if we could detect recursion at the parsing phase, but we
can't because a template can refer to a keyword of the same name. For example,
"rev = {rev}" is valid if rev is a keyword, and we don't know if rev is a
keyword or a template while parsing.
In 7a1ccfe03f74 (treemanifests: set bundle2 part parameter indicating
treemanifest, 2016-01-08), I didn't realize I had to set the parameter
separately for getbundle and unbundle. Having the parameter there on
push allows us to push to an empty repo and have the requirements
updated correctly.
Before this patch, the help bar in crecord wouldn't be printed correctly when
the terminal window didn't have enough column to display it. This patch adds
logic to make sure that the help bar message is always displayed. We use an
ellipsis when it is not possible to display the complete message.
In the crecord help dialog, the toggle all option was wrongfully documented.
Instead of using 'a', one must use 'A' to toggle all the hunks. The crecord
header that is always displayed on the screen contains the right shortcut and
does not need to be changed.
This patch improves the error messages raised when an OSError occurs, since
simply re-raising the exception can be both confusing and misleading. For
example, if "hg identify" is run inside a repository that contains a Git
subrepository and the git binary could not be found, it'll exit with the message
"abort: No such file or directory". That implies "identify" has a problem
reading the repository itself. There's no way for the user to know what the
real problem is unless they dive into the Mercurial source, which is what I
ended up doing after spending hours debugging errors while provisioning a VM
with Ansible (turns out I forgot to install Git on it).
Descriptive errors are especially important on Windows, since it's common for
Windows users to forget to set the "Path" system variable after installing Git.
After 313b8d61b548 graph canvas width is decided once on the initial rendering.
However, after graph page gets scrolled down to load more, it might need more
horizontal space to draw, so it needs to resize the canvas dynamically.
The exact problem that this patch solves can be seen using:
hg init testfork
cd testfork
echo 0 > foo
hg ci -Am0
echo 1 > foo
hg ci -m1
hg up 0
echo 2 > foo
hg ci -m2
hg gl -T '{rev}\n'
@ 2
|
| o 1
|/
o 0
hg serve
And then by navigating to http://127.0.0.1:8000/graph/tip?revcount=1
"revcount=1" makes sure the initial graph contains only revision 2. And because
the initial canvas width takes only that one revision into count, after the
(immediate) AJAX update revision 1 will be cut off from the graph.
We can safely set canvas width to the new value we get from the AJAX request
because every time graph is updated, it is completely redrawn using all the
requested nodes (in the case above it will use /graph/2?revcount=61), so the
value is guaranteed not to decrease.
P.S.: Sorry for parsing HTML with regexes, but I didn't start it.
The newly created helper changegroup.safeversion() knows to pick
version 03 if the repo uses treemanifests, so just using that means we
pick the right changegroup version.
In a few places (at least repair.py and shelve.py), we want to find
the best changegroup version that we can assume users of the repo will
understand. For example, we choose version 01 by default, but if it's
a generaldelta repo, we expect clients to support version 02 anyway,
so we choose that for new bundles (for e.g. "hg strip"). Let's create
a helper for this functionality in changegroup, so we can reuse it
elsewhere later.
Since it would be terribly expensive to convert between flat manifests
and treemanifests, we have decided to simply not support changegroup
version 01 and 02 with treemanifests. Therefore, let's stop announcing
that we support these versions on treemanifest repos.
Note that this means that older clients that try to clone from a
treemanifest repo will fail. What happens is that the server, after
this patch, finds that there are no common versions and raises
"ValueError: no common changegroup version". This results in "abort:
HTTP Error 500: Internal Server Error" on the client.
Before this patch, it was no better: The server would instead find
that there were directory manifest nodes to put in the changegroup 01
or 02 and raise an AssertionError on changegroup.py#668 (assert not
tmfnodes), which would also appear as a 500 to the client.
The new transaction context did not handle the case where an exception during
close should still call release. This cause pretxnclose hooks that failed to
cause the transaction to fail without aborting, thus requiring a hg recover.
I've added a test.
changegroup.getchunks() determines the end of the stream by looking
for an empty chunk group (two consecutive empty chunks). It ignores
empty groups in the first two groups. Changegroup 3 introduced an
empty chunk between the manifests and the files, which confuses
getchunks(). Since it comes after the first two, getchunks() will stop
there.
Fix by rewriting getchunks so it first counts two groups (empty or
not) and then keeps antostarts counting empty groups. With this counting,
changegroup 1 and 2 have exactly one empty group after the first two
groups, while changegroup 3 has two (one for directories and one for
files).
It's a little hard to test this at this point, but I have verified
that this patch fixes narrowhg (which was broken before this
patch). Also, future patches will fix "hg strip" with treemanifests,
and once that's done, getchunks() will be tested through tests of "hg
strip".
Before this patch if the hiddencache existed but was empty, it would crash
mercurial. This patch adds exception handling when reading the hiddencache to
avoid the issue.
When encountering a corrupted cache file we print a devel warning. There would
be no point in issuing a normal warning as the user wouldn't be able to do
anything about the situation.
The warning looks like:
devel-warn: corrupted hidden cache, removing it at: /path/to/repoview.py
These operations are obviously invalid for file-like channels because they
will read or write protocol headers.
This patch works around the issue that "hg archive" generates a corrupted
zip file on Windows commandserver because of unusable tell() implementation.
But the problem still occurs without using a commandserver.
$ hg archive -R not-small-repo -t zip - | cat > invalid.zip
So, this patch cannot fix the issue5049 completely.
This patch adds a variable to keep track of what hunk was selected
before the edit. We use that variable to select the hunk or its
replacement after the edit.
Before this patch, template files for "graph" web page use fixed width
size "480" for canvas element.
This causes pruned lanes and invisible vertexes, if there are 16 or
more vertical lanes at once. In such case, part of graph in right side
area over 480 is invisible, even though corresponded summary text
blocks are visible correctly.
This limitation isn't reasonable for workflow using many branches at
once (e.g. "one branch per issue" workflow).
There were changes below related to width of canvas:
- 6c855f5350cd (templates: widen the graph canvas (issue2683)),
released as a part of Mercurial 1.8.2
According to the description, this assumed that 15 parallel
branches was enough for ordinary workflow, and bumped width of
canvas up from 224 to 480.
- f5506d2a674c (hgweb: make graph data suitable for template usage),
released as a part of Mercurial 2.3
This introduced "canvaswidth" template keyword as a part of
refactoring around graph rendering.
But 'width="480"' of canvas element in template files wasn't
replaced by 'width="{canvaswidth}"' in it (or subsequent one).
This patch uses dynamic value "{canvaswidth}" instead of fixed width
size "480" for canvas element.
This is posted for "stable", because:
- this is re-fixing issue2683
- this is simple enough for stable
- using "{canvaswidth}" doesn't require any additional cost
Calculation of canvaswidth is already implied as a part of "graph"
web command.
Previously, if you passed a revset that resolved to no nodes, it would get
interpreted by the changegroup discovery logic as 'bundle all my heads', which
is not what the user asked.
Let's exit early when we notice this case.
It could be argued that the changeset discovery logic should be smarter and only
assume 'all heads' if the incoming heads parameter is None, but that's a much
riskier change.
Before, the hook() closure (which is called as part of locking hooks)
would maintain a reference to a transaction instance (which should be
finalized by the time lock hooks are called). Because we accumulate
hook() instances when there are multiple transactions per lock, this
would result in holding references to the transaction instances which
would lead to higher memory utilization.
Creating a reference to the hook arguments dict minimizes the number
of objects that are kept alive until the lock release hook runs,
minimizing memory "leaks."
util.cachefunc stores all arguments as the cache key. For filectxfn
functions, the arguments include the memctx instance. This creates a
cycle where memctx._filectxfn references self. This causes a memory
leak.
We break the cycle by implementing our own memoizing function that
only uses the path as the cache key. Since each memctx has its own
cache instance, there is no concern about invalid cache hits.
Similar to what was explained in the previous commit, the diff code
expected copy source to be in "ctx1", which is not always the case
during a merge. This has been broken since before hg 2.0.
Also similar to the previous commit, we fix the problem by fixing up
the copy dict.
During a merge, if the user removes a file that came from parent 2 and
did not exist in parent 1, the file's status will be "removed". This
surprises the diff code, which crashes because it expects removed
files exist in parent 1. This has been broken since ff976121fb34
(trydiff: use 'not in addedset' for symmetry with 'not in removedset',
2014-12-23).
Fix by fixing up the list of removed file, similar to how we currently
fix up the list of modified and added files during a merge.
This prepares for future patches, and it also lets us remove the ugly
"ctx1" argument to _filepairs() (ugly because of its assymmetry --
there's no "ctx2" argument).
The behaviour in this case is undefined. Instead of silently doing something
"random" and surprising, at least issue a warning.
(This should perhaps be considered a "deprecation" and turned into an error in
a future release.)
Positional parameters are also treated as revisions, but the order of revisions
matters and it will often be wrong if the user understands it as `-r` taking
multiple revisions as `-r REV1 REV2`.
(Alternatively, `-r` could be turned into a no-op flag as the documentation
suggests. That would however be less "semantic markup" and I agree with the
implementation in 40cbb25097c8 but not the documentation.)
Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.
This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!
The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).
Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:
Linux: ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)
Windows is ~27.5x slower. Yikes!
After this patch:
Linux: ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)
Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.
This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.
Closing files that have been appended to is relatively slow on
Windows/NTFS. This makes several Mercurial operations slower on
Windows.
The workaround to this issue is conceptually simple: use multiple
threads for I/O. Unfortunately, Python doesn't scale well to multiple
threads because of the GIL. And, refactoring our code to use threads
everywhere would be a huge undertaking. So, we decide to tackle this
problem by starting small: establishing a thread pool for closing
files.
This patch establishes a mechanism for closing file handles on separate
threads. The coordinator object is basically a queue of file handles to
operate on and a thread pool consuming from the queue.
When files are opened through the VFS layer, the caller can specify
that delay closing is allowed.
A proxy class for file handles has been added. We must use a proxy
because it isn't possible to modify __class__ on built-in types. This
adds some overhead. But as future patches will show, this overhead
is cancelled out by the benefit of closing file handles on background
threads.
This provides a general-purpose interface to all custom namespaces.
The {namespaces} keyword honors the definition order of namespaces as they
are kept by sortdict.
This is necessary to obtain a _hybrid object from a dict. If get() yields
a value, it would be stringified.
I see no benefit to make get() lazy, so this patch just changes "yield" to
"return".
In templater, a callable symbol exists for lazy evaluation, which should have
f(**mapping) signature. On the other hand, _hybrid.__call__(), which was
introduced by 4e182fb53989, generates mapping for each element.
This patch renames _hybrid.__call__() to _hybrid.itermaps() so that a _hybrid
object can be a value of a mapping dict.
{namespaces % "{namespace}: {names % "{name }"}\n"}
~~~~~
a _hybrid object
The added content is inside a verbose container.
I figure it makes sense to explicitly document behavior, including
with the caveat it may change later. People can't say they weren't
warned!
We don't currently have a mechanism for inferring bundle spec strings
from bundle files. This patch adds one.
This will eventually be used to make the producing of clone bundles
manifests easier.
RFC 7159 does not state that U+007F must be escaped, but it is widely
considered a control character. As '\x7f' is invisible on a terminal, and
Python's json.dumps() escapes '\x7f', let's do the same.
Some evil evil awful tool adds resource forks to files it's comparing.
Our Mac-specific code to do bulk stats was accidentally using "total
size" which includes those forks in the file size, causing them to be
reported as modified. This changes it to only care about the normal
data size and thus agree with what Mercurial's expecting.
If we move all the files out of one directory, but into two different
directories, we should not consider it a directory rename. The
detection of this case was broken.
This lets us greatly simply acquire/release cycles.
If the block completes without raising an exception, the transaction
is closed.
Code pattern before:
try:
tr = repo.transaction('x')
# zillions of lines of code
tr.close()
finally:
tr.release()
And after:
with tr.transaction('x'):
# ...
