statichttprepo inherits from localrepository. In doing so, it
obtains default implementations of various methods, like wlock().
Before this change, tags cache writing would call repo.wlock().
This failed on statichttprepo due to localrepository's wlock()
looking for an instance attribute that doesn't exist on statichttprepo
(statichttprepo doesn't call localrepository.__init__).
We /could/ define missing attributes until the base wlock() works.
However, a statichttprepo is remote and read-only and can't be
locked. The class already has a lock() that short circuits. So
it makes sense to implement a short-circuited wlock() as well. That
is what this patch does.
LockError is expected to be raised when locking fails. The constructor
takes a number of arguments that are local repository centric. Rather
than rework LockError to not require them (which would not be
appropriate for stable), this commit populates dummy values. I don't
believe they'll ever be seen by the user, as lock failures on
static http repos should be limited to well-defined (and tested)
scenarios. We can and should revisit the LockError type to improve
this.
When we changed basematcher.visitdir() in 0ca205268beb (match: make
base matcher return True for visitdir, 2017-07-14), we forgot to add
an override in nevermatcher. This led to tests failing in narrowhg.
As Durham pointed out, it's high time to add unit tests for the
matcher, so this patch also adds a first unit test.
Differential Revision: https://phab.mercurial-scm.org/D151
Without this, multiple spaces or tabs in the commit message aren't
preserved and things like tables don't align properly.
As part of adding the CSS rule, we had to cuddle the content
with the <div> to not introduce leading and trailing whitespace.
The "addbreaks" filter was also removed because it would insert
an additional newline, effectively double spacing content.
Differential Revision: https://phab.mercurial-scm.org/D113
The presence of a sparse checkout can confuse legacy clients or
clients without sparse enabled for reasons that should be obvious.
This commit introduces a new repository requirement that tracks
whether sparse is enabled. The requirement is added when a sparse
config is activated and removed when the sparse config is reset.
The localrepository constructor has been taught to not open repos
with this requirement unless the sparse feature is enabled. It yields
a more actionable error message than what you would get if the
lockout were handled strictly at the requirements verification phase.
Old clients that aren't sparse aware will see the generic
"repository requires features unknown to this Mercurial" error,
however.
The new requirement has "exp" in its name to reflect the
experimental nature of sparse. There's a chance that the eventual
non-experimental feature won't change significantly and we could
have squatted on the "sparse" requirement without ill effect. If
that happens, we can teach new clients to still recognize the old
name. But I suspect we'll sneak in some BC and we'll want a new
requirement to convey new meaning.
Differential Revision: https://phab.mercurial-scm.org/D110
In 3 functions we were writing the sparse config and updating the
working directory. In two of them we had a transaction-like process
for restoring the sparse config in case of wdir update fail.
Because the pattern is common, we've already made mistakes, and the
complexity will increase in the near future, let's consolidate the
code into a reusable function.
As part of this refactor, we end up reading the "sparse" file twice
when updating it. This is a bit sub-optimal. But I don't think it
is worth the code complexity to pass around the variables to avoid
the redundancy.
Differential Revision: https://phab.mercurial-scm.org/D109
The repo instance is currently only used to provide a changeset
lookup function as part of parsing revsets. I /think/ this allows
node fragments to resolve. I'm not sure why we wouldn't want this
to always "just work" if parsing a revset string.
Plus, an upcoming commit will introduce a new consumer that needs a
handle on the repo. So passing it more often will make that code
work more.
Passing a repo instance in all callers of revset.match* results in
a bunch of test changes. Notably, branch and tags caches get
populated as part of evaluating revsets. I'm not sure if this is
desirable. So this patch takes the conservative approach and only
passes the repo if we're passing a ui instance.
Differential Revision: https://phab.mercurial-scm.org/D97
Previously, [include] was implicit and pattern lines before a
[section] were added to includes.
Because the format may change in the future and explicit behavior,
well, more explicit, this commit changes the config parser to
reject pattern lines that don't occur in a [section].
Differential Revision: https://phab.mercurial-scm.org/D96
This adds an experimental flag -t/--terse which will terse the output. The terse flag
will respect other flags which filters the output. The flag takes a string
whose value can be a subsequence of "marduic" (the order does not matter here.)
Ignored files are not considered while tersing unless -i flag is passed or 'i'
is there is the terse flag value.
The flag is experimental for testing as there may be cases which will produce
strange results with the flag. We can set the terse on by default by simply
passing 'u' to the cmdutil.tersestatus().
This patch also adds a test file with tests covering the new feature.
Experimental because given the possible complexity, it may be worth figuring out
how to load this from a file, similar to the style files for the log command,
instead of trying to stuff it on the command line.
There are no visible changes here.
I'm starting to wonder if adding the '+' to the 'node' line instead of a
separate key line in a4d42f6edc09 was the right thing to do. The '{node}'
keyword never includes '+' elsewhere, and the way setup.py works, it would
truncate it anyway. Additionally, the file is missing '{p2node}' when 'wdir()'
merges are archived. I thought about adding an 'identify' line that would
correspond to `hg id -n`. But the other nodes are the full 40 characters, and
the output most useful for versioning is the short form. All of this cries out
for customization via templating. (Although maybe having the short identify
line by default is still a good idea.)
Now that we records "all" changes happening in a transaction (in tr.changes)
we will be able to provide better report on various changes (phases turned
public, changeset obsoleted, branch merged or created, etc..)
This is far too late in the cycle to play with this, but having this existing
method called more widely will help extensions to play around with various
options during the 4.4 cycle.
Instead of calling registersummarycallback only for transactions we want, we
always call it and use the transaction name to decide when to report (eg: we
do not want `hg amend` to report new obsoleted changesets). Filtering on
transaction name does not seems great, but seems good enough for the moment.
We can change the API during the next cycle.
The previous manual call during unbundling of the bundle2 "obsmarkers" part is
no longer necessary and has been dropped.
Share extension now also share caches reads and writes. Not sharing caches
results in costly caches recomputations which can takes up to minutes when
using shares on large repositories.
There are a couple of file in the '.hg/cache/' that depends of the current
visibility. Visibility can be affected by the working copy location, something
which is specific to each share. We ignores them for this series because they:
* are the minority,
* already have a good fallback to other precomputed caches,
* are only affected when people use the experimental evolution feature.
Most of the cache content lives in '.hg/cache/'. Moreover they are computed
exclusively from data in the '.hg/store' directory. This creates issues with
the share extension as the '.hg/store' directory is shared but the '.hg/cache'
is not. On large repositories, this makes this prevent some usage of the share
extension inefficient as some caches can take minutes to be recomputed.
To improve the situation, we introduce a new 'cachevfs' that will be dedicated
to cache reading and writing. In the next patches of this series, we'll
migrate the 4 existing caches to it and update the share extension.
Update 'write with no lock' warnings in order to be better controlled by the
config. We reuse the option used for lock order for these other lock related
message.
The message can now be disabled using 'devel.check-locks = no' (in addition to
the usual 'devel.all-warnings = no').
The transaction has now a 'bookmarks' dictionary in tr.changes. The structure
of the dictionary is {BOOKMARK_NAME: (OLD_NODE, NEW_NODE)}. If a bookmark is
deleted NEW_NODE will be None. If a bookmark is created OLD_NODE will be None.
If the bookmark is updated multiple time, the initial value is preserved.
Now that we have migrated all in-core caller of 'recordchange' to
'applychanges', deprecate 'recordchange' so external callers will move to the
new unified method.
checkconflict used to also do some bookmark deletion in case of divergence. It
is a bit suspicious given the function name, but it's not the goal of this
series.
In order to unify bookmarks changing, checkconflict now return the list of
divergent bookmarks to clean up and the callers must clean them by calling
applyphases.
We want to use applychanges in order to unify bookmark movement. We need a way
to compute divergence deletion without actually removing them.
We split the function in two in this patch while we migrate the existing users
of this code on next patches.
We do not touch the bookmarks store in this code, just the active bookmark, not
covered by the transaction. So it seems we can safely drop this call and the
tests agree with us.
Using a matcher for this command allows processing the named file(s) as
relative to cwd. It also leverages the icasefs normalization logic the same
way the status command does. (However, a false indicator is given for a
nonexistent file in some cases, e.g. passing 'foo.REJ' when that file doesn't
exist, and the rule is '*.rej'. Maybe the regex itself needs to be case
insensitive on these platforms, at least for the debug command.) Finally, the
file printed is relative to cwd and uses platform specific slashes, so a few
(glob)s were needed in seemingly unrelated tests.
The selector does not have a __del__ method and needs a manual close. We can
also use "with selector" but that makes the code too indented. Therefore
append a "selector.close()" after the end of the main loop for now.
This allows us to read a customized range of markers, instead of loading all
of them.
The condition of stop is made consistent across C and Python implementation
so we will still read marker when offset=a, stop=a+1.
Previously, commandserver was using select.select. That could have issue if
_sock.fileno() >= FD_SETSIZE (usually 1024), which raises:
ValueError: filedescriptor out of range in select()
We got that in production today, although it's the code opening that many
files to blame, it seems better for commandserver to work in this case.
There are multiple way to "solve" it, like preserving a fd with a small
number and swap it with sock using dup2(). But upgrading to a modern
selector supported by the system seems to be the most correct way.
This library was a backport of the Python 3 "selectors" library. It is
useful to provide a better selector interface for Python2, to address some
issues of the plain old select.select, mentioned in the next patch.
The code [1] was ported using the MIT license, with some minor modifications
to make our test happy:
1. "# no-check-code" was added since it's foreign code.
2. "from __future__ import absolute_import" was added.
3. "from collections import namedtuple, Mapping" changed to avoid direct
symbol import.
[1]: d27dbd2fdc/selectors2.py
# no-check-commit
I was several directories deep in the kernel tree, ran `hg add`, and got the
warning about the size of one of the files. I noticed that it suggested undoing
the add with a specific revert command. The problem is, it would have failed
since the path printed was relative to the repo root instead of cwd. While
here, I just fixed the other messages too. As an added benefit, these messages
now look the same as the verbose/inexact messages for the corresponding command.
I don't think most of these messages are reachable (typically the corresponding
cmdutil function does the check). I wasn't able to figure out why the keyword
tests were failing when using pathto()- I couldn't cause an absolute path to be
used by manipulating the --cwd flag on a regular add. (I did notice that
keyword is adding the file without holding wlock.)
This is done by a script [2] using RedBaron [1], a tool designed for doing
code refactoring. All "default" values are decided by the script and are
strongly consistent with the existing code.
There are 2 changes done manually to fix tests:
[warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal
[warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal
Since RedBaron is not confident about how to indent things [2].
[1]: https://github.com/PyCQA/redbaron
[2]: https://github.com/PyCQA/redbaron/issues/100
[3]:
#!/usr/bin/env python
# codemod_configitems.py - codemod tool to fill configitems
#
# Copyright 2017 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function
import os
import sys
import redbaron
def readpath(path):
with open(path) as f:
return f.read()
def writepath(path, content):
with open(path, 'w') as f:
f.write(content)
_configmethods = {'config', 'configbool', 'configint', 'configbytes',
'configlist', 'configdate'}
def extractstring(rnode):
"""get the string from a RedBaron string or call_argument node"""
while rnode.type != 'string':
rnode = rnode.value
return rnode.value[1:-1] # unquote, "'str'" -> "str"
def uiconfigitems(red):
"""match *.ui.config* pattern, yield (node, method, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
obj = node[-3].value
method = node[-2].value
args = node[-1]
section = args[0].value
name = args[1].value
if (obj in ('ui', 'self') and method in _configmethods
and section.type == 'string' and name.type == 'string'):
entry = (node, method, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def coreconfigitems(red):
"""match coreconfigitem(...) pattern, yield (node, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
args = node[1]
section = args[0].value
name = args[1].value
if (node[0].value == 'coreconfigitem' and section.type == 'string'
and name.type == 'string'):
entry = (node, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def registercoreconfig(cfgred, section, name, defaultrepr):
"""insert coreconfigitem to cfgred AST
section and name are plain string, defaultrepr is a string
"""
# find a place to insert the "coreconfigitem" item
entries = list(coreconfigitems(cfgred))
for node, args, nodesection, nodename in reversed(entries):
if (nodesection, nodename) < (section, name):
# insert after this entry
node.insert_after(
'coreconfigitem(%r, %r,\n'
' default=%s,\n'
')' % (section, name, defaultrepr))
return
def main(argv):
if not argv:
print('Usage: codemod_configitems.py FILES\n'
'For example, FILES could be "{hgext,mercurial}/*/**.py"')
dirname = os.path.dirname
reporoot = dirname(dirname(dirname(os.path.abspath(__file__))))
# register configitems to this destination
cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py')
cfgred = redbaron.RedBaron(readpath(cfgpath))
# state about what to do
registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred))
toregister = {} # {(section, name): defaultrepr}
coreconfigs = set() # {(section, name)}, whether it's used in core
# first loop: scan all files before taking any action
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
iscore = ('mercurial' in path) and ('hgext' not in path)
red = redbaron.RedBaron(readpath(path))
# find all repo.ui.config* and ui.config* calls, and collect their
# section, name and default value information.
for node, method, args, section, name in uiconfigitems(red):
if section == 'web':
# [web] section has some weirdness, ignore them for now
continue
defaultrepr = None
key = (section, name)
if len(args) == 2:
if key in registered:
continue
if method == 'configlist':
defaultrepr = 'list'
elif method == 'configbool':
defaultrepr = 'False'
else:
defaultrepr = 'None'
elif len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
# try to understand the "default" value
dnode = args[2].value
if dnode.type == 'name':
if dnode.value in {'None', 'True', 'False'}:
defaultrepr = dnode.value
elif dnode.type == 'string':
defaultrepr = repr(dnode.value[1:-1])
elif dnode.type in ('int', 'float'):
defaultrepr = dnode.value
# inconsistent default
if key in toregister and toregister[key] != defaultrepr:
defaultrepr = None
# interesting to rewrite
if key not in registered:
if defaultrepr is None:
print('[note] %s: %s.%s: unsupported default'
% (path, section, name))
registered.add(key) # skip checking it again
else:
toregister[key] = defaultrepr
if iscore:
coreconfigs.add(key)
# second loop: rewrite files given "toregister" result
for path in argv:
# reconstruct redbaron - trade CPU for memory
red = redbaron.RedBaron(readpath(path))
changed = False
for node, method, args, section, name in uiconfigitems(red):
key = (section, name)
defaultrepr = toregister.get(key)
if defaultrepr is None or key not in coreconfigs:
continue
if len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
try:
del args[2]
changed = True
except Exception:
# redbaron fails to do the rewrite due to indentation
# see https://github.com/PyCQA/redbaron/issues/100
print('[warn] %s: %s.%s: default needs manual removal'
% (path, section, name))
if key not in registered:
print('registering %s.%s' % (section, name))
registercoreconfig(cfgred, section, name, defaultrepr)
registered.add(key)
if changed:
print('updating %s' % path)
writepath(path, red.dumps())
if toregister:
print('updating configitems.py')
writepath(cfgpath, cfgred.dumps())
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
This was only used by the sparse extension's dirstate._ignore
override, which no longer exists.
Differential Revision: https://phab.mercurial-scm.org/D60
This is a Windows only thing. Unfortunately, the socket is closed at this point
(so the certificate is unavailable to check the chain). That means it's printed
out when verification fails as a guess, on the assumption that 1) most of the
time verification won't fail, and 2) sites using expired or certs that are too
new will be rare. Maybe this is an argument for adding more functionality to
debugssl, to test for problems and print certificate info. Or maybe it's an
argument for bundling certificates with the Windows builds. That idea was set
aside when the enhanced SSL code went in last summer, and it looks like there
were issues with using certifi on Windows anyway[1].
This was tested by deleting the certificate out of certmgr.msc > "Third-Party
Root Certification Authorities" > "Certificates", seeing `hg pull` fail (with
the new message), trying this command, and then successfully performing the pull
command.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089573.html
This is only useful on Windows, and avoids the need to use Internet Explorer to
build the certificate chain. I can see this being extended in the future to
print information about the certificate(s) to help debug issues on any platform.
Maybe even perform some of the python checks listed on the secure connections
wiki page. But for now, all I need is 1) a command that can be invoked in a
setup script to ensure the certificate is installed, and 2) a command that the
user can run if/when a certificate changes in the future.
It would have been nice to leverage the sslutil library to pick up host specific
settings, but attempting to use sslutil.wrapsocket() failed the
'not sslsocket.cipher()' check in it and aborted.
The output is a little more chatty than some commands, but I've seen the update
take 10+ seconds, and this is only a debug command.
I started a thread[1] on the mailing list awhile ago, but the short version is
that Windows doesn't ship with a full list of certificates[2]. Even if the
server sends the whole chain, if Windows doesn't have the appropriate
certificate pre-installed in its "Third-Party Root Certification Authorities"
store, connections mysteriously fail with:
abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
Windows expects the application to call the methods invoked here as part of the
certificate verification, triggering a call out to Windows update if necessary,
to complete the trust chain. The python bug to add this support[3] hasn't had
any recent activity, and isn't targeting py27 anyway.
The only work around that I could find (besides figuring out the certificate and
walking through the import wizard) is to browse to the site in Internet
Explorer. Opening the page with FireFox or Chrome didn't work. That's a pretty
obscure way to fix a pretty obscure problem. We go to great lengths to
demystify various SSL errors, but this case is clearly lacking. Let's try to
make things easier to diagnose and fix.
When I had trouble figuring out how to get ctypes to work with all of the API
pointers, I found that there are other python projects[4] using this API to
achieve the same thing.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096501.html
[2] https://support.microsoft.com/en-us/help/931125/how-to-get-a-root-certificate-update-for-windows
[3] https://bugs.python.org/issue20916
[4] 3b86bce206/source/updateCheck.py (L511)
There is still some use of 'deletedivergent' bookmark here. They will be taken
care of later. The 'deletedivergent' code needs some rework before fitting in
the new world.
We want to track bookmark movement within a transaction. For this we need a
more centralized way to update bookmarks.
For this purpose we introduce a new 'applychanges' method that apply a list of
changes encoded as '(name, node)'. We'll cover all bookmark updating code to
this new method in later changesets and add bookmark move in the transaction
when all will be migrated.
If a matcher doesn't implement visitdir, we should be returning True so that
tree traversals are not prematurely pruned. The old value of False would prevent
tree traversals when using any matcher that didn't implement visitdir.
Differential Revision: https://phab.mercurial-scm.org/D83
Thinking a bit further about list/dict subscript operation (proposed by
issue 5534), I noticed the current data structure, a dict of dicts, might
not be ideal.
For example, if there were "'[' index ']'" and "'.' key" operators,
"{parents[0]}" would return "{p1rev}:{p1node}", and we would probably want to
write "{parents[0].desc}" to get the first element of "{parents % "{desc}"}".
This will basically execute parents[0].makemap()['desc'] in Python.
Given the rule above, "{peerpaths.default.pushurl}" will be translated to
peerpaths['default'].makemap()['pushurl'], which means {peerpaths} should
be a single-level dict and sub-options should be makemap()-ed.
"{peerpaths % "{name} = {url}, {pushurl}, ..."}"
(Well, it could be peerpaths['default']['pushurl'], but in which case,
peerpaths['default'] should be a plain dict, not a hybrid object.)
So, let's mark the current implementation experimental and revisit it later.
find_deepest is used to find the "best" ancestors given a list. In the main
loop it keeps an invariant called 'ninteresting' which is supposed to contain
the number of non-zero entries in the 'interesting' array. This invariant is
incorrectly maintained, however, which leads the the algorithm returning an
empty result for certain graphs. This has been fixed.
Also, the 'interesting' array is supposed to fit 2^ancestors values, but is
incorrectly allocated to twice that size. This has been fixed as well.
The tests in test-ancestor.py compare the Python and C versions of the code,
and report the error correctly, since the Python version works correct. Even
so, I have added an additional test against the expected result, in the event
that both algorithms have an identical error in the future.
This fixes issue5623.
In some case, the default of one value is derived from other value. We add a
way to register them anyway and an associated devel-warning.
The registration is very naive for the moment. We might be able to have a
better way for registering each of these cases but it could be done later.
cg.apply used to returns the added nodes. Callers doesn't have a use for it
anymore, remove the added node and stops recording it in the current
operation.
This information was added in the current release cycle so no extensions
breakage should happens.
updatephases have no use of the 'addednodes' parameter since 44be3dc1fec8.
However caller are still passing it for nothing, remove the parameter and
remove computing of the added nodes in caller.
We adds new computation to find and record the revision affected by the
boundary retraction. This add more complication to the function but this seems
fine since it is only used in a couple of rare and explicit cases (`hg phase
--force` and `hg qimport`).
Having strong tracking of phase changes is worth the effort.
It is useful to detect noop and avoid expensive operations in this case.
We return the information to inform the caller of a possible update. Top level
function might need to react to the phase update (eg: invalidating some
caches, tracking phase change).
We rework the code to call 'registernew' before any other phase advancement.
This make 'changegroup.apply' register correct phase movement for the added
and bundled nodes.
This new function will be used by code that adds new changesets. It ajusts the
phase boundary to make sure added changesets are at least in their target
phase (they end up in an higher phase if their parents are in a higher phase).
Having a dedicated function also simplify the phases tracking. All the new
nodes are passed as argument, so we know that all of them needs to have their
new phase registered. We also know that no other nodes will be affected, so no
extra computation are needed.
This function differ from 'retractboundary' where some nodes might change
phase while some other might not. It can also affect nodes not passed as
parameters.
These simplification also apply to the computation itself. For now we use
'_retractboundary' there by convenience, but we may introduces simpler code
later.
While registering new revisions, we still need to check the actual phases of
the added node because it might be higher than the target phase (eg: target is
draft but parent is secret).
We will migrate users over the next changesets.
At the moment the 'retractboundary' function is called for multiple reasons:
First, actually retracting boundaries. There are only two cases for theses:
'hg phase --force' and 'hg qimport'. This will need extra graph computation to
retrieve the phase changes.
Second, setting the phases of newly added changesets. In this case we already
know all the affected nodes and we just needs to register different
information (old phase is None).
Third, when reducing the set of roots when advancing phase. The phase are
already properly tracked so we do not needs anything else in this case.
To deal with this difference in phase tracking, we extract the core logic into
a private method that all three cases can use.
Makes advanceboundary record the phase movement of affected revisions in
tr.changes['phases'].
The tracking is not usable yet because the 'retractboundary' function can also
affect phases.
We'll improve that in the coming changesets.
When advancing phases, we compute the new roots for the phases above. During
this process, we need to compute all the revisions that change phases (to the
new target phases). Extract these revisions into a separate variable. This
will be useful to record the phase changes in the transaction.
It seems that we were calling retractboundary for each phases to process.
Putting the retractboundary out of the loop reduce the number of calls,
helping tracking the phases changes.
unionmatcher is currently used where only a limited subset of its
functions will be called. Specifically, visitdir() is never
called. The next patch will pass it to dirstate.walk() where it will
matter that visitdir() is correctly implemented, so let's fix
that. Also add the explicitdir etc that will also be assumed by
dirstate.walk() to exist on a matcher.
Differential Revision: https://phab.mercurial-scm.org/D58
The forceincludematcher is simply a unionmatcher of a includematcher
(matching paths recursively) with the given matcher. Since the
forceincludematcher is only used by sparse, move it there.
I don't have a good sparse repo setup to test performance impact on.
Differential Revision: https://phab.mercurial-scm.org/D57
rebase will have similar logic, so let's extract it. Besides, it makes
the histedit code more readable.
We may want to parametrize acceptintervention() by the exception(s)
that should result in transaction close.
Differential Revision: https://phab.mercurial-scm.org/D66
Update the dirstate functions so that the caller supplies the full backup
filename rather than just a prefix and suffix.
The localrepo code was already hard-coding the fact that the backup name must
be (exactly prefix + "dirstate" + suffix): it relied on this in _journalfiles()
and undofiles(). Making the caller responsible for specifying the full backup
name removes the need for the localrepo code to assume that dirstate._filename
is always "dirstate".
Differential Revision: https://phab.mercurial-scm.org/D68
This was meant as a substitute for Python's "with" with multiple
context managers before we moved to Python 2.7. We're now on 2.7, so
we should have no reason to keep ctxmanager. "hg grep --all
ctxmanager" says that it was never used anyway.
Differential Revision: https://phab.mercurial-scm.org/D73
This is the result of running:
python codemod_nestedwith.py **/*.py
where codemod_nestedwith.py looks like this:
#!/usr/bin/env python
# codemod_nestedwith.py - codemod tool to rewrite nested with
#
# Copyright 2017 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function
import sys
import redbaron
def readpath(path):
with open(path) as f:
return f.read()
def writepath(path, content):
with open(path, 'w') as f:
f.write(content)
def main(argv):
if not argv:
print('Usage: codemod_nestedwith.py FILES')
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
changed = False
red = redbaron.RedBaron(readpath(path))
processed = set()
for node in red.find_all('with'):
if node in processed or node.type != 'with':
continue
top = node
child = top[0]
while True:
if len(top) > 1 or child.type != 'with':
break
# estimate line length after merging two "with"s
new = '%swith %s:' % (top.indentation, top.contexts.dumps())
new += ', %s' % child.contexts.dumps()
# only do the rewrite if the end result is within 80 chars
if len(new) > 80:
break
processed.add(child)
top.contexts.extend(child.contexts)
top.value = child.value
top.value.decrease_indentation(4)
child = child[0]
changed = True
if changed:
print('updating %s' % path)
writepath(path, red.dumps())
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Differential Revision: https://phab.mercurial-scm.org/D77
we wrap 'repo.svfs.audit' to check for the store lock when accessing file in
'.hg/store' for writing. This caught a couple of instance where the transaction
was released after the lock, we should probably have a dedicated checker for
that case.
When the appropriate developer warnings are enabled, We wrap 'repo.vfs.audit' to
check for locks when accessing file in '.hg' for writing. Another changeset will
add a 'ward' for the store vfs (svfs).
This check system has caught a handful of locking issues that have been fixed
in previous series (mostly in 4.0). I expect another batch to be caught in third
party extensions.
We introduce two real exceptions from extensions 'blackbox.log' (because a lot of
read-only operations add entry to it), and 'last-email.txt' (because 'hg email'
is currently a read only operation and there is value to keep it this way).
In addition we are currently allowing bisect to operate outside of the lock
because the current code is a bit hard to get properly locked for now. Multiple
clean up have been made but there is still a couple of them to do and the freeze
is coming.
We want to be able to do more precise check when auditing a path depending of
the intend of the file access (eg read versus write). So we now pass the 'mode'
value to 'audit' and update the audit function to accept them.
This will be put to use in the next changeset.
This function already does an excellent job of reading from context objects;
we simply need to change the single write call to eliminate all uses of the
wvfs.
As with past changes, the effect should be a no-op but opens the door to
in-memory merge later by using different context objects.
I ran into this ctypes bug while working with the Crypto API. While this could
be an issue with any Win32 API in theory, the handful of things that we call are
older functions that are unlikely to return COM errors, so I didn't retrofit
this everywhere.
The proposed syntax [1] was originally 'set{n rel}', but it seemed slightly
confusing if template is involved. On the other hand, we want to keep 'set[n]'
for future extension. So this patch introduces 'set#rel[n]' ternary operator.
I chose '#' just because it looks like applying an attribute.
This also adds stubs for 'set[n]' and 'set#rel' operators since these syntax
elements are fundamental for constructing 'set#rel[n]'.
[1]: https://www.mercurial-scm.org/wiki/RevsetOperatorPlan#ideas_from_mpm
It's sometimes useful to show hyperlinks in log output.
"{get(peerpaths, "default")}/rev/{node}"
Since each path may have sub options, "{peerpaths}" is structured as a dict
of dicts, but the inner dict is rendered as if it were a string URL. The
implementation is ad-hoc, so there are some weird behaviors described in
the test. We might need to introduce a proper way of handling a hybrid
scalar object.
This patch adds _hybrid.__getitem__() so d['path']['url'] works.
The keyword is named as "peerpaths" since "paths" seemed too generic in
log context.
The new 'phase-heads' forced all added node to secret before advancing the
boundary to work around the fact changesets were added as draft by default.
This is no longer necessary since the changegroup part can now use the
'targetphase' parameter.
Not doing this retract boundary call has a couple of advantages:
* This makes implementing phases change tracking in the transaction much
simpler since retract boundary can become a rare case.
* Bundling secret changesets is not the norm. Exchange never does that and
even for strip, the use-case is not common.Skipping the retract boundary
will avoid useless work here.
* Sending phase update on push can be simplified since we can rely on the
behavior of 'cg.apply' for most of it.
This means less phases update send for example.
* We no longer needs to track and use the addednodes during unbundling. This
make it possible to have multiple 'changegroup' and 'phase-heads' parts in the
same bundle without them interfering with each others.
The new part has not been part of any release yet so we do not offer backward
compatibility yet. It is important to update this semantic before the 4.3
freeze happens.
If we are bundling secret changeset and the bundle will contain phase, we
request the changegroup to be applied as secret.
It will be useful for next patch as we are now sure that secrets changesets
are applied as secret and not applied as draft then forced to secret.
By default unbundled changesets are drafts. We want to reduce the number of
phases changes during unbundling by giving the possibility to the bundle to
indicate the phase of unbundled changesets.
The longer terms goal is to add phase movement tracking in tr.changes and the
'retractboundary' call is making it more complicated than we want.
Since 26d535788092, the strip bundle includes the phases of the stripping
node. Hence we don't need this special case anymore.
Dropping it will helps make the phase behavior more consistent across all
exchanges medium.
This changeset attempts to solve two issues with the "followlines" UI in
hgweb. First the "followlines" action is currently not easily discoverable
(one has to hover on a line for some time, wait for the invite message to
appear and then perform some action). Second, it gets in the way of natural
line selection, especially in filerevision view.
This changeset introduces an additional markup element (a <button
class="btn-followlines">) alongside each content line of the view. This button
now holds events for line selection that were previously plugged onto content
lines directly. Consequently, there's no more action on content lines, hence
restoring the "natural line selection" behavior (solving the second problem).
These buttons are hidden by default and get displayed upon hover of content
lines; then upon hover of a button itself, a text inviting followlines section
shows up. This solves the first problem (discoverability) as we now have a
clear visual element indicating that "some action could be perform" (i.e. a
button) and that is self-documented.
In followlines.js, all event listeners are now attached to these <button>
elements. The custom "floating tooltip" element is dropped as <button>
elements are now self-documented through a "title" attribute that changes
depending on preceding actions (selection started or not, in particular).
The new <button> element is inserted in followlines.js script (thus only
visible if JavaScript is activated); it contains a "+" and "-" with a
"diff-semantics" style; upon hover, it scales up.
To find the parent element under which to insert the <button> we either rely
on the "data-selectabletag" attribute (which defines the HTML tag of children
of class="sourcelines" element e.g. <span> for filerevision view and <tr> for
annotate view) or use a child of the latter elements if we find an element
with class="followlines-btn-parent" (useful for annotate view, for which we
have to find the <td> in which to insert the <button>).
On noticeable change in CSS concerns the "margin-left" of span:before
pseudo-elements in filelog view that has been increased a bit in order to
leave space for the new button to appear between line number column and
line content one.
Also note the "z-index" addition for "annotate-info" box so that the latter
appears on top of new buttons (instead of getting hidden).
In some respect, the UI similar to line commenting feature that is implemented
in popular code hosting site like GitHub, BitBucket or Kallithea.
Python introduces a reference cycle on dynamically created types
via __mro__, making them very easy to leak. See
https://bugs.python.org/issue17950.
Previously, repo.filtered() created a type on every invocation.
Long-running processes (like `hg convert`) could call this
function thousands of times, leading to a steady memory leak.
Since we're Unable to stop the leak because this is a bug in
Python, the next best thing is to contain it.
This patch adds a cache of of the dynamically generated repoview/filter
types on the localrepo object. Since we only generate each type
once, we cap the amount of memory that can leak to something
reasonable.
After this change, `hg convert` no longer leaks memory on every
revision. The process will likely grow memory usage over time due
to e.g. larger manifests. But there are no leaks.
isfilecached() encapsulates internal implementation of filecache-ed
property.
"name in repo.unfiltered().__dict__" or so can't be used for this
purpose, because corresponded entry in __dict__ might be discarded by
repo.invalidate(), repo.invalidatedirstate() or so (fsmonitor does so,
for example).
This patch makes isfilecached() return not only whether filecache-ed
property is already cached, but also already cached value (or None),
in order to avoid subsequent access to cached object via "repo.NAME",
which prevents main Mercurial procedure after reposetup() from
validating cache.
Currently, sslutil._hostsettings() performs validation that web.cacerts
exists. However, client certificates are passed in to the function
and not all callers may validate them. This includes
httpconnection.readauthforuri(), which loads the [auth] section.
If a missing file is specified, the ssl module will raise a generic
IOException. And, it doesn't even give us the courtesy of telling
us which file is missing! Mercurial then prints a generic
"abort: No such file or directory" (or similar) error, leaving users
to scratch their head as to what file is missing.
This commit introduces explicit validation of all paths passed as
arguments to wrapsocket() and wrapserversocket(). Any missing file
is alerted about explicitly.
We should probably catch missing files earlier - as part of loading
the [auth] section. However, I think the sslutil functions should
check for file presence regardless of what callers do because that's
the only way to be sure that missing files are always detected.
The matchers that were recently moved into core from the sparse
extension override __call__, while the previously existing matchers
override matchfn. Let's switch to the latter for consistency.
When I added prefix() in 559ee9ecae07 (match: introduce boolean
prefix() method, 2014-10-28), we already had always(), isexact(), and
anypats(), so it made sense to write it in terms of them (a prefix
matcher is one that isn't any of the other types). It's only now that
I realize that it's much more natural to define prefix() explicitly
(it's one that uses path: patterns, roughly speaking) and let
anypats() be defined in terms of the others. Remember that these
methods are all used for determining which fast paths are
possible. anypats() simply means that no fast paths are possible (it
could be called complex() instead). Further evidence is that
rootfilesin:some/dir does not have any patterns, but it's still
considered to be an anypats() matcher. That's because anypats() really
just means that it's not a prefix() matcher (and not always() and not
isexact()).
This patch thus changes prefix() to return False by default and
anypats() to return True only if the other three are False. Having
anypats() be True by default also seems like a good thing, because it
means forgetting to override it will lead only to performance bugs,
not correctness bugs.
Since the base class's implementation changes, we're also forced to
update the subclasses. That change exposed and fixed a bug in the
differencematcher: for example when both its two input matchers were
prefix matchers, we would say that the result was also a prefix
matcher, which is incorrect, because e.g "path:dir - path:dir/foo" no
longer matches everything under "dir" (which is what prefix() means).
The m.isexact() and m.prefix() methods are used by callers to
determine whether m.files() can be used for fast paths. It seems safe
to let callers to any fast paths it can that rely on the empty
m.files().
This revset returns all successors, including transit nodes and the source
nodes (to be consistent with existing revsets like "ancestors").
To filter out transit nodes, use `successors(X)-obsolete()`.
To filter out divergent case, use `successors(X)-divergent()-obsolete()`.
The revset could be useful to define rebase destination, like:
`max(successors(BASE)-divergent()-obsolete())`. The `max` is to deal with
splits.
There are other implementations where `successors` returns just one level of
successors, and `allsuccessors` returns everything. I think `successors`
returning all successors by default is more user friendly. We have seen
cases in production where people use 1-level `successors` while they really
want `allsuccessors`. So it seems better to just have one single revset
returning all successors by default to avoid user errors.
In the future we might want to add `depth` keyword argument to it and for
other revsets like `ancestors` etc. Or even build some flexible indexing
syntax [1] to satisfy people having the depth limit requirement.
[1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101140.html
* Use context manager for wlock
* Rename oldsparsematch to oldmatcher
* Always call parseconfig() because parsing an empty string yields
the same result as the old code
The sparse extension performs a lot of monkeypatching of dirstate
to make it sparse aware. Essentially, various operations need to
take the active sparse config into account. They do this by obtaining
a matcher representing the sparse config and filtering paths through
it.
The monkeypatching is done by stuffing a reference to a repo on
dirstate and calling sparse.matcher() (which takes a repo instance)
during each function call. The reason this function takes a repo
instance is because resolving the sparse config may require resolving
file contents from filelogs, and that requires a repo. (If the
current sparse config references "profile" files, the contents of
those files from the dirstate's parent revisions is resolved.)
I seem to recall people having strong opinions that the dirstate
object not have a reference to a repo. So copying what the sparse
extension does probably won't fly in core. Plus, the dirstate
modifications shouldn't require a full repo: they only need a matcher.
So there's no good reason to stuff a reference to the repo in
dirstate.
This commit exposes a sparse matcher to dirstate via a property that
when looked up will call a function that eventually calls
sparse.matcher(). The repo instance is bound in a closure, so it
isn't exposed to dirstate.
This approach is functionally similar to what the sparse extension does
today, except it hides the repo instance from dirstate. The approach
is not optimal because we have to call a proxy function and
sparse.matcher() on every property lookup. There is room to cache
the matcher instance in dirstate. After all, the matcher only changes
if the dirstate's parents change or if the sparse config changes. It
feels like we should be able to detect both events and update the
matcher when this occurs. But for now we preserve the existing
semantics so we can move the dirstate sparseness bits into core. Once
in core, refactoring becomes a bit easier since it will be clearer how
all these components interact.
The sparse extension has been updated to use the new property.
Because all references to the repo on dirstate have been removed,
the code for setting it has been removed.
This is a pretty straightforward port. Some code cleanup was
performed. But no major changes to the logic were made.
I'm not a huge fan of this function because it does multiple
things. I'd like to get things into core first to facilitate
refactoring later.
Please also note the added inline comment about the oddities
of writeconfig() and the try..except to undo it. This is because
of the hackiness in which the sparse matcher is obtained by
various consumers, notably dirstate. We'll need a massive
refactor to address this. That refactor is effectively blocked
on having the sparse dirstate hacks live in core.
activeprofiles() is a special case of a more generic function.
Furthermore, that generic function is essentially already
implemented inline in the sparse extension.
So, refactor activeprofiles() to a generic activeconfig(). Change
the only consumer of activeprofiles() to use it. And have the
inline implementation in the sparse extension use it.
Well, mostly. The annotation on subrepo functions tacks on a parenthetical to
the abort message, which seems reasonable for a generic mechanism. But now all
messages consistently spell out 'subrepository', and double quote the name of
the repo. I noticed the inconsistency in the change for the last commit.
This simply passes the 'missing' argument down from the context of the parent
repo, so the same rules apply. subrepo.bailifchanged() is hardcoded to care
about missing files, because cmdutil.bailifchanged() is too.
In the end, it looks like this addresses inconsistencies with 'archive',
'identify', blackbox logs, 'merge', and 'update --check'. I wasn't sure how to
implement this in git, so that's left for someone more familiar with it.
Since the identify command adds a '+' for missing files, it's reasonable that
this does too. Perhaps the node field's hex value should be p1+p2 for merges?
This is equivalent to the previous code. But it seems to me that if the user is
going to be prompted that a commit is needed, missing files should be ignored,
but branch and merge changes shouldn't be.
This is equivalent to the previous code, but it seems better to be explicit
about what aspects of dirty are being ignored. Perhaps they shouldn't be, since
the help text says 'followed by a "+" if the working directory has uncommitted
changes'. Both merges and branch changes are committable, even if the files are
unchanged.
Additionally, this will make the `identify` command notice missing subrepo
files, once subrepos are taught to look for missing files.
The regexes for path: and relpath: patterns are the same (since the
paths have already been normalized at the point we create the
regexes).
I don't think the "if pat == '.'" will have any effect relpath:
because relpath: patterns will have the root directory already
normalized to '' by pathutil.canonpath() (unlike path:, for which the
root gets normalized to '.' by util.normpath()).
The regexes are passed to re.match(), which matches against the
beginning of the input, so the '^' doesn't do anything.
Note that unrooted patterns, such as globs and regexes from .hgignore
are instead achieved by adding '.*' to the expression given by the
user. (That's unless the user's expression started with '^', in which
case the '.*' is not added, perhaps to keep the regex cleaner?)
Instead of wrapping committablectx.markcommitted(), we inline
the call into workingctx.markcommitted().
Per smf's review, workingctx is the proper location for this
code, as committablectx is the shared base class for it and
memctx. Since this code touches the working directory, it belongs
in workingctx.
The 'successors' part of the mappings used of be a tuple. This avoid issue from
code consuming the generator "by mistake". For example, an extension inspecting the
mapping content used to be able to iterate over the successors mapping without
consequence.
Since the mapping are small we do not expect any performance impact we use tuple
again for this.
Previously repo.anyrevs only expand aliases in [revsetalias] config. This
patch makes it more flexible to accept a customized dict defining aliases
without having to couple with ui.
revsetlang.expandaliases now has the signature (tree, aliases, warn=None)
which is more consistent with templater.expandaliases. revsetlang.py is now
free from "ui", which seems to be a good thing.
cleanupnodes takes care of bookmark movement, and bookmark movement could
cause bookmark divergent resolution as a side effect. This patch adds such
bookmark divergent resolution logic so future rebase migration will be
easier.
The revset is carefully written to be equivalent to what rebase does today.
Although I think it might make sense to remove divergent bookmarks more
aggressively, for example:
F book@1
|
E book@2
|
| D book
| |
| C
|/
B book@3
|
A
When rebase -s C -d E, "book@1" will be removed, "book@3" will be kept,
and the end result is:
D book
|
C
|
F
|
E book@2 (?)
|
B book@3
|
A
The question is should we keep book@2? The current logic keeps it. If we
choose not to (makes some sense to me), the "deleterevs" revset could be
simplified to "newnode % oldnode".
For now, I just make it compatible with the existing behavior. If we want to
make the "deleterevs" revset simpler, we can always do it in the future.
In some valid usecases, the "mapping" received by scmutil.cleanupnodes have
filtered nodes. Use unfiltered repo to access them correctly.
The added test case will fail with the old cleanupnodes code.
This is important to migrate histedit to use the cleanupnodes API.
Aliases define optional alternatives to existing options. For example the old
option ui.user was deprecated and replaced by ui.username. With this mechanism,
it's even possible to create an alias to an option in a different section.
Add ui.user as alias to ui.username as an example of this concept.
The old alternates principle in ui.config is removed as it was used only for
this option.
If the matching command lives in an in-tree extension (which is all we
scan for), and the user has disabled that extension with
"extensions.<name>=!", we were not finding it, because the path in
_disabledextensions was the empty string. If the user had set
"extensions.<name>=!<valid path>" it would work, so it seems like just
a mistake that it didn't work.
This is a pretty straightforward move of the code.
I converted the "force" argument to a keyword argument.
Like other recent changes, this code is tightly coupled with
working directory update code in merge.py. I suspect the code
will become more tightly coupled over time, possibly even moved
to merge.py. For now, let's get the code in core.
merge.calculateupdates() now filters the update actions through sparse
by default.
The filtering no-ops if sparse isn't enabled or no sparse config
is defined.
The function has been refactored to behave more like a filter
instead of a wrapper of merge.calculateupdates().
We should arguably take sparse into account earlier in
merge.calculateupdates(). This patch preserves the old behavior
of applying sparse at the end of update calculation, which is the
simplest and safest approach.
This was our last method on the custom repo type, meaning we could
remove that custom type and inline the 2 lines of code into
reposetup().
As part of the move, instead of wrapping merge.update() from
the sparse extension, we inline the function call. The ported
function now no-ops if sparse isn't enabled, making it safe to
always call.
The call site in update() may not be the most appropriate. But
it matches the previous behavior, which is the safest thing
to do. It can be improved later.
As part of the move, the function arguments changed so revs are
passed as a list instead of *args. This allows us to use keyword
arguments properly.
Since the plan is to integrate sparse into core and have it
enabled by default, we need to prepare for a sparse matcher
to always be obtained and operated on. As part of the move,
we inserted code that returns an always matcher if sparse
isn't enabled. Some callers in the sparse extension take this
into account and conditionally perform matching depending on
whether the special always matcher is seen. I /think/ this
may have sped up some operations where the extension is
installed but no sparse config is activated.
One thing I'm ensure of in this code is whether os.path.dirname()
is semantically correct. os.posixpath.dirname() (which is
exported as pathutil.dirname) might be a better choise because
all patterns should be using posix directory separators (/)
instead of Windows (\). There's an inline comment that implies
Windows was tested. So hopefully it won't be a problem. We
can improve this in a follow-up. I've added a TODO to track it.
The sparse extension contains some matcher types that are
generic and can exist in core.
As part of the move, the classes now inherit from basematcher.
always(), files(), and isexact() have been dropped because
they match the default implementations in basematcher.
Before, 0 was being used as the default signature value and we cast
the int to a string. We also handled I/O exceptions manually.
The new code uses cfs.tryread() so we always feed data into the
hasher. The empty string does hash and and should be suitable
for input into a cache key.
The changes made the code simple enough that the separate checksum
function could be inlined.
The goal is to allow us to easily access to nested data. The dot operator
will be introduced later so we can write '{p1.files}' instead of
'{revset("p1()") % "{files}"}' for example.
In the example above, 'p1' needs to carry a mapping dict along with its
string representation. If it were a list or a dict, it could be wrapped
semi-transparently with the _hybrid class, but for non-list/dict types,
it would be difficult to proxy all necessary functions to underlying value
type because several core operations may conflict with the ones of the
underlying value:
- hash(value) should be different from hash(wrapped(value)), which means
dict[wrapped(value)] would be invalid
- 'value == wrapped(value)' would be false, breaks 'ifcontains'
- len(wrapped(value)) may be either len(value) or len(iter(wrapped(value)))
So the wrapper has no proxy functions and its scope designed to be minimal.
It's unwrapped at eval*() functions so we don't have to care for a wrapped
object unless it's really needed:
# most template functions just call evalfuncarg()
unwrapped_value = evalfuncarg(context, mapping, args[n])
# if wrapped value is needed, use evalrawexp()
maybe_wrapped_value = evalrawexp(context, mapping, args[n])
Another idea was to wrap every template variable with a tagging class, but
which seemed uneasy without a static type checker.
This patch updates {manifest} to a mappable as an example.
This makes 'foo|bar%baz' parsed as '(foo|bar)%baz', not 'foo|(bar%baz)'.
Perhaps it was a mistake that '%' preceded '|'. Both '|' and '%' can be
considered a kind of function application, and '|' is more like a '.' operator
seen in OO languages. So IMHO '|' should have the same (or higher) binding as
'%'.
The BC breakage should be minimal since both '|' and '%' operators have
strict requirements for their operands and 'foo|bar%baz' was invalid:
- right-hand side of '|' must be a symbol
- left-hand side of '%' must be a dict or list
- right-hand side of '%' must be a string or symbol
Before, iter(hybrid) was proxied to hybrid.gen, which generated formatted
strings. That's why we had to apply joinfmt() to the dicts generated by
hybrid.itermaps(). Since this weird API was fixed at 1906bcc0f923, we can
get rid of the makemap() calls from join().
We do "'%d:%s' % (ctx...)" at several places, so let's formalize it. A low-
level function, formatrevnode(ui, rev, node), is extracted so we can pass
a manifest rev/node pair.
Note that hex() for manifest output can be replaced with hexfunc() because
it is printed only when debugflag is set.
i18n/de.po is updated so test-log.t passes with no error.
The 'stepdone' set is design to be a client side mechanism. If the client used
some advanced capabilities to request necessary information (changeset,
obsmarkers, phases, etc). It marks the steps as done to avoid having a less
advanced mechanism issue a duplicated request.
So, the "stepdone.add('phases')" should be the result of a client choice,
because only the client can know it has requested all it needed to request. In
4a08cf1a2cfe this principle was broken because any phase-heads part sent by
the server to the client would declare the phases retrieval complete.
Now that there is an official phases related capability and code associated to
it. We do not need the change in 4a08cf1a2cfe anymore and we can back it out.
This brings back 'stepdone' management for 'phases' in line with the rest of
the code (including other phases handing).
Here is an example of potential misbehavior that 4a08cf1a2cfe introduced:
Imagine a server that pre-computes bundles. The bundles contains a changegroup
part and an (advisory) 'phase-heads' part. When a pull occurs, precomputed
bundled are reused if available. As the phase part is advisory it can be sent
to all clients. However they could be relevant changesets without phase
information. Either because they are already common or because they had no
precomputed bundle for them yet.
If receiving any 'phase-heads' parts disable subsequent phases re-trivial
parts, the client will not request phase data for all relevant changesets. For
example common changesets will not turn public.
A new bundle2 capability 'phases' has been added. If 'heads' is part of the
supported value for 'phases', the server supports reading and sending 'phase-
heads' bundle2 part.
Server is now able to process a 'phases' boolean parameter to 'getbundle'. If
'True', a 'phase-heads' bundle2 part will be included in the bundle with phase
information relevant to the whole pulled set. If this method is available the
phases listkey namespace will no longer be listed.
Beside the more efficient encoding of the data, this new method will greatly
improve the phase exchange efficiency for repositories with non-served
changesets (obsolete, secret) since we'll no longer send data about the
filtered heads.
Add a new 'devel.legacy.exchange' config item to allow fallback to the old
'listkey in bundle2' method.
Reminder: the pulled set is not just the changesets bundled by the pull. It
also contains changeset selected by the "pull specification" on the client
side (eg: everything for bare pull). One of the reason why the 'pulled set' is
important is to make sure we can move -common- nodes to public.
The next patch will use the 'phase-heads' part to exchange phase data relevant to
the pulled set.
'handlephases' currently acquires a transaction even in case of no-op pull,
which would results in an empty transaction and messing with the existing
journal.
Pass the transaction fetcher to updatephases so it can fetch it if necessary.
An extra post processing was added to recognize remote heads that are hidden
locally as "common" instead of "unknown". However, this processing was
removing such hidden heads from the remote heads sets.
It had no impact because we used to pull phase information from all remote
heads.
This series will replace the phase pulling operation to a more efficient
process but requires the unmodified pulled set information.
This patch limits the estimate time interval to roughly the last minute
(configurable by `estimateinterval`) to be more practical. See the test
change for why this is better.
.. feature:: Estimated time is more accurate with non-linear progress
Differential Revision: https://phab.mercurial-scm.org/D820
It was introduced by 98e4d39 ("progress: add speed format" 2011-5-9) and was
intended to hide ETA information for the first few seconds.
Later 5d261fd ("progress: add a changedelay to prevent parallel topics from
flapping (issue2698)" 2011-6-23) introduced `changedelay` config which hides
the entire progress bar for the first few seconds. So `progress.estimate` seems
somehow duplicated feature-wise. Since it's experimental and duplicated, let's
just remove it. This makes the next patch simpler - it no longer needs to make
sure `starttimes` is the real start time.
Differential Revision: https://phab.mercurial-scm.org/D828
The heuristics options tries to the default full copytracing algorithm if both
the source and destination branches contains of non-public changesets only. But
this can be slow in cases when we have a lot of drafts.
This patch adds a new config option experimental.copytrace.sourcecommitlimit
which defaults to 100. This value will be the limit of number of drafts from c1
to base. Incase there are more changesets even though they are draft, the
heuristics algorithm will be used.
Differential Revision: https://phab.mercurial-scm.org/D763
af9f7f376059 introduced a bug: emails Content-Transfer-Encoding
is silently replaced with 'quoted-printable' while any other
encoding could be used by underlying code. The problem is revealed
when a long unicode line is encoded.
The patch implements proper check which works for any text and
encoding.
As the TODO in the test said, the previous error message was not
very helpful. Let's improve things.
Differential Revision: https://phab.mercurial-scm.org/D811
With many aliases, resolving them could have some visible overhead. Below is
part of traceprof [1] output of `hg bookmark --hidden`:
(time unit: ms)
37 \ addaliases dispatch.py:526
37 | __init__ (60 times) dispatch.py:402
33 | findcmd (108 times) cmdutil.py:721
16 | findpossible (49 times) cmdutil.py:683
It may get better by optimizing `findcmd` to do a bisect, but we don't
really need to resolve an alias if it's not used, so let's make those
command entries lazy.
After this patch, `addalias` takes less than 1ms.
.. perf:: improved performance when many aliases are defined
[1]: 9aca0dbdbd/hgext3rd/traceprof.pyx
Differential Revision: https://phab.mercurial-scm.org/D805
This patch moves the old definition checking logic introduced by
df4ba5915933 earlier. So that the test itself does not depend on `aliasdef`.
The check is to avoid wrapping a same alias multiple times. It can be done
by checking the config name and value (`definition` in code), without
constructing a `cmdalias` instance.
This makes the next patch easier to review.
Differential Revision: https://phab.mercurial-scm.org/D804
This will allow anyone to enable the first in-menmory merge milestone
by wrapping merge.update in an extension and creating an overlayworkingctx.
Differential Revision: https://phab.mercurial-scm.org/D682
Following on from Jun Wu's patch last October[1], we have found that using mmap
for the revlog index in repos with large revlogs gives a noticable performance
improvment (~110ms on each hg invocation), particularly for commands that don't
touch the index very much.
This changeset adds this as an option, activated by a new experimental config
option so that it can be enabled on a per-repo basis. The configuration option
specifies an index size threshold at which Mercurial will switch to using mmap
to access the index.
If the configuration option is not specified, the default remains to load the
full file, which seems to be the best option for smaller repos.
Some initial performance numbers for average of 5 invocations of `hg log -l 5`
for different cache states:
| Repo: | HG | FB |
|---|---|---|
| Index size: | 2.3MB | much bigger |
| read (warm): | 237ms | 432ms |
| mmap (warm): | 227ms | 321ms |
| | (-3%) | (-26%) |
| read (cold): | 397ms | 696ms |
| mmap (cold): | 410ms | 888ms |
| | (+3%) | (+28%) |
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/088737.html
Test Plan:
`hg log --config experimental.mmapindex=true`
Differential Revision: https://phab.mercurial-scm.org/D477
This is useful for large files that are only partly touched.
Test Plan:
Will be used and tested in a later patch.
Differential Revision: https://phab.mercurial-scm.org/D476
Previously delta chunk returned a dictionary. Now that we consume deltachunk
within changegroup (instead of outside in revlog) we can just return a tuple and
have it be returned directly by deltaiter.
Differential Revision: https://phab.mercurial-scm.org/D746
Changegroup now has a deltaiter api for easy iteration over a series of deltas.
Let's use that in the debugbundle command.
Differential Revision: https://phab.mercurial-scm.org/D744
The recent e85296920485 patch removed the linkmapper argument from addgroup, as
part of trying to make addgroup more agnostic from the changegroup format. It
turns out that the changegroup can't resolve linkrevs while iterating over the
deltas, because applying the deltas might affect the linkrev resolution. For
example, when applying a series of changelog entries, the linkmapper just
returns len(cl). If we're iterating over the deltas without applying them to the
changelog, this results in incorrect linkrevs. This was caught by the hgsql
extension, which reads the revisions before applying them.
The fix is to return linknodes as part of the delta iterator, and let the
consumer choose what to do.
Differential Revision: https://phab.mercurial-scm.org/D730
There has report of flakiness in test-status-terse.t. In the terse code, we are
iterating on a dictionary without sorting and since python dicts are unordered,
that can be a reason behind the flakiness. Before we have a better
implementation for the terse thing, let's make sure current implementation
possess minimum bugs.
Differential Revision: https://phab.mercurial-scm.org/D740
This patch adds the functionality to use the full copytracing even if
`experimental.copytrace = heuristics` in cases when drafts are involved.
This is also a part of copytrace extension in fbext.
This also adds tests which are also taken from fbext.
.. feature::
The `heuristics` option for `experimental.copytrace` performs full
copytracing if both source and destination branches contains non-public
changsets only.
Differential Revision: https://phab.mercurial-scm.org/D625
We added support for including the operation responsible for creating
the obsmarker in 44ba6434eaf4 (obsolete: add operation metadata to
rebase/amend/histedit obsmarkers, 2017-05-09). However, soon
thereafter, in 819cf35e629a (obsmarker: add an experimental flag
controlling "operation" recording, 2017-05-20), it was hidden behind a
config that was off by default. It seems unlikely that people will
manually turn it on, and obsmarkers/evolution as a whole is still
experimental anyway, so let's turn on the tracking by default.
Differential Revision: https://phab.mercurial-scm.org/D722
Valid part names are restricted to [a-zA-Z0-9_:-]+, so I'm not worried
about having quoting present in places where we should have
predominantly valid part names. This will significantly ease the
Python 3 transition, and simultaneously isn't a BC because this is
only in error messages that should never be shown.
Previously revset weight is hardcoded and cannot be modified. This patch
moves it to predicate so newly registered revsets could define their weight
to properly give static optimization some hint.
Differential Revision: https://phab.mercurial-scm.org/D657
`_optimize` calculates weights of subtrees. "small" affects some weight
calculation (either 1 or 0.5). The weights are now only useful in `and`
optimization where we might swap two arguments and use `andsmally`.
In the real world, it seems unlikely that revsets with weight of 0.5 or 1
matters the `and` order optimization. I think the important thing is to get
weights of expensive revsets right (ex. `contains`).
This patch removes the `small` argument to simplify the interface.
As for choosing between 0.5 vs 1, things returning a single revision
(`ancestor`, `string`) has a weight of 0.5. Things returning multiple
revisions returns 1. This could be sometimes useful in the `andsmally`
optimization, ex.
(((:)-2) & expensive()) & ((1-2) & expensive())
^^^ ^
^^^^^^^ ^^^^^
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
weight=1 weight=0.5
would have an `andsmally` optimization so `1-2` gets executed first, which
seems to be desirable.
Differential Revision: https://phab.mercurial-scm.org/D656
Now that the part processing loop is tiny, let's move it to a separate function.
This will allow extensions to completely replace the part processing logic,
without having to replace the overall bundle processing logic or the stream
maintenance logic.
This will be useful for the infinitepush extension, so it can completely take
over receiving a bundle and rerouting it to a side store. This will also make it
easier to upstream the infinitepush functionality later.
Differential Revision: https://phab.mercurial-scm.org/D709
As part of refactoring bundle part processing let's move handler validation to
its own function.
Differential Revision: https://phab.mercurial-scm.org/D707
The processpart function also did some stream maintenance, so let's move it to
the part iterator as well, as part of moving all part iteration logic into the
class.
There is one place processpart is called outside of the normal loop, so we
manually handle the seek there.
The now-empty try/finally will be removed in a later patch, for ease of review.
Differential Revision: https://phab.mercurial-scm.org/D706
