The 'stepdone' set is design to be a client side mechanism. If the client used
some advanced capabilities to request necessary information (changeset,
obsmarkers, phases, etc). It marks the steps as done to avoid having a less
advanced mechanism issue a duplicated request.
So, the "stepdone.add('phases')" should be the result of a client choice,
because only the client can know it has requested all it needed to request. In
4a08cf1a2cfe this principle was broken because any phase-heads part sent by
the server to the client would declare the phases retrieval complete.
Now that there is an official phases related capability and code associated to
it. We do not need the change in 4a08cf1a2cfe anymore and we can back it out.
This brings back 'stepdone' management for 'phases' in line with the rest of
the code (including other phases handing).
Here is an example of potential misbehavior that 4a08cf1a2cfe introduced:
Imagine a server that pre-computes bundles. The bundles contains a changegroup
part and an (advisory) 'phase-heads' part. When a pull occurs, precomputed
bundled are reused if available. As the phase part is advisory it can be sent
to all clients. However they could be relevant changesets without phase
information. Either because they are already common or because they had no
precomputed bundle for them yet.
If receiving any 'phase-heads' parts disable subsequent phases re-trivial
parts, the client will not request phase data for all relevant changesets. For
example common changesets will not turn public.
A new bundle2 capability 'phases' has been added. If 'heads' is part of the
supported value for 'phases', the server supports reading and sending 'phase-
heads' bundle2 part.
Server is now able to process a 'phases' boolean parameter to 'getbundle'. If
'True', a 'phase-heads' bundle2 part will be included in the bundle with phase
information relevant to the whole pulled set. If this method is available the
phases listkey namespace will no longer be listed.
Beside the more efficient encoding of the data, this new method will greatly
improve the phase exchange efficiency for repositories with non-served
changesets (obsolete, secret) since we'll no longer send data about the
filtered heads.
Add a new 'devel.legacy.exchange' config item to allow fallback to the old
'listkey in bundle2' method.
Reminder: the pulled set is not just the changesets bundled by the pull. It
also contains changeset selected by the "pull specification" on the client
side (eg: everything for bare pull). One of the reason why the 'pulled set' is
important is to make sure we can move -common- nodes to public.
The next patch will use the 'phase-heads' part to exchange phase data relevant to
the pulled set.
'handlephases' currently acquires a transaction even in case of no-op pull,
which would results in an empty transaction and messing with the existing
journal.
Pass the transaction fetcher to updatephases so it can fetch it if necessary.
An extra post processing was added to recognize remote heads that are hidden
locally as "common" instead of "unknown". However, this processing was
removing such hidden heads from the remote heads sets.
It had no impact because we used to pull phase information from all remote
heads.
This series will replace the phase pulling operation to a more efficient
process but requires the unmodified pulled set information.
This patch limits the estimate time interval to roughly the last minute
(configurable by `estimateinterval`) to be more practical. See the test
change for why this is better.
.. feature:: Estimated time is more accurate with non-linear progress
Differential Revision: https://phab.mercurial-scm.org/D820
It was introduced by 98e4d39 ("progress: add speed format" 2011-5-9) and was
intended to hide ETA information for the first few seconds.
Later 5d261fd ("progress: add a changedelay to prevent parallel topics from
flapping (issue2698)" 2011-6-23) introduced `changedelay` config which hides
the entire progress bar for the first few seconds. So `progress.estimate` seems
somehow duplicated feature-wise. Since it's experimental and duplicated, let's
just remove it. This makes the next patch simpler - it no longer needs to make
sure `starttimes` is the real start time.
Differential Revision: https://phab.mercurial-scm.org/D828
The heuristics options tries to the default full copytracing algorithm if both
the source and destination branches contains of non-public changesets only. But
this can be slow in cases when we have a lot of drafts.
This patch adds a new config option experimental.copytrace.sourcecommitlimit
which defaults to 100. This value will be the limit of number of drafts from c1
to base. Incase there are more changesets even though they are draft, the
heuristics algorithm will be used.
Differential Revision: https://phab.mercurial-scm.org/D763
af9f7f376059 introduced a bug: emails Content-Transfer-Encoding
is silently replaced with 'quoted-printable' while any other
encoding could be used by underlying code. The problem is revealed
when a long unicode line is encoded.
The patch implements proper check which works for any text and
encoding.
As the TODO in the test said, the previous error message was not
very helpful. Let's improve things.
Differential Revision: https://phab.mercurial-scm.org/D811
With many aliases, resolving them could have some visible overhead. Below is
part of traceprof [1] output of `hg bookmark --hidden`:
(time unit: ms)
37 \ addaliases dispatch.py:526
37 | __init__ (60 times) dispatch.py:402
33 | findcmd (108 times) cmdutil.py:721
16 | findpossible (49 times) cmdutil.py:683
It may get better by optimizing `findcmd` to do a bisect, but we don't
really need to resolve an alias if it's not used, so let's make those
command entries lazy.
After this patch, `addalias` takes less than 1ms.
.. perf:: improved performance when many aliases are defined
[1]: 9aca0dbdbd/hgext3rd/traceprof.pyx
Differential Revision: https://phab.mercurial-scm.org/D805
This patch moves the old definition checking logic introduced by
df4ba5915933 earlier. So that the test itself does not depend on `aliasdef`.
The check is to avoid wrapping a same alias multiple times. It can be done
by checking the config name and value (`definition` in code), without
constructing a `cmdalias` instance.
This makes the next patch easier to review.
Differential Revision: https://phab.mercurial-scm.org/D804
This will allow anyone to enable the first in-menmory merge milestone
by wrapping merge.update in an extension and creating an overlayworkingctx.
Differential Revision: https://phab.mercurial-scm.org/D682
Following on from Jun Wu's patch last October[1], we have found that using mmap
for the revlog index in repos with large revlogs gives a noticable performance
improvment (~110ms on each hg invocation), particularly for commands that don't
touch the index very much.
This changeset adds this as an option, activated by a new experimental config
option so that it can be enabled on a per-repo basis. The configuration option
specifies an index size threshold at which Mercurial will switch to using mmap
to access the index.
If the configuration option is not specified, the default remains to load the
full file, which seems to be the best option for smaller repos.
Some initial performance numbers for average of 5 invocations of `hg log -l 5`
for different cache states:
| Repo: | HG | FB |
|---|---|---|
| Index size: | 2.3MB | much bigger |
| read (warm): | 237ms | 432ms |
| mmap (warm): | 227ms | 321ms |
| | (-3%) | (-26%) |
| read (cold): | 397ms | 696ms |
| mmap (cold): | 410ms | 888ms |
| | (+3%) | (+28%) |
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/088737.html
Test Plan:
`hg log --config experimental.mmapindex=true`
Differential Revision: https://phab.mercurial-scm.org/D477
This is useful for large files that are only partly touched.
Test Plan:
Will be used and tested in a later patch.
Differential Revision: https://phab.mercurial-scm.org/D476
Previously delta chunk returned a dictionary. Now that we consume deltachunk
within changegroup (instead of outside in revlog) we can just return a tuple and
have it be returned directly by deltaiter.
Differential Revision: https://phab.mercurial-scm.org/D746
Changegroup now has a deltaiter api for easy iteration over a series of deltas.
Let's use that in the debugbundle command.
Differential Revision: https://phab.mercurial-scm.org/D744
The recent e85296920485 patch removed the linkmapper argument from addgroup, as
part of trying to make addgroup more agnostic from the changegroup format. It
turns out that the changegroup can't resolve linkrevs while iterating over the
deltas, because applying the deltas might affect the linkrev resolution. For
example, when applying a series of changelog entries, the linkmapper just
returns len(cl). If we're iterating over the deltas without applying them to the
changelog, this results in incorrect linkrevs. This was caught by the hgsql
extension, which reads the revisions before applying them.
The fix is to return linknodes as part of the delta iterator, and let the
consumer choose what to do.
Differential Revision: https://phab.mercurial-scm.org/D730
There has report of flakiness in test-status-terse.t. In the terse code, we are
iterating on a dictionary without sorting and since python dicts are unordered,
that can be a reason behind the flakiness. Before we have a better
implementation for the terse thing, let's make sure current implementation
possess minimum bugs.
Differential Revision: https://phab.mercurial-scm.org/D740
This patch adds the functionality to use the full copytracing even if
`experimental.copytrace = heuristics` in cases when drafts are involved.
This is also a part of copytrace extension in fbext.
This also adds tests which are also taken from fbext.
.. feature::
The `heuristics` option for `experimental.copytrace` performs full
copytracing if both source and destination branches contains non-public
changsets only.
Differential Revision: https://phab.mercurial-scm.org/D625
We added support for including the operation responsible for creating
the obsmarker in 44ba6434eaf4 (obsolete: add operation metadata to
rebase/amend/histedit obsmarkers, 2017-05-09). However, soon
thereafter, in 819cf35e629a (obsmarker: add an experimental flag
controlling "operation" recording, 2017-05-20), it was hidden behind a
config that was off by default. It seems unlikely that people will
manually turn it on, and obsmarkers/evolution as a whole is still
experimental anyway, so let's turn on the tracking by default.
Differential Revision: https://phab.mercurial-scm.org/D722
Valid part names are restricted to [a-zA-Z0-9_:-]+, so I'm not worried
about having quoting present in places where we should have
predominantly valid part names. This will significantly ease the
Python 3 transition, and simultaneously isn't a BC because this is
only in error messages that should never be shown.
Previously revset weight is hardcoded and cannot be modified. This patch
moves it to predicate so newly registered revsets could define their weight
to properly give static optimization some hint.
Differential Revision: https://phab.mercurial-scm.org/D657
`_optimize` calculates weights of subtrees. "small" affects some weight
calculation (either 1 or 0.5). The weights are now only useful in `and`
optimization where we might swap two arguments and use `andsmally`.
In the real world, it seems unlikely that revsets with weight of 0.5 or 1
matters the `and` order optimization. I think the important thing is to get
weights of expensive revsets right (ex. `contains`).
This patch removes the `small` argument to simplify the interface.
As for choosing between 0.5 vs 1, things returning a single revision
(`ancestor`, `string`) has a weight of 0.5. Things returning multiple
revisions returns 1. This could be sometimes useful in the `andsmally`
optimization, ex.
(((:)-2) & expensive()) & ((1-2) & expensive())
^^^ ^
^^^^^^^ ^^^^^
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
weight=1 weight=0.5
would have an `andsmally` optimization so `1-2` gets executed first, which
seems to be desirable.
Differential Revision: https://phab.mercurial-scm.org/D656
Now that the part processing loop is tiny, let's move it to a separate function.
This will allow extensions to completely replace the part processing logic,
without having to replace the overall bundle processing logic or the stream
maintenance logic.
This will be useful for the infinitepush extension, so it can completely take
over receiving a bundle and rerouting it to a side store. This will also make it
easier to upstream the infinitepush functionality later.
Differential Revision: https://phab.mercurial-scm.org/D709
As part of refactoring bundle part processing let's move handler validation to
its own function.
Differential Revision: https://phab.mercurial-scm.org/D707
The processpart function also did some stream maintenance, so let's move it to
the part iterator as well, as part of moving all part iteration logic into the
class.
There is one place processpart is called outside of the normal loop, so we
manually handle the seek there.
The now-empty try/finally will be removed in a later patch, for ease of review.
Differential Revision: https://phab.mercurial-scm.org/D706
This is a trivial backport of 9f516a26a962 performed by
augie@google.com, but the change is still really Durham's not mine, so
I [augie] am leaving him as the author.
It seems like we used to pick the oldest possible version of the
changegroup to use for bundles created by the repair module (used
e.g. by "hg strip" and for temporary bundles by "hg rebase"). I tried
to preserve that behavior when I created the changegroup.safeversion()
method in 77f74106b264 (changegroup: introduce safeversion(),
2016-01-19).
However, we have recently chagned our minds and decided that these
commands are only used locally and downgrades are unlikely. That
decicion allowed us to start adding obsmarker and phase information to
these bundles. However, as the bug report shows, it means we get
different behavior e.g. when generaldelta is not enabled (because when
it was enabled, it forced us to use bundle2). The commit that actually
caused the reported bug was 26d535788092 (strip: include phases in
bundle (BC), 2017-06-15).
So, since we now depend on having more information in the bundles,
let's make sure we instead pick the newest possible changegroup
version.
Differential Revision: https://phab.mercurial-scm.org/D715
It can be useful for extensions to be able to produce the shortest
unambiguous hash (including the in-tree "show" extension). That logic
is currently inside the shortest() template function. Let's move it
out of the templater. I've put it on revlog since it's closely related
to revlog._partialmatch. We may also want a convenience method on
context, but I'll leave that for a later patch.
Differential Revision: https://phab.mercurial-scm.org/D724
I've tripped on this several times now, and am tired of debugging. Now
the header parts are part of the error message when the ''.join()
fails, which makes debugging obvious.
