We are about to give 'strip' the ability to remove obsmarkers. Before we start
removing data we must make sure it is preserved somewhere. So the backup bundle
created by 'strip' now contains obsmarkers.
The improvement in time complexitty and the speed-up in computation is large
enough that the has little use now. Its update time can even gets in the way. So
we drop it.
This will allow us to unify the static/dynamic blockers logic in the next
changeset.
Also use the default-date when creating obsmarkers. Currently they are created
with the current date and without any option to force their value.
To test the feature, we remove some of the many 'glob' used to match obsmarker
date in the tests.
Receiving markers affecting changeset we'll receives later is legitimate and
not so uncommon case. Working on cache highlighted that this was only testing
in the evolve extension. We add a test for this case in core.
It seems better to introduce the experiment behind a flag for now as there are
multiple concerns around the feature:
* Storing operation increase the size of obsolescence markers significantly
(+10-20%).
* It performs poorly when exchanging markers (cannot combine command names,
command name might be unknown remotely, etc)
cmdutil.command wasn't a member of the registrar framework only for a
historical reason. Let's make that happen. This patch keeps cmdutil.command
as an alias for extension compatibility.
By recording what operation created the obsmarker, we can show very intuitive
messages to the user in various UIs. For instance, log output could have
messages like "Amended as XXX" to show why a commit is old and has an 'x' on it.
@ ac28e3 durham
/ First commit
|
| o d4afe7 durham
| | Second commit
| |
| x 8e9a5d (Amended as ac28e3) durham
|/ First commit
|
The conditional was updating the repository, which wasn't reflected in
subsequent logs on Windows, so the conditional is narrowed to just the serve
commands. The serve operation generates log files, so those are deleted to keep
the output of summary consistent.
The "troubles" template keyword returns a list of evolution troubles.
It is EXPERIMENTAL, as anything else related to changeset evolution.
Test it in test-obsolete.t which has troubled changesets.
It appears that computing index isn't cheap if --rev is specified. That's
why "index" field is available only if --index is specified.
I've named marker.flags() as "flag" because "flags" implies a list or dict
in template world.
Thanks to Piotr Listkiewicz for the initial implementation of this patch.
The only remaining usage of the experimental config were enforcing bundle2 on.
These are very old remains of when bundle2 was off by default. This was also
useful to highlight the fact that this was a bundle2 run and that a bundle1 one
was nearby. However, we want a future developer working on bundle3 to notice
possible output/behavior change on these tests and take them in account. So we
do not enforce bundle2 for these runs. We leave a comment around to make sure
dev still notice the bundle1 version.
This hasn't been testing anything since partway through the 3.7 cycle
due to unrelated refactoring. Sadly, the behavior it was trying to
prevent reemerged in the codebase at that time. A fix is in the next
patch, because proving that the fix was actually correct ended up
being trickier than I expected.
A bigger picture is the ability to be delete an arbitrary marker form the
repo's obsstore. This is a useful debug ability and it needs a way to indentify
the marker one wants to delete. Having a marker's index provides such an
ability.
Some tests fail while running with chg because they do not flush their output
streams. chgserver will make sure ui.flush is called after dispatch, but not
after {ui,repo}setup. For other non-ui streams, it should be explicitly
flushed since the request handler will use os._exit.
This patch adds explicit flushes in test-bundle2-format.t, test-extension.t
and test-obsolete.t. It will fix most test cases of them when running with chg.
The bundlerepository have to do some special magic to handle linkrev of the
bundlerepo filerev. That logic was done from a repoview and obsolescence marker
affecting bundled changeset could lead to a crash. We now ensure we operate on
unfiltered repository.
Before this patch if the hiddencache existed but was empty, it would crash
mercurial. This patch adds exception handling when reading the hiddencache to
avoid the issue.
When encountering a corrupted cache file we print a devel warning. There would
be no point in issuing a normal warning as the user wouldn't be able to do
anything about the situation.
The warning looks like:
devel-warn: corrupted hidden cache, removing it at: /path/to/repoview.py
This aligns with the unconditional plural output for the update line contents,
as well as the incoming/outgoing bookmarks line. It also matches the message
in evolve's summary hook as of 4f83b2d2d20d. (Though I thought this was removed
recently?)
The 'peer.known' call (handled at the repository level) was applying its own
manual filtering (looking at phases) instead of relying on the repoview
mechanism. This led to the discovery finding more "common" node that
'getbundle' was willing to recognised. From there, bad things happen, issue4982
is a symptom of it. While situations like described in issue4982 can still
happen because of race conditions, fixing 'peer.known' is important for
consistency in all cases.
We update the code to use 'repoview' filtering. This lead to small changes in
the tests for exchanging obsolescence marker because the discovery yields
different results.
The test affected in 'test-obsolete-changeset-exchange.t' is a test for
issue4982 getting back to its expected state.
A fix to issue4982 (not fixed in this patch) will reinforce the filtering
during discovery. This will makes two of our test repositories appear
unrelated (because all common content is properly hidden). To avoid this, we
introduce an extra base changeset that will not get obsoleted. This affects
various test output so we put this addition in its own changeset.
Context: the result of computehidden, used to compute the 'visible' revisions
is cached. Its output can change when:
1) new obsolete commits are created
2) new bookmarks are created or deleted
3) new tags are created or deleted
4) the parents of the working copy change
We currently correctly invalidate the cache only in the case 1).
This patch fixes the second case (bookmarks) by invalidating the cache once
a bookmark is added or removed.
$TESTDIR is added to the path, so this is superfluous. Also,
inconsistent use of quotes means we might have broken on tests with
paths containing spaces.
The phase of the pending commit depends on the parent of the working directory
and on the phases.newcommit configuration.
First, this information rather depend on the commit line which describe the
pending commit.
Then, we only want to be advertised when the pending phase is going to be higher
than the default new commit phase.
So the format will change from
$ hg summary
parent: 2:ab91dfabc5ad
foo
parent: 3:24f1031ad244 tip
bar
branch: default
commit: 1 modified, 1 unknown, 1 unresolved (merge)
update: (current)
phases: 1 secret (secret)
to
parent: 2:ab91dfabc5ad
foo
parent: 3:24f1031ad244 tip
bar
branch: default
commit: 1 modified, 1 unknown, 1 unresolved (merge) (secret)
update: (current)
phases: 1 secret
The bundle2 version of obsmarkers exchange is more informative. Switching to
bundle2 by default will change the output of this tests. To reduce the noise
when switching bundle2 to the default protocol, we migrate this tests early.
This patch refactors the native computation of heads. It fixes a bug where
filtered heads in the pending index could be returned by the native code
despite their filtering.
Because bundle2 allows a more precise exchange of obsmarkers during pull, it
sends them in a different order (previously unstable because of sets.) As
a result, they are added to the repository in a different order. To stabilize
the order and ensure tests are unchanged when moving from bundle1 to bundle2 we
sort markers when exchanging them.
In the long run, the obsstore will probably not use a linear storage.
The number of draft and secret changesets are currently not summarized.
This is an important information because the number of drafts give some rough
idea of the number of outgoing changesets in typical workflows, without needing
to probe a remote repository. And a non-zero number of secrets means that
those changeset will not be pushed.
If the repository is "dirty" - some draft or secret changesets exists - then
summary will display a line like:
phases: X draft, Y secret (public)
The phase in parenthesis corresponds to the highest phase of the parents of
the working directory, i.e. the current phase.
By default, the line is not printed if the repository is "clean" - all
changesets are public - but if verbose is activated, it will display:
phases: (public)
On the other hand, nothing will be printed if quiet is in action.
A few tests have been added in test-phases.t to cover the -v and -q cases.
This avoids the following error that happened if base revision of bundle file
was hidden. bundlerevlog needs it to construct revision texts from bundle
content as revlog.revision() does.
File "mercurial/context.py", line 485, in _changeset
return self._repo.changelog.read(self.rev())
File "mercurial/changelog.py", line 319, in read
text = self.revision(node)
File "mercurial/bundlerepo.py", line 124, in revision
text = self.baserevision(iterrev)
File "mercurial/bundlerepo.py", line 160, in baserevision
return changelog.changelog.revision(self, nodeorrev)
File "mercurial/revlog.py", line 1041, in revision
node = self.node(rev)
File "mercurial/changelog.py", line 211, in node
raise error.FilteredIndexError(rev)
mercurial.error.FilteredIndexError: 1
The obsolescence markers exchange is still experimental. We (developer) need
more information about what is going on. I'm adding an experimental flag to add
display the amount of data exchanged during bundle2 exchanges.
Running the tags function filtered will lead to different results with different
filter levels. This seems too dangerous to be done blindly as 39c37a1a9e2d did.
As per fullreposet.__and__, it can omit the range check of rev. Therefore,
"null" revision is accepted automagically.
It seems this can fix many query results involving null symbol. Originally,
the simplest "(null)" query did fail if there were hidden revisions. Tests
are randomly chosen.
fullreposet mimics the behavior of localrepo, where "null" revision is not
listed but contained.
I ran into a case when adding a test where there were cryptic hg command line
errors. I eventually traced it back to 'hg id' printing debug messages before
the hash:
invalid branchheads cache (served): tip differs <hash>
This method should eliminate any other output except the node.
The issue is titled "filtered revision 'XXX' (not in 'served' subset)" and that
is the error message you sometimes get when trying to look at a file (/file or
/annotate) in hgweb. For example:
http://hg.intevation.org/mercurial/crew/file/8414f8487b33/mercurial/cmdutil.py
This happens when a parent revision for a file is hidden, thus it is
not 'served' and isn't accessible in hgweb by default. When hgweb tries to
access such changeset, it produces the error and HTTP status code 404.
Another detail is that the parents() function, that is used in multiple places
in hgweb, sometimes returned changesets that were obsoleted by the current
changeset for the file. For example, when using rebase with evolve and rebasing
a divergent changeset that introduces a file on top of current branch. Or
grafting a change and making the new grafted changeset obsolete the source
(shown in the test case). The result is the same - the obsoleted changeset was
mistakingly returned from parents(), even though it's not a parent and the only
link to the new changeset is an obsoletion marker (and rebase/graft metadata?
not sure it matters).
The problem is fixed by using introrev() instead of linkrev() for finding
parents. This prevents parents() function from returning unrelated obsolete
changesets.
The test case prepares a separate repo because (afaict) all other test cases
never reuse file names, so there are no files that were changed in multiple
changesets. So no previously available files have obsolete changesets in their
history.
If a commit and a followup tag commit are pruned, there are no references to it
in any non obsolete version of .hgtags. Without this change however, the next
time a tag is added to another branch, the obsolete references are appended in
.hgtags before the new entries for the current tag command.
The annotation to unfilter localrepo._tag() has been around since 3da49fd631fb.
The log message for it mentions computing the tag cache though, so I'm not sure
if this was misplaced? It looks like branchmap was aware of filtering then, and
now tracks a cache per view.
This patch makes bundrepo retract the phase boundary for new commits to 'draft'
status, which is consistent with the behavior of 'hg unbundle'. The old
behavior was for commits to appear with the same phase as their nearest
ancestor in the base repository.
This affects several classes of operation:
* Inspecting a bundle with the -B flag
* Treating a bundle file as a peer (old: everything public, new: everything draft)
* Incoming command (neither old or new behavior is sensible -- fixed in next patch)
This is a very silly case and not particularly likely to happen in the wild,
but it turns out we can hit it in a couple of places. As we tune the storage
parameters we're likely to hit more such cases.
The affected test cases all have smaller revlogs now.
Before this patch, "test-obsolete.t" fails on Windows environment,
because strings corresponded to "tm_wday" (day of the week) field are
incorrect.
On POSIX environment, "gmtime()" returns correct "tm_wday" value even
for negative "time_t" value. On the other hand, it returns incorrect
one on Windows environment. At least, "gmtime()" of the Windows
runtime library bundled with Python 2.7.3 does.
According to f18f840c2b6e introducing original timestamp value '56
120', it shouldn't cause negative "time_t" value.
test-obsolete: remove subminute timezone in test
Obsmarker format "1" does not supports sub minute timezone. So we
change the test to something slightly more sensible.
It replaced "-d '56 12'" by "-d '56 120'".
The recent optimization of "and" operation relies on the assumption that
the rhs set does not contain invalid revisions. So rev() has to remove
invalid revisions.
This is still faster than using `.filter(lambda r: r == l)`.
revset #0: rev(25)
0) wall 0.026341 comb 0.020000 user 0.020000 sys 0.000000 (best of 113)
1) wall 0.000038 comb 0.000000 user 0.000000 sys 0.000000 (best of 66567)
2) wall 0.000062 comb 0.000000 user 0.000000 sys 0.000000 (best of 43699)
(0: 428fa22fb2d1^, 1: 3.2-rc, 2: this patch)
Hidden changesets are by far the most common error case and is the only one[1]
that can reach the user. We move to a friendlier message with a hint about how
to access the data anyway. We should probably point to a help topic instead but
we do not have such a topic yet.
Example of the new output
abort: hidden revision '4'!
(use --hidden to access hidden revisions)
[1] Actually, filtering from "served" can also reach the user during certain
exchange operations.
This will help user to debug. A more precise message will be issued
for the most common case ("visible" filter) in the next changesets.
example output:
- abort: filtered revision '4'!
+ abort: filtered revision '4' (not in 'visible' subset)!
We capture FilteredxxxError and issue a FilteredRepoLookupError instead with a
sightly different messsge. The message will likely get more improvement in the
future.
error: filtered revision '4'
The obsolete._enabled flag has become a config option. This updates all but one
of the tests to use the minimal number of flags necessary for them to pass. For
most tests this is just 'createmarkers', for a couple tests it's
'allowunstable', and for even fewer it's 'exchange'.
Previously, obstore read the obsolete._enabled flag to determine whether to
allow writes to the obstore. Since obsolete._enabled will be moving into a repo
specific config, we can't read it globally, and therefore must pass the
information into the constructor.
The set of relevant markers is currently unordered. Therefore the
markers will be added in arbitrary order. We sort the list of markers
beforehand to ensure stable output for testing.
We are going to only exchange markers relevant to the exchanged
changesets. So we need to change this marker to use a known changeset as
a successor instead of a precursor.
We add a ``--record-parents`` flag to debugobsolete. This can be used to record
parent information in the marker when the precursors are known locally. This
will be useful to test the "relevant markers" computation.
The discovery of necessary bookmark updates is now done within the "discovery
phase". This opens the door to the inclusion of bookmarks in a unified bundle2
push.
This fixes a crash that may happen when using mercurial 3.0.x.
The _gethiddenblockers function assumed that the output of tags.readlocaltags()
was a dict mapping tags to of valid nodes. However this was not necessarily the
case. When a repository had obsolete revisions and had local tag pointing to a
non existing revision was found, many mercurial commands would crash.
This revision fixes the problem by removing any tags from the output of
tags.readlocaltags() which point to invalid nodes.
We may want to add a warning when this happens (although it might be
annoying to get that warning for every command, possibly even more than once per
command).
A test for this problem has been added to test-obsolete.t. Without this fix the
test would output:
$ hg tags
abort: 00changelog.i@3816541e5485: no node!
[255]
Instead of:
$ hg tags
tiptag 2:3816541e5485
tip 2:3816541e5485
visible 0:193e9254ce7e
Previously, only bookmarks would be considered for blocking a changeset from
being hidden. Now, we also consider non-global tags. This is helpful if we have
local tags that might be hard to find once they are hidden, or tag that are
added by extensions (e.g. hggit or remotebranches).
If there is no outgoiing changesets but we have filtered revision in outgoing.excluded
We run into a filtering related crash. The excluded revision should not be there
in the first place but discovery need cleanup in default, not stable.
In its current state discovery may return (remotely) filtered elements
in "common". This has usually no impact as "missing" is kept clear of
filtered elements. However when the "remote" repo is a local repo (disk
accessible, and directly created in memory) the incoming code takes a
shortcut and directly uses the "remote" repo to generate the incoming
output. When some common elements are filtered this led to a crash. We
now ensure we use an unfiltered repository to generate the incoming
output. This does not change the behavior as missing is clear of
filtered revision.
Now that we have proper low level filtering, incoming code needs a
deeper cleanup but it is already planned.
This options add a new `web.view` to control filter level of hgweb.
This option have two purposes:
1) Allow fall back to unfiltered version in case a yet undetected by critical
bug is found in filtering after 2.5 release
2) People use hgweb as a local repoviewer. When they have secret changesets,
they wants to use "visible" filter not "served"
(modified by mpm, documentation deferred)
We noa pass an unfiltered repo in the same way `localrepo.push` does. This does
not alter outgoing behavior and prevents possible crash with computing
common/missing.
The `findcommonincoming` code could be simplified to make this unnecessary, but
this is too much change for the freeze.
We can not use `len(repo,changelog)`, it may be a filtered revision. We now use
`repo,changelog.tip()` to fetch this information.
The `tip` command is also fixed and tested
Thanks goes to Idan Kamara for the initial report.
If there is no outgoiing changesets but we have filtered revision in outgoing.excluded
We run into a filtering related crash. The excluded revision should not be there
in the first place but discovery need cleanup in default, not stable.
This is necessary to enforce filtering. The result is a bit buggy (may provide
less changeset than expected, but it will stop crashing on filtered revision
access.
Note that changelog.revs can not represents empty iteration like xrange did. So
we have to explicitly prevent call when there is nothing to do.
This changeset checks that a revision is known before adding it to the
navigation.
This will prevent traceback on filtered repository. This changeset result in an
incorrect behaviors, Navigation link may be dropped without any replacement.
However this bad navigation generation is much better than a crash
Since 3230dd238cf7 hgweb is broken with filtering. This changeset add test that
should pass once it is fixed. Test currently broken are commented and will be
uncommented by changeset that fix them.
The filelog test is currently passing because we already have some hack in core
regarding filelog (see 83a1b777fc02).
The dispatch code now enables filtering of "hidden" changesets globally. The
filter is installed before command and extension invocation. The `--hidden`
switch is now global and disables this filtering for any command.
Code in log dedicated to changeset exclusion is removed as this global filtering
has the same effect.
This is the second real use of changelog filtering. The change is very small to
allow testing the new filter with a setup close to the original one.
We replace custom post processing on `heads`function by call to the standard
code pass on a filtering repo.
In later coming will have wider usage of filtering that will make the dedicated
function useless.
We push between two repo which once filtered looks unrelated. Weakness in the
current implementation allows this push to be done without -f. But later
improvement with filtering will make this push fails for unrelatedness. However
we want this push to fail for including bumped changeset. So we had a smaller
push --force to make them related.
This applies the same logic as used for `obsolete` and `unstable` changesets.
Refuse to push them without force.
We'll probably want to factor this logic with two new functions
`pctx.troubled()` and `ctx.troubles()`. But I'm waiting for the third
"trouble" to make it into core.
The first obsolescence flag is introduced to allows for fixing the "bumped"
changeset situation.
bumpedfix == 1.
Creator of new changesets intended to fix "bumped" situation should not forget
to add this flag to the marker. Otherwise the newly created changeset will be
bumped too. See inlined documentation for details.
This options allows to specify the `flag` part of obsolete markers. For details
about marker flags, check the `mercurial/obsolete.py` documentation. Some random
flag are added to a marker to test this feature.
