The general delta heuristic to select a delta do not scale with the number of
branch. The delta base is frequently too far away to be able to reuse a chain
according to the "distance" criteria. This leads to insertion of larger delta (or
even full text) that themselves push the bases for the next delta further away
leading to more large deltas and full texts. This full text and frequent
recomputation throw Mercurial performance in disarray.
For example of a slightly large repository
280 000 files (2 150 000 versions)
430 000 changesets (10 000 topological heads)
Number below compares repository with and without the distance criteria:
manifest size:
with: 21.4 GB
without: 0.3 GB
store size:
with: 28.7 GB
without 7.4 GB
bundle last 15 00 revisions:
with: 800 seconds
971 MB
without: 50 seconds
73 MB
unbundle time (of the last 15K revisions):
with: 1150 seconds (~19 minutes)
without: 35 seconds
Similar issues has been observed in other repositories.
Adding a new option or "feature" on stable is uncommon. However, given that this
issues is making Mercurial practically unusable, I'm exceptionally targeting
this patch for stable.
What is actually needed is a full rework of the delta building and reading
logic. However, that will be a longer process and churn not suitable for stable.
In the meantime, we introduces a quick and dirty mitigation of this in the
'experimental' config space. The new option introduces a way to set the maximum
amount of memory usable to store a diff in memory. This extend the ability for
Mercurial to create chains without removing all safe guard regarding memory
access. The option should be phased out when core has a more proper solution
available.
Setting the limit to '0' remove all limits, setting it to '-1' use the default
limit (textsize x 4).
It isn't needed that storecache is derived from repofilecache.
Changes in this patch allow repofilecache and storecache to do in own
__init__() differently from each other.
Some extensions like fsmonitor need to run code after dirstate.status is
called, but while the wlock is held. The extensions could grab the wlock again,
but that has its own peculiar race issues. For example, fsmonitor would not
like its state to be written out if the dirstate has changed underneath (see
issue5581 for what can go wrong in that sort of case).
To protect against these sorts of issues, allow extensions to declare that they
would like to run some code to run at fixup time.
fsmonitor will switch to using this in the next patch in the series.
This is obviously an instance attribute, not a type attribute. The
modern Python style is to use __init__ for defining these.
This exposes statichttprepo as inheriting from localrepository
without calling its __init__. As a result, its __init__ defines
a lot of variables that methods on localrepository's methods need.
But factoring the common bits into a separate class is for another
day.
There are a number of improvements we want to make to revlogs
that will require a new version - version 2. It is unclear what the
full set of improvements will be or when we'll be done with them.
What I do know is that the process will likely take longer than a
single release, will require input from various stakeholders to
evaluate changes, and will have many contentious debates and
bikeshedding.
It is unrealistic to develop revlog version 2 up front: there
are just too many uncertainties that we won't know until things
are implemented and experiments are run. Some changes will also
be invasive and prone to bit rot, so sitting on dozens of patches
is not practical.
This commit introduces skeleton support for version 2 revlogs in
a way that is flexible and not bound by backwards compatibility
concerns.
An experimental repo requirement for denoting revlog v2 has been
added. The requirement string has a sub-version component to it.
This will allow us to declare multiple requirements in the course
of developing revlog v2. Whenever we change the in-development
revlog v2 format, we can tweak the string, creating a new
requirement and locking out old clients. This will allow us to
make as many backwards incompatible changes and experiments to
revlog v2 as we want. In other words, we can land code and make
meaningful progress towards revlog v2 while still maintaining
extreme format flexibility up until the point we freeze the
format and remove the experimental labels.
To enable the new repo requirement, you must supply an experimental
and undocumented config option. But not just any boolean flag
will do: you need to explicitly use a value that no sane person
should ever type. This is an additional guard against enabling
revlog v2 on an installation it shouldn't be enabled on. The
specific scenario I'm trying to prevent is say a user with a
4.4 client with a frozen format enabling the option but then
downgrading to 4.3 and accidentally creating repos with an
outdated and unsupported repo format. Requiring a "challenge"
string should prevent this.
Because the format is not yet finalized and I don't want to take
any chances, revlog v2's version is currently 0xDEAD. I figure
squatting on a value we're likely never to use as an actual revlog
version to mean "internal testing only" is acceptable. And
"dead" is easily recognized as something meaningful.
There is a bunch of cleanup that is needed before work on revlog
v2 begins in earnest. I plan on doing that work once this patch
is accepted and we're comfortable with the idea of starting down
this path.
changectx.__init__() is slightly modified to take str(wdirrev) as a valid
integer revision (and raise WdirUnsupported exception.)
Test will be added by the next patch.
The function use the length of the repository, something affected by filtering.
It seems better to use the unfiltered length here.
Credit for finding this goes to Durham Goode.
This method is only used internally by destutil, and it's obscure
enough I'm willing to just move it without a deprecation warning,
especially since the new method has more constrained functionality.
Design-wise I'd also like to get active bookmark handling folded into
the bookmark store, so that we don't squirrel away an extra attribute
for the active bookmark on the repository object.
When changelogs are written, a copy of the index (or inline revlog)
may be written to an 00changelog.i.a file to facilitate hooks and
other processes having access to the pending data before it is
finalized.
The way it works today, the localrepo class loads the changelog
like normal. Then, if it detects a pending transaction, it asks
the changelog class to load a pending changelog. The changelog
class looks for a 00changelog.i.a file. If it exists, it is
loaded and internal data structures on the new revlog class are
copied to the original instance.
The existing mechanism is inefficient because it loads 2 revlog
files. The index, node map, and chunk cache for 00changelog.i
are thrown away and replaced by those for 00changelog.i.a.
The existing mechanism is also brittle because it is a layering
violation to access the data structures being accessed. For example,
the code copies the "chunk cache" because for inline revlogs
this cache contains the raw revision chunks and allows the original
changelog/revlog instance to access revision data for these pending
revisions. This whole behavior of course relies on the revlog
constructor reading the entirety of an inline revlog into memory
and caching it. That's why it is brittle. (I discovered all this
as part of modifying behavior of the chunk cache.)
This patch streamlines the loading of a pending 00changelog.i.a
revlog by doing it directly in the changelog constructor if told
to do so. When this code path is active, we no longer load the
00changelog.i file at all.
The only negative outcome I see from this change is if loading
00changelog.i was somehow facilitating a role. But I can't imagine
what that would be because we throw away its data (the index data
structures are replaced and inline revision data is replaced via
the chunk cache) and since 00changelog.i.a is a copy of
00changelog.i, file content should be identical, so there should
be no meaninful file integrity checking at play. I think this was
all just sub-optimal code.
We are about to remove the branchmap cache update in changegroup application.
There is a debug message alongside this update that we do not want to loose. We
move the message beforehand to simplify the test update in the next changeset.
The message move is quite noisy and isolating that noise is useful.
Most tests update are just line reordering since the message is issued at a
later point during the transaction.
After this changes, the message is displayed in more case since local commit
creation also issue it.
Now that we garantee that branchmap cache are updated at the end of the
transaction we can drop that one. This removes a problematic case with nested
transaction where the new cache could be written on disk before the transaction
is finished.
The test change is harmless, since we update the cache at a later point, the
dirstate have been updated in between.
Regenerating the cache after a 'strip' or a 'rollback' is useful. So we call the
generic cache warming function as other caches than just branchmap will be
updated there in the future.
To do so, we have to make 'repo.updatecache()' able to take no arguments. In
such cases, we reload all caches.
We have multiple caches that gain from being kept up to date. For example in a
server setup, we want to make sure the branchcache cache is hot for other
read-only clients.
Right now each cache tries to update themself in place where new data have been
added. However the approach is error prone (we might miss some spot) and
fragile. When nested transaction are involved, such cache updates might happen
before a top level transaction is committed. Writing caches for uncommitted
data on disk.
Having a single entry point, run at the end of each successful transaction,
helps to ensure the cache is up to date and refreshed at the right time.
We start with updating the branchmap cache but other will come.
Tracking revisions is not the data that will unlock the most new capability.
However, they are the simplest thing to track and still unlock some nice
improvements in regard with caching.
We plug ourself at the changelog level to make sure we do not miss any revision
additions.
The 'revs' set is configured at the repository level because the transaction
itself does not needs to know that much about the business logic.
It seems like localrepo.getbundle() is trying to do the same thing, so
let's just call the method. That way we get the same condition as
there (matching any "HG2" prefix, not only "HG20").
The tag changes information we compute is now written to disk. This gives
hooks full access to that data.
The format picked for that file uses a 2 characters prefix for the action:
-R: tag removed
+A: tag added
-M: tag moved (old value)
+M: tag moved (new value)
This format allows hooks to easily select the line that matters to them without
having to post process the file too much. Here is a couple of examples:
* to select all newly tagged changeset, match "^+",
* to detect tag move, match "^.M",
* to detect tag deletion, match "-R".
Once again we rely on the fact the tag tests run through all possible
situations to test this change.
We now compute the proper actuall differences between tags before and after the
transaction. This catch a couple of false positives in the tests.
The compute the full difference since we are about to make this data available
to hooks in the next changeset.
This changeset introduces detection of tags changes during transaction. When
this happens a 'tag_moved=1' argument is set for hooks, similar to what we do
for bookmarks and phases.
This code is disabled by default as there are still various performance
concerns. Some require a smarter use of our existing tag caches and some other
require rework around the transaction logic to skip execution when unneeded.
These performance improvements have been delayed, I would like to be able to
experiment and stabilize the feature behavior first.
Later changesets will push the concept further and provide a way for hooks to
know what are the actual changes introduced by the transaction. Similar work
is needed for the other families of changes (bookmark, phase, obsolescence,
etc). Upgrade of the transaction logic will likely be performed at the same
time.
The current code can report some false positive when .hgtags file changes but
resulting tags are unchanged. This will be fixed in the next changeset.
For testing, we simply globally enable a hook in the tag test as all the
possible tag update cases should exist there. A couple of them show the false
positive mentioned above.
See in code documentation for more details.
This is minor update along the way. We simplify the 'findglobaltags' function to
only return the tags. Since no existing data is reused, we know that all tags
returned are global and we can let the caller get that information if it cares
about it.
As far as I understand, that function do not needs to be on the local repository
class, so we extract it in the 'tags' module were it will be nice and
comfortable. We keep the '_' in the name since its only user will follow in the
next changeset.
At the beginning of March, I promised Yuya that I would follow up a comment I
made on a patch with improved documention for these vfs objects. Also hat tip
to Pierre-Yves for adding the documentation here in the first place.
On Python 3, keys() is more like iterkeys(), so we got in trouble for
mutating the dict while we're iterating here. Since the list of caches
should be relatively small, work around this difference by just
forcing a copy of the key list.
