It turns out that the files list is not sufficient to identify with revlogs have
changed. In a merge commit, no files could've changed but directories would
have. For now let's just backout this optimization.
When changelogs are written, a copy of the index (or inline revlog)
may be written to an 00changelog.i.a file to facilitate hooks and
other processes having access to the pending data before it is
finalized.
The way it works today, the localrepo class loads the changelog
like normal. Then, if it detects a pending transaction, it asks
the changelog class to load a pending changelog. The changelog
class looks for a 00changelog.i.a file. If it exists, it is
loaded and internal data structures on the new revlog class are
copied to the original instance.
The existing mechanism is inefficient because it loads 2 revlog
files. The index, node map, and chunk cache for 00changelog.i
are thrown away and replaced by those for 00changelog.i.a.
The existing mechanism is also brittle because it is a layering
violation to access the data structures being accessed. For example,
the code copies the "chunk cache" because for inline revlogs
this cache contains the raw revision chunks and allows the original
changelog/revlog instance to access revision data for these pending
revisions. This whole behavior of course relies on the revlog
constructor reading the entirety of an inline revlog into memory
and caching it. That's why it is brittle. (I discovered all this
as part of modifying behavior of the chunk cache.)
This patch streamlines the loading of a pending 00changelog.i.a
revlog by doing it directly in the changelog constructor if told
to do so. When this code path is active, we no longer load the
00changelog.i file at all.
The only negative outcome I see from this change is if loading
00changelog.i was somehow facilitating a role. But I can't imagine
what that would be because we throw away its data (the index data
structures are replaced and inline revision data is replaced via
the chunk cache) and since 00changelog.i.a is a copy of
00changelog.i, file content should be identical, so there should
be no meaninful file integrity checking at play. I think this was
all just sub-optimal code.
Previously, "hg verify" verifies everything, which could be undesirable when
there are expensive flag processor contents.
This patch adds a "verify.skipflags" developer config. A flag processor will
be skipped if (flag & verify.skipflags) == 0.
In the LFS usecase, that means "hg verify --config verify.skipflags=8192"
will not download all LFS blobs, which could be too large to be stored
locally.
Note: "renamed" is also skipped since its default implementation may call
filelog.data() which will trigger the flag processor.
Commit 9233182ea547d0aa removed the unused bundlecaps argument from the
changegroup code. While it is unused in core Mercurial, it was an important
feature for the remotefilelog extension because it allowed the exchange layer to
communicate to the changegroup packer that this was a shallow repo and that
filelogs should not be included. Without bundlecaps, there is currently no other
way to pass that information along without a more extensive refactor of
exchange, bundle, and changegroup code.
This patch backs out the original removal, and merges it with some recent
changes to changegroup apis.
locale.getpreferredencoding() was buggy in OS X for Python <2.7.
Since we no longer support Python <2.7, we no longer need this
workaround.
This essentially reverts 4a8b821a69fb.
Environment variables like HGUSER, HGEDITOR, HGEDITFROM should not trigger
a new chgserver. This patch uses a whitelist for environ variables starting
with "HG" to reduce the number of servers.
I have went through `grep -o "[\"']HG[A-Z_0-9]*['\"]" -hR . | sort -u` so
the list should be up-to-date.
To ease migration from files with version numbers in their first lines,
we want simplekeyvaluefile to support a non-key-value first line. In this
way, old versions of Mercurial will read such files, discover a newer version
than the one they know how to handle and fail gracefully, rather than with
exception. Shelve's shelvestate file is an example.
Now that we garantee that branchmap cache is updated at the end of the
transaction we can drop this update. This removes a problematic case with
nested transaction where the new cache could be written on disk before the
transaction is finished (and even roll-backed)
Such premature cache write was visible in the following test:
* tests/test-acl.t
* tests/test-rebase-conflicts.t
In addition, running the cache update later means having more date about the
state of the repository (in particular: phases). So we can generate caches with
more information. This creates harmless changes to the following tests:
* tests/test-hardlinks-whitelisted.t
* tests/test-hardlinks.t
* tests/test-phases.t
* tests/test-tags.t
* tests/test-inherit-mode.t
We are about to remove the branchmap cache update in changegroup application.
There is a debug message alongside this update that we do not want to loose. We
move the message beforehand to simplify the test update in the next changeset.
The message move is quite noisy and isolating that noise is useful.
Most tests update are just line reordering since the message is issued at a
later point during the transaction.
After this changes, the message is displayed in more case since local commit
creation also issue it.
Now that we garantee that branchmap cache are updated at the end of the
transaction we can drop that one. This removes a problematic case with nested
transaction where the new cache could be written on disk before the transaction
is finished.
The test change is harmless, since we update the cache at a later point, the
dirstate have been updated in between.
That command make sure caches are updated. This is based on
'localrepo.updatecaches' so when we move support for new cache in that function this
command will benefit from it.
Regenerating the cache after a 'strip' or a 'rollback' is useful. So we call the
generic cache warming function as other caches than just branchmap will be
updated there in the future.
To do so, we have to make 'repo.updatecache()' able to take no arguments. In
such cases, we reload all caches.
We have multiple caches that gain from being kept up to date. For example in a
server setup, we want to make sure the branchcache cache is hot for other
read-only clients.
Right now each cache tries to update themself in place where new data have been
added. However the approach is error prone (we might miss some spot) and
fragile. When nested transaction are involved, such cache updates might happen
before a top level transaction is committed. Writing caches for uncommitted
data on disk.
Having a single entry point, run at the end of each successful transaction,
helps to ensure the cache is up to date and refreshed at the right time.
We start with updating the branchmap cache but other will come.
Tracking revisions is not the data that will unlock the most new capability.
However, they are the simplest thing to track and still unlock some nice
improvements in regard with caching.
We plug ourself at the changelog level to make sure we do not miss any revision
additions.
The 'revs' set is configured at the repository level because the transaction
itself does not needs to know that much about the business logic.
The transaction is already tracking some data intended for hooks (in
'hookargs'). However, that information is minimal as we optimise for
passing data to other processes through environment variables. There are
multiple places were we could use more complete and lower level
information locally (eg: cache update, better report of changes to
hooks, etc...).
For this purpose we introduces a 'changes' dictionary on the
transaction. It is intended to track every changes happening to the
repository (eg: new revs, bookmarks move, phases move, obs-markers,
etc).
For now we just adds the 'changes' dictionary. We'll adds more tracking
and usages over time.
For large enough repositories, pull-based clones take too long, and an attempt
to use them indicates some sort of configuration or other issue or maybe an
outdated Mercurial. Add a config option to disable them.
This helps both users and the people who support them figure out why
a stream clone couldn't be performed.
In an upcoming patch we're going to add a way for servers to hard
abort on a full getbundle. In those cases servers might expect
clients to perform a stream clone, so it's important to communicate
why one couldn't be done.
At the moment this isn't used and all stream clones use the legacy protocol.
In an upcoming diff, canperformstreamclone will print out a message if a stream
clone was requested but couldn't happen for some reason. Removing this call
ensures the message isn't printed twice.
Internal merge tool :dump implies premerge. Therefore, files aren't
dumped, if premerge runs successfully.
This undocumented behavior might confuse users, if they want to always
dump files. But just making :dump omit premerge might cause backward
compatibility issue for existing automation.
This patch adds new internal merge tool :forcedump, which works as
same as :dump, but omits premerge always.
Internal tools annotated with "nomerge" should merge "change and
delete" correctly, but _forcedump() can't. Therefore, it is annotated
with "mergeonly" to always omit premerge, even though it doesn't merge
files actually.
This patch also adds explanation about premerge to :dump, to clarify
how :dump actually works.
BTW, this patch specifies internal tools with "internal:" prefix in
newly added test scenario in test-merge-tools.t, even though this
prefix is already deprecated. This is only for similarity to other
tests in test-merge-tools.t.
Before this patch, internal merge tool :prompt shows "no tool found to
merge FILE" line, even if :prompt is explicitly specified as a tool to
be used.
This patch shows warning message about choice of :prompt only at an
actual fallback, in which case any tool is rejected by capability for
binary or symlink.
It is for backward compatibility to omit warning message in
"changedelete" case.
Previously we relied on the opener options to tell the revlog to be a tree
manifest. This makes it complicated for extensions to create treemanifests and
normal manifests at the same time. Let's add a construtor argument to create a
treemanifest revlog as well.
I considered removing the options['treemanifest'] logic from manifestrevlog
entirely, but doing so shifts the responsibility to the caller which ends up
requiring changes in localrepo, bundlerepo, and unionrepo. I figured having the
dual mechanism was better than polluting other parts of the code base with
treemanifest knowledge.
We're going to make the 'c' policy more strict, where no missing attribute
will be allowed. Since we want to run 'hg bisect' without rebuilding the C
extension modules, we'll need a looser policy for development environment.
The default for system installation isn't changed.
Note that the current 'c' policy is practically 'allow'-ish as we have lots
of adhoc fallbacks to pure functions.
Previously, verify only checks "rawsize == len(rawtext)", if
"len(fl.read()) != fl.size()".
With flag processor, "len(fl.read()) != fl.size()" does not necessarily mean
"rawsize == len(rawtext)". So we may miss a useful check.
This patch removes the "if len(fl.read()) != fl.size()" condition so the
rawsize check is always performed.
With the condition removed, "fl.read(n)" looks unnecessary so a comment was
added to explain the side effect is wanted.
The end goal is to make it possible to avoid potential expensive fctx.data()
when unnecessary.
While memctx is useful for creating new file contexts, there are many cases
where we could reuse an existing raw file revision (amend, histedit, rebase,
process a revision constructed by a remote peer, etc). The overlayfilectx
class is made to support such reuse cases. Together with a later patch, hash
calculation and expensive flag processor could be avoided.
The new method returns the low-level revlog flag. We already have "rawdata"
so a "rawflags" makes sense.
Both "rawflags" and "rawdata" will be used in a later patch.
Make basefilectx._flags a property cache, so basefilectx.flags() could be
reasonably reused. This avoids code duplication between memfilectx and a
class added in a later patch.
Graft used findmissingrevs to find the candidates for graft duplicates in the
destination. That function operates with the constraint:
1. N is an ancestor of some node in 'heads'
2. N is not an ancestor of any node in 'common'
For our purpose, we do however have to work correctly in cases where the graft
set has multiple roots or where merges between graft ranges are skipped. The
only changesets we can be sure doesn't have ancestors that are grafts of any
changeset in the graftset, are the ones that are common ancestors of *all*
changesets in the graftset. We thus need:
2. N is not an ancestor of all nodes in 'common'
This change will graft more correctly, but it will also in some cases make
graft slower by making it search through a bigger and unnecessary large sets of
changes to find duplicates. In the general case of grafting individual or
linear sets, we do the same amount of work as before.
