Before this patch, "hg commit" (process A) executes steps below:
1. get current branch heads via 'repo.branchheads()'
- cache 'repo.changelog'
2. invoke 'repo.commit()'
3. acquire wlock
- invalidate 'repo.dirstate'
4. access 'repo.dirstate'
- re-read '.hg/dirstate'
- check validity of parent revisions with 'repo.changelog'
5. invoke 'repo.commitctx()'
6. acquire store lock (slock)
- invalidate 'repo.changelog'
7. do committing
8. release slock
9. release wlock
10. check new branch head (via 'cmdutil.commitstatus()')
If acquisition of wlock at (3) above waits for another "hg commit"
(process B) or so running parallelly to release wlock, process A
causes creating orphan revision, because:
- '.hg/dirstate' refers the revision, which is newly added by
process B, as its parent
- but already cached 'repo.changelog' doesn't contain such revision
- therefore, validating parents of '.hg/dirstate' at (4) above
replaces such revision with 'nullid'
Then, process A creates "orphan" revision, of which parent is "null"
revision.
In addition to it, "created new head" may be shown at the end of
process A unintentionally, if store is updated parallelly, because
both getting branch heads (1) and checking new branch head (10) are
executed outside slock scope.
To avoid this issue, this patch makes "hg commit" acquire wlock and
slock before processing.
This patch resolves the issue between "hg commit" processes, but not
one between "hg commit" and other commands. Subsequent patches resolve
the latter.
Even after this patch, there are still corner case problems below:
- filecache may overlook changes of '.hg/dirstate', and it causes
similar issue (see below for detail)
https://bz.mercurial-scm.org/show_bug.cgi?id=4368#c10
- 3rd party extension may cause similar issue, if it directly uses
'repo.commit()' without acquisition of wlock and slock
This can be fixed by acquisition of slock at the beginning of
'repo.commit()', but it seems suitable for "default" branch
In fact, acquisition of slock itself is already introduced at
"default" branch by ec227b188932, but acquisition is not at the
beginning of 'repo.commit()'.
This patch also changes some tests:
- test-fncache.t needs this tricky wrapping, to release (= forced
failure of) wlock certainly
- order of "hg commit" output is changed by widening scope of locks,
because some hooks are fired after releasing wlock
Before this patch, making a commit on a local repo could move a bookmark and
both operations would not be grouped as one transaction. This patch makes both
operations part of one transaction. This is necessary to switch to the new api
to save bookmarks repo._bookmarks.recordchange if we don't want to change the
current behavior of rollback.
Dirstate change happening after the commit is done is now part of the
transaction mentioned above. This leads to a change in the expected output of
several tests.
The change to test-fncache happens because both lock are now released in the
same finally clause. The lock release is made explicitly buggy in this test.
Previously releasing lock would crash triggering release of wlock that crashes
too. Now lock release crash does not directly result in the release of wlock.
Instead wlock is released at garbage collection time and the error raised at
that time "confuses" python.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
This is part of a series that will allow locks to be inherited by subprocesses
in limited circumstances.
When allowed, the parent process will pass down requisite information to the
child process by way of this environment variable.
Currently, there is no way to recover from a missing or corrupt fncache
file in place (a clone is required). For certain use cases such as
servers and with large repositories, an in-place repair may be
desirable. This patch adds functionality for in-place repair of the
fncache.
The `hg debugrebuildfncache` command is introduced. It ensures the
fncache is up to date by reconstructing the fncache from all seen files
encountered during a brute force traversal of the repository's entire
history.
The command will add missing entries and will prune excess ones.
Currently, the command no-ops unless the repository has the fncache
requirement. The command could later grow the ability to "upgrade" an
existing repository to be fncache enabled, if desired.
When testing this patch on a local clone of the Firefox repository, it
removed a bunch of entries. Investigation revealed that removed entries
belonged to empty (0 byte size) .i filelogs. The functionality for
pruning fncache of stripped revlogs was introduced in 93ba76bfbe8a, so
the presence of these entries likely predates this feature.
Once the transaction is closed, we now write transaction related data for
possible future undo. For now, we only do it for full file "backup" because
their were not handle at all in that case. In the future, we could move all the
current logic to set undo up (that currently exists in localrepository) inside
transaction itself, but it is not strictly requires to solve the current
situation.
The revbranchcache is read on demand before it will be used for updating the
branch map. It is written back when the branchmap is written and it will thus
use the same locking as branchmap. The revbranchcache instance is short-lived;
it is only stored in the branchmap from .update() is invoked and until .write()
is invoked. Branchmap already assume that the repo is locked in that case.
The use of revbranchcache for branch map updates will make sure that the
revbranchcache "always" is kept up-to-date.
The perfbranchmap benchmark is somewhat bogus, especially when we can see that
the caching makes a significant difference between the realistic case of a
first run and the rare case of rerunning it with a full cache. Here are some
'base' numbers on mozilla-central:
Before:
! wall 6.912745 comb 6.910000 user 6.840000 sys 0.070000 (best of 3)
After - initial, cache is empty:
! wall 7.792569 comb 7.790000 user 7.720000 sys 0.070000 (best of 3)
After - cache is full:
! wall 0.879688 comb 0.880000 user 0.870000 sys 0.010000 (best of 4)
The overhead when running with empty cache comes from checking, missing and
updating it every time.
Most of the performance improvement comes from not having to extract the branch
info from the changelog. The last doubling of performance comes from no longer
having to convert all branch names to local encoding but reuse the few already
converted branch names.
On the hg repo:
Before:
! wall 0.715703 comb 0.710000 user 0.710000 sys 0.000000 (best of 14)
After:
! wall 0.105489 comb 0.110000 user 0.110000 sys 0.000000 (best of 87)
The transaction backupfiles logic was broken for 'hg recover'. The file format
is XXX\0XXX\0YYY\0YYY\0 but the parser did a couple things wrong. 1) It went one
step beyond the final \0 and tried to read past the end of the array. 2)
array[i:i+1] returns a single item, instead of two items as intended.
Added a test to catch it, which turns out to be the first actual 'hg recover'
test.
Previously the fncache was written at lock.release time. This meant it was not
tracked by a transaction, and if an error occurred during the fncache write it
would fail to update the fncache, but would not rollback the transaction,
resulting in an fncache that was not in sync with the files on disk (which
causes verify to fail, and causes streaming clones to not copy all the revlogs).
This uses the new transaction backup mechanism to make the fncache transacted.
It also moves the fncache from being written at lock.release time, to being
written at transaction.close time.
Now that changelog filtering is in place, it's become evident that
naming the filters according to the set of revs _not_ included in the
filtered changelog is confusing. This is especially evident in the
collaborative branch cache scheme.
This changes the names of the filters to reflect the revs that _are_
included:
hidden -> visible
unserved -> served
mutable -> immutable
impactable -> base
repoview.filteredrevs is renamed to filterrevs, so that callers read a
bit more sensibly, e.g.:
filterrevs('visible') # filter revs according to what's visible
Filtered repository are *subset* of unfiltered repository. This means that a
filtered branchmap could be use to compute the unfiltered version.
And filtered version happen to be subset of each other:
- "all() - unserved()" is a subset of "all() - hidden()"
- "all() - hidden()" is a subset of "all()"
This means that branchmap with "unfiltered" filter can be used as a base for
"hidden" branchmap that itself could be used as a base for unfiltered
branchmap.
unserved < hidden < None
This changeset implements this mechanism. If the on disk branchcache is not valid
we use the branchcache of the nearest subset as base instead of computing it from
scratch. Such fallback can be cascaded multiple time is necessary.
Note that both "hidden" and "unserved" set are a bit volatile. We will add more
stable filtering in next changesets.
This changeset enables collaboration between no filtering and "unserved"
filtering. Fixing performance regression introduced by 7bff5f37cb97
test-mq-cache.t did apparently look at stale cache content.
Testing with different locking mechanism happened to update the cache more
frequently and thus caused a test failure.
Globbing is usually used for filenames, so on windows it is reasonable and very
convenient that glob patterns accepts '\' or '/' when the pattern specifies
'/'.
Before this patch undo.bookmarks was created on bookmarks write and
not with other transaction-related files. There were two issues: first
is that if you have changed bookmarks few times after a transaction
happened, rollback will give you a state which can point to
non-existing revision. Second is that if you have not changed
bookmarks after a transaction, rollback will touch your state anyway.
This change also adds `localrepo._writejournal` method, which can be
used by other extensions to save their transaction-related backup in
right time.
