When debugging in a Python shell, the type of "repo" is "proxycls", which
could confuse new people.
In [1]: repo
Out[1]: <mercurial.localrepo.proxycls at 0x7f65d4b976d0>
Let's rename it to "filteredrepo" to make it clearer.
Now that the 'vfs' classes moved in their own module, lets use the new module
directly. We update code iteratively to help with possible bisect needs in the
future.
In "aftertrans", we rename "journal.*" to "undo.*". We expect "journal.*"
files to disappear after renaming.
However, if "journal.foo" and "undo.foo" refer to a same file (hardlink),
rename may be a no-op, leaving both files on disk, according to Linux
manpage [1]:
If oldpath and newpath are existing hard links referring to the same
file, then rename() does nothing, and returns a suc‐ cess status.
The POSIX specification [2] is not very clear about what to do.
To be safe, remove "undo.*" before the rename so "journal.*" cannot be left
on disk.
[1]: http://man7.org/linux/man-pages/man2/rename.2.html
[2]: http://pubs.opengroup.org/onlinepubs/9699919799/
Storing a relative path the source repository is useful when exporting
repositories over the network or when they're located on external
drives where the mountpoint isn't always fixed.
Currently, Mercurial interprets paths in `.hg/shared` relative to
$PWD. I suspect this is very much unintentional, and you have to
manually edit `.hg/shared` in order to trigger this behaviour.
However, on the off chance that someone might rely on it, I added a
new capability called 'relshared'. In addition, this makes earlier
versions of Mercurial fail with a graceful error.
I should note that I haven't tested this patch on Windows.
The 'ui' object dedicated to a 'localrepo' is independent from the one available
in dispatch (and 'uisetup'). In addition, it is created from the 'baseui'
(apparently for good reason). As a result, we need to run the color setup on
it after the local repository config is read.
This was overlooked when the rest of the initialization changed but did not
had impact yet because all setup is still global. We fix it before it is too
late.
Before this patch, checking HG_PENDING for changelog in localrepo.py
might cause unintentional reading unrelated '00changelog.i.a' in,
because HG_PENDING is checked by str.startswith().
An external hook spawned by inner repository in nested ones satisfies
this condition.
This patch uses txnutil.mayhavepending() to check HG_PENDING strictly.
BTW, this patch may cause failure of bisect in the repository of
Mercurial itself, if examination at bisecting assumes that an external
hook can see all pending changes while nested transactions across
repositories.
This invisibility issue will be fixed by subsequent patch, which
allows HG_PENDING to refer multiple repositories.
This was one of the hardest import cycles as scmutil is widely used and
revset functions are likely to depend on a variety of modules.
New repo.anyrevs() does not expand user aliases by default to copy the
behavior of the existing repo.revs(). I don't want to add new function to
localrepository, but this function is quite similar to repo.revs() so it
won't increase the complexity of the localrepository class so much.
New revsetlang module hosts parser, tokenizer, and miscellaneous functions
working on parsed tree. It does not include functions for evaluation such as
getset() and match().
2288 mercurial/revset.py
684 mercurial/revsetlang.py
2972 total
get*() functions are aliased since they are common in revset.py.
The stateful chg plan [1] requires a special repo object, where ideally all
side effects caused by loading the repo object could be reverted by just
dropping (gabbage collect) the loaded repo object.
Currently, that is impossible because repo.__init__ calls
"extensions.loadall", which may have unpredictable side-effects that cannot
be reverted by dropping the repo object.
This patch moves "extensions.loadall" to a separate method, so chg could
subclass localrepository and make extensions loading a no-op.
[1]: mercurial-scm.org/pipermail/mercurial-devel/2017-February/092547.html
changelog.heads() first calls headrevs then converts them to nodes.
localrepo.heads() then sorts them using self.changelog.rev function and makes
useless conversion back to revs. Instead let's call changelog.headrevs() from
localrepo.heads(), sort the output and then convert to nodes. Because headrevs
does not support start parameter this optimization only works if start is None.
The final part of integrating the compression manager APIs into
revlog storage is the plumbing for repositories to advertise they
are using non-zlib storage and for revlogs to instantiate a non-zlib
compression engine.
The main intent of the compression manager work was to zstd all
of the things. Adding zstd to revlogs has proved to be more involved
than other places because revlogs are... special. Very small inputs
and the use of delta chains (which are themselves a form of
compression) are a completely different use case from streaming
compression, which bundles and the wire protocol employ. I've
conducted numerous experiments with zstd in revlogs and have yet
to formalize compression settings and a storage architecture that
I'm confident I won't regret later. In other words, I'm not yet
ready to commit to a new mechanism for using zstd - or any other
compression format - in revlogs.
That being said, having some support for zstd (and other compression
formats) in revlogs in core is beneficial. It can allow others to
conduct experiments.
This patch introduces *highly experimental* support for non-zlib
compression formats in revlogs. Introduced is a config option to
control which compression engine to use. Also introduced is a namespace
of "exp-compression-*" requirements to denote support for non-zlib
compression in revlogs. I've prefixed the namespace with "exp-"
(short for "experimental") because I'm not confident of the
requirements "schema" and in no way want to give the illusion of
supporting these requirements in the future. I fully intend to drop
support for these requirements once we figure out what we're doing
with zstd in revlogs.
A good portion of the patch is teaching the requirements system
about registered compression engines and passing the requested
compression engine as an opener option so revlogs can instantiate
the proper compression engine for new operations.
That's a verbose way of saying "we can now use zstd in revlogs!"
On an `hg pull` conversion of the mozilla-unified repo with no extra
redelta settings (like aggressivemergedeltas), we can see the impact
of zstd vs zlib in revlogs:
$ hg perfrevlogchunks -c
! chunk
! wall 2.032052 comb 2.040000 user 1.990000 sys 0.050000 (best of 5)
! wall 1.866360 comb 1.860000 user 1.820000 sys 0.040000 (best of 6)
! chunk batch
! wall 1.877261 comb 1.870000 user 1.860000 sys 0.010000 (best of 6)
! wall 1.705410 comb 1.710000 user 1.690000 sys 0.020000 (best of 6)
$ hg perfrevlogchunks -m
! chunk
! wall 2.721427 comb 2.720000 user 2.640000 sys 0.080000 (best of 4)
! wall 2.035076 comb 2.030000 user 1.950000 sys 0.080000 (best of 5)
! chunk batch
! wall 2.614561 comb 2.620000 user 2.580000 sys 0.040000 (best of 4)
! wall 1.910252 comb 1.910000 user 1.880000 sys 0.030000 (best of 6)
$ hg perfrevlog -c -d 1
! wall 4.812885 comb 4.820000 user 4.800000 sys 0.020000 (best of 3)
! wall 4.699621 comb 4.710000 user 4.700000 sys 0.010000 (best of 3)
$ hg perfrevlog -m -d 1000
! wall 34.252800 comb 34.250000 user 33.730000 sys 0.520000 (best of 3)
! wall 24.094999 comb 24.090000 user 23.320000 sys 0.770000 (best of 3)
Only modest wins for the changelog. But manifest reading is
significantly faster. What's going on?
One reason might be data volume. zstd decompresses faster. So given
more bytes, it will put more distance between it and zlib.
Another reason is size. In the current design, zstd revlogs are
*larger*:
debugcreatestreamclonebundle (size in bytes)
zlib: 1,638,852,492
zstd: 1,680,601,332
I haven't investigated this fully, but I reckon a significant cause of
larger revlogs is that the zstd frame/header has more bytes than
zlib's. For very small inputs or data that doesn't compress well, we'll
tend to store more uncompressed chunks than with zlib (because the
compressed size isn't smaller than original). This will make revlog
reading faster because it is doing less decompression.
Moving on to bundle performance:
$ hg bundle -a -t none-v2 (total CPU time)
zlib: 102.79s
zstd: 97.75s
So, marginal CPU decrease for reading all chunks in all revlogs
(this is somewhat disappointing).
$ hg bundle -a -t <engine>-v2 (total CPU time)
zlib: 191.59s
zstd: 115.36s
This last test effectively measures the difference between zlib->zlib
and zstd->zstd for revlogs to bundle. This is a rough approximation of
what a server does during `hg clone`.
There are some promising results for zstd. But not enough for me to
feel comfortable advertising it to users. We'll get there...
os.environ is a dictionary which has string elements on Python 3. We have
encoding.environ which take care of all these things. This is the first patch
of 5 patch series which tend to replace the occurences of os.environ with
encoding.environ as using os.environ will result in unusual behaviour.
This is an example usage of ProgrammingError. Let's start migrating
RuntimeError to ProgrammingError.
The code only runs when devel.all-warnings or devel.check-locks is set, so
it does not affect the end-user experience.
The path variable in localrepository.__init__() has a default value None. So
it gives us a option to create an object to localrespository class without
path variable. But things break if you try to do so. The second line in the
init which will be executed when we try to create a localrepository object
will call os.path.expandvars(path) which returns
TypeError: argument of type 'NoneType' is not iterable
I checked occurrences when it is called and can't find any piece of code
which calls it without path variable. Also if something is calling it, its
should break.
This removes one more dependency on the manifest class by moving the write
functionality onto the memmanifestctx classes and changing the one consumer to
use the new API.
By moving the write path to a manifestctx, we now give the individual manifests
control over how they're read and serialized. This will be useful in developing
new manifest formats and storage systems.
localrepo.commit had code to check for unresolved merge conflicts,
it would be helpful for at least rebase to be able to use that
code without calling commit().
The old @property on manifestlog was broken. It meant that we would always
recreate the manifestlog instance, which meant the cache was never hit. Since
we'll eventually remove repo.manifest and make manifestlog the only property,
let's go ahead and make manifestlog the @storecache property, have manifestlog
own the manifest instance, and have repo.manifest refer to it via manifestlog.
This means all accesses go through repo.manifestlog, which is now invalidated
correctly.
A future patch will be moving manifest creation to be inside manifestlog as part
of improving our cache guarantees. bundlerepo and unionrepo currently rely on
being able to hook into manifest creation, so let's temporarily move the actual
manifest creation to a helper function for them to intercept.
In the future manifest.manifest() will disappear entirely and this can
disappear.
Currently, exchange.getbundle() returns either a cg1unpacker or a
util.chunkbuffer (in the case of bundle2). This is kinda OK, as
both expose a .read() to consumers. However, localpeer.getbundle()
has code inferring what the response type is based on arguments and
converts the util.chunkbuffer returned in the bundle2 case to a
bundle2.unbundle20 instance. This is a sign that the API for
exchange.getbundle() is not ideal because it doesn't consistently
return an "unbundler" instance.
In addition, unbundlers mask the fact that there is an underlying
generator of changegroup data. In both cg1 and bundle2, this generator
is being fed into a util.chunkbuffer so it can be re-exposed as a
file object.
util.chunkbuffer is a nice abstraction. However, it should only be
used "at the edges." This is because keeping data as a generator is
more efficient than converting it to a chunkbuffer, especially if we
convert that chunkbuffer back to a generator (as is the case in some
code paths currently).
This patch refactors exchange.getbundle() into
exchange.getbundlechunks(). The new API returns an iterator of chunks
instead of a file-like object.
Callers of exchange.getbundle() have been updated to use the new API.
There is a minor change of behavior in test-getbundle.t. This is
because `hg debuggetbundle` isn't defining bundlecaps. As a result,
a cg1 data stream and unpacker is being produced. This is getting fed
into a new bundle20 instance via bundle2.writebundle(), which uses
a backchannel mechanism between changegroup generation to add the
"nbchanges" part parameter. I never liked this backchannel mechanism
and I plan to remove it someday. `hg bundle` still produces the
"nbchanges" part parameter, so there should be no user-visible
change of behavior. I consider this "regression" a bug in
`hg debuggetbundle`. And that bug is captured by an existing
"TODO" in the code to use bundle2 capabilities.
This adds a simple add() function to manifestlog. This lets us convert more
uses of repo.manifest to use repo.manifestlog, so we can further break our
dependency on the manifest class.
Before this patch, if transaction is started via "filtered repo"
object, _refreshfilecachestats() at closing transaction doesn't
refresh file stat of any @filecache properties correctly, because:
- _refreshfilecachestats() omits refreshing file stat of a
@filecache property, if it doesn't appear in self.__dict__
- if transaction is started via "filtered repo",
_refreshfilecachestats() is applied on "filtered repo"
because repo.transaction() adds "self._refreshfilecachestats" to
post close procedures. repo.transaction() isn't unfiltered method,
and "self" in it means "filtered repo" in this case.
Transactions started by explicit repo.transaction() easily causes
this situation.
- _refreshfilecachestats() applied on "filtered repo" omits whole
refreshing
because @filecache properties are stored into "unfiltered repo",
and appear only in self.__dict__ of "unfiltered repo".
This incorrect refreshing causes unnecessary reloading from files.
To refresh file stat of @filecache properties at closing transaction
correctly, this patch makes _refreshfilecachestats() unfiltered
method.
This patch chooses making _refreshfilecachestats() unfiltered method
instead of making transaction() unfiltered method, to reduce
unexpected side effect.
Before this patch, invalidate() discards in-memory fncache changes,
even inside transaction scope.
Such changes should be written out at closing transaction. Otherwise,
fncache might overlook newly added files. A file overlooked by
fncache isn't accessible via store vfs, even if it actually exists in
store.
On the other hand, a non-existing file in fncache is less harmful,
because fncachestore always examines whether a file actually exists or
not before access. Therefore, discarding in-memory changes can be
safely omitted.
It is typical case that repo.invalidate() in streamclone is executed
inside nested transaction.
This patch makes invalidate() avoid invalidating store inside
transaction.
This patch focuses on describing only how invalidate() changes own
behavior according to activity of transaction. Describing other detail
of invalidate() in docstr will be done in another series, which
refactors invalidate*() functions.
Improve the lock waiting warning message by explicitly saying that a host and
process are holding the lock. This nudges confused new users in the direction
of investigating the other process instead of removing the lock.
The file caches we're using to avoid reloading the manifest from disk everytime
has an annoying bug that causes the in memory structure to not be reloaded if
the mtime and the size haven't changed. This causes a breakage in the tests
because the manifestlog is not being reloaded after a commit+strip operation in
mq (the mtime is the same because it all happens in the same second, and the
resulting size is the same because we add 1 and remove 1). The only reason this
doesn't affect the manifest itself is because we touch it so often that we
had already reloaded it after the commit, but before the strip.
Once the entire manifest has migrated to manifestlog, we can get rid of these
properties, since then the manifestlog will be touched after the commit, but
before the strip, as well.
This is the start of a large refactoring of the manifest class. It introduces
the new manifestlog and manifestctx classes which will represent the collection
of all manifests and individual instances, respectively.
Future patches will begin to convert usages of repo.manifest to
repo.manifestlog, adding the necessary functionality to manifestlog and instance
as they are needed.
I can never remember the differences between the various revset
APIs. I can never remember that scmutil.revrange() is the one I
want to use from user-facing commands.
Add some documentation to clarify this.
While we're here, the argument name for revrange() is changed to
"specs" because that's what it actually is.
To make it easier to patch the wrapped function, make it possible to access the
filecache descriptor directly on the class (rather than have to use
ClassObject.__dict__['attributename']). Returning `self` when the first
argument to `__get__` is `None` makes the descriptor behave the same way
`property` objects do.
Rollback of previous transaction restores contents of files below by
renaming from 'undo.*' file. If renaming keeps ctime, mtime and size
of a file, restoring is overlooked, and old contents cached before
restoring isn't invalidated as expected.
- .hg/bookmarks
- .hg/phaseroots
To avoid ambiguity of file stat at restoring, this patch invokes
vfs.rename() with checkambig=True.
BTW, .hg/dirstate is also restored at rollback. But it is restored by
dirstate.restorebackup(), and previous patch already made it invoke
vfs.rename() with checkambig=True.
This patch is a part of "Exact Cache Validation Plan":
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
All versions of Python we support or hope to support make the hash
functions available in the same way under the same name, so we may as
well drop the util forwards.
If you have just executable-bit change and amend it twice it will vanish:
* After the first amend the commit will have the proper executable bit set
in manifest but it won't have the the file on the list of files in
changelog.
* The second amend will read the wrong list of files from changelog and it
will copy the manifest entry from parent for this file.
* Voila! The change is lost.
This change repairs the bug in localrepo causing this and adds a test for it.
This is one step towards having dirstate manage its own storage. It will
be useful for the implementation of sql dirstate [1].
This introduced a small test change: now we always write the dirstate before
saving backup so in some cases where dirstate file didn't exist yet
savebackup can create it.
[1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan
This is one step towards having dirstate manage its own storage. It will
be useful for the implementation of sqldirstate [1].
I'm deleting two of the dirstate.invalidate() calls in localrepo because
restorebackup method does that for us.
[1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan
We have been warning about transactions without locks for about a year (and
three releases), third party extensions had a fair grace period to fix their
code, we are moving lack of locking to a hard failure in order to protect users
against repository corruption.
