Python garbage collection is triggered by container creation. So code that
creates a lot of tuples tends to trigger GC a lot. We disable the gc during
obsolescence marker parsing and associated initialization. This provides an
interesting speedup (25%).
Load marker function on my 58758 markers repo:
before: 0.468247 seconds
after: 0.344362 seconds
The benefit is a bit less visible overall. With python2.6 on my system I see:
after: 0.60
before: 0.53
The difference is probably explained by the delaying of a costly GC. (but there
is still a win). Marking involved tuples, lists and dicts as ignorable by the
garbage collector should give us more benefit. But this is another adventure.
Thanks goes to Siddharth Agarwal for the lead.
Garbage collection behave pathologically when creating a lot of containers. As
we do that more than once it become sensible to have a decorator for it. See
inline documentation for details.
Most merge action messages don't describe the action itself, they
describe the reason the action was taken. The only exeption is the 'k'
action, for which the message is just "keep" and instead there is a
code comment folling it that says "remote unchanged". Let's move that
comment into the merge action message.
This fixes the previously mentioned issue with 7d5fcea60c78, and undoes its
corresponding test change.
The test change demonstrates the correctness when a file is specified (i.e. the
glob is required on Windows because relative paths use '\' and absolute paths
use '/'). It is admittedly very subtle, but there will be a more robust test in
the addremove -S v3 series.
Several methods print files relative to the repo root, unless files are named on
the command line, in which case they are printed relative to cwd. Since the
check relies on the 'pats' parameter, which needs to be replaced by a matcher
when adding subrepo support, this logic gets folded into the matcher to tidy up
the callers.
Prior to 7d5fcea60c78, this style decision was based off of whether or not the
'pats' list was empty. That change altered the check to test match.anypats()
instead, in order to make paths printed consistent when -I/-X is specified.
That however, changed the style when a file is given to the command. So now we
test the pattern list to get the old behavior for files, as well as test -I/-X
to get the consistency for patterns.
When the local side has renamed a directory from a/ to b/ and added a
file b/c in it, and the remote side has added a file a/c, we end up
overwriting the local file b/c with the contents of remote file
a/c. Add a check for this case and use the merge ('m') action in this
case instead of the directory rename get ('dg') action.
When the remote side has renamed a directory from a/ to b/ and added a
file b/c in it, and the local side has added a file a/c, we end up
moving a/c to b/c without considering the remote version of b/c. Add a
check for this case and use the merge ('m') action in this case
instead of the directory rename ('dm') action.
There are three high-level cases that are of interest in
manifestmerge(): 1) The file exists on both sides, 2) The file exists
only on the local side, and 3) The file exists only on the remote
side. Let's make this clearer in the code.
The 'if f in copied' case will be broken up into the two applicable
branches in the next patch.
Currently, the revlog index C implementation assumes its node tree will be
initialized before a new element is inserted by revnum. For example, revlog.py
executes 'self.index.insert(-1, e)' in _addrevision(). This is only safe
because the node tree has been initialized by a "node in self.nodemap"
check made in addrevision().
(For context, this was discovered while developing an experimental revlog
mixin which stores "elided nodes" via a separate code path from
_addrevision(); that new code path segfaults without this patch.)
The order is determined by manifest.diff(), which currently is not
sorted. There are currently no tests for this, but we will soon add
some that would be flaky without this patch.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
This diff adds an experimental config option "bundle2.pushback" which provides
a transaction to the reply unbundler during a push operation. This behavior is
opt-in because of potential security issues: the response can contain any part
type that has a handler defined, allowing the server to make arbitrary changes
to the local repository.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
The default transaction getter for processbundle is a private function that
raises an exception; this diff lets calling code pass None as the transaction
getter to explicitly request this default behavior.
The next diff will check a config option to determine whether to provide a
transaction to the reply bundle processor. If one shouldn't be provided, the
code needs a way to specify that the default behavior should be used.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
Most pushes already open a transaction in order to sync phase information.
This diff replaces that transaction with one that spans the entire push
operation.
This transaction will be used in a later patch to guard repository changes
made during the reply handler.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
Aside from the transaction logic, the pulloperation class is used primarily as
a logic-free data structure for storing state information. This diff extracts
the transaction logic into its own class that can be shared with push
operations.
These aren't exactly format-breaking features -- just ones for which patches
applied to a repo will produce incorrect commits, In any case, some commands
like record and annotate only care about this feature.
Not all callers are interested in all diffopts -- for example, commands like
record (which use diff internally) break when diffopts like noprefix are
enabled. This function will allow us to add flags that callers can use to
enable only the features they're interested in.
For "hg addremove 'glob:*.py'", we print any paths added or removed as
relative to the current directory, but when "hg addremove -I
'glob:*.py'" is used, we use the absolute path (relative from the repo
root). It seems like they should be the same, so change it so we use
relative paths in both cases. Continue to use absolute paths when no
patterns are given.
Instead of using a file that we know is not in the common ancestor's
maniffest, let's use None. This is safe as the only place that cares
about the value (applyupdates) already checks if the item exists in
the ancestor.
We can further limit the scope of the 2-way merge case by breaking out
the case where the file was not created from scratch on both sides but
rather renamed in the same way (and is therefore a 3-way merge). This
involves copying some code, but it makes it clearer which case the
"Note:" in the code refers to.
When 'f' is not in 'ma', 'a' will be 'nullid' and all the if/elif
conditions that check whether some one nodeid is equal to 'a' will
fail, and the else-clause will instead apply. We can make that more
explicit by creating a separate 'm' action for the case where 'a' is
'nullid'. While it does mean copying some code, perhaps it makes it a
little clearer which codepaths are possible, and which cases the
"Note:" in the code refers to. It also lets us make the debug action
messages a little more specific.
The change in 02ecc94fb657 created a problem on Windows and OS X:
--- /usr/local/mercurial/tests/test-issue660.t
+++ /usr/local/mercurial/tests/test-issue660.t.err
@@ -47,6 +47,8 @@
Should succeed - shadow removed:
$ hg add b
+ adding b/b
+ b/b does not exist!
Prior to the failing 'hg add', the file 'b/b' was added and committed, then 'b'
was recursively deleted from the filesystem, file 'b' was created and the delete
was recorded with 'hg rm --after'. This add is attempting to record the
existence of file 'b'.
A filesystem that is not case sensitive prevents dirstate.walk() from skipping
its step 3, and step 3 has the effect of inserting removed files into the walk
list. The Linux code doesn't run through step 3, and didn't exhibit the
problem. It's not clear why a non case sensitive filesystem triggers step 3,
given that the path normalization occurs in step 2.
Prior to 02ecc94fb657, part of the check here was 'f not in repo.dirstate'
instead of 'f not in wctx'. Files in the 'r' state are filtered out of
context.__contains__() but not dirstate.__contains__(). Therefore the removed
file name wasn't added to the list of files to add when checking against
dirstate. That change was to allow removed files to be readded, but adding a
file that doesn't exist is nonsensical. If the user specifies a missing file,
it will be an exact match and will still fail.
Since 4a56fba99974 (merge: don't use unknown(), 2012-02-09), untracked
files are no longer included in the manifest diff, so there is no need
to check exclude them when renaming files for directory moves with the
'dm' action.
calculateupdates() happens before applyupdates(), so move it before in
the code. That also moves it close to manifestmerge(), which is a good
location as calculateupdates() is the only caller of manifestmerge().
In a mozilla repo with tip at bb3ff09f52fe,
hg update tip~1000 && time hg revert -nq -r tip .
displays ~4:20 minutes. With tip~100, it runs in ~11 s. With revision
100000, it did not finish in 12 minutes.
Revert calls dirstate.status() with a matcher that matches each file
in the target revision. The main problem [1] lies in
dirstate._walkexplicit(), which looks for matching deleted directories
by checking whether each path is prefix of any path in the
dirstate. With m files in the dirstate and n files in the target
revision that are not in the dirstate, this is clearly O(m*n). Let's
improve by keeping a lazily initialized set of all the directories in
the dirstate, so the time becomes O(m+n).
After this patch, the 4:20 minutes become 5.5 s, while for a single
missing path, it slows down from 1.092 s to 1.150 s (best of 4). The
>12 min case becomes 5.8 s.
[1] A narrower optimization would be to make revert take the fast
path for '.' and '--all'.
This patch also replaces "self._getstorehashcachepath" (building
absolute path up) by "self._getstorehashcachename" (building relative
path up), because "vfs.writelines" requires relative path.
This patch uses "False" as default value of "notindexed" argument,
even though "vfs.makedir()" uses "True" for it, because "os.mkdir()"
doesn't set "_FILE_ATTRIBUTE_NOT_CONTENT_INDEXED" attribute to newly
created directories.
This patch also replaces "self._getstorehashcachepath" (building
absolute path up) by "self._getstorehashcachename" (building relative
path up), because "vfs.tryreadlines" requires relative path.
This patch makes "_readstorehashcache()" return "[]" (returned by
"vfs.tryreadlines()"), when cache file doesn't exist, even though
"_readstorehashcache()" returned '' (empty string) in such case before
this patch.
"_readstorehashcache()" is invoked only by the code path below in
"_storeclean()":
for filehash in self._readstorehashcache(path):
if filehash != itercache.next():
clean = False
break
In this case, "[]" and '' don't differ from each other, because both
of them cause avoiding iteration of "for loop".
This patch allows "readlines" and "tryreadlines" to take "mode"
argument, because "subrepo" requires to read files not in "rb"
(binary, default for vfs) but in "r" (text) mode in subsequent patch.
This "vfs" object will be used by subsequent patches to handle cache
store hash files without direct file APIs.
This patch decorates "_cachestorehashvfs" with "@propertycache" to
delay vfs creation, because it is used only for cooperation with other
repositories.
In this patch, "/" is used as the path separator, even though
"self._repo.join" uses platform specific path separator (e.g. "\\" on
Windows). But it is reasonable enough, because "store" and other
management file handling already include such implementation, and they
work well.
"_calcfilehash" can be completely replaced by simple "vfs.tryread"
invocation.
def _calcfilehash(filename):
data = ''
if os.path.exists(filename):
fd = open(filename, 'rb')
data = fd.read()
fd.close()
return util.sha1(data).hexdigest()
Building absolute path "absname" up by "self._repo.join" for files in
"filelist" is avoided, because "vfs.tryread" does so internally.
Existance of specified "path" should be examined by "exists" via wvfs
of the parent repository, because the working directory of the parent
repository may be in UTF-8 mode. Wide API should be used via wvfs in
such case.
In this patch, "/" is used as the path separator, even though "path"
uses platform specific path separator (e.g. "\\" on Windows). But it
is reasonable enough, because "store" and other management file
handling already include such implementation, and they work well.
"util.makedirs" for the (sub-)repository root of "hgsubrepo" is also
executed in the constructor of "localrepository", if "create" is True
and ".hg" of it doesn't exist.
This patch avoids redundant "util.makedirs" invocation in the
constructor of "hgsubrepo".
manifestmerge() has a piece of code that's roughly:
if not force and different:
abort
else:
# if different: old untracked f may be overwritten and lost
...
The comment only talks about what happens when 'different' is true,
and in combination with the if-block above, that must mean that it is
only about what happens when 'force and different'. It seems quite
fine that files are overwritten when 'force' is true, so let's remove
the comment. As it stands, it can easily be interpreted as a TODO
(which is how I interpreted it at first).
Such file are generated with a .pending prefix. It is up to the reader to
implement the necessary logic for reading pending files.
We add a test to ensure pending files are properly cleaned-up in both success and
error cases.
This will allow us to generate temporary pending files. Files
generated with a suffix are assumed temporary and will be cleaned up
at the end of the transaction.
As far as I and the test suite can tell, the checks in manifestmerge()
already report the errors (whether or not --check is given), so we
don't need to call merge.checkunknown(). Since this is the last call
to the method, also remove the method.
We were definitely being suboptimal here: we were constructing two full sets,
one with the full set of common nodes (i.e. a graph traversal) and one with all
nodes. Then we subtract one set from the other. This whole process is
O(commits) and causes discovery to be significantly slower than it should be.
Instead, keep track of common incrementally and keep undecided as small as
possible.
This makes discovery massively faster on large repos: on one such repo, 'hg
debugdiscovery' over SSH with one commit missing on the client and five on the
server went from 4.5 seconds to 1.5. (An 'hg debugdiscovery' with no commits
missing on the client, i.e. connection startup time, was 1.2 seconds.)
This allows multiple efficient missing ancestor queries against the same set of
bases. In upcoming patches we'll also define ways to grow the set of bases.
The fact that the test output hasn't changed establishes this patch's
correctness.
Any revs that are filtered out are also in basesvisit, which means they
wouldn't be returned in the missing list anyway. There's no need to explore
such revs or their ancestors.
The 'if not revsvisit' check moves down because we can't call max() on an empty
set.
We only actually care about whether revsvisit is empty, so we can let
basesvisit grow to arbitrary size.
It turns out that this actually helps performance. For a large repo with
hundreds of thousands of commits, hg perfrevset 'only(0, tip)' (basically the
worst case, involving a full DAG traversal) goes from 1.63 seconds to 1.50. hg
perfrevset 'only(tip, 0)' remains unchanged at 1.98 seconds.
Previously, any files relative to the root of the repo that match the -I
patterns would be deleted, but the command exited with 1 after printing a
warning:
$ hg remove -S -I 're:.*.txt' .
removing sub1/sub2/folder/test.txt
removing sub1/sub2/test.txt
not removing .: no tracked files
Like 'forget', git and svn subrepos are currently not supported. Unfortunately
the name 'remove' is already used in the subrepo classes, so we break the
convention of naming the subrepo function after the command.
Because pipe-mode server uses stdio as IPC channel, other modules should not
touch stdio directly and use ui instead. However, this strategy is brittle
because several Python functions read and write stdio implicitly.
print 'hello' # should use ui.write()
# => ch = 'h', size = 1701604463 'ello', data = '\n'
This patch adds protection for such mistakes. Both stdio files and low-level
file descriptors are redirected to /dev/null while command server uses them.
The method has been called from commands.py since 8d9ca2ac2fe8
(update: just merge unknown file collisions, 2012-02-09), so drop the
underscore prefix that suggests that it's private.
We are still doing double backups, but now that we have proper
location handling this is less of an issue. Dropping this simplifies
the code before we add some pending-related logic.
This also ensures we actually test the new 'location' mechanism.
The current naming scheme ('journal.backups.<file>') resulted is bad directory
name when 'file' was in a subdirectory. We now extract the directory name and
create the backupfile within it.
We plan to use file in a subdirectory for cachefile.
We do not want to abort if anything wrong happen while handling a cache file.
Cache file have way to be invalidated and if old/bad version stay no
misbehavior will happen. Proper value will eventually be computed and the wrong
will be righten.
This changeset use the transaction reporter (usually writing on stderr) to write
details about failed cache handling. This will only apply to write operation
using a transaction. The usual update during read only operation will stay a
debug message.
I was on the way to bring these message back to debug level when I realised it
could be a feature. People with write access to the repository are likely to
have the power to fix error related to cache (and it is valuable to fix them).
So let the things as is for now.
The goal is to allow access to file outside ofthe store directory from the
transaction. The obvious target are the `bookmarks` file. But we can envision
usage for cache too.
We keep passing a main opener explicitly because a lot of code rely on this
default opener. The main opener (operating on store) is using an empty key ''.
We need to store new data to improve the current transaction logic:
- location: We want to generate and backup file outside of the 'store' (eg:
bookmarks, or various cache files). This requires knowing and preserving where
each file is located. The value of this new field is a string. It will be used
as a key for a vfs mapping.
- cache: We would like to handle cache file in the transaction code. This
Will help to have cache consistent with the repository state and avoid
performance issue on big repository like Mozilla. However, failure to handle
cache file should not result in a transaction failure. We add a new field that
carry this information. The value is boolean, A True value mean any error
while handling this file can be ignored.
Those two mechanisms are not implemented yet, but they are now persisted in the
on disk file. Support for new mechanisms is coming in later changeset.
We update the file format now and will introduce the new features in later
changeset. The format version is set to 0 until we actually support the new feature.
This will prevent misunderstanding between incomplete and final client.
Support for reading both version 1 and (future) version 2 could be achieved
(using default value when reading version 1) but has not been seen as necessary
for now.
This dumb cache works surprisingly well: on a repository with typical delta
chains ~50k in length, unbundling a linear series of 5000 revisions (changelogs
and manifests only) went from 60 seconds to 3.
Move the code in context._manifestmatches() into a new
manifest.matches(). It's a natural place for the code to live and it
allows other callers to easily use it. It should also make it easier
to optimize the new method in alternative implementations of the
manifest (same reasoning as with manifest.diff()).
We can just modify the status tuple we got from dirstate.status()
instead of deconstructing it and constructing a new instance, thereby
simplifying the code a little.
Letting _dirstatestatus() return an scmutil.status instance also means
that _buildstatus() will always return such an instance, so we can
remove the conversion from the call sites.
It makes no sense to request reverse status (i.e. changes from the
working copy to its parent) and then look at the deleted, unknown or
ignored fields. If you do, you would get the result from the forward
status (changes from parent to the working copy). Instead of giving a
nonsensical answer to a nonsensical question, it seems a little saner
to return empty lists. It might be best if we could prevent the caller
accessing these lists, but it's doubtful it's worth the trouble.
See previous patch descriptions for the motivation.
The tests reflect the current state of the world -- as we add support we'll see
changes in the test output.
In an upcoming patch we'll enable support as an option to 'hg diff' as well.
The tests reflect the current state of the world -- as we add support we'll see
changes in the test output.
Upcoming patches will add an option that will almost certainly break diff
output parsers when enabled. Add support for forcing an option to something in
plain mode, as a fallback. Options passed in via the CLI are not affected,
though -- it is assumed that any script passing the option in explicitly knows
what it is doing.
By popular demand, we introduce an option to disable the 'a/' and 'b/'
prefixes in diff output. This makes copying and pasting filenames from diff
output easier.
This option will be implemented and documented in upcoming patches. To ensure
that existing scripts that parse output don't break, we will ensure that this
prefix is disabled in plain mode. A straight 'hg export | hg import' without
HGPLAIN=1 will still be broken though, but there's little that can be done
about that.
During the transaction, files may be created to store or expose data
involved in the transaction (eg: changelog index data are written in
a 'changelog.i.a' for hooks). But we do not have an official way to
record such file creation and make sure they are cleaned up. The lack
of clean-up is currently okay because there is a single file involved
and a single producer/consumer.
However, as we want to expose more data (bookmarks, phases, obsmarker)
we need something more solid. The 'backupentries' mechanism could
handle that. Temporary files can be encoded as a backup of nothing
'('', <temporarypath>)'. We "need" to attach it to the same mechanism
as we use to be able to use temporary transaction files outside of
.'store/' and 'backupentries' is expected to gain such feature.
This changeset makes it clear that we should rename 'backupentries' to
something more generic.
This doesn't affect normal clones since they'd be bound by the CPU bound below
anyway -- it does, however, improve generaldelta clones significantly.
This also results in better deltaing for generaldelta clones -- in generaldelta
clones, we calculate deltas with respect to the closest base if it has a higher
revision number than either parent. If the base is on a significantly different
branch, this can result in pointlessly massive deltas. This reduces the number
of bases and hence the number of bad deltas.
Empirically, for a highly branchy repository, this resulted in an improvement
of around 15% to manifest size.
This is a very silly case and not particularly likely to happen in the wild,
but it turns out we can hit it in a couple of places. As we tune the storage
parameters we're likely to hit more such cases.
The affected test cases all have smaller revlogs now.
We are about to use the 'backupentry' mechanism to allow cleaning up
transaction-related temporary files (such as 'changelog.i.a'). We start
by extracting the entry registration into its own method for easy reuse.
At that point, I would like to rename the backup-file related variable to
something generic but I'm a bit short of ideas.
This mirrors the API for 'pending' and 'finalize' callbacks. I do not have
immediate usage planned for it, but I'm sure some callback will be happy to
access transaction related data.
The callback will likely need to perform some operation related to the
transaction (eg: registering file update). So we better pass the current
transaction as the callback argument. Otherwise callback that needs it has to
rely on horrible weak reference trick.
This allow already allow us to slay a wild weak reference usage.
The callback will likely need to perform some operation related to the
transaction (eg: backing files up). So we better pass the current transaction as
the callback argument. Otherwise callback that needs it has to rely on horrible
weak reference trick.
The first foreseen user of this is changelog._writepending. We would like it to
register the temporary file it create for cleanup purpose.
The case where a backup of a missing file was requested was previously
handled by the 'entries' list. As the 'backupentries' is about to gain
ability to backup files outside of '.hg/store', we want it to be able
to handle the missing file too.
Reminder: using 'addbackup' on a missing file means that such file needs to be
deleted if we rollback the transaction.
This change is intended to avoid future problem of data corruption under
command server. out=ui.fout is mandatory as long as command server uses
stdout as IPC channel.
Future patches will allow extensions to choose which order a namespace should
output in the log, so we add a way for sortdict to insert to a specific
location.
Future patches will start using sortdict for log operations where order is
important. Adding iteritems removes the headache of having to remember to use
items() if the object is a sortdict.
After running "hg forget README && hg addremove", README will still be
reported as removed, while "hg forget README && hg add README" adds it
back so it gets reported as clean. It seems like they should behave
the same. Furthermore, it seems like no files should remain untracked
after 'hg addremove && hg commit' (or 'hg commit -A'). For these
reasons, change the behavior of addremove so it does add forgotten
files back.
The problem is with scmutil._interestingfiles(), which reports the
file as removed, so scmutil.addremove() does not add it. Fix by
teaching _interestingfiles() to report forgotten files separately from
removed files and make addremove() add forgotten files back. However,
do not treat forgotten files as sources for rename detection. Note
that since removed and forgotten files are treated the same before
this change, forgotten files were considered sources for rename
detection.
Also update the other caller, marktouched(), in the same way as
addremove().
I accidentally did 'hg forget .' and tried to undo the operation with
'hg add .'. I expected the files to be reported as either modified or
clean, but they were still reported as removed. It turns out that
forgotten files are only added back if they are listed explicitly, as
shown by the following two invocations. This makes it hard to recover
from the mistake of forgetting a lot of files.
$ hg forget README && hg add README && hg status -A README
C README
$ hg forget README && hg add . && hg status -A README
R README
The problem lies in cmdutil.add(). That method checks that the file
isn't already tracked before adding it, but it does so by checking the
dirstate, which does have an entry for forgotten files (state 'r'). We
should instead be checking whether the file exists in the
workingctx. The workingctx is also what we later call add() on, and
that method takes care of transforming the add() into a normallookup()
on the dirstate.
Since we're changing repo.dirstate into wctx, let's also change
repo.walk into wctx.walk for consistency (repo.walk calls wctx.walk,
so we're simply inlining the call).
The current heuristic for deciding between storing delta and full texts
is based on ratio of (sizeofdeltas)/(sizeoffulltext).
In some cases (for example a manifest for ahuge repo) this approach
can result in extremely long delta chains (~30,000) which are very slow to
read. (In the case of a manifest ~500ms are added to every hg command because of that).
This commit introduces "revlog.maxchainlength" configuration variable that will
limit delta chain length.
The chain length was computed correctly only when generaldelta
feature was enabled. Now it's fixed.
When generaldelta is disabled the base revision in revlog index is not
the revision we have delta against - it's always previous revision.
Instead of incorrect chainbaseandlen in command.py we are now using two
single-responsibility functions in revlog.py:
- chainbase(rev)
- chainlen(rev)
Only chainlen(rev) was missing so it was written to mimic the way the
chain of deltas is actually found during file reconstruction.
The `startgroup` and `endgroup` methods are used in a very specific
context to wrap a very specific operation (revlog truncation). It does
not make sense to perform any other operations during such a "group"
(eg:file backup). There is currently no user of backupfile during a
"group" so we drop the group-specific code and restrict authorized
operations during "group".
We don't care about filtering out symlinks that have already been
committed with full content, only those that have been accidentally
resolved in the working directory.
In basectx._buildstatus(), we read the manifests for the two revisions
being compared. For "caching reasons" unknown to me, it is better to
read the earlier manifest first, which basectx._prestatus() takes care
of. However, if the 'self' context is a committablectx and the 'other'
context is the parent of the working directory (as in the very common
case of plain "hg status"), there is no need to read any manifests at
all -- all that's needed is the dirstate status. To avoid reading the
manifests, _prestatus() is overridden in committablectx and avoids
calling its super method, and _buildstatus() calls its super method
only if the 'other' context is not the parent of the working
directory.
It seems easier to follow what's happening if we move the pre-fetching
to _buildstatus() just before the place where the manifests are
fetched. We just need to add an extra check that the revision is not
None to handle the case that was previously handled by subclass
overriding. That also makes it safe for committablectx._prestatus() to
call its parent, although the latter now becomes empty, so we won't
bother.
The workingctx method simply calls the super method. The only effect
it has is that it uses a different default argument for the 'other'
argument. The only in-tree caller is patch.diff, which always passes
an argument to the method, so it should be safe to remove the
overriding. Having the default argument depend on the type seems
rather dangerous anyway.
The post-transaction hooks run after the lock release (because hooks may want to
touch the repository), but they must only run if the transaction is successfully
closed.
We use the new 'addpostclose' method on transaction to register a callback
installing this post-lock-release call.
The addchangegroup code considers the transaction done after a 'tr.close()' call
and schedules the hook's execution for after lock release. In the nested transaction
case, the transaction is not yet committed and we must delay this scheduling.
We add an 'addpostclose' method (like the 'addpending' and 'addfinalize' ones) that
registers code to be run if the transaction is successfully committed.
Changeset d79feb65f3ee added advertising of supported changegroup version
through the new 'b2x:changegroup' capability. However, this capability is not
new and has been around since 3.1 with an empty value. This makes new clients
unable to push to 3.2 servers through bundle2 as they cannot find a common
changegroup version to use from and empty list.
Treating empty 'b2x:changegroup' value as old client fixes it.
Instead of calling 'cl.finalize()' by hand (possibly at a bogus time) we
register it in the transaction during 'delayupdate' and rely on 'tr.close()' to
call it at the right time.
The new 'addfinalize' method allows people to register a callback to
be triggered when the transaction is closed. This aims to get rid of
explicit calls to 'changelog.finalize'. This also obsoletes the
'onclose' function but removing it is not in the scope of this series.
The 'delayupdate' method now takes a transaction object and registers its
'_writepending' method for execution in 'transaction.writepending()'. The hook can then
use 'transaction.writepending()' directly.
At some point this will allow the addition of other file creation
during writepending.
The contents of the transaction must be flushed to disk before running
a hook. But it must be flushed to a special file so that the normal
reader does not use it. This logic is currently in the changelog only.
We add some facility to register such operations in the transaction
itself.
The current way we use the 'delayupdate' mechanism is wrong. We call
'delayupdate' right after the transaction retrieval, then we call 'finalize'
right before calling 'tr.close()'. The 'finalize' call will -always- result in a
flush to disk, making the data available to all readers. But the 'tr.close()' may
be a no-op if the transaction is nested. This would result in data:
1) exposed to reader too early,
2) rolled back by other part of the transaction after such exposure
So we need to end up in a situation where we call 'finalize' a single time when
the transaction actually closes. For this purpose we need to be able to call
'delayupdate' and '_writepending' multiple times and 'finalize' once. This was
not possible with the previous state of the code.
This changeset refactors the code to makes this possible. We buffer data in memory
as much as possible and fall-back to writing to a ".a" file after the first call
to '_writepending'.
cg2 supports generaldelta in changegroups, to be used in bundle2.
Since generaldelta is handled directly in cg2, reordering is switched
off by default.
When using bundle2, we find the common subset of supported changegroup-packers
and we pick the max of them. This allow to use generaldelta aware changegroups through
bundle2.
When using bundle2, we find the common subset of supported changegroup-packers
and we pick the max of them. This allow to use generaldelta aware changegroup
through bundle2.
The commands getchangegroup, getlocalchangegroup and getsubset now each
have a version ending in -raw. The raw versions return the chunk generator
from the changegroup packer directly, without wrapping it in a chunkbuffer
and unpacker. This avoids extra chunkbuffers in the bundle2 code path.
Also, the raw versions can be extended to support alternative packers
in the future, to be used from bundle2.
This will let the bundle2 client and server detect what packer they should be using.
This detection part is not done. I expect it to be done with the addition of the
second packer (with generaldelta support).
We only have "01" right now, but we should get general delta in soon.
Bundle2 is expected to make use of this to advertise and select the right packer
to use on both sides.
Calling 'baseset(repo.changelog)' builds a list for all revisions in
the repo. And we already have the lazy and efficient 'fullreposet'
class for this purpose.
This gives us the usual benefits of the fullreposet but it is less visible
because the matching process itself is very expensive:
revset) matching(100)
before) wall 6.413281 comb 6.420000 user 5.910000 sys 0.510000 (best of 3)
after) wall 6.173608 comb 6.170000 user 5.750000 sys 0.420000 (best of 3)
However for some complex list, this provide a massive speedup
revset) matching(parents(100))
before) wall 23.890740 comb 23.890000 user 23.450000 sys 0.440000 (best of 3)
after) wall 6.382280 comb 6.390000 user 5.930000 sys 0.460000 (best of 3)
Calling 'baseset(repo.changelog)' builds a list for all revisions in
the repo. And we already have the lazy and efficient 'fullreposet'
class for this purpose.
This gives us the usual benefits of the fullreposet:
revset) 100^1
before) wall 0.002694 comb 0.000000 user 0.000000 sys 0.000000 (best of 897)
after) wall 0.000997 comb 0.000000 user 0.000000 sys 0.000000 (best of 2324)
revset) parents(100)^1
before) wall 0.003832 comb 0.000000 user 0.000000 sys 0.000000 (best of 587)
after) wall 0.001034 comb 0.000000 user 0.000000 sys 0.000000 (best of 2309)
revset) (100^1)^1
before) wall 0.005616 comb 0.000000 user 0.000000 sys 0.000000 (best of 405)
after) wall 0.001030 comb 0.000000 user 0.000000 sys 0.000000 (best of 2258)
Calling 'baseset(repo.changelog)' builds a list for all revisions in the
repo. And we already have the lazy and efficient 'fullreposet' class
for this purpose.
This gives us the usual benefits of the fullreposet:
revset) children(tip~100)
before) wall 0.007469 comb 0.010000 user 0.010000 sys 0.000000 (best of 338)
after) wall 0.003356 comb 0.000000 user 0.000000 sys 0.000000 (best of 755)
Calling 'baseset(repo.changelog)' builds a list for all revisions in
the repo. And we already have the lazy and efficient 'fullreposet'
class for this purpose.
This gives us the usual benefits of the fullreposet:
revset) 100~5
before) wall 0.002712 comb 0.000000 user 0.000000 sys 0.000000 (best of 918)
after) wall 0.000996 comb 0.000000 user 0.000000 sys 0.000000 (best of 2493)
revset) parents(100)~5
before) wall 0.003812 comb 0.010000 user 0.010000 sys 0.000000 (best of 667)
after) wall 0.001038 comb 0.000000 user 0.000000 sys 0.000000 (best of 2361)
revset) (100~5)~5
before) wall 0.005614 comb 0.000000 user 0.000000 sys 0.000000 (best of 446)
after) wall 0.001035 comb 0.000000 user 0.000000 sys 0.000000 (best of 2424)
Calling 'baseset(repo.changelog)' builds a list for all revisions in
the repo. And we already have the lazy and efficient 'fullreposet'
class for this purpose.
This gives us the usual benefit ofs the fullreposet:
revset) 10:100
before) wall 0.002774 comb 0.000000 user 0.000000 sys 0.000000 (best of 797)
after) wall 0.001977 comb 0.000000 user 0.000000 sys 0.000000 (best of 1244)
revset) parents(10):parents(100)
before) wall 0.005054 comb 0.000000 user 0.000000 sys 0.000000 (best of 481)
after) wall 0.002060 comb 0.000000 user 0.000000 sys 0.000000 (best of 1056)
The merge tool configuration is an essential part of a good initial user
experience. 'make osx' installers and direct 'make' installation did not have
merge tool configuration. Now they have.
Note: The installer fixes for windows have been done blindly and might require
additional changes.
This helps providing a more consistent user experience on all platforms and
with all packaging.
The exact location of default.d depends on how Mercurial is installed and
whether it is 'frozen'. The exact location should never be relevant to users
and is intentionally not explained in details in the documentation. It will
however always be next to the help and templates files.
Note that setting HGRCPATH also disables these defaults. I don't know if that
should be considered a bug or a feature.
Further digging on this issue show that the limit on the sample size used in
discovery never works for heads. Here is a quote from the code itself:
desiredlen = size - len(always)
if desiredlen <= 0:
# This could be bad if there are very many heads, all unknown to the
# server. We're counting on long request support here.
The long request support never landed and evolution make the "very many heads,
all unknown to the server" case quite common.
We implement a simple and stupid hard limit of sample size for all query. This
should prevent HTTP 414 error with the current state of the code.
History rewriting commands like histedit tend to use temporary
commits. They may schedule hook execution on these temporary commits
for after the lock has been released. But temporary commits are likely
to have been stripped before the lock is released (and the hook run).
Hook executed for missing revisions leads to various crashes.
We disable hooks execution for revision missing in the repo. This
provides a dirty but simple fix to user issues.
On streaming clone, we were priming the local branch cache with the
remote branchmap, without checking which heads were closed.
This fixes an issue introduced in:
changeset: 17740:f8d7aaf86507
user: Tomasz Kleczek <tomasz.kleczek@fb.com>
date: Wed Oct 03 13:19:53 2012 -0700
summary: branchcache: fetch source branchcache during clone (issue3378)
that was exposed in 2.9 by:
changeset: 20192:6c385e85aa05
user: Brodie Rao <brodie@sf.io>
date: Mon Sep 16 01:08:29 2013 -0700
summary: branches: simplify with repo.branchmap().iterbranches()
8a92e6790099 broke bookmarks getting copied during uncompressed clones. Since
most of the pull logic has been moved into exchange.py, lets just call
exchange.pull to fix up the repo with the latest bits after the streaming clone
has bootstrapped the repo. This keeps us from having to duplicate the bookmark
logic.
The matcher variable 'm' in checkstatus() is reset to None on each
call, so the caching of the matcher no longer happens as it was
intended. This seems to be a regression in 6b9fbae54476 (revset: added
lazyset implementation to checkstatus, 2014-01-03).
Fix by moving the cached matcher into the enclosing function so it's
actually cached across calls. This speeds up
hg log -r 'modifies(mercurial/context.py)' >/dev/null
from 7.5s to 4s.
Also see similar fix in 5ff5c5c9e69f (revset: avoid recalculating
filesets, 2014-10-22).
We extract the code generating files into its own function. We are
about to move this code around to fix a bug. We'll need it in a
function soon to reuse it for "pending" logic. So we move the code
into a function instead of moving it twice.
Amend will reuse the original username if a new one is not provided
with -U, but obsolete.createmarkers() only considers ui.username() for
the obsolete marker's metadata. Allowing the metadata field to be
spoofed seems to defeat the point of the field in the first place.
This covers 'evolve amend' and 'ci --amend' with evolve enabled.
Without this, the transaction aborts but the parent changeset is set to -1. The
corresponding test will be added to evolve separately.
hg log -r 1 ... -r 100 was never returning due to a regression in the
way addset computes __nonzero__. It used 'bool(self._r1 or self._r2)'
which required executing self._r1.__nonzero__ twice (once for the or,
once for the bool). hg log with a lot of -r's happens to build a one
sided addset tree of N length, which ends up being 2^N performance.
This patch fixes it by converting to bool before or'ing.
This problem can be repro'd with something as simple as:
hg log `for x in $(seq 1 50) ; do echo "-r $x "; done`
Adding '1 + 2 + ... + 20' to the revsetbenchmark.txt didn't seem to repro the
problem, so I wasn't able to add a revset benchmark for this issue.
Previously, we'd scan through the entire directory listing looking for a
normalized match. This is O(N) in the number of files in the directory. If we
decide to call util.fspath on each file in it, the overall complexity works out
to O(N^2). This becomes a problem with directories a few thousand files or
larger.
Switch to using a dictionary instead. There is a slightly higher upfront cost
to pay, but for cases like the above this is amortized O(1). Plus there is a
lower constant factor because generator comprehensions are faster than for
loops, so overall it works out to be a very small loss in performance for 1
file, and a huge gain when there's more.
For a large repo with around 200k files in it on a case-insensitive file
system, for a large directory with over 30,000 files in it, the following
command was tested:
ls | shuf -n $COUNT | xargs hg status
This command leads to util.fspath being called on $COUNT files in the
directory.
COUNT before after
1 0.77s 0.78s
100 1.42s 0.80s
1000 6.3s 0.96s
I also tested with COUNT=10000, but before took too long so I gave up.
The phrase "revision or range" comes from a pre-revset era. Since the
documentation for ranges now is under the revset docs, and as a
helpful hint nudging users towards revsets, I think it's better to say
"revision or revset"
5d1adb6683fa introduced use of the new filtering headrevs C implementation. It
caught TypeError to detect when to fall back to the implementation that was
compatible with old extensions. That method was however not reliable.
Instead, use the new headrevsfiltered function when passing a filter. It will
reliably fail with AttributeError when an old extension that predates
headrevsfiltered is used.
All extensions that have this function do support filtering. The existing
headrevs function may support filtering but we cannot reliably detect whether
it does.
This fixes several push tests in test-bundle2-exchange.t that were failing on
Windows with messages like the following:
$ hg -R main push http://localhost:$HGPORT2/ -r 32af7686d403 \
--bookmark book_32af
pushing to http://localhost:$HGPORT2/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: 1 new obsolescence markers
updating bookmark book_32af
abort: The process cannot access the file because it is being used by another
process: 'C:\path\to\tmp\bundle.hg'
[255]
In order not to avoid listing files as both added and deleted, for
example, we check for every file in the manifest if it is in the
_list_ of deleted files. This can get quite slow when there are many
deleted files. Change it to a set to make the containment check
faster. On a somewhat contrived example of the Mozilla repo with the
entire testing/ directory deleted (~14k files), this makes
'hg status --rev .^' go from 26s to 2s.
The set discovery start by sending a "known" command with all local heads. When
the number of local heads is massive (eg: using hidden changesets) such request
becomes too large. This lead to 414 error over http, aborting the whole
process.
We limit the size of the sample used by the first query to fix this.
The test are impacted because they do test massive number of heads. But they do
not test it over real world http setup.
48062b2d0f30 regressed the behavior of pushing an unchanged bookmark to
a remote. Before that commit, pushing a unchanged bookmark would result
in "exporting bookmark @" being printed. After that commit, we now see
an incorrect message "bookmark %s does not exist on the local or remote
repository!"
This patch fixes the regression introduced by 48062b2d0f30 by having
the bookmark error reporting code filter identical bookmarks and adds
a test for the behavior.
bookmarks.compare() previously lumped identical bookmarks in the
"invalid" bucket. This patch adds a "same" bucket.
An 8-tuple for holding this state is pretty gnarly. The return value
should probably be converted into a class to increase readability. But
that is beyond the scope of a patch intended to be a late arrival to
stable.
The comment in workingctx.status() says that "calling 'super' subtly
reveresed the contexts", but that is simply not true, so we should not
be swapping added and removed fields.
0cc5c10d5dc7 was not the final version of that patch. It was really slow
because `l not in repo.changelog` iterates revisions up to `l`. Instead,
rev() should utilize spanset.__contains__().
revset #0: rev(210000)
0) wall 0.000039 comb 0.000000 user 0.000000 sys 0.000000 (best of 67978)
1) wall 0.002721 comb 0.000000 user 0.000000 sys 0.000000 (best of 1055)
2) wall 0.000059 comb 0.000000 user 0.000000 sys 0.000000 (best of 45599)
(0: 3.2-rc, 1: 0cc5c10d5dc7, 2: this patch)
Note that the benchmark result described in 0cc5c10d5dc7 is wrong because
it is the one of the initial version.
'n' was introduced in Mercurial in 5d1adb6683fa and broke Python 2.4 support in
mysterious ways that only showed failure in test-glog.t. Py_BuildValue failed
because of the unknown format and a TypeError was thrown ... but it never
showed up on the Python side and it happily continued processing with wrong
data.
Quoting https://docs.python.org/2/c-api/arg.html :
n (integer) [Py_ssize_t]
Convert a Python integer or long integer to a C Py_ssize_t.
New in version 2.5.
k (integer) [unsigned long]
Convert a Python integer or long integer to a C unsigned long without
overflow checking.
This will use unsigned long instead of Py_ssize_t. That is not a good solution,
but good is not an option when we have to support Python 2.4.
This addresses the bug described in issue4405: when obsolescence markers are
enabled, amending a commit with a file move can lead to the copy information
being lost.
However, the bug is more general and can be reproduced without obsmarkers as
well, as demonstracted by Pierre-Yves and put into the updated test.
Specifically, graph topology divergences between the filelogs and the changelog
can cause copy information to be lost during amends.
In light of the POODLE[0] attack on SSLv3, let's just drop the ability to
use anything older than TLSv1 entirely.
This only fixes the client side. Another commit will fix the server
side. There are still a few SSLv[23] constants hiding in httpclient,
but I'll fix those separately upstream and import them when we're not
in a code freeze.
0: http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html
If an exception is raised during a bundle2 part payload generation it is now
recorded in the bundle. If such exception occurs, we capture it, transmit an
abort exception through the bundle, cleanly close the current part payload and
raise it again. This allow to generate valid bundle even in case of exception so
that the consumer does not wait forever for a dead producer. This also allow to
raise the exception during unbundling at the exact point it happened during
bundling make debugging easier.
It is now possible to emit a single part in the middle of a payload production.
This part will be processed with limitation (only access to a `ui` object). The
goal is to let the server raise exception and output while a part is being
processed. The source motivation is to transmit exception that occurs while
generating a part.
This change is was the motivation to bump the bundle2 format from HG2X to HG2Y.
Somehow, the format bump made it into 3.2 without it. So this change go on
stable. It is low risk as bundle2 is still disabled by default.
Previously the journal.backupfiles file was delimited by \0. Now we delimit it
using \n (same as the journal file). This allows us to change the number of
values in each line more easily, rather than relying on the count of \0's.
The transaction format will be changing a bit over the next releases, so let's
go ahead and add a version number to make backwards compatibility easier. This
whole file format was broken prior to 3.2 (see previous patch), so changing it
now is pretty low risk.
The transaction backupfiles logic was broken for 'hg recover'. The file format
is XXX\0XXX\0YYY\0YYY\0 but the parser did a couple things wrong. 1) It went one
step beyond the final \0 and tried to read past the end of the array. 2)
array[i:i+1] returns a single item, instead of two items as intended.
Added a test to catch it, which turns out to be the first actual 'hg recover'
test.
The recent optimization of "and" operation relies on the assumption that
the rhs set does not contain invalid revisions. So rev() has to remove
invalid revisions.
This is still faster than using `.filter(lambda r: r == l)`.
revset #0: rev(25)
0) wall 0.026341 comb 0.020000 user 0.020000 sys 0.000000 (best of 113)
1) wall 0.000038 comb 0.000000 user 0.000000 sys 0.000000 (best of 66567)
2) wall 0.000062 comb 0.000000 user 0.000000 sys 0.000000 (best of 43699)
(0: 428fa22fb2d1^, 1: 3.2-rc, 2: this patch)
The problem in commandserver was addressed by 766cfbe766dc, but it is tricky
to reuse ui.nontty option to disable echo back. Instead, this patch introduces
new option to enable echoing of prompt response.
Prompt echoing is changed to be off by default, which should avoid possible
breakage of output parsing in user scripts.
For now, this option is undocumented because it exists for internal use.
Hooks that run after the transaction need to be able to touch the
repository. So we need to run them after the lock release. This is
similar to what the "changegroup" hook is doing in the
`addchangegroup` function.
Hidden changesets are by far the most common error case and is the only one[1]
that can reach the user. We move to a friendlier message with a hint about how
to access the data anyway. We should probably point to a help topic instead but
we do not have such a topic yet.
Example of the new output
abort: hidden revision '4'!
(use --hidden to access hidden revisions)
[1] Actually, filtering from "served" can also reach the user during certain
exchange operations.
This will help user to debug. A more precise message will be issued
for the most common case ("visible" filter) in the next changesets.
example output:
- abort: filtered revision '4'!
+ abort: filtered revision '4' (not in 'visible' subset)!
The hack for using certificate store in addition to the provided CAs resides in
Apple's OpenSSL. Apple's own Pythons will use it, but other custom built
Pythons might use a custom built OpenSSL without that hack and will fail when
exposed to the dummy cacert introduced in ee8b7fe5e119.
There do not seem to be a simple way to check from Python if we are using a
patched OpenSSL or if it is an Apple OpenSSL.
Instead, check if the Python executable resides in /usr/bin/python* or in
/System/Library/Frameworks/Python.framework/ and assume that all Pythons found
there will be native Pythons using the patched OpenSSL.
Custom built Pythons will not get the benefit of using the CAs from the
certificate store.
Because unix-mode server forks child process per connection, client does not
know the pid of the server that will handle requests. The pid is necessary
to interrupt hung process:
1. client connects to socket server
2. server accepts the connection, forks, and tells pid
3. client requests "runcommand pull"
.. hung ..
4. client sends SIGINT to the (forked) server
5. server returns from I/O wait
Note that getsockopt(SO_PEERCRED) of Linux cannot be used because the server
fork()s after accept().
Consider a hypothetical bug in the release function that causes it to raise an
exception. Also consider the bisect command, which saves its state in a finally
clause. Saving the state requires acquiring the wlock.
If we don't unlink the lockfile when the exception is thrown, we'll try to
acquire the wlock again. We're going to try and acquire a lock again while our
old lockfile is on disk. The PID on disk is our own, and of course we're still
running, so we won't take over the lock. Hence we'll be stuck waiting for a
lock that we left behind ourselves.
To avoid this, always unlink the lockfile. This preserves the invariant that
self.held > 0 is equivalent to the lockfile existing on disk.
util.system() copies subprocess' output through pipe if output file is not
stdout. Because a file iterator has internal buffering, output won't be
flushed until enough data is available. Therefore, it could easily miss
important messages such as "waiting for lock".
Bundle2 opens doors to advanced features allowing to reduce load on
mercurial servers, and improve clone experience for users on unstable or
slow networks.
For instance, it could be possible to pre-generate a bundle of a
repository, and give a pointer to it to clients cloning the repository,
followed by another changegroup with the remainder. For significantly
big repositories, this could come as several base bundles with e.g. 10k
changesets, which, combined with checkpoints (not part of this change),
would prevent users with flaky networks from starting over any time
their connection fails.
While the server-side support for those features doesn't exist yet, it
is preferable to have client-side support for this early-on, allowing
experiments on servers only requiring a vanilla client with bundle2
enabled.
The phase retrieval is fast enough to not require caching the result of the
functions.
draft()
0) wall 0.017209 comb 0.020000 user 0.020000 sys 0.000000 (best of 149)
1) wall 0.011654 comb 0.010000 user 0.010000 sys 0.000000 (best of 186)
public()
0) wall 0.018687 comb 0.010000 user 0.010000 sys 0.000000 (best of 128)
1) wall 0.013290 comb 0.010000 user 0.010000 sys 0.000000 (best of 181)
secret()
0) wall 0.017464 comb 0.020000 user 0.020000 sys 0.000000 (best of 127)
1) wall 0.011499 comb 0.000000 user 0.000000 sys 0.000000 (best of 196)
draft() - ::bookmark()
0) wall 0.020099 comb 0.020000 user 0.020000 sys 0.000000 (best of 127)
1) wall 0.014399 comb 0.020000 user 0.020000 sys 0.000000 (best of 169)
We capture FilteredxxxError and issue a FilteredRepoLookupError instead with a
sightly different messsge. The message will likely get more improvement in the
future.
error: filtered revision '4'
We are going to introduce more precise exception classes for filtered nodes. So
we will have to upgrade them to the `RepoLookupError` level here. We wrap the
whole thing into a try/except to ease this future catching. Some of the current
exception catching will be moved in this one. But the current changeset focuses
on code movement only.
We are changing all integers that denote the size of a chunk to read to int32.
There are two main motivations for that.
First, we change everything to the same width (32 bits) to make it possible for
a reasonably agnostic actor to forward a bundle2 without any extra processing.
With this change, this could be achieved by just reading int32s and forwarding
chunks of the size read. A bit a smartness would be logic to detect the end of
stream but nothing too complicated.
Second, we need some capacity to transmit special information during the bundle
processing. For example we would like to be able to raise an exception while a
part is being read if this exception happend while this part was generated.
Having signed integer let us use negative numbers to trigger special events
during the parsing of the bundle.
The format is renamed for B2X to B2Y because this breaks binary
compatibility. The B2X format support is dropped. It was experimental to
allow this kind of things. All elements not directly related to the binary
format remain flagged "b2x" because they are still compatible.
This is code movement only. This will be useful to have it separated for reuse
purposes. We plan to introduce a new feature to the bundle format that allow
inserting a part in the middle of another part payload. This will be useful to
transmit a exception raised during a part generation.
Instead of checking all elements of the subset against a single rev, just check
if this rev is in the subset. The old way was inherited from when the subset was
a list.
Non surprise, this provide massive speedup.
id("b7dc31e4baa4")
before) wall 0.008205 comb 0.000000 user 0.000000 sys 0.000000 (best of 302)
after) wall 0.000069 comb 0.000000 user 0.000000 sys 0.000000 (best of 34518)
revset #1: public() and id("b7dc31e4baa4")
before) wall 0.019763 comb 0.020000 user 0.020000 sys 0.000000 (best of 124)
after) wall 0.000101 comb 0.000000 user 0.000000 sys 0.000000 (best of 20130)
The & version is more likely to be optimised.
only(.)
before) wall 0.003216 comb 0.000000 user 0.000000 sys 0.000000 (best of 768)
after) wall 0.001086 comb 0.000000 user 0.000000 sys 0.000000 (best of 2231)
only(default, stable)
before) wall 0.018469 comb 0.020000 user 0.020000 sys 0.000000 (best of 138)
after) wall 0.015888 comb 0.010000 user 0.010000 sys 0.000000 (best of 156)
For python struct module, "d" is double. But for python string
formating, "d" is integer. We want to preserve the floating point
nature of the data, so we store it in the metadata as floating
point. We use "%r" to make sure we get as many significant digitis as
necessary to restore the float to the exact same value on the other
side.
The fm1 is transmitting the information as float. The lack of this made
fm1-stored markers not survive a round-trip to fm0 leading to duplicated
markers (or two markers very alike).
We need a wider set of hooks to process all the changes that happened during the
pull transaction. We reuse the experimental `b2x-transactionclose` hook set
from server's unbundle for consistency. This hook is experimental and will not
remains as-is forever, but this will open the door for experimentation in 3.2.
Typical use case of 'unix' mode is a background hg daemon.
$ hg serve --cmdserver unix --cwd / -a /tmp/hg-`id -u`.sock
Unlike 'pipe' mode in which parent process keeps stdio channel, 'unix' server
can be detached. So clients can freely connect and disconnect from server,
saving Python start-up time.
It might be better to write "--cmdserver socket -a unix:/sockpath" instead
of "--cmdserver unix -a /sockpath" in case hgweb gets the ability to listen
on unix domain socket.
This is the stub for new mode that will listen for connections on unix domain
socket.
Though --daemon option is not banned in 'pipe' mode, it is useless because
the detached 'pipe' mode server exits immediately due to null stdin. Should
it abort if --daemon is specified with --cmdserver pipe or --stdio?
39fbe33f95fa brought "asciilower" and "import parsers" into
"encoding.py".
This works fine with "parsers" module in C implementation, but doesn't
with one in pure Python implementation, because the latter causes
cyclic dependency below and aborting execution:
util => i18n => encoding => parsers => util
This patch delays importing "parsers" module until it is really
needed, to avoid cyclic dependency around "parsers" in pure Python
build.
The source information can, should be applied once when opening the transaction
for the pull. This will lets element processed within a bundle2 be aware of them
and open the door to running a set of hooks when closing this pull transaction.
This is similar to what is done in server's unbundle call.
We store the source and url of the current data into `transaction.hookargs` this
let us inherit it from upper layers that may have created a much wider
transaction. We have to modify bundle2 at the same time to register the source
and url in the transaction. We have to do it in the same patch otherwise, the
`addchangegroup` call would fill these values and the hook calling will crash
because of the duplicated 'source' and 'url' arguments passed to the hook call.
We want to reused some possible information stored in the transaction
`hookargs` dict that may be stored by something handling the transaction at an
upper level (eg: bundle2) So we move the running of the hooks after transaction
creation. This has no visible effects (but an empty transaction roolback if the
hook fails) because nothing had happened in the transaction yet.
The transaction is now carrying hook-related informations. So we use it to
retrieve the `node` argument. This will also carry around all kinds of other useful
informations (like: "are we in a bundle2 processing")
From manifest.diff(), we return a dict from filename to pairs of pairs
of file nodeids and flags (values of the form ((n1,n2),(fl1,fl2))). To
create this dict, we currently generate one dict for files (with
(n1,n2) values) and one for flags (with (fl1,fl2) values) and then
join these dicts. Missing files are represented by None and missing
flags by '', but due to the dict joining, the inner pairs themselves
can also be None. The only caller, merge.manifestmerge(), then unpacks
these values while checking for None values.
By inlining the calls to dicthelpers and simplifying it to only
iterate over files (ignoring flags-only differences), we can simplify
life for our caller.
The manifestdict class already has a method for diff flags between two
manifests (presumably because there is no full access to the private
_flags field). The only caller is merge.manifestmerge(), which also
wants a diff of files between the same manifests. Let's combine the
code for diffing files and flags into a single method on
manifestdict. This puts all the manifest diffing in one place and will
allow for further simplification. It might also be useful for it to be
encapsulated in manifestdict if we later decide to to shard
manifests. The docstring is intentionally unclear about missing
entries for now.
A bundle2 may contain multiple parts adding changegroups, in which case there
are multiple operation records for changegroups, each with its own return
value. Those multiple return values are aggregated in a single cgresult value
for the whole operation.
As can be seen in the associated test case, the situation with hooks is not
really the best, but without deeper thoughts and changes, we can't do much
better. Hopefully, things will be improved before bundle2 is enabled by default.
In the meanwhile, multiple changegroups is not expected to be in widespread
use, and even less expected to be used for pushes. Also, not many clients
cloning or pulling bundle2 with multiple changesets are not expected to have
changegroup hooks anyways.
addchangegroup creates a runhook function that is used to invoke the
changegroup and incoming hooks, but at the time the function is called,
the contents of hookargs associated with the transaction may have been
modified externally. For instance, bundle2 code affects it with
obsolescence markers and bookmarks info.
It also creates problems when a single transaction is used with multiple
changegroups added (as per an upcoming change), whereby the contents
of hookargs are that of after adding a latter changegroup when invoking
the hook for the first changegroup.
This effectively backs out changeset 7582042d6cce.
The API change is done so that both util.sha1 and util.md5 can be called the
same way. The function is moved in order to use it for md5 checksumming for
an upcoming bundle2 feature.
The basic obsolete option is allowing the creation of obsolete markers. This
does not enable other features, such as allowing unstable commits or exchanging
obsolete markers.
Previously, obstore read the obsolete._enabled flag to determine whether to
allow writes to the obstore. Since obsolete._enabled will be moving into a repo
specific config, we can't read it globally, and therefore must pass the
information into the constructor.
Previously, obsolete used the module level _enabled flag to determine whether it
was on or off. We need a bit more granular control, so we'll be introducing
toggle options. The isenabled() function is how you check if a particular option
is enabled for the given repository.
Future patches will add options such as 'createmarkers', 'allowunstable', and
'exchange' to enable various features of obsolete markers.
The push process uses a `stepsdone` attribute instead of a `todosteps` one (with
the logic swapped). We unify the two process by picking the `stepsdone` version.
I feel like `stepsdone` better fits extensions that would want to extend the push
exchange process.
We apply the same approach as for push and make the discovery extensible. There
is only one user in core right now, but we already know we'll need something
smarter for obsmarkers. In fact the evolve extension could use this to cleanly
extend discovery.
The main motivation for this change is consistency between push and pull.
This change allows a revision log to not fail integrity checks when applying a
changegroup delta (eg from a bundle) results in a censored file tombstone. The
tombstone is inserted as-is, so future integrity verification will observe the
tombstone. Deltas based on the tombstone will also remain correct.
The new code path is encountered for *exactly* the cases where _addrevision is
importing a tombstone from a changegroup. When committing a file containing
the "magic" tombstone text, the "text" parameter will be non-empty and the
checkhash call is not executed (and when committing, the node will be computed
to match the "magic" tombstone text).
Two possible behaviors are defined for handling censored data: abort, and
ignore. When we ignore censored data we return an empty file to callers
requesting the file data.
We do modify the lists that make up the status in several places, so
it seems risky to use the same instance of a list for several
different status types. Use a separate empty list for each type
instead.
In commands.summary(), we currently zip a list of labels with a list
of statuses. This means the order of the status list has to match the
list of the labels, which in turn means the status elements have to be
inserted into specific places in the list. Let's instead group the
labels and status data we want to display in a single list of pairs.
The various status types are currently documented on the
dirstate.status() method. Now that we have a class for the status
types, it makese sense to document the status types there
instead. Only leave the bits related to lookup/unsure in the status()
method documentation.
Callers of various status() methods (on dirstate, context, repo) get a
tuple of 7 elements, where each element is a list of files. This
results in lots of uses of indexes where names would be much more
readable. For example, "status.ignored" seems clearer than "status[4]"
[1]. So, let's introduce a simple named tuple containing the 7 status
fields: modified, added, removed, deleted, unknown, ignored, clean.
This patch introduces the class and updates the status methods to
return instances of it. Later patches will update the callers.
[1] Did you even notice that it should have been "status[5]"?
(tweaked by mpm to introduce the class in scmutil and only change one user)
The status tuple returned from dirstate.status() has an additional
field compared to the other status tuples: lookup/unsure. This field
is just an optimization and not something most callers care about
(they want the resolved value of 'modified' or 'clean'). To prepare
for a single future status type, let's separate out the 'lookup' field
from the rest by having dirstate.status() return a pair: (lookup,
status).
Now that we have a separate variable for the original 'm1' manifest,
we can safely update the nodeid of the file in the new manifest in the
same place as we update the flags.
In localrepo.commitctx(), p1's manifest is copied and used as the
basis for the manifest that is about to be committed. The way the copy
is updated makes it safe to use it where the original p1's manifest is
wanted. For readability, though, a separate variable for each purpose
would be clearer. Make it so.
This moves the initial root phase assignment to it's own function. Future
patches which make phase calculations lazy will use this function to pre-fill
certain phases which can be deduced from the roots.
A future patch is going to make phase computation lazy, so the phase command can
no longer read and diff the entire phase list directly. This changes the phase
command to build it's own list for diff purposes.
