Before this patch, it was possible for run-tests to crash on a race condition.
The race condition happens in the following case:
- the last test finishes and calls: done.put(None)
- the context switches to the main thread that clears the channels list
- the context switches to the last test mentioned above, it tries to access
channels[channel] and crashes
This happened to me while running run-tests.
This patch fixes the issue by clearing the channel before considering that the
test is done.
The new transaction context did not handle the case where an exception during
close should still call release. This cause pretxnclose hooks that failed to
cause the transaction to fail without aborting, thus requiring a hg recover.
I've added a test.
changegroup.getchunks() determines the end of the stream by looking
for an empty chunk group (two consecutive empty chunks). It ignores
empty groups in the first two groups. Changegroup 3 introduced an
empty chunk between the manifests and the files, which confuses
getchunks(). Since it comes after the first two, getchunks() will stop
there.
Fix by rewriting getchunks so it first counts two groups (empty or
not) and then keeps antostarts counting empty groups. With this counting,
changegroup 1 and 2 have exactly one empty group after the first two
groups, while changegroup 3 has two (one for directories and one for
files).
It's a little hard to test this at this point, but I have verified
that this patch fixes narrowhg (which was broken before this
patch). Also, future patches will fix "hg strip" with treemanifests,
and once that's done, getchunks() will be tested through tests of "hg
strip".
Before this patch if the hiddencache existed but was empty, it would crash
mercurial. This patch adds exception handling when reading the hiddencache to
avoid the issue.
When encountering a corrupted cache file we print a devel warning. There would
be no point in issuing a normal warning as the user wouldn't be able to do
anything about the situation.
The warning looks like:
devel-warn: corrupted hidden cache, removing it at: /path/to/repoview.py
These operations are obviously invalid for file-like channels because they
will read or write protocol headers.
This patch works around the issue that "hg archive" generates a corrupted
zip file on Windows commandserver because of unusable tell() implementation.
But the problem still occurs without using a commandserver.
$ hg archive -R not-small-repo -t zip - | cat > invalid.zip
So, this patch cannot fix the issue5049 completely.
This patch adds a variable to keep track of what hunk was selected
before the edit. We use that variable to select the hunk or its
replacement after the edit.
Before this patch, template files for "graph" web page use fixed width
size "480" for canvas element.
This causes pruned lanes and invisible vertexes, if there are 16 or
more vertical lanes at once. In such case, part of graph in right side
area over 480 is invisible, even though corresponded summary text
blocks are visible correctly.
This limitation isn't reasonable for workflow using many branches at
once (e.g. "one branch per issue" workflow).
There were changes below related to width of canvas:
- 6c855f5350cd (templates: widen the graph canvas (issue2683)),
released as a part of Mercurial 1.8.2
According to the description, this assumed that 15 parallel
branches was enough for ordinary workflow, and bumped width of
canvas up from 224 to 480.
- f5506d2a674c (hgweb: make graph data suitable for template usage),
released as a part of Mercurial 2.3
This introduced "canvaswidth" template keyword as a part of
refactoring around graph rendering.
But 'width="480"' of canvas element in template files wasn't
replaced by 'width="{canvaswidth}"' in it (or subsequent one).
This patch uses dynamic value "{canvaswidth}" instead of fixed width
size "480" for canvas element.
This is posted for "stable", because:
- this is re-fixing issue2683
- this is simple enough for stable
- using "{canvaswidth}" doesn't require any additional cost
Calculation of canvaswidth is already implied as a part of "graph"
web command.
Previously, if you passed a revset that resolved to no nodes, it would get
interpreted by the changegroup discovery logic as 'bundle all my heads', which
is not what the user asked.
Let's exit early when we notice this case.
It could be argued that the changeset discovery logic should be smarter and only
assume 'all heads' if the incoming heads parameter is None, but that's a much
riskier change.
hgweb_mod.hgweb.repo disappeared in 4d6522c4ac11 (hg: establish
a cache for localrepository instances) and the code for accessing repo
instances from hgweb was later refactored to go through a cache-aware
context manager.
Adapt zeroconf to access the repo instance via the new API.
If an emergency comes in while you're in the middle of an experimental
change, it can be useful to shelve not just files hg already tracks but
also your unknown files while you handle the emergency. This is
especially true if you have hooks intended to prevent you from
forgetting to add new code before you push.
Teach "hg shelve" to optionally shelve unknown files, not just tracked
files. This is functionally similar to --addremove, but with two
differences:
1) Deleted files are not removed.
2) Files added during shelve creation are tracked in extra so that they
can be forgotten by "hg unshelve".
When unshelving, we take care to only forget files if they've been
created during the unshelve operation; if you add a file that's being
tracked in a shelve as an unknown file, it should not become unknown
again when the shelve is unshelved.
Before, the hook() closure (which is called as part of locking hooks)
would maintain a reference to a transaction instance (which should be
finalized by the time lock hooks are called). Because we accumulate
hook() instances when there are multiple transactions per lock, this
would result in holding references to the transaction instances which
would lead to higher memory utilization.
Creating a reference to the hook arguments dict minimizes the number
of objects that are kept alive until the lock release hook runs,
minimizing memory "leaks."
util.cachefunc stores all arguments as the cache key. For filectxfn
functions, the arguments include the memctx instance. This creates a
cycle where memctx._filectxfn references self. This causes a memory
leak.
We break the cycle by implementing our own memoizing function that
only uses the path as the cache key. Since each memctx has its own
cache instance, there is no concern about invalid cache hits.
92117e4f6f8d improved merging of standin files referencing missing largefiles.
It did however not test or fix commits of such merges; it would abort.
To fix that, change copytostore to skip and warn about missing largefiles
with a message similar the one for failing get from remote filestores. (It
would perhaps in both cases be better to emit a more helpful warning like
"warning: standin file for large1 references 58e24f733a which can't be found in
the local store".)
To test this, make sure commit doesn't find the "missing" largefile in the global
usercache. For further testing, verify that update and status works as expected
after this.
This will also effectively backout 159c82dd6523.
Similar to what was explained in the previous commit, the diff code
expected copy source to be in "ctx1", which is not always the case
during a merge. This has been broken since before hg 2.0.
Also similar to the previous commit, we fix the problem by fixing up
the copy dict.
During a merge, if the user removes a file that came from parent 2 and
did not exist in parent 1, the file's status will be "removed". This
surprises the diff code, which crashes because it expects removed
files exist in parent 1. This has been broken since ff976121fb34
(trydiff: use 'not in addedset' for symmetry with 'not in removedset',
2014-12-23).
Fix by fixing up the list of removed file, similar to how we currently
fix up the list of modified and added files during a merge.
This prepares for future patches, and it also lets us remove the ugly
"ctx1" argument to _filepairs() (ugly because of its assymmetry --
there's no "ctx2" argument).
The behaviour in this case is undefined. Instead of silently doing something
"random" and surprising, at least issue a warning.
(This should perhaps be considered a "deprecation" and turned into an error in
a future release.)
Positional parameters are also treated as revisions, but the order of revisions
matters and it will often be wrong if the user understands it as `-r` taking
multiple revisions as `-r REV1 REV2`.
(Alternatively, `-r` could be turned into a no-op flag as the documentation
suggests. That would however be less "semantic markup" and I agree with the
implementation in 40cbb25097c8 but not the documentation.)
Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.
This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!
The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).
Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:
Linux: ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)
Windows is ~27.5x slower. Yikes!
After this patch:
Linux: ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)
Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.
This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.
Closing files that have been appended to is relatively slow on
Windows/NTFS. This makes several Mercurial operations slower on
Windows.
The workaround to this issue is conceptually simple: use multiple
threads for I/O. Unfortunately, Python doesn't scale well to multiple
threads because of the GIL. And, refactoring our code to use threads
everywhere would be a huge undertaking. So, we decide to tackle this
problem by starting small: establishing a thread pool for closing
files.
This patch establishes a mechanism for closing file handles on separate
threads. The coordinator object is basically a queue of file handles to
operate on and a thread pool consuming from the queue.
When files are opened through the VFS layer, the caller can specify
that delay closing is allowed.
A proxy class for file handles has been added. We must use a proxy
because it isn't possible to modify __class__ on built-in types. This
adds some overhead. But as future patches will show, this overhead
is cancelled out by the benefit of closing file handles on background
threads.
This provides a general-purpose interface to all custom namespaces.
The {namespaces} keyword honors the definition order of namespaces as they
are kept by sortdict.
This is necessary to obtain a _hybrid object from a dict. If get() yields
a value, it would be stringified.
I see no benefit to make get() lazy, so this patch just changes "yield" to
"return".
In templater, a callable symbol exists for lazy evaluation, which should have
f(**mapping) signature. On the other hand, _hybrid.__call__(), which was
introduced by 4e182fb53989, generates mapping for each element.
This patch renames _hybrid.__call__() to _hybrid.itermaps() so that a _hybrid
object can be a value of a mapping dict.
{namespaces % "{namespace}: {names % "{name }"}\n"}
~~~~~
a _hybrid object
The added content is inside a verbose container.
I figure it makes sense to explicitly document behavior, including
with the caveat it may change later. People can't say they weren't
warned!
Before the existence of `hg debugbundle --spec`, the process for
defining the BUNDLESPEC value in manifests was not very clear and not
trivial to automate, especially in the case of stream clone bundles.
This patch adds documentation to note the existence of
`hg debugbundle --spec`. We drop the reference to stream clone
requirements handling because it is now redundant with
`hg debugbundle --spec`. While we are here, we further reinforce the
importance of defining BUNDLESPEC.
We don't currently have a mechanism for inferring bundle spec strings
from bundle files. This patch adds one.
This will eventually be used to make the producing of clone bundles
manifests easier.
RFC 7159 does not state that U+007F must be escaped, but it is widely
considered a control character. As '\x7f' is invisible on a terminal, and
Python's json.dumps() escapes '\x7f', let's do the same.
Some evil evil awful tool adds resource forks to files it's comparing.
Our Mac-specific code to do bulk stats was accidentally using "total
size" which includes those forks in the file size, causing them to be
reported as modified. This changes it to only care about the normal
data size and thus agree with what Mercurial's expecting.
If we move all the files out of one directory, but into two different
directories, we should not consider it a directory rename. The
detection of this case was broken.