Before this change, when histedit exited to interactive session (during edit
command for example), user could introduce obsolescence markers that would not
be known to histedit. For example, user could've amended one of the commits.
The fact of this amendment would not be stored in histedit's state file
and later, when histedit would try to process all the replacements,
one of the final successors (in histedit's opinion) would turn out to be hidden.
This behavior is described in issue4800. This commit fixes it.
When using treemanifests, an on-disk manifest entry with the 't' flag
set means that that entry is a directory and not a file. When read
into memory, these become instances of the treemanifest class. The 't'
flag should therefore never be visible to outside of manifest.py, so
setflag() checks that it is not called with the 't' flag. However, it
turns out that it will be useful for the narrowhg extension to expose
the 't' flag to the user (see below), so let's drop the assertion.
The narrowhg extension allows cloning only a given set of files and
directories. Filelogs and dirlogs that don't match that set will not
be included in the clone. The extension currently doesn't work with
treemanifests. I plan on changing it so directories outside the narrow
clone appear in the manifest. For example, if a directory 'outside/'
is not part of the narrow clone, it will look like a file 'outside'
with the 't' flag set. That will make e.g. manifestmerge() just work
in most cases (and make it well prepared to handle the other
cases).
This is necessary to preserve filename encoding over JSON. Instead, this
patch inserts "|utf8" where non-ascii local-encoding texts can be passed
to "|json".
See also the commit that introduced "utf8" filter.
Because future patches will change "|json" filter to handle input bytes
transparently, i.e. use UTF-8b encoding, "{jsdata}" must keep data in UTF-8
bytes, whereas "{nodes}" are text.
This patch inserts encodestr() where localstr is likely to survive.
This will be applied prior to "|json" filter. This sounds like odd, but it
is necessary to handle local-encoding text as well as raw filename bytes.
Because filenames are bytes in Mercurial and Unix world, {filename|json} should
preserve the original byte sequence, which implies
{x|json} -> '"' toutf8b(x) '"'
On the other hand, most template strings are in local encoding. Because
"|json" filter have to be byte-transparent to filenames, we need something to
annotate an input as a local string, that's what "|utf8" will do.
{x|utf8|json} -> '"' toutf8b(fromlocal(x)) '"'
"|utf8" is an explicit call, so aborts if input bytes can't be converted to
UTF-8.
It's been unused, undocumented and flawed in that it expects a unicode input,
never works correctly if an input has non-ascii character. We should use "json"
filter instead.
In repos with treemanifests, the non-root-directory dirlogs often have
many more total revisions than the root manifest log has. This change
adds progress out to that part of 'hg verify'. Since the verification
is recursive along the directory tree, we don't know how many total
revisions there are at the beginning of the command, so instead we
report progress in units of directories, much like we report progress
for verification of files today.
I'm not very happy with passing both 'storefiles' and 'progress' into
the recursive calls. I tried passing in just a 'visitdir(dir)'
callback, but the results did not seem better overall. I'm happy to
update if anyone has better ideas.
We already report orphaned filelogs, i.e. revlogs for files that are
not mentioned in any manifest. This change adds checking for orphaned
dirlogs, i.e. revlogs that are not mentioned in any parent-directory
dirlog.
Note that, for fncachestore, only files mentioned in the fncache are
considered, there's not check for files in .hg/store/meta that are not
mentioned in the fncache. This is no different from the current
situation for filelogs.
In repos with treemanifests, there is no specific verification of
directory manifest revlogs. It simply collects all file nodes by
reading each manifest delta. With treemanifests, that's means calling
the manifest._slowreaddelta(). If there are missing revlog entries in
a subdirectory revlog, 'hg verify' will simply report the exception
that occurred while trying to read the root manifest:
manifest@0: reading delta 1700e2e92882: meta/b/00manifest.i@67688a370455: no node
This patch changes the verify code to load only the root manifest at
first and verify all revisions of it, then verify all revisions of
each direct subdirectory, and so on, recursively. The above message
becomes
b/@0: parent-directory manifest refers to unknown revision 67688a370455
Since the new algorithm reads a single revlog at a time and in order,
'hg verify' on a treemanifest version of the hg core repo goes from
~50s to ~14s. As expected, there is no significant difference on a
repo with flat manifests.
When pooled storage is enabled, `hg clone` will initialize a repo
from a local repo using the store sharing mechanism then pull from
the originally requested repo.
Before this patch, the working directory update occurred between
these steps. This meant that we would only update to revisions that
were already present in the local pooled storage.
This patch moves the update to after we pull from the originally
requested repository so we may check out a revision that didn't yet
exist locally. In other words, it makes the behavior like normal
`hg clone`.
As 70dbbec70c4c demonstrated with stream clones, closing files on
background threads on Windows can yield a significant speedup
because closing files that have been created/appended to is slow
on Windows/NTFS.
Working directory updates can write thousands of files. Therefore it
is susceptible to excessive slowness on Windows due to slow file
closes.
This patch enables background file closing when performing working
directory file writes. The impact when performing an `hg up tip` on
mozilla-central (136,357 files) from an empty working directory is
significant:
Before: 535s (8:55)
After: 133s (2:13)
Delta: -402s (6:42)
That's a 4x speedup!
By comparison, that same machine can perform the same operation
in ~15s on Linux. So Windows went from ~35x to ~9x slower. Not bad
but there's still work to do.
As a reminder, background file closing is only activated on Windows
because it is only beneficial on that platform. So this patch
shouldn't change non-Windows behavior at all.
It's worth noting that non-Windows systems perform working directory
updates with multiple processes. Unfortunately, worker.py doesn't
yet support Windows. So, there is still plenty of room for making
working directory updates faster on Windows. Even if multiple
processes are used on Windows, I believe background file closing
will still provide a benefit, as individual processes will still
be slowed down by the file close bottleneck (assuming the I/O system
isn't saturated).
This is a part of the one server per config series. In multiple-server setup,
multiple clients may try to start different servers (on demand) at the same
time. The old lock will not guarantee a client to connect to the server it
just started, and is not crash friendly.
This patch addressed above issues by using flock and does not release the lock
until the client actually connects to the server or times out.
This patch changes the format of value of --daemon-postexec. Now it can be
either to unlink a file, or a no-op. The following patch will make chg to
use the no-op option.
Initially we use --daemon-pipefds to pass file descriptors for synchronization.
Later, in order to support Windows, --daemon-pipefds is changed to accept a
file path to unlink instead. The name is outdated since then.
chg client is designed to use flock, which will be held before starting a
server and until the client actually connects to the server it started. The
unlink synchronization approach is not so helpful in this case.
To address the issues, this patch renames pipefds to postexec and the following
patch will allow the value of --daemon-postexec to be things like
'unlink:/path/to/file' or 'none'.
We want to explicitly test the next behavior. We add this test in its own
changeset to make the test change from the behavior change as clean as possible.
This changeset finally make 'hg rebase' choose its default destination using the
same logic as 'hg merge'. The previous default was "tipmost changeset on the
current branch", the new default is "the other head if there is only one". This
change has multiple consequences:
- Multiple tests which were not rebasing anything (rebasing from tipmost head)
are now rebasing on the other "lower" branch. This is the expected new
behavior.
- A test is now explicitly aborting when there is too many heads on the branch.
This is the expected behavior.
- We gained a better detection of the "nothing to rebase" case while performing
'hg pull --rebase' so the message have been updated. Making clearer than an
update was performed and why. This is beneficial side-effect.
- Rebasing from an active bookmark will behave the same as 'hg merge' from a
bookmark.
I've updated the script to reflect changes in Mercurial and to include a much
more through installation guide with configuration examples and details on how
to configure IIS. I've used the script to set up a working server from scratch.
The bundlerepository have to do some special magic to handle linkrev of the
bundlerepo filerev. That logic was done from a repoview and obsolescence marker
affecting bundled changeset could lead to a crash. We now ensure we operate on
unfiltered repository.
Before this patch collapse message wasn't stored so when
you ran into the merge conflict while rebasing, running
rebase --continue didn't remember the message and
always opened editor to fill commit message.
This patch adds saving collapse message in
.hg/last-message.txt and restoring it later
when needed.
The motivation for the change from 100 to 95 is included in a comment.
* Updated the tests to include a change to a moved file that still should be
caught as a move.
* Use ui.configint() to non-integer configuration entries more gracefully. Also
complain if a similarity outside of the acceptable range is set.
The only consumer of the worker pool code today is `hg update`.
Previously, the algorithm to partition work to each worker process
preserved input list ordering. We'd take the first N elements, then
the next N elements, etc. Measurements on mozilla-central demonstrate
this isn't an optimal partitioning strategy.
I added debug code to print when workers were exiting. When performing
a working copy update on a previously empty working copy of
mozilla-central, I noticed that process lifetimes were all over the
map. One worker would complete after 7s. Many would complete after
12s. And another worker would often take >16s. This behavior occurred
for many worker process counts and was more pronounced on some than
others.
What I suspect is happening is some workers end up with lots of
small files and others with large files. This is because the update
code passes in actions according to sorted filenames. And, directories
under tend to accumulate similar files. For example, test directories
often consist of many small test files and media directories contain
binary (often larger) media files.
This patch changes the partitioning algorithm to select every Nth
element from the input list. Each worker thus has a similar composition
of files to operate on.
The result of this change is that worker processes now all tend to exit
around the same time. The possibility of a long pole due to being
unlucky and receiving all the large files has been mitigated. Overall
execution time seems to drop, but not by a statistically significant
amount on mozilla-central. However, repositories with directories
containing many large files will likely show a drop.
There shouldn't be any regressions due to partial manifest decoding
because the update code already iterates the manifest to determine
what files to operate on, so the manifest should already be decoded.
When reloading tests, run-tests.py was assuming that it could look
up the test by the basename, which only works if you are running
tests which are in the current directory.
This patch changes that lookup to use the full path. This is all
that was needed, and does not appear to cause any problems for
any of the existing testing work flows based on running the
suggested commands at the top of run-tests.py.
Motivation: In order to test Mercurial with Hypothesis (according
to https://www.mercurial-scm.org/wiki/HypothesisPlan) it is
useful to be able to generate temporary test files and execute
them. Generating temporary files in the tests/ directory leads to
a lot of suboptimal clutter.
Because templater.revset() returns a list of strings, repo["-1"] was mapped to
the tipmost revision. Ideally, we should make revset() return a list of integer
revisions, but it turned out not simple. If revset() is a list of integers,
"{ifcontains(rev, revset(), ...)}" would fail because "ifcontains" casts "rev"
to a string.
So this patch just converts a string back to an integer revision.
Importing sqlalchemy.events cannot be delayed as it registers classes to
their event mechanism. It worked fine before faecf59a4184, since they use
new-style imports. But now we have to blacklist it because our demandimport
can handle new-style imports.
This patch series isn't intended for stable as we don't guarantee API
compatibility with 3rd-party extensions. They can temporarily disable the
demand importer to work around the issue.
If a module is loaded as "from . import x" form, there has been no way to
disable demand loading for that module because name is ''. This patch makes
it possible to prevent demand loading by '<package-name>.x'.
We don't use _hgextimport(_origimport) here since attr is known to be a
sub-module name. Adding hgext_ to attr wouldn't make sense.
Without this, the paths were in the form "C:\\Users\\Matt\\AppData...". The
test runner wouldn't collapse it down to $TESTTMP, which uses single
backslashes.
This patch changes "progress.shouldprint()" so a feature name is provided to
"ui.plain()" to determine if there is an exception specificed in HGPLAINEXCEPT
for the progress extension.
This will allow user-facing scripts to provide progress output while HGPLAIN
is enabled.
The only consumer was test-treemanifest.t, which has been fixed.
In general, you should be able to use killdaemons.py to recycle
ports instead of going over 3 ports (HGPORT, HGPORT1, HGPORT2).
In the future, if you want to add a port, be sure to change
portneeded in _getport.
Adding a port reservation was too hard and someone did it wrong.
By refactoring, such reservations can be managed more safely.
This also adds documentation so that the next person who tries
is more likely to update all the places correctly.
Note that in this commit the reservation and consumers do not
match, that will be fixed in the next commit.
test-treemanifest.t had introduced HGPORT3 and HGPORT4,
which were improperly added to run-tests.py.
It also was not using HGPORT1.
This recycles HGPORT, and shifts everything into HGPORT1 + HGPORT2.
We are going to make chgserver load repo config, remember what it is, and
load repo config again to detect config change. This is the first step that
passes config, repo, cwd options to server. Traceback is passed as well to
cover errors before hitting chgserver.runcommand.
They are like malloc and realloc but will abort the program on error.
A lot of places use {m,re}alloc and check their results. This patch
can simplify them.
