Summary:
The inline revlog format merges `.i` and `.d` into one `.i` file. It was intended to reduce the
number of files for filelogs. For the changelog one extra file does not hurt.
This makes it easier to write native code parsing the changelog revlog index.
Reviewed By: xavierd
Differential Revision: D17125922
fbshipit-source-id: f48ffe0d2df71abec007a80e05b684dcbac71883
Summary:
Now that all our repos are treemanifest, let's enable the extension by
default in tests. Once we're certain no one needs it in production we'll also
make it the default in core Mercurial.
This diff includes a minor fix in treemanifest to be aware of always-enabled
extensions. It won't matter until we actually add treemanifest to the list of
default enabled extensions, but I caught this while testing things.
Reviewed By: ikostia
Differential Revision: D15030253
fbshipit-source-id: d8361f915928b6ad90665e6ed330c1df5c8d8d86
Summary:
The postincoming checks prints out advice of the following forms:
* `(run 'hg heads' to see heads)`
* `(run 'hg heads' to see heads, 'hg merge' to merge)`
* `(run 'hg heads .' to see heads, 'hg merge' to merge)`
* `(run 'hg update' to get a working copy)`
This advice is no longer useful, so remove it.
Reviewed By: DurhamG, farnz
Differential Revision: D15317185
fbshipit-source-id: 50ba576406c96715fa058399da53462be9b7a3bf
Summary:
Move the strip extension to core. Rename the command to `hg debugstrip` as it
is not intended for use by users. Users should use `hg hide` instead.
Reviewed By: quark-zju
Differential Revision: D14185822
fbshipit-source-id: ef096488cb94b72a7bb79f5bf153c064e0555b34
Summary: Traditional shelve did not require `obsstore` to be enabled, which means that `saved bundle` messages were printed a lot. This change fixes all tests that expect these messages.
Reviewed By: singhsrb
Differential Revision: D10261258
fbshipit-source-id: 2b4b10ee94ae9b5723cb69fc8f7c29bc65613774
Summary:
Before this commit, `treemanifest` registered a post-transaction hook to
prefetch data for the parents of new commits. Shelve produces a hidden
commit, so this logic failed, since it used a filtered repo instance.
Let change it to use unfiltered repo and remove all of the hidden commits
from the revset to avoid surprises.
Reviewed By: markbt
Differential Revision: D9613454
fbshipit-source-id: 5caddd2948e5d2e982462bebdb950b359995baa1
Summary:
When running with a Python runtime with a slightly different zlib module,
some `zlib.compress` outputs are different. Some tests are testing the
length, or the content of `zlib.compress` output, directly or indirectly.
That's causing issues.
This patch adds a `common-zlib` hghave test so it can be used to gate tests
checking zlib output. Some lengths are also changed to glob patterns to be
compatible.
Reviewed By: ryanmce
Differential Revision: D6937735
fbshipit-source-id: 2328a39d7f2022f16d51f61b6178568b26dfe2fb
Summary:
Previously `hg server` uses `HGPORT` that might be in use. This patch uses
`-p 0 --port-file ...` so `hg server` always gets assigned a free port.
The change was first made by the following Ruby script:
```
re = /^ \$ hg serve(.*) -p \$(HGPORT[12]?) (.*[^\\])$\n \$/
Dir['*.t'].each do |path|
old = File.read(path)
new = old.lines.map do |l|
next l if l[/\(glob\)/] or not l['$HGPORT'] or l[/^ [$>]/]
"#{l.chomp} (glob)\n"
end.join.gsub re, <<-'EOS'.chomp
$ hg serve\1 -p 0 --port-file $TESTTMP/.port \3
$ \2=`cat $TESTTMP/.port`
$
EOS
File.write(path, new) if old != new
end
```
Then there are some manual changes:
run-tests.py: It now treats `$HGPORT` in output as glob pattern `*`, since
it does not know the assigned value in tests.
test-bookmarks-pushpull.t, test-https.t: Some `hg pull`s were changed to use
explicit paths instead of relying on `.hgrc` since the test restarts the
server and `.hg/hgrc` having an outdated URL.
test-schemes.t: The test writes `$HGPORT` to `.hgrc` before assigning it.
Changed the order so the correct `$HGPORT` is written.
test-patchbomb-tls.t: Changed `(?) (glob)` to `(glob) (?)`.
Reviewed By: DurhamG
Differential Revision: D6925398
fbshipit-source-id: d5c10476f43ce23f9e99618807580cf8ba92595c
Summary: Also change the internal API so it no longer accepts the "heads" argument.
Reviewed By: ryanmce
Differential Revision: D6745865
fbshipit-source-id: 368742be49b192f7630421003552d0a10eb0b76d
# skip-blame because this was mechanically rewritten the following script. I
ran it on both *.t and *.py, but none of the *.py changes were proper. All *.t
ones appear to be, and they run without addition failures on both Windows and
Linux.
import argparse
import os
import re
ap = argparse.ArgumentParser()
ap.add_argument('path', nargs='+')
opts = ap.parse_args()
globre = re.compile(r'^(.*) \(glob\)(.*)$')
for p in opts.path:
tmp = p + '.tmp'
with open(p, 'rb') as src, open(tmp, 'wb') as dst:
for line in src:
m = globre.match(line)
if not m or '$LOCALIP' in line or '*' in line:
dst.write(line)
continue
if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
dst.write(line)
continue
dst.write(m.group(1) + m.group(2) + '\n')
os.unlink(p)
os.rename(tmp, p)
Upon pull or unbundle, we display a message with the range of new revisions
fetched. This revision range could readily be used after a pull to look out
what's new with 'hg log'. The algorithm takes care of filtering "obsolete"
revisions that might be present in transaction's "changes" but should not be
displayed to the end user.
--uncompressed isn't a very good name and its description in the
help documentation isn't very useful. We refer to this concept as
"stream clones" in a number of places. I think it makes sense to
change the user-facing argument to use the mode --stream. So this
commit does that.
We keep --uncompressed around for backwards compatibility.
While we're here, we overhaul the help docs for streaming clones
to be somewhat useful.
All tests have been updated to reflect the new preferred --stream
argument. A test for backwards compatibility of --uncompressed has
been added.
.. bc::
`hg clone --stream` should now be used instead of --uncompressed.
--uncompressed is marked as deprecated and is an alias for --stream.
There is no schedule for elimination of --uncompressed.
Differential Revision: https://phab.mercurial-scm.org/D864
There was an extra commit made during the amend operation to track the
changes to the working copy. However, this logic was written a long time back
and newer API's make this extra commit redundant. Therefore, I am removing the
extra commit. After this change, I noticed that
- Execution time of the cmdutil.amend improved by over 40%.
- Execution time of "hg commit --amend" improved by over 20%.
Test Plan:
I ensured that the all the hg tests passed after the change. I had
to fix a few tests which were aware of the extra commit made during the amend.
Differential Revision: https://phab.mercurial-scm.org/D636
In a flat manifest, a node with the same content but different parents is still
considered a new node. In the current tree manifests however, if the content is
the same, we ignore the parents entirely and just reuse the existing node.
In our external treemanifest extension, we want to allow having one treemanifest
for every flat manifests, as a way of easeing the migration to treemanifests. To
make this possible, let's change the root node treemanifest behavior to match
the behavior for flat manifests, so we can have a 1:1 relationship.
While this sounds like a BC breakage, it's not actually a state users can
normally get in because: A) you can't make empty commits, and B) even if you try
to make an empty commit (by making a commit then amending it's changes away),
the higher level commit logic in localrepo.commitctx() forces the commit to use
the original p1 manifest node if no files were changed. So this would only
affect extensions and automation that reached passed the normal
localrepo.commit() logic straight into the manifest logic.
Previously the hg files tests also covered the logic (i.e.
treemanifest.matches) that governed how hg diff limited its diff. In a future
patch we will be switching treemanifest.diff() to have a custom implementation,
so let's go ahead and add equivalent test coverage for hg diff.
The POSIX documentation about "cp" [1] says:
....
RATIONALE
....
Earlier versions of this standard included support for the -r option to
copy file hierarchies. The -r option is historical practice on BSD and
BSD-derived systems. This option is no longer specified by POSIX.1-2008
but may be present in some implementations. The -R option was added as a
close synonym to the -r option, selected for consistency with all other
options in this volume of POSIX.1-2008 that do recursive directory
descent.
The difference between -R and the removed -r option is in the treatment
by cp of file types other than regular and directory. It was
implementation-defined how the - option treated special files to allow
both historical implementations and those that chose to support -r with
the same abilities as -R defined by this volume of POSIX.1-2008. The
original -r flag, for historic reasons, did not handle special files any
differently from regular files, but always read the file and copied its
contents. This had obvious problems in the presence of special file
types; for example, character devices, FIFOs, and sockets.
....
....
Issue 6
The -r option is marked obsolescent.
....
Issue 7
....
The obsolescent -r option is removed.
....
(No "Issue 8" yet)
Therefore it's clear that "cp -R" is strictly better than "cp -r".
The issue was discovered when running tests on OS X after 2e4d149e62aa.
[1]: pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html
This is a little messier than I'd like, and I'll probably come back
and do some more refactoring later, but as it is this unblocks
narrowhg. An alternative approach (which I may do as part of the
mentioned refactoring) would be to construct *all* dirlog instances up
front, so that we don't have to keep track of the linkmapper
method. This would avoid a reference cycle between the bundlemanifest
and the bundlerepository, but I was hesitant to do all the work up
front like that.
With this change, it's possible to do 'hg incoming' and 'hg pull' from
bundles in .hg/strip-backup in a treemanifest repository. Sadly, this
doesn't make it possible to 'hg clone' one of those (if you do 'hg
strip 0'), because the cg3 in the bundle gets written without a
treemanifest flag. Since that's going to be an involved refactor in a
different part of the code (which I *suspect* won't touch any of the
code I've just written here), let's leave it as an idea for Later.
Stripping has only partly worked since f41815302d49 (repair: use cg3
for treemanifests, 2016-01-19): the bundle seems to have been created
correctly, but revlog entries in subdirectory revlogs were not
stripped. This meant that e.g. "hg verify" would fail after stripping
in a tree manifest repo.
To find the revisions to strip, we simply iterate over all directories
in the repo (included in store.datafiles()). This is inefficient for
stripping few commits, but efficient for stripping many commits. To
optimize for stripping few commits, we could instead walk the tree
from the root and find modified subdirectories, just like we do in the
changegroup code. I'm leaving that for another day.
When grafting/rebasing, it is common for multiple changesets to make
the same change to a subdirectory. When writing the revlog for the
directory, the revlog code already takes care of not writing the entry
again. In 3eb9fa4180d3 (changegroup: prune subdirectory dirlogs too,
2016-02-12), I added the corresponding code in changegroup (not
sending entries the client already has), but I forgot to avoid sending
the entire changegroup if no nodes remained in the pruned
set. Although that's harmless besides the wasted network traffic, the
receiving side was checking for it (copied from the changegroup code
for handling files). This resulted in the client crashing with:
abort: received dir revlog group is empty
Fix by simply not emitting a changegroup for the directory if there
were no changes is it. This matches how files are handled.
When using treemanifests, only changegroup3 bundles can be
created. However, there is currently no way of requesting a
changegroup3 bundle, so we run into an assertion in
changegroup.getbundler() when trying to get a changroup2
bundler. Let's avoid the traceback and print a short error message
instead.
The treemanifest tests use the -T option to cp in order to ensure that the
two directories named on the commandline are treated as peers, rather than
the usual behavior when the final argument is a directory. GNU cp has this
option, but other implementations may not. Thankfully, there's no pressing
reason to use it. We can simply copy the contents of the first directory
into the target directory, since we know that the target directory already
exists.
The current code for generating treemanifest revisions takes the list
of files in the changeset and finds the directories from them. This
does not work for merges, since a merge may pick file A from one side
and file B from another and neither of them would appear in the
changeset's "files" list, but the manifest would still change.
Fix this by instead walking the root manifest log for all needed
revisions, storing all needed file and subdirectory revisions, then
recursively visiting the subdirectories. This also turns out to be
faster: cloning a version of hg core converted to treemanifests went
from ~28s to ~19s (timing somewhat unfair: before this patch, timed
until crash; after this patch, timed until manifests complete).
The new algorithm is used only on treemanifest repos. Although it
works equally well on flat manifests, we leave the iteration over
files in the changeset for flat manifests for now.
We already report orphaned filelogs, i.e. revlogs for files that are
not mentioned in any manifest. This change adds checking for orphaned
dirlogs, i.e. revlogs that are not mentioned in any parent-directory
dirlog.
Note that, for fncachestore, only files mentioned in the fncache are
considered, there's not check for files in .hg/store/meta that are not
mentioned in the fncache. This is no different from the current
situation for filelogs.
In repos with treemanifests, there is no specific verification of
directory manifest revlogs. It simply collects all file nodes by
reading each manifest delta. With treemanifests, that's means calling
the manifest._slowreaddelta(). If there are missing revlog entries in
a subdirectory revlog, 'hg verify' will simply report the exception
that occurred while trying to read the root manifest:
manifest@0: reading delta 1700e2e92882: meta/b/00manifest.i@67688a370455: no node
This patch changes the verify code to load only the root manifest at
first and verify all revisions of it, then verify all revisions of
each direct subdirectory, and so on, recursively. The above message
becomes
b/@0: parent-directory manifest refers to unknown revision 67688a370455
Since the new algorithm reads a single revlog at a time and in order,
'hg verify' on a treemanifest version of the hg core repo goes from
~50s to ~14s. As expected, there is no significant difference on a
repo with flat manifests.
test-treemanifest.t had introduced HGPORT3 and HGPORT4,
which were improperly added to run-tests.py.
It also was not using HGPORT1.
This recycles HGPORT, and shifts everything into HGPORT1 + HGPORT2.
When I taught debugrebuildfncache about dirlogs in ebe9dacc63ba
(treemanifests: fix streaming clone, 2016-02-04), I added a
last-minute "if 'treemanifest' in repo" guard. That should have been
checking for "... in repo.requirements". Fix that and add tests for
it.
Similar to the previous patch, the .hg/store/meta/ directory does not
get copied when when using "hg clone --uncompressed". Fix by including
"meta/" in store.datafiles(). This seems safe to do, as there are only
a few users of this method. "hg manifest" already filters the paths by
"data/" prefix. The calls from largefiles also seem safe. The use in
verify needs updating to prevent it from mistaking dirlogs for
orphaned filelogs. That change is included in this patch.
Since the dirlogs will now be in the fncache when using fncachestore,
let's also update debugrebuildfncache(). That will also allow any
existing treemanifest repos to get their dirlogs into the fncache.
Also update test-treemanifest.t to use an a directory name that
requires dot-encoding and uppercase-encoding so we test that the path
encoding works.
When doing a local clone with treemanifests, the .hg/store/meta/
directory currently does not get copied. To fix it, all we need to do
is to add it to the list of directories to copy.
In b89de5ee5b31 (changegroup: don't support versions 01 and 02 with
treemanifests, 2016-01-19), I stopped supporting use of cg1 and cg2
with treemanifest repos. What I had not considered was that it's
perfectly safe to pull *to* a treemanifest repo using any changegroup
version. As reported in issue5066, I therefore broke pull from old
repos into a treemanifest repo. It was not covered by the test case,
because that pulled from a local repo while enabling treemanifests,
which enabled treemanifests on the source repo as well. After
switching to pulling via HTTP, it breaks.
Fix by splitting up changegroup.supportedversions() into
supportedincomingversions() and supportedoutgoingversions().
There were two mistakes: one was accidental reuse of the fclnode
variable from the loop gathering file nodes, and the other (masked by
that bug) was not correctly handling deleted directories. Both cases
are now fixed and the test passes.
In 7a1ccfe03f74 (treemanifests: set bundle2 part parameter indicating
treemanifest, 2016-01-08), I didn't realize I had to set the parameter
separately for getbundle and unbundle. Having the parameter there on
push allows us to push to an empty repo and have the requirements
updated correctly.
The newly created helper changegroup.safeversion() knows to pick
version 03 if the repo uses treemanifests, so just using that means we
pick the right changegroup version.
In order to give us the freedom to change the changegroup3 format,
let's hide it behind an experimental config. Since it is required by
treemanifests, that will override the cg3 config.
I'm not entirely happy with using a trailing / on a "file" entry for
transferring a treemanifest. We've discussed putting some flags on
each file header[0], but I'm unconvinced that's actually any better:
if we were going to add another feature to the cg format we'd still be
doing a version bump anyway to cg4, so I'm inclined to not spend time
coming up with a more sophisticated format until we actually know what
the next feature we want to stuff in a changegroup will be.
Test changes outside test-treemanifest.t are only due to the new CG3
bundlecap showing up in the wire protocol.
Many thanks to adgar@google.com and martinvonz@google.com for helping
me with various odd corners of the changegroup and treemanifest API.
0: It's not hard refactoring, nor is it a lot of work. I'm just
disinclined to do speculative work when it's not clear what the
customer would actually be.
We currently use 'd' to indicate that a manifest entry is a
directory. Let's switch to 't', since that's not a valid hex digit and
therefore easier to spot in the raw manifest data.
This will break any existing repos with tree manifests, but it's still
an experimental feature and there are probably only a few test repos
in existence with 'd' flags.
In the most complex case, we try using the incoming delta base, then
we try both parents, and then we try the previous revlog entry. If
none of these result in a good delta, we natually use the null
revision as base. However, we sometimes consider the nullrev before we
have exhausted our other options. Specifically, when both parents are
null, we use the nullrev as delta base if it produces a good delta
(according to _isgooddelta()), and we fail to try the previous revlog
entry as delta base. After e60126c6093d (addrevision: use general
delta when the incoming base delta is bad, 2015-12-01), it can also
happen for non-merge commits when the incoming delta is not good.
The Firefox repo (from many months back) shrinks a tiny bit with this
patch: from 1.855GB to 1.830GB (1.4%). The hg repo itself shrinks even
less: by less than 0.1%. There may be repos that get larger instead.
This undoes the unexplained test change in e60126c6093d.
For globs like 'foo/ba?', match._roots() will return 'foo'. Since
visitdir(), excludes directories in the excluded roots, it would skip
the entire foo directory. This is incorrect, since 'foo/ba?' doesn't
mean that everything in foo/ should be exluded. Note that visitdir()
is called only from the treemanifest class, so this only affects tree
manifests. Fix by adding roots to the set of excluded roots only if
there are no excluded patterns.
Since 'glob' is the default pattern type for globs, we also need to
update some -X patterns in the tests to be of 'path' type to take
advantage of the visitdir tricks. For consistency, also update the -I
patterns.
It seems a little unfortunate that 'foo' in 'hg files -X foo' is
considered a pattern because of the implied 'glob' type, but improving
that is left for another day.
match.visitdir() used to only look at the match's primary pattern roots to
decide if a treemanifest traverser should descend into a particular directory.
This change logically makes visitdir also consider the match's include and
exclude pattern roots (if applicable) to make this decision.
This is especially important for situations like using narrowhg with multiple
treemanifest revlogs.
Most operations on treemanifests already visit only relevant
submanifests. Notable examples include __getitem__, __contains__,
walk/matches with matcher, diff. By making submanifests lazily loaded,
we speed up all these operations.
The lazy loading is achieved by adding a _load() method that gets
defined where we currently eagerly parse the manifest. We make sure to
call it before any access to _dirs, _files or _flags.
Some timings on the Mozilla repo (with flat manifest timings for
reference):
hg cat -r . README.txt: 1.644s -> 0.096s (0.255s)
hg diff -r .^ -r . : 1.746s -> 0.137s (0.431s)
hg files -r . python : 1.508s -> 0.146s (0.335s)
hg files -r . : 2.125s -> 2.203s (0.712s)
