There was a flakey test because the remote stderr output was getting eaten part
of the time. This fixes it by capturing all the output and then sending it to
stderr manually.
Tested it by running all the tests in parallel multiple times and not seeing the
issue anymore.
Also we now send that output to the debug output. We're already throwing a more
useful error message, so let's hide the noise.
Differential Revision: https://phab.mercurial-scm.org/D1276
Adds a test and improves output for dealing pulling a normal scratch branch into
a treeonly repo (so the manifest doesn't exist).
In the future we could fix this by having the server generate the trees on the
fly for infinitepush branches that don't already have them.
Differential Revision: https://phab.mercurial-scm.org/D1268
Previously the sendflat config decided if we should send flat manifests or not.
To reduce the number of config permutations we have to handle, let's change the
code to always send the flat manifests, if it exists.
Differential Revision: https://phab.mercurial-scm.org/D1266
Previously we were relying on config values to determine when to send or not
send tree and flat manifests. This resulted in complicated permutations to test.
Let's remove these config values and always send trees if we have them. A future
patch will do the same for flat manifests.
Differential Revision: https://phab.mercurial-scm.org/D1265
Previously test-treemanifest-treeonly.t was hanging. Now it doesn't hang (but
still fails).
It obviously doesn't fix the real issue, but at least it doesn't make test
hang.
Test Plan:
Run test-treemanifest-treeonly.t
Differential Revision: https://phab.mercurial-scm.org/D1205
In hybrid tree/flat mode, if a local commit accidentally got marked as public,
then we would attempt to access the server, which of course didn't have the
commit and failed. This caused an error on the client, when in reality we
should've just fallen back to the flat manifest. Let's eat the error and the
message in this case.
Differential Revision: https://phab.mercurial-scm.org/D772
Previously, if during the middle a transaction something needed to access a tree
for a commit that was created during the transaction, the manifestlog could find
that tree in a side store it kept for current transaction trees. Unfortunatley
we were only storing the data, and not the parents, so if logic needed the
parents it failed with a key error. The fix is to just store the parents as
well.
This was caught by the sparse extension, which looks at the current tree during
a commit when a sparse profile changes.
Upstream has changed local bundles to use the highest available changegroup
version, which means we're using changegroup3 for strips now. So we need to
handle that in shallowbundle.
Summary:
If the treemanifest is loaded, but not enabled for a repo, it was still
executing tree code for that repo, which would break. This adds checks to
prevent tree code from running in non-tree repositories.
Reviewers: #fbhgext, simonfar
Reviewed By: #fbhgext, simonfar
Subscribers: simonfar
Differential Revision: https://phab.mercurial-scm.org/D568
I made a last minute change to the treemanifest code to match a change that went
into @, but I forgot to run the tests and I broke it pretty badly. This fixes
the code.
Adds a test for rebasing in tree only mode and fixes a bug in readdelta where it
was returning the parent's file nodes instead of the current file nodes. The
test covers this case.
In order to support pulling from the main server, we need to:
- ignore the flat manifest part of the changegroup
- change our treeonlymanifestlog to do store.get() instead of
store.getmissing(), since only get() fetches from the remote server.
Previously we set repo.name during reposetup. Soon we're going to support
bundlerepositories though, which will need access to the tree store (and
therefore the repo.name) during repo contruction (before reposetup). So let's
have the tree store setup logic manually read the repo name from the config
instead of waiting for the reposetup to do it.
When performing a strip or other bundle operation, the bundle2 generator doesn't
use the normal part generator functions. Instead it has a custom function that
adds each part individually. We need to hook into that code to add our tree
packs to bundle files.
When writing a series of treemanifests, future trees may need to reference trees
that have already been written. It can't look in the pack file like normal
because the pack file hasn't been finalized yet. To make those written trees
accessible, let's put them in a temporary memory set that the treemanifestlog
knows to look in.
Summary:
Now that all the users of the datastore access it via manifestlog, let's remove
the setters for the old repo.svfs version
Test Plan: Ran the tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D236
Summary:
This adds the data and history stores to the manifestlog, instead of storing
them on the opener. It also moves construction of the stores to manifestlog
creation time, instead of repo creation time. This is important for bundlerepo
support, since the bundlerepo needs to access the manifestlog as part of the
initial repo setup, and previously the stores weren't added to the opener until
after repo setup.
This patch does not remove the stores from the opener yet. Future patches will
migrate other extensions that depend on the datastores, and then we'll delete
them from the opener.
Test Plan: Ran the tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D234
Summary:
In a future patch we will move tree store setup to be part of manifestlog
creation, instead of part of repo setup. Let's move the logic to it's own
function for now.
Test Plan: Ran the tests
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D232
This is the result of running a codemod script:
python ~/hg/contrib/codemod/codemod_nestedwith.py **/*.py
Plus a manual cleanup that removes the comment in `absorb/__init.py`.
See D76 for the codemod script.
Differential Revision: https://phab.mercurial-scm.org/D78
Summary:
We want to reuse the treeonly logic in both the server and the client, so we
need to move the treeonly version of the manifest log to it's own class that
doesn't depend on revlogs. This is actually a lot cleaner too.
Test Plan: Ran the tests
Reviewers: #fbhgext, simpkins
Reviewed By: #fbhgext, simpkins
Subscribers: simpkins
Differential Revision: https://phab.mercurial-scm.org/D12
Summary:
Adds a config option that forces all manifest accesses to use treemanifests.
This will be useful in enforcing the final stage of the treemanifest migration
where we use tree manifests for everything.
Test Plan:
A subsequent patch adds a test that runs hooks that touch the manifest while
treeonly is set.
Reviewers: #fbhgext, simpkins
Reviewed By: #fbhgext, simpkins
Differential Revision: https://phab.mercurial-scm.org/D7
Summary:
When requesting trees from the server, the client sends a list of manifest nodes
it wants and base nodes it has, then the server sends back any parts of the tree
that are in the desired manifests but not in the base nodes. If no base nodes
are sent, then the server tries to pick appropriate base nodes from the list of
manifests being requested.
Previously, when requesting manifest X, it would only try to pick X^ as the base
node (assuming X^ was in the request set). This meant that if you requested X
and X^^, it wouldn't realize they were very similar and instead it would send
the entire X and entire X^^ trees (with lots of duplication).
With this patch, it will always use the previously requested manifest as the
base node for the next manifest, if no base nodes are specified. So requests X
and X^^ will return the minimal set of nodes.
Test Plan: Added a test. It fails before and passes afterwards
Reviewers: #fbhgext, simpkins
Reviewed By: #fbhgext, simpkins
Differential Revision: https://phab.mercurial-scm.org/D75
The compatibility shim for the old protocol was just plain broken. This fixes
it. It's hard to write a test for this since the code that produced the old
format is no longer around. But I tested it internally against our servers with
the old format.
Summary:
Eventually we will want to only send trees instead of flat manifests. Let's add
a config to disable sending them. For now it will still loop over the flat
manifests, since that's needed to discover which files to send, but it won't
actually send the flat manifests.
This makes it easier to test the pushrebase pack handling code in a future
patch.
Test Plan:
A future diff adds testing for pushing only tree packs and ensuring
they are received correctly.
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mitrandir, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5330005
Signature: t1:5330005:1499713117:7e52dd79ab6759717dbac8431d01ce2e55d1f683
Summary:
A future patch will use this same logic to create pack parts, so let's make it
reusable.
Test Plan: Ran the tests
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5326175
Signature: t1:5326175:1498751898:bf71081ab6309aa48c5ccadfd7e46d7d67588367
Summary:
In a future patch we will reuse pack part generation for another part type.
Let's refactor our existing tree group part type to be more generic. We can now
specify what category the pack belongs to (manifests, files, etc) and treecache
has become cache.
Since argument changes in bundle2 are breaking changes, we need to deploy this
as a new bundle2 part type first, then once all old clients are gone, we can
remove the old part type and rename it.
Test Plan: Ran the tests
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mitrandir, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5326081
Signature: t1:5326081:1499704314:59e53a6fb573d9b4594c5ac64e6c4b6f8301df6e
Summary:
Previously the wireprotocol allowed for fetching a specific sub-tree,
but the server's implementation didn't handle it. This caused issues when the
user made a local commit whose trees had references to public trees, then the
user deleted the cache trees and refilling the cache failed.
The current solution is to just serve the entire tree, since the current native
code treemanifest implementation can't serve sub-parts of the tree at the
moment. This is expensive, but should be pretty rare though since the client
usually the entirety of a tree if it's going to keep the root.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5296211
Signature: t1:5296211:1498086921:3e7be99b8581c1fdf507a2fe560b4c796835cdf3
Summary:
Previously, if we were fetching a treemanifest we would only search downwards
for commits for possible base trees (so we could download only the new parts of
the tree). With this patch we will also search upwards in the changelog. This
means that running 'hg log -r master --stat' will efficiently download the tree
for `.^` since it will be based against `.`.
Also fixes an issue with on demand downloads looking for base revs by starting
at the wrong changelog rev (they used the manifest rev number instead of the
changelog linkrev number). The same test covers this.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mitrandir, medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5279552
Signature: t1:5279552:1497980967:a2d6003b126858cc05eb0095cd68c12db9670e29
Summary:
If the repo being repacked has commits being pushed, it's possible that the
repack will encounter file revisions who's linkrevs don't exist in the inmemory
changelog. Let's set an upper bound on what linkrevs to repack so we can only
process revisions who's commits we can see.
Test Plan: Added test
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5279171
Signature: t1:5279171:1497911997:a48c56abebd14a1c066c9fc1ee4098f813d062df
Summary:
Adds output that tells how many trees were downloaded, and how long it took.
Only appears if remotefilelog.debug=True is set.
Test Plan: Updated the tests
Reviewers: #mercurial, phillco
Reviewed By: phillco
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5278596
Signature: t1:5278596:1497926493:82a01b3b49e9ff87e51ac3c5cd0e4be952020ba0
Summary:
`@bundle2.parthandler` and `@exchange.b2partsgenerator` are not side-effect
free like other things used by the registrar framework. Registering a same
type twice will cause assertion errors. We have `import treemanifest` in
`fastmanifest` which should be side-effect free.
Therefore move bundle2 part registering to uisetup so importing the module
multiple times from different places won't cause assertion errors.
Test Plan: arc unit
Reviewers: durham, #mercurial, stash
Reviewed By: stash
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5271419
Signature: t1:5271419:1497861881:9b97dcf80ca893d7c08780cd5eed789e445dc52c
Summary:
Previously, if a tree needed to be downloaded on demand (like during hg
show), it would download the entire tree. With this patch, we look backwards in
the changelog for a tree we already have and then download only the new parts of
the tree.
Test Plan:
The existing demanddownload test changed because of this. I've added
more output to the test to show that only new items are downloaded (to be
specific subdir/ is not downloaded again since it did not change).
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D5210268
Signature: t1:5210268:1497299426:195517184d53396849d0ba8c75216d07741e5863
Summary:
Previously, we would only look at the node parents to determine what the base
node would be for a prefetch. This didn't work so well when we started
prefetching arbitrary revs, like only prefetching master. This patch allows us
to scan the changelog back a bit to find the nearest base tree.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5108686
Signature: t1:5108686:1495565589:1729bfae7198e8db2223446b81e3bdbf77cf3db6
Summary:
This adds a config option that allows clients to prefetch only certain trees
after a pull. This is useful if the user wants to download just master. A future
patch will make it only download the portion of the master tree that is new.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5108677
Signature: t1:5108677:1495565526:a443c41ccbf55e529c93aa0c021849396d611ac6
Summary: Only do the tree prefetch if there are nodes we don't have.
Test Plan: manual testing
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5108667
Signature: t1:5108667:1495580154:393125986ce4ce307f299f8ee3856fc76c52a25f
Summary: Upstream has deprecated cmdutil.commands() in favor of registrar.commands()
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5106486
Signature: t1:5106486:1495485074:0e20f00622cc651e8c9dda837f84dd84cc51099e
Summary:
When the server performs a repack, it would read all the data from all the
revlogs. This was very slow and expensive. Let's add a --incremental option that
makes it only read the revlog entries that have a linkrev newer than the latest
linkrev data that's already in a pack file.
Test Plan: Adds a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4997260
Signature: t1:4997260:1493904216:c4f5b6c9652bbf8f66c1bc2b2547c898324d22cd
Summary:
This hooks into the strip code to allow stripping revisions from treemanifests
as well. This will make it easier to clean up server repos when mistakes happen.
Test Plan: Adds a test
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5031489
Summary:
Previously, when creating a new manifestrevlog for the trees, we attempted to
copy the opener options and insert the treemanifest value for the manifestrevlog
class to pick up. It turns out that the opener we copied was actually just a
wrapper around the same underlying object that the real repo used, so when we
did opener.options = oldopener.options.copy(), we were actually putting the new
copy on both the old and new opener. This meant we enabled treemanifest for the
entire repo.
This wasn't a problem in most cases because the manifestlog had already been
loaded. But in a pushrebase world, all the data structures are refreshed when
the lock is obtained (after waiting for the other push to finish). In this
situation, the refreshed repo.manifestlog was now a treemanifest so the second
commit was pushed to the main manifest as a tree.
The fix is to not use the opener.options hack to configure the tree. Instead we
are refactoring upstream to take an explicit options to the manifestrevlog
constructor. This patch makes our internal treemanifest extension use this new
api and adds a test for push contention.
Test Plan: Added a test. It failed before, and passes after.
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5026019
The bundle2 part was always being generated, even for non-treemanifest repos
(because fastmanifest imports the treemanifest module, so the b2partsgenerator
gets registered). This cause pushes to break in any repo with fastmanifest
enabled.
