Summary:
Previously, if a tree needed to be downloaded on demand (like during hg
show), it would download the entire tree. With this patch, we look backwards in
the changelog for a tree we already have and then download only the new parts of
the tree.
Test Plan:
The existing demanddownload test changed because of this. I've added
more output to the test to show that only new items are downloaded (to be
specific subdir/ is not downloaded again since it did not change).
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D5210268
Signature: t1:5210268:1497299426:195517184d53396849d0ba8c75216d07741e5863
Summary:
Previously, we would only look at the node parents to determine what the base
node would be for a prefetch. This didn't work so well when we started
prefetching arbitrary revs, like only prefetching master. This patch allows us
to scan the changelog back a bit to find the nearest base tree.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5108686
Signature: t1:5108686:1495565589:1729bfae7198e8db2223446b81e3bdbf77cf3db6
Summary:
This adds a config option that allows clients to prefetch only certain trees
after a pull. This is useful if the user wants to download just master. A future
patch will make it only download the portion of the master tree that is new.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5108677
Signature: t1:5108677:1495565526:a443c41ccbf55e529c93aa0c021849396d611ac6
Summary:
This patch just bumps the histpack version number to 1 and adds a config flag to
enable writing v1 pack files. The format hasn't actually changed in this patch,
I'm just doing the verison bump so I can update all the hashes in the tests
without working about functionality change.
In the next patch I will modify the index format, which won't affect the hashes.
Test Plan:
Ran the tests. I also ran the tests with some debug code to manually
force the sha to include 0 instead of 1 and verified that the hash didn't change
(which confirms that all of these hash changes are just because of that one byte
version change).
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4983675
Signature: t1:4983675:1493692444:5d88df4d46ce487f1b791417754ba000ecf10a1e
Summary:
Adds a config option that will automatically download the last N trees from the
server during pull. It efficiently decides what bases to set, based on what
commit parents it already has.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4923211
Signature: t1:4923211:1493200730:39872db6aabb48c12131ff8501a950c8ee60e2b0
Summary:
BSD and GNU ls behave differently in case of a missing dir:
- exit code is [1] vs [2]
- message is also slightly different
We want this to be unified for both platforms.
Test Plan:
- run the test on Linux
- hope for the best on OSX
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4921369
Summary: Now that prefetching supports base nodes, let's add a simple test covering it.
Test Plan: Ran the test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4911537
Signature: t1:4911537:1492641400:752c52acfb264ec42c5d05865bf87ebfc2558801
Summary:
The gettreepack protocol already had the arguments for the client to specify
what nodes they want and what nodes they have, but it wasn't implemented (we
just always returned the entire tree). This patch adds support for only sending
parts of the tree that are new. In addition, if the client requests many nodes,
we will only send the portion of each node that is not contained in the other
nodes. For instance if you ask for X and Y, where X is the parent of Y, then we
will send all of X and only the part of Y that is new.
The order changed a bit, probably because we stopped over delivering data.
Test Plan:
Ran and updated the tests. The next patch will actually add a test
for this code.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4911531
Signature: t1:4911531:1492643897:58aabe5a32f156dd7424cc07c990301dbd5dc1c7
Summary:
A future patch will make tree prefetching more efficient. Let's change the
prefetch test to have changes in two separate directories so we can observe that
one directory is not downloaded when doing an efficient prefetch.
Test Plan: Ran the test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4911523
Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
Summary:
The name being passed to the store was wrong, because it had a trailing slash.
This wasn't caught before because the code was reading and writing the paths
with a slash at the end, so it matched as long as we only interacted with packs
produced by the code. The issue became more obvious when I tried to have packs
generated from revlogs interact with this code.
All the tests are affected since the entry keys changed.
Also use 'const ManifestFetcher &x' to pass the ref to avoid the copy.
Test Plan: Tests updated
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901274
Signature: t1:4901274:1492638476:ff28f8976657baec99effbd82ecd436f6282ea5b
Summary:
The server treemanifest reading code was super slow because it used the pure
python treemanifest implementation. Let's switch it to use the ctreemanifest
implementation on top of a revlog store class.
The tests change because the output is now in reverse topological order (deepest
trees first).
Test Plan: Tests updated
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901268
Signature: t1:4901268:1492637601:371a2303ac4f6e2b81f69a85ca657849150abcf7
Summary:
This adds a remote tree datastore to the store, so if we don't have a tree on
local disk, we can reach out to the server for it.
This also makes a small change to fastmanifest to use get() instead of
getmissing() to test if the tree is available. getmissing() would indicate the
tree is missing if it wasn't local, while get() will attempt to fetch it from
the server.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4864645
Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
Summary:
This adds a new prefetchtrees command and gettrees wireprotocol command for
receiving trees from the server. The actual wire protocol is a custom request
format, with a normal pack file response format (same as used in remotefilelog).
The request args can specify which manifest nodes are wanted, for which
directory and which subdirectories.
I attempted to use the changegroup3 response format instead of the pack format,
but changegroup is extremely intertwined with revlogs and with producing
changegroups for linear swaths of the repo history, instead of random parts of
just the manifest.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: stash, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4864435
Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
