Summary:
To enable pushing between peers (and eventually pushing to the server), let's
teach bundle creation to include the trees being pushed.
Test Plan: Adds a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957456
Signature: t1:4957456:1493266296:67f98a2b3d691644bde9098a713d05266f349cde
Summary:
Previously the server would access the tree data in an adhoc manner. Sometimes
it would talk straight to revlogs, sometimes it would create stores and talk to
data packs. This patch makes it access trees the same way clients do, through
repo.svfs.manifestdatastore and manifesthistorystore.
This also cleans up the client store creation just a little and adds a
unionmetadatastore for unified history access.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957441
Signature: t1:4957441:1493263349:e76d177f7a9f45343e6f984d6c0ae2c7cacba035
Summary:
Previously, the logic that added data to a mutable history pack was required to
add it in the correct order (all entries for a certain file at once, and in
newest-first order). This required the callers to jump through weird hoops if
the data came in out of order or at different times in the transaction.
This patch moves the ordering logic to be inside MutableHistoryPack, so callers
can add the data in any order they wish, and it will get sorted before being
serialized.
This does add memory pressure to things that read a lot of history, like repack.
If this becomes a problem we may want to add a 'historypack.flush()' api that
let's us tell the history pack it's ok to flush it's current contents to disk.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956096
Signature: t1:4956096:1493264693:a2275a49e35565d4b11244e3e5dd82c25de7e16e
Summary:
The remote store needs to throw a KeyError if it can't find the node. Returning
None was just causing breakages up stream and meant the union store did not move
on to the next store. This was caught by peer-to-peer pushing tests in a future
patch.
Test Plan: Future tests caught this
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956062
Signature: t1:4956062:1493264179:432a3add7c0501804520e9abc92e33ba8fec4384
Summary:
The old func api parsed the getpack arguments internally. Let's move that out to
the wire protocol level, so in future patches we can reuse generatepackstream
more easily.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956038
Signature: t1:4956038:1493265461:e7ed1ad659fb0639c325df3f8352dbe06f650740
Summary:
Previously the wire pack protocol was a custom stream of bytes. This
patch moves it to be within bundle2 so we can take advantage of features it has
(versioning, arguments, mid-stream error messaging, including packs in other
bundles, etc). This will be useful in future patches where we want to allow
peer-to-peer exchange of client only tree manifest data.
This also removes all the custom argument encoding/decoding.
Test Plan:
Ran the tests. Order changed but the content remains the same and the
order is consistent.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4924718
Signature: t1:4924718:1493050942:2734cfdbca0b0b0a3f3be1ee8e224f634553c305
Summary:
A future patch will move the pack wireprotocol to use bundle2. In this new world
we'll be given a file handle instead of a remote peer, so let's switch the
utility methods to work on a file handle instead.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4924553
Signature: t1:4924553:1493050882:7a9ee8b282bf47ef393362dd0114d801dc2a68d5
Summary:
Adds a config option that will automatically download the last N trees from the
server during pull. It efficiently decides what bases to set, based on what
commit parents it already has.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4923211
Signature: t1:4923211:1493200730:39872db6aabb48c12131ff8501a950c8ee60e2b0
Summary: Now that prefetching supports base nodes, let's add a simple test covering it.
Test Plan: Ran the test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4911537
Signature: t1:4911537:1492641400:752c52acfb264ec42c5d05865bf87ebfc2558801
Summary:
The gettreepack protocol already had the arguments for the client to specify
what nodes they want and what nodes they have, but it wasn't implemented (we
just always returned the entire tree). This patch adds support for only sending
parts of the tree that are new. In addition, if the client requests many nodes,
we will only send the portion of each node that is not contained in the other
nodes. For instance if you ask for X and Y, where X is the parent of Y, then we
will send all of X and only the part of Y that is new.
The order changed a bit, probably because we stopped over delivering data.
Test Plan:
Ran and updated the tests. The next patch will actually add a test
for this code.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4911531
Signature: t1:4911531:1492643897:58aabe5a32f156dd7424cc07c990301dbd5dc1c7
Summary:
Accessing treemanifests in revlogs is incredibly slow. This patch adds the
ability for `hg repack` to repack the revlog content into a data pack file, and
teaches the getserverpack code to look in the pack file first.
Test Plan: Need to add a test. Sent out anyway for review
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901298
Signature: t1:4901298:1492640706:2850b9f0e9bbee77952f46af3b784aa81253e626
Summary:
Future tests showed that backfilltree was not populating the fncache, which
meant the files were not streamed during a clone or read during a repack. This
fixes this by using the actual repo store (which handles fncache serialization)
instead of a creating a new store.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4916070
Signature: t1:4916070:1492647129:4deeb70c63dd414f6d5c06e4259a0bbf899f3425
Summary: This is a generally useful store class. Let's move it to the store code.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901293
Signature: t1:4901293:1492647203:0f76a6e78fd0035b61c4e45e18cd9fce59359768
Summary:
The server treemanifest reading code was super slow because it used the pure
python treemanifest implementation. Let's switch it to use the ctreemanifest
implementation on top of a revlog store class.
The tests change because the output is now in reverse topological order (deepest
trees first).
Test Plan: Tests updated
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901268
Signature: t1:4901268:1492637601:371a2303ac4f6e2b81f69a85ca657849150abcf7
Summary:
This adds a remote tree datastore to the store, so if we don't have a tree on
local disk, we can reach out to the server for it.
This also makes a small change to fastmanifest to use get() instead of
getmissing() to test if the tree is available. getmissing() would indicate the
tree is missing if it wasn't local, while get() will attempt to fetch it from
the server.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4864645
Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
Summary:
This adds a new prefetchtrees command and gettrees wireprotocol command for
receiving trees from the server. The actual wire protocol is a custom request
format, with a normal pack file response format (same as used in remotefilelog).
The request args can specify which manifest nodes are wanted, for which
directory and which subdirectories.
I attempted to use the changegroup3 response format instead of the pack format,
but changegroup is extremely intertwined with revlogs and with producing
changegroups for linear swaths of the repo history, instead of random parts of
just the manifest.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: stash, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4864435
Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
Summary:
This adds 'hg debuggentrees' that takes an optional --verify flag. This makes it
easier to debug tree creation and catch errors.
Test Plan:
Ran `PYTHONPATH=/data/users/durham/fb-hgext python ../hg/hg --pager=off debuggentrees --verify master~xxxx master` to generate 6 months
worth of trees. It found bugs which are fixed in other patches.
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724502
Summary:
We are relying on pushrebase to replay each of the incoming commits so they
produce both flat and treemanifests. This patch blocks any commits that don't
come through pushrebase, by intercepting manifest.addgroup calls (which is what
a normal push would use) and throwing an exception.
Test Plan: Added a test
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4093745
Signature: t1:4093745:1478100170:79b1eea6668cd3efb27742323e11a45bde930440
Summary:
When a server is a treemanifest enabled server, we want to ensure that every
commit that comes in will be added to the treemanifest as well as the flat
manifest. This patch accomplishes that by hooking into the addmanifest logic and
applying the new manifest to the tree as well.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4093738
Signature: t1:4093738:1478260442:58f77bd88ab09ccda4ac238a065388be8f4a271d
Summary:
This adds a command for producing tree manifests for existing manifests in the
repository. The tree manifests are stored similar to how vanilla mercurial
stores them, but the root revs are in 00manifestree.i instead of 00manifest.i
and we fake the root node to be the same as the flat manifest node, so we can
reference either the flat or the tree easily.
Test Plan: Added a test. Also began converting fbsource
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4092252
Signature: t1:4092252:1478260394:a55360a8ba83ad85b1694cb93f0788f1f672ef61
Summary:
As part of the migration, we need both flat and tree manifests to exist at once
on the server. To do this, we need to store the tree manifest root revisions
somewhere other than 00manifest.i.
This patch divides the treemanifest logic into client and server bits, and on
the server creates a repo.manifesttree property that contains the tree manifest.
Future patches will write to this to fill in new tree entries, and eventually
read from it to seed clients with tree data.
Test Plan:
Future patches add tests that for features that use this
functionality
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4092217
Signature: t1:4092217:1478260318:7edada2ea39a5de565955a4180bfad62f6126352
Summary:
Previously, tree repacks did not take into account tree history. It would just
look at the delta base and if the base existed, it would just reuse the delta.
This would A) result in very long chains, and B) result in chains where the full
text was the oldest version, instead of the newest (recent full texts means
faster access to recent versions).
This patch threads tree history into the repacker, which already knows how to
use history for repacks.
Test Plan:
Updated the tests, and inspected the new test results to ensure tree
entries that were not deltas before the repack became reverse deltas during the
repack.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4647359
Signature: t1:4647359:1488882710:dba72cf488766ce827b7641735164fa0efc9a303
Summary:
Previously, when treemanifest would create packs of trees during pull, we
allowed trees to be delta'd against trees in other packs. This resulted in
smaller packs, but if the other pack disappeared for some reason (since it's a
cache), the chain broke.
This patch ensures that the first version of every tree added to a pack is a
full text.
This temporarily makes repacks worse, since the repacker doesn't know about
history to produce deltas when combining packs. The next patch adds history
awareness which improves the repack deltafication.
Test Plan:
Updated the tests, and inspected the new test results to ensure that
all packs only had deltas within the pack.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4647348
Signature: t1:4647348:1488882214:e850622a853a534fc60caeef604c88c30740c60d
Summary:
Previously the treemanifest code itself would create the text deltas when
writing a tree out. This meant we couldn't make the delta decision based on
other data, like if the p1 commit was in the same pack file.
This patch removes treemanifest.write() and moves all calls over to
treemanifest.finalize() which gives the python/pack layer control over delta
choices. A future patch will use this to ensure tree packs always contain
complete delta chains.
Test Plan: All tests pass
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4645942
Signature: t1:4645942:1488880851:d0c8c902e7e849072a53344630a9184b6d8e1e7f
Summary:
Previously the treemanifest auto-tree-creation logic only produced data packs
containing the actual contents of the tree blobs. This lost history information
which is important for our ability to efficiently repack the data files.
This patch creates history packs during pull as well. A future patch will also
create history packs for the local tree blob store.
Test Plan: Updated the tests to cover this
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4638865
Signature: t1:4638865:1488449992:48b60961b50b90b6d0e75a64af1f36fb29944e7a
Summary:
Adds a treemanifest.usecunionstore config flag for enabling and disabling use of
the native code uniondatapackstore.
Since we haven't implemented the repack APIs on the native datapack stores, we
currently have to force repack to use the old python implementations. Instead of
trying to expose just the appropriate APIs through the python interface, I think
we'll rewrite all of repack to be in C++ at a future time, since we can take
advantage of parallelism, etc.
Test Plan:
Updated test-treemanifest.t to use the c datapackstore. Also run all
the tests with --extra-config-opt=treemanifest.usecunionstore=True.
These tests caught a missing null check in the C++ code as well.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609795
Signature: t1:4609795:1488365341:203362db5f470b613c4d6484686cd32c3fa8458f
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
treemanifest requires fastmanifest, and fastmanifest.usetree requires
treemanifest. Let's make these dependencies explicit in the code and error out
if they are incorrect.
Test Plan: Added a test
Reviewers: #mercurial, wez
Reviewed By: wez
Subscribers: wez, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4417512
Signature: t1:4417512:1484347447:7e18340813fac0b298aa51a7cc2f89fc6953680f
Summary:
Treemanifest had a bug where the pack files it created were 400 instead of 444.
This meant people sharing a cache on the same machine couldn't access them. In
most of the remotefilelog code we had set opener.createmode before creating the
pack but we forgot to for treemanifest.
This patch moves the opener creation and createmode setting into the mutable
pack class so we can't screw this up again.
Test Plan:
Tests that wrote to the packs before, now failed and had to be
updated to chmod the packs before writing to them.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4411580
Tasks: 15469140
Signature: t1:4411580:1484321293:9aa78254677548a6dc2270c58cee0ec6f57dd089
Summary:
We have run into some cases where users ended up with empty pack file in their
packs directory. Just log a warning in this case and skip this pack file,
rather than letting the exception propagate up and crashing the command.
Test Plan:
Created empty 0000000000000000000000000000000000000000.histpack and
0000000000000000000000000000000000000000.histidx files in my repository's
hgcache directory, and confirmed that "hg log" now simply warns about them
instead of crashing.
I didn't really test the perftest.py or treemanifest_correctness.py extensions
much. They seem to throw exceptions, and look like they have maybe gotten a
bit stale. I fixed one minor typo, but didn't dig into the other exceptions
too much.
Reviewers: durham
Reviewed By: durham
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4402516
Tasks: 15428659
Signature: t1:4402516:1484155254:96d2980efcec2d735257b08910e1ca437ef1dad6
Summary:
In 676596f945ea2166820ef92e692ef7fe6a6247f0 were added comments with
lines > 80. In aec81a9a80d22989bbdc8c74c1dfec9dcbbe6866 default config value
was changed.
Test Plan: arc unit
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377188
Previously we were relying on mfrevlog.revdiff() to produce the delta for us.
This only showed us what files were added/modified, as compared to p1, and we
had to use a heuristic to know what files were deleted (by looking at the list
of files in the commit metadata). Merge commits have a different criteria for
what is in the commit metadata (it only contains entries where the file is
different from both parents), so we can't use it for the same heuristic. So
let's fall back to a normal manifest diff for merge commits, since they are
rare.
Adds a test for verifying that conversion of merge commits into a tree works.
Commit 24515b72d5 attempted to optimize writes by checking if the file in the
delta was also in the list of files in the commit metadata. This doesn't work for merge
commits since the only files the commit metadata are the ones that differ from
both (so set(changes in metadata) != set(changes in node-diff-against-p1)). This
caused the verify code to catch the issue. The fix is to just remove the
allfiles optimization.
Summary:
This adds a simple test that verifies hg repack will pack two tree manifest
packs into one.
It caught a bug where creating a treemanifest for a commit with a null parent
produced incorrect output because it constructed an empty tree and tried to use
it's node as the parent of the delta, when there should not have been any delta
in the first place. This is fixed by this diff as well.
Test Plan: Ran the new test
Reviewers: #mercurial, dsyang, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4261591
Signature: t1:4261591:1480705822:ef21fb8cebd8b89f92f58f11bb1dab59bf97664d
Summary:
Since treemanifest uses the same storage locations as remotefilelog, it needs
access to the repo name as well. If a given repo has treemanifest enabled but is
not a remotefilelog repo, it won't have the repo.name member already. So let's
add it ourselves in treemanifest.
Eventually we should probably refactor this out to be a more global concept of
repo name.
Test Plan: A future patch adds a test that caught this
Reviewers: #mercurial, dsyang, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4261581
Signature: t1:4261581:1480705457:8fb4b86ce8abeed62cc7c8f787868359c3cf8abc
Summary:
Previous hg repack would only repack the shared manifest cache store. This patch
makes it also repack the local manifest store too.
Test Plan:
Made a local commit in my test repo with treemanifests enabled.
Rebased the commit to master so now there were two local tree packs. Ran hg
repack and verified they were combined into one pack in
.hg/store/packs/manifests
I'll add a test later today
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4260580
Signature: t1:4260580:1480696151:8f989e299dda50281ca63489e870202eb195d714
Summary:
Previously, when we wrote each tree entry into a pack file, it wasn't delta'd in
any way. This patch makes it store the delta against p1 in the pack file.
Testing in a large repo shows this reduces tree pack size by about 22x.
Test Plan:
Ran the tests. Did a pull in a large repo and saw the pack file was
22x smaller than before (and still usable).
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D4202088
Summary:
Creating a tree for a commit whose parent is not already a tree is expensive.
Let's optimize the autocreatetrees option (which converts manifests to trees
during pull) to only create trees if A) the parent is already a tree, or B) the
parent is master. This way we only pay the expensive part once. It also means
that as new branches fork off master, they will be trees too, since all commits
in the new branch will meet criteria (A).
Test Plan:
Ran hg pull in a large repo over a large pull with 40 different
branches and verified it only paused for the initial master commit.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4129010
Signature: t1:4129010:1478260239:1698ada4e3c6a38ab77a94317e75daee4812276f
Summary:
Upstream has refactored the manifest class into several classes, so we need to
update treemanifest to work with the new structure. Notably, the factory add
function previously relied on the ability for the revlog class to create a new
manifestdict (via manifest.maniest.read()), since this isn't possible anymore,
we have to construct the hybridmanifest ourselves and provide an appropriate
loadflat function to get the flat manifest if necessary.
Test Plan: Ran the tests
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4180891
Signature: t1:4180891:1479285991:82bc546a1eb682d3cfd8b4724bda575410405d0f
Summary:
This makes local commits get written to trees (as well as flat manifest) when a
commit happens where the parent commit has a tree manifest already. During a
transaction where multiple trees are written (like when rebasing multiple
nodes), we reuse the same pack file for all the trees produced by tieing into
the transaction abort and close hooks.
Test Plan:
Ran the tests. Ran hg commit with the extension enabled. A future
patch will add an integration test for the treemanifest extension.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4055851
Signature: t1:4055851:1477059659:91b1c2f93ef986e910cea752ebf2466cb20ac921
Summary:
In a future patch we will start writing user local tree data into a local
directory. This patches add the local store to the union store so the contents
will be accessible once we start writing it.
It also renames manifest to manifests for consistency with the other store names
(like 'packs').
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4055827
Signature: t1:4055827:1477059457:0d5b0d999b47d88c73f5ab2721d8d27deacc01bc
Summary:
In a future diff we will be introducing packs into .hg/store, so we need to
differentiate between cache packs and local packs. This patch renames
getpackpath to getcachepackpath. A future diff will add getlocalpackpath.
This exposed a pyflakes error, so we fix that too.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4055815
Signature: t1:4055815:1477059415:e0221557bbeec6701820c826f00390d3a71cd2d3
We compute which files have changed by looking at the manifest deltas. It's
possible that a manifest delta may contain an entry that deletes a file, then
replaces that file with the exact same content. This results in an unnecessary
set operation. Let's catch that earlier and avoid the set.
Previously, autocreate would keep every tree we created in memory. This result
in a memory explosion such that the process was eating 10's of GB of memory.
Let's optimize this to keep track of how many times each tree will be needed,
then throw the tree away once it is no longer needed.
Testing this via 'hg pull' showed that memory stayed constant even when pulling
and converting thousands of commits.
Summary:
This adds a treemanifest.autocreatetrees config option. When it is set, hg pull
will automatically create a pack file that contains tree contents during an hg
pull.
We'll need to wait until the setitem and dirtybit logic is landed before we land
this, since that's required for us to test the full iteration logic here.
Test Plan:
Ran hg pull and verified a datapack was produced with the correct
manifest contents. The contents currently only contained the root manifest,
since we don't have the setitem + dirtybit logic necessary to actually modify
tree yet.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3838836
Summary:
Adds the initial extension that sets up the ctreemanifest. It currently relies
on the fastmanifest extension to hook into all the manifest APIs to construct
ctreemanifests.
Test Plan:
In a future patch, I was able to run 'hg manifests' on a commit and
have it return the manifest contents by reading the treemanifest.
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Subscribers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3755327
Signature: t1:3755327:1472114482:0c5862cba68ed4db643d28c2fae01f33f5352970