We compute which files have changed by looking at the manifest deltas. It's
possible that a manifest delta may contain an entry that deletes a file, then
replaces that file with the exact same content. This results in an unnecessary
set operation. Let's catch that earlier and avoid the set.
Previously, autocreate would keep every tree we created in memory. This result
in a memory explosion such that the process was eating 10's of GB of memory.
Let's optimize this to keep track of how many times each tree will be needed,
then throw the tree away once it is no longer needed.
Testing this via 'hg pull' showed that memory stayed constant even when pulling
and converting thousands of commits.
Summary:
This adds a treemanifest.autocreatetrees config option. When it is set, hg pull
will automatically create a pack file that contains tree contents during an hg
pull.
We'll need to wait until the setitem and dirtybit logic is landed before we land
this, since that's required for us to test the full iteration logic here.
Test Plan:
Ran hg pull and verified a datapack was produced with the correct
manifest contents. The contents currently only contained the root manifest,
since we don't have the setitem + dirtybit logic necessary to actually modify
tree yet.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3838836
Summary:
Adds the initial extension that sets up the ctreemanifest. It currently relies
on the fastmanifest extension to hook into all the manifest APIs to construct
ctreemanifests.
Test Plan:
In a future patch, I was able to run 'hg manifests' on a commit and
have it return the manifest contents by reading the treemanifest.
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Subscribers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3755327
Signature: t1:3755327:1472114482:0c5862cba68ed4db643d28c2fae01f33f5352970
