Commit Graph

59 Commits

Author SHA1 Message Date
Durham Goode
e34660b057 commands: update to use registrar instead of cmdutil
Summary: Upstream has deprecated cmdutil.commands() in favor of registrar.commands()

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D5106486

Signature: t1:5106486:1495485074:0e20f00622cc651e8c9dda837f84dd84cc51099e
2017-05-22 13:38:37 -07:00
Durham Goode
e205d14095 treemanifest: only use two compare trees
Limit basetrees to only 2 trees, since the current walksubtrees calculation can
only handle 2 trees at the moment.
2017-05-21 17:09:08 -07:00
Durham Goode
0af34bfb0c treemanifest: make base tree calculation cleaner
Make the base tree calculation cleaner. Only parents for which we already have
the tree are considered base nodes.
2017-05-21 17:09:08 -07:00
Durham Goode
12f99274cb treemanifest: add incremental server repack
Summary:
When the server performs a repack, it would read all the data from all the
revlogs. This was very slow and expensive. Let's add a --incremental option that
makes it only read the revlog entries that have a linkrev newer than the latest
linkrev data that's already in a pack file.

Test Plan: Adds a test

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4997260

Signature: t1:4997260:1493904216:c4f5b6c9652bbf8f66c1bc2b2547c898324d22cd
2017-05-16 15:28:13 -07:00
Durham Goode
12891668ee treemanifest: don't prefetch trees for draft commits
The prefetch logic is only meant to prefetch public commits, which have trees
available on the server. So let's exclude non-public from the prefetch.
2017-05-16 11:40:02 -07:00
Durham Goode
8ebfa4993f treemanifest: enable stripping tree revlogs
Summary:
This hooks into the strip code to allow stripping revisions from treemanifests
as well. This will make it easier to clean up server repos when mistakes happen.

Test Plan: Adds a test

Reviewers: #mercurial

Differential Revision: https://phabricator.intern.facebook.com/D5031489
2017-05-10 13:48:34 -07:00
Durham Goode
e079e7c52d treemanifest: fix issue with normal manifests being written as trees
Summary:
Previously, when creating a new manifestrevlog for the trees, we attempted to
copy the opener options and insert the treemanifest value for the manifestrevlog
class to pick up. It turns out that the opener we copied was actually just a
wrapper around the same underlying object that the real repo used, so when we
did opener.options = oldopener.options.copy(), we were actually putting the new
copy on both the old and new opener. This meant we enabled treemanifest for the
entire repo.

This wasn't a problem in most cases because the manifestlog had already been
loaded. But in a pushrebase world, all the data structures are refreshed when
the lock is obtained (after waiting for the other push to finish). In this
situation, the refreshed repo.manifestlog was now a treemanifest so the second
commit was pushed to the main manifest as a tree.

The fix is to not use the opener.options hack to configure the tree. Instead we
are refactoring upstream to take an explicit options to the manifestrevlog
constructor. This patch makes our internal treemanifest extension use this new
api and adds a test for push contention.

Test Plan: Added a test. It failed before, and passes after.

Reviewers: #mercurial

Differential Revision: https://phabricator.intern.facebook.com/D5026019
2017-05-10 13:48:34 -07:00
Durham Goode
3ca5e9b375 treemanifest: put bundle sending behind a config flag
The bundle2 part was always being generated, even for non-treemanifest repos
(because fastmanifest imports the treemanifest module, so the b2partsgenerator
gets registered). This cause pushes to break in any repo with fastmanifest
enabled.
2017-04-30 22:05:09 -07:00
Durham Goode
2b12f514c7 treemanifest: include tree pack during pushes
Summary:
To enable pushing between peers (and eventually pushing to the server), let's
teach bundle creation to include the trees being pushed.

Test Plan: Adds a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4957456

Signature: t1:4957456:1493266296:67f98a2b3d691644bde9098a713d05266f349cde
2017-04-27 10:44:34 -07:00
Durham Goode
afacc9433b treemanifest: unify client and server tree history access patterns
Summary:
Previously the server would access the tree data in an adhoc manner. Sometimes
it would talk straight to revlogs, sometimes it would create stores and talk to
data packs. This patch makes it access trees the same way clients do, through
repo.svfs.manifestdatastore and manifesthistorystore.

This also cleans up the client store creation just a little and adds a
unionmetadatastore for unified history access.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4957441

Signature: t1:4957441:1493263349:e76d177f7a9f45343e6f984d6c0ae2c7cacba035
2017-04-27 10:44:34 -07:00
Durham Goode
df97f609bf treemanifest: move history sorting responsibility into MutableHistoryPack
Summary:
Previously, the logic that added data to a mutable history pack was required to
add it in the correct order (all entries for a certain file at once, and in
newest-first order). This required the callers to jump through weird hoops if
the data came in out of order or at different times in the transaction.

This patch moves the ordering logic to be inside MutableHistoryPack, so callers
can add the data in any order they wish, and it will get sorted before being
serialized.

This does add memory pressure to things that read a lot of history, like repack.
If this becomes a problem we may want to add a 'historypack.flush()' api that
let's us tell the history pack it's ok to flush it's current contents to disk.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4956096

Signature: t1:4956096:1493264693:a2275a49e35565d4b11244e3e5dd82c25de7e16e
2017-04-27 10:44:34 -07:00
Durham Goode
df6bf53021 treemanifest: fix bug with remote store
Summary:
The remote store needs to throw a KeyError if it can't find the node. Returning
None was just causing breakages up stream and meant the union store did not move
on to the next store. This was caught by peer-to-peer pushing tests in a future
patch.

Test Plan: Future tests caught this

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4956062

Signature: t1:4956062:1493264179:432a3add7c0501804520e9abc92e33ba8fec4384
2017-04-27 10:44:34 -07:00
Durham Goode
ba9a6b0e28 treemanifest: change generatepackstream api
Summary:
The old func api parsed the getpack arguments internally. Let's move that out to
the wire protocol level, so in future patches we can reuse generatepackstream
more easily.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4956038

Signature: t1:4956038:1493265461:e7ed1ad659fb0639c325df3f8352dbe06f650740
2017-04-27 10:44:33 -07:00
Durham Goode
198bc173fb treemanifest: move pack wire protocol to operate within bundle2
Summary:
Previously the wire pack protocol was a custom stream of bytes. This
patch moves it to be within bundle2 so we can take advantage of features it has
(versioning, arguments, mid-stream error messaging, including packs in other
bundles, etc).  This will be useful in future patches where we want to allow
peer-to-peer exchange of client only tree manifest data.

This also removes all the custom argument encoding/decoding.

Test Plan:
Ran the tests. Order changed but the content remains the same and the
order is consistent.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4924718

Signature: t1:4924718:1493050942:2734cfdbca0b0b0a3f3be1ee8e224f634553c305
2017-04-27 10:44:33 -07:00
Durham Goode
2b154613e2 pack: switch readers to read from file handle
Summary:
A future patch will move the pack wireprotocol to use bundle2. In this new world
we'll be given a file handle instead of a remote peer, so let's switch the
utility methods to work on a file handle instead.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4924553

Signature: t1:4924553:1493050882:7a9ee8b282bf47ef393362dd0114d801dc2a68d5
2017-04-27 10:44:33 -07:00
Durham Goode
7e4775c6ff treemanifest: enable auto prefetching trees during hg pull
Summary:
Adds a config option that will automatically download the last N trees from the
server during pull. It efficiently decides what bases to set, based on what
commit parents it already has.

Test Plan: Added a test

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4923211

Signature: t1:4923211:1493200730:39872db6aabb48c12131ff8501a950c8ee60e2b0
2017-04-27 10:44:33 -07:00
Durham Goode
f1a7d04b25 treemanifest: add test for prefetch with base nodes
Summary: Now that prefetching supports base nodes, let's add a simple test covering it.

Test Plan: Ran the test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4911537

Signature: t1:4911537:1492641400:752c52acfb264ec42c5d05865bf87ebfc2558801
2017-04-19 21:14:04 -07:00
Durham Goode
248785548b treemanifest: add support for minimal tree pack sending
Summary:
The gettreepack protocol already had the arguments for the client to specify
what nodes they want and what nodes they have, but it wasn't implemented (we
just always returned the entire tree). This patch adds support for only sending
parts of the tree that are new. In addition, if the client requests many nodes,
we will only send the portion of each node that is not contained in the other
nodes. For instance if you ask for X and Y, where X is the parent of Y, then we
will send all of X and only the part of Y that is new.

The order changed a bit, probably because we stopped over delivering data.

Test Plan:
Ran and updated the tests. The next patch will actually add a test
for this code.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4911531

Signature: t1:4911531:1492643897:58aabe5a32f156dd7424cc07c990301dbd5dc1c7
2017-04-19 21:14:04 -07:00
Durham Goode
dae50fc99e treemanifest: support repacking revlogs into packs
Summary:
Accessing treemanifests in revlogs is incredibly slow. This patch adds the
ability for `hg repack` to repack the revlog content into a data pack file, and
teaches the getserverpack code to look in the pack file first.

Test Plan: Need to add a test. Sent out anyway for review

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901298

Signature: t1:4901298:1492640706:2850b9f0e9bbee77952f46af3b784aa81253e626
2017-04-19 21:14:04 -07:00
Durham Goode
42344a282f treemanifest: fix fncache integration
Summary:
Future tests showed that backfilltree was not populating the fncache, which
meant the files were not streamed during a clone or read during a repack. This
fixes this by using the actual repo store (which handles fncache serialization)
instead of a creating a new store.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4916070

Signature: t1:4916070:1492647129:4deeb70c63dd414f6d5c06e4259a0bbf899f3425
2017-04-19 21:14:04 -07:00
Durham Goode
3214c9a4df treemanifest: move revlogdatastore to contentstore.py
Summary: This is a generally useful store class. Let's move it to the store code.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901293

Signature: t1:4901293:1492647203:0f76a6e78fd0035b61c4e45e18cd9fce59359768
2017-04-19 21:14:04 -07:00
Durham Goode
b30215f840 treemanifest: change server to use ctreemanifest for reading revlogs
Summary:
The server treemanifest reading code was super slow because it used the pure
python treemanifest implementation. Let's switch it to use the ctreemanifest
implementation on top of a revlog store class.

The tests change because the output is now in reverse topological order (deepest
trees first).

Test Plan: Tests updated

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901268

Signature: t1:4901268:1492637601:371a2303ac4f6e2b81f69a85ca657849150abcf7
2017-04-19 21:14:04 -07:00
Durham Goode
8da9d7b37f treemanifest: automatically download missing trees on demand
Summary:
This adds a remote tree datastore to the store, so if we don't have a tree on
local disk, we can reach out to the server for it.

This also makes a small change to fastmanifest to use get() instead of
getmissing() to test if the tree is available. getmissing() would indicate the
tree is missing if it wasn't local, while get() will attempt to fetch it from
the server.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864645

Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-19 21:14:03 -07:00
Durham Goode
3633627260 treemanifest: add prefetchtrees for receiving trees from the server
Summary:
This adds a new prefetchtrees command and gettrees wireprotocol command for
receiving trees from the server. The actual wire protocol is a custom request
format, with a normal pack file response format (same as used in remotefilelog).
The request args can specify which manifest nodes are wanted, for which
directory and which subdirectories.

I attempted to use the changegroup3 response format instead of the pack format,
but changegroup is extremely intertwined with revlogs and with producing
changegroups for linear swaths of the repo history, instead of random parts of
just the manifest.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: stash, quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864435

Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-19 21:14:03 -07:00
Kostia Balytskyi
aeef0ad5cc compatibility: migrate from scmutil.vfs to mercurial.vfs.vfs
Differential Revision: https://phabricator.intern.facebook.com/D4908906
2017-04-18 14:42:33 -07:00
Durham Goode
0942e94dcc treemanifest: reverse historypack entries
History pack entries need to be in newest-to-oldest order, so let's reverse
them.
2017-03-20 18:58:09 -07:00
Durham Goode
efed1f1209 treemanifest: add tree generation and verification command
Summary:
This adds 'hg debuggentrees' that takes an optional --verify flag. This makes it
easier to debug tree creation and catch errors.

Test Plan:
Ran `PYTHONPATH=/data/users/durham/fb-hgext python ../hg/hg --pager=off  debuggentrees --verify master~xxxx master` to generate 6 months
worth of trees. It found bugs which are fixed in other patches.

Reviewers: #mercurial

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4724502
2017-03-18 19:38:45 -07:00
Durham Goode
67da0498fb treemanifest: block non-pushrebase pushes to treemanifest servers
Summary:
We are relying on pushrebase to replay each of the incoming commits so they
produce both flat and treemanifests. This patch blocks any commits that don't
come through pushrebase, by intercepting manifest.addgroup calls (which is what
a normal push would use) and throwing an exception.

Test Plan: Added a test

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4093745

Signature: t1:4093745:1478100170:79b1eea6668cd3efb27742323e11a45bde930440
2017-03-09 14:45:23 -08:00
Durham Goode
e2b422dc98 treemanifest: server - write trees at the same time as flat manifests
Summary:
When a server is a treemanifest enabled server, we want to ensure that every
commit that comes in will be added to the treemanifest as well as the flat
manifest. This patch accomplishes that by hooking into the addmanifest logic and
applying the new manifest to the tree as well.

Test Plan: Added a test

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4093738

Signature: t1:4093738:1478260442:58f77bd88ab09ccda4ac238a065388be8f4a271d
2017-03-09 14:45:23 -08:00
Durham Goode
911ea4ace5 treemanifest: add backfilltree command
Summary:
This adds a command for producing tree manifests for existing manifests in the
repository. The tree manifests are stored similar to how vanilla mercurial
stores them, but the root revs are in 00manifestree.i instead of 00manifest.i
and we fake the root node to be the same as the flat manifest node, so we can
reference either the flat or the tree easily.

Test Plan: Added a test. Also began converting fbsource

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4092252

Signature: t1:4092252:1478260394:a55360a8ba83ad85b1694cb93f0788f1f672ef61
2017-03-09 14:45:23 -08:00
Durham Goode
29dc858324 treemanifest: create a separate manifest revlog on the server
Summary:
As part of the migration, we need both flat and tree manifests to exist at once
on the server. To do this, we need to store the tree manifest root revisions
somewhere other than 00manifest.i.

This patch divides the treemanifest logic into client and server bits, and on
the server creates a repo.manifesttree property that contains the tree manifest.
Future patches will write to this to fill in new tree entries, and eventually
read from it to seed clients with tree data.

Test Plan:
Future patches add tests that for features that use this
functionality

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4092217

Signature: t1:4092217:1478260318:7edada2ea39a5de565955a4180bfad62f6126352
2017-03-09 14:45:23 -08:00
Durham Goode
70ce116529 treemanifest: add history data to tree repacks
Summary:
Previously, tree repacks did not take into account tree history. It would just
look at the delta base and if the base existed, it would just reuse the delta.
This would A) result in very long chains, and B) result in chains where the full
text was the oldest version, instead of the newest (recent full texts means
faster access to recent versions).

This patch threads tree history into the repacker, which already knows how to
use history for repacks.

Test Plan:
Updated the tests, and inspected the new test results to ensure tree
entries that were not deltas before the repack became reverse deltas during the
repack.

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4647359

Signature: t1:4647359:1488882710:dba72cf488766ce827b7641735164fa0efc9a303
2017-03-07 11:15:26 -08:00
Durham Goode
3f65bd2532 treemanifest: don't build pack files that depend on other pack files
Summary:
Previously, when treemanifest would create packs of trees during pull, we
allowed trees to be delta'd against trees in other packs. This resulted in
smaller packs, but if the other pack disappeared for some reason (since it's a
cache), the chain broke.

This patch ensures that the first version of every tree added to a pack is a
full text.

This temporarily makes repacks worse, since the repacker doesn't know about
history to produce deltas when combining packs. The next patch adds history
awareness which improves the repack deltafication.

Test Plan:
Updated the tests, and inspected the new test results to ensure that
all packs only had deltas within the pack.

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4647348

Signature: t1:4647348:1488882214:e850622a853a534fc60caeef604c88c30740c60d
2017-03-07 11:15:25 -08:00
Durham Goode
5bc368c71d treemanifest: move tree delta logic up to python layer
Summary:
Previously the treemanifest code itself would create the text deltas when
writing a tree out. This meant we couldn't make the delta decision based on
other data, like if the p1 commit was in the same pack file.

This patch removes treemanifest.write() and moves all calls over to
treemanifest.finalize() which gives the python/pack layer control over delta
choices. A future patch will use this to ensure tree packs always contain
complete delta chains.

Test Plan: All tests pass

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4645942

Signature: t1:4645942:1488880851:d0c8c902e7e849072a53344630a9184b6d8e1e7f
2017-03-07 11:15:25 -08:00
Durham Goode
943afede25 treemanifest: add support for creating history packs
Summary:
Previously the treemanifest auto-tree-creation logic only produced data packs
containing the actual contents of the tree blobs. This lost history information
which is important for our ability to efficiently repack the data files.

This patch creates history packs during pull as well. A future patch will also
create history packs for the local tree blob store.

Test Plan: Updated the tests to cover this

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4638865

Signature: t1:4638865:1488449992:48b60961b50b90b6d0e75a64af1f36fb29944e7a
2017-03-07 11:15:25 -08:00
Durham Goode
940796814d treemanifest: add option for using native store
Summary:
Adds a treemanifest.usecunionstore config flag for enabling and disabling use of
the native code uniondatapackstore.

Since we haven't implemented the repack APIs on the native datapack stores, we
currently have to force repack to use the old python implementations. Instead of
trying to expose just the appropriate APIs through the python interface, I think
we'll rewrite all of repack to be in C++ at a future time, since we can take
advantage of parallelism, etc.

Test Plan:
Updated test-treemanifest.t to use the c datapackstore. Also run all
the tests with --extra-config-opt=treemanifest.usecunionstore=True.

These tests caught a missing null check in the C++ code as well.

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4609795

Signature: t1:4609795:1488365341:203362db5f470b613c4d6484686cd32c3fa8458f
2017-03-01 16:55:19 -08:00
Durham Goode
2cd1eeb08e ctreemanifest: move treemanifest into cstore
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.

Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4602484

Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
2017-02-23 14:03:03 -08:00
Durham Goode
760ab3473c treemanifest: error out if invalid configuration
Summary:
treemanifest requires fastmanifest, and fastmanifest.usetree requires
treemanifest. Let's make these dependencies explicit in the code and error out
if they are incorrect.

Test Plan: Added a test

Reviewers: #mercurial, wez

Reviewed By: wez

Subscribers: wez, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4417512

Signature: t1:4417512:1484347447:7e18340813fac0b298aa51a7cc2f89fc6953680f
2017-01-13 14:58:20 -08:00
Durham Goode
d3c6def7b8 remotefilelog: move pack file permissions to be in mutablebasepack
Summary:
Treemanifest had a bug where the pack files it created were 400 instead of 444.
This meant people sharing a cache on the same machine couldn't access them. In
most of the remotefilelog code we had set opener.createmode before creating the
pack but we forgot to for treemanifest.

This patch moves the opener creation and createmode setting into the mutable
pack class so we can't screw this up again.

Test Plan:
Tests that wrote to the packs before, now failed and had to be
updated to chmod the packs before writing to them.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4411580

Tasks: 15469140

Signature: t1:4411580:1484321293:9aa78254677548a6dc2270c58cee0ec6f57dd089
2017-01-13 09:42:25 -08:00
Adam Simpkins
ab81106b08 [remotefilelog] don't crash on invalid pack files
Summary:
We have run into some cases where users ended up with empty pack file in their
packs directory.  Just log a warning in this case and skip this pack file,
rather than letting the exception propagate up and crashing the command.

Test Plan:
Created empty 0000000000000000000000000000000000000000.histpack and
0000000000000000000000000000000000000000.histidx files in my repository's
hgcache directory, and confirmed that "hg log" now simply warns about them
instead of crashing.

I didn't really test the perftest.py or treemanifest_correctness.py extensions
much.  They seem to throw exceptions, and look like they have maybe gotten a
bit stale.  I fixed one minor typo, but didn't dig into the other exceptions
too much.

Reviewers: durham

Reviewed By: durham

Subscribers: net-systems-diffs@, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4402516

Tasks: 15428659

Signature: t1:4402516:1484155254:96d2980efcec2d735257b08910e1ca437ef1dad6
2017-01-12 09:47:29 -08:00
Stanislau Hlebik
e03071ee64 treemanifest: fix unit tests
Summary:
In 676596f945ea2166820ef92e692ef7fe6a6247f0 were added comments with
lines > 80. In aec81a9a80d22989bbdc8c74c1dfec9dcbbe6866 default config value
was changed.

Test Plan: arc unit

Reviewers: #sourcecontrol

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4377188
2017-01-03 05:09:06 -08:00
Durham Goode
8041fb2a9c treemanifest: debug verification refactor 2016-12-31 18:22:38 -08:00
Durham Goode
5f0176753a treemanifeset: fix auto-tree creation for merges
Previously we were relying on mfrevlog.revdiff() to produce the delta for us.
This only showed us what files were added/modified, as compared to p1, and we
had to use a heuristic to know what files were deleted (by looking at the list
of files in the commit metadata). Merge commits have a different criteria for
what is in the commit metadata (it only contains entries where the file is
different from both parents), so we can't use it for the same heuristic. So
let's fall back to a normal manifest diff for merge commits, since they are
rare.

Adds a test for verifying that conversion of merge commits into a tree works.
2016-12-31 18:22:38 -08:00
Durham Goode
bab8d2e0a5 treemanifest: remove allfiles optimization
Commit 24515b72d5 attempted to optimize writes by checking if the file in the
delta was also in the list of files in the commit metadata. This doesn't work for merge
commits since the only files the commit metadata are the ones that differ from
both (so set(changes in metadata) != set(changes in node-diff-against-p1)). This
caused the verify code to catch the issue. The fix is to just remove the
allfiles optimization.
2016-12-12 18:59:45 -08:00
Durham Goode
51ae957ffc treemanifest: add simple test for tree repack
Summary:
This adds a simple test that verifies hg repack will pack two tree manifest
packs into one.

It caught a bug where creating a treemanifest for a commit with a null parent
produced incorrect output because it constructed an empty tree and tried to use
it's node as the parent of the delta, when there should not have been any delta
in the first place. This is fixed by this diff as well.

Test Plan: Ran the new test

Reviewers: #mercurial, dsyang, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4261591

Signature: t1:4261591:1480705822:ef21fb8cebd8b89f92f58f11bb1dab59bf97664d
2016-12-02 14:37:55 -08:00
Durham Goode
fe149d0caa treemanifest: set repo.name
Summary:
Since treemanifest uses the same storage locations as remotefilelog, it needs
access to the repo name as well. If a given repo has treemanifest enabled but is
not a remotefilelog repo, it won't have the repo.name member already. So let's
add it ourselves in treemanifest.

Eventually we should probably refactor this out to be a more global concept of
repo name.

Test Plan: A future patch adds a test that caught this

Reviewers: #mercurial, dsyang, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4261581

Signature: t1:4261581:1480705457:8fb4b86ce8abeed62cc7c8f787868359c3cf8abc
2016-12-02 14:37:51 -08:00
Durham Goode
046fb2ca08 repack: enable repacking of local manifest stores
Summary:
Previous hg repack would only repack the shared manifest cache store. This patch
makes it also repack the local manifest store too.

Test Plan:
Made a local commit in my test repo with treemanifests enabled.
Rebased the commit to master so now there were two local tree packs. Ran hg
repack and verified they were combined into one pack in
.hg/store/packs/manifests

I'll add a test later today

Reviewers: #mercurial, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4260580

Signature: t1:4260580:1480696151:8f989e299dda50281ca63489e870202eb195d714
2016-12-02 14:37:45 -08:00
Durham Goode
a0dc16174d treemanifest: write deltas for trees
Summary:
Previously, when we wrote each tree entry into a pack file, it wasn't delta'd in
any way. This patch makes it store the delta against p1 in the pack file.

Testing in a large repo shows this reduces tree pack size by about 22x.

Test Plan:
Ran the tests. Did a pull in a large repo and saw the pack file was
22x smaller than before (and still usable).

Reviewers: #mercurial

Differential Revision: https://phabricator.intern.facebook.com/D4202088
2016-11-29 15:37:58 -08:00
Durham Goode
3ad41b8578 treemanifest: limit autocreatetrees to commits on master
Summary:
Creating a tree for a commit whose parent is not already a tree is expensive.
Let's optimize the autocreatetrees option (which converts manifests to trees
during pull) to only create trees if A) the parent is already a tree, or B) the
parent is master. This way we only pay the expensive part once. It also means
that as new branches fork off master, they will be trees too, since all commits
in the new branch will meet criteria (A).

Test Plan:
Ran hg pull in a large repo over a large pull with 40 different
branches and verified it only paused for the initial master commit.

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4129010

Signature: t1:4129010:1478260239:1698ada4e3c6a38ab77a94317e75daee4812276f
2016-11-16 13:51:48 -08:00
Durham Goode
b1109deca8 treemanifest: update to work with manifestlog
Summary:
Upstream has refactored the manifest class into several classes, so we need to
update treemanifest to work with the new structure. Notably, the factory add
function previously relied on the ability for the revlog class to create a new
manifestdict (via manifest.maniest.read()), since this isn't possible anymore,
we have to construct the hybridmanifest ourselves and provide an appropriate
loadflat function to get the flat manifest if necessary.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4180891

Signature: t1:4180891:1479285991:82bc546a1eb682d3cfd8b4724bda575410405d0f
2016-11-16 12:11:15 -08:00