Which packs get cleaned up is non-deterministic in some cases, so grepping for
the 'x' in this test did not catch when a non-x pack was left behind, which
caused flakey output. Let's instead check that 3 packs were left behind and one
of them was from the small generation.
If a bunch of packs are the same size, the ones repack chooses to pack is
non-deterministic. This was causing flakiness in the test as different packs
were being left behind. Let's remove that dependency.
Summary:
Previously incremental repack would only take the top 3 packs in a generation of
packs. For instance, if there were 100 packs between 1KB and 1MB, it would only
repack the first 3, instead of repacking a whole bunch because it's cheap. This
patch fixes it to grab as many packs in a generation as it can fit under the
limit (default of 100MB of pack data).
This will fix an issue where we're seeing users with 100's of tiny pack files
that aren't getting repacked fast enough.
Test Plan: Added a test
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Subscribers: quark
Differential Revision: https://phab.mercurial-scm.org/D500
Summary:
Previously the code required that sizes be of type int. Since python plays loose
with integer types, we also need to support long.
Test Plan:
The existing test-remotefilelog-repack-fast.t test was completely
broken. It only enabled fast datapacks for the server repo, not the clients.
Enabling it for the clients as well catches this issue and verifies the fix.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D54
Summary:
If the repo being repacked has commits being pushed, it's possible that the
repack will encounter file revisions who's linkrevs don't exist in the inmemory
changelog. Let's set an upper bound on what linkrevs to repack so we can only
process revisions who's commits we can see.
Test Plan: Added test
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5279171
Signature: t1:5279171:1497911997:a48c56abebd14a1c066c9fc1ee4098f813d062df
Summary:
Enhance check-ext script to be more strict:
- Only one foreign extension is allowed: `remotenames`
- Require explicit path for in-repo extensions to avoid wrong extensions
being tested
This would make the test more predicatable since system extensions
will be less likely to be imported. Explicit path is better than
setting `PYTHONPATH` since `hgext/name.py` could override
`hgext3rd/name.py` regardless of `PYTHONPATH`.
Test Plan: arc unit
Reviewers: phillco, durham, ikostia, #mercurial, stash
Reviewed By: stash
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5271430
Signature: t1:5271430:1497861776:7dd35ec7c522cd9b26aa0871cb4306b4f1b8993a
This test had non-deterministic ordering, which caused it to fail sometimes.
Let's make the test check more explicitly, by checking that the bad commit hash
is not present.
Summary:
When the server performs a repack, it would read all the data from all the
revlogs. This was very slow and expensive. Let's add a --incremental option that
makes it only read the revlog entries that have a linkrev newer than the latest
linkrev data that's already in a pack file.
Test Plan: Adds a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4997260
Signature: t1:4997260:1493904216:c4f5b6c9652bbf8f66c1bc2b2547c898324d22cd
Summary:
This patch just bumps the histpack version number to 1 and adds a config flag to
enable writing v1 pack files. The format hasn't actually changed in this patch,
I'm just doing the verison bump so I can update all the hashes in the tests
without working about functionality change.
In the next patch I will modify the index format, which won't affect the hashes.
Test Plan:
Ran the tests. I also ran the tests with some debug code to manually
force the sha to include 0 instead of 1 and verified that the hash didn't change
(which confirms that all of these hash changes are just because of that one byte
version change).
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4983675
Signature: t1:4983675:1493692444:5d88df4d46ce487f1b791417754ba000ecf10a1e
Summary:
Previously, a local commit would only write data packs, and it just threw away
the history data entirely. Let's add history packs and record them on commit.
Test Plan:
The tests are updated to show these new packs. In some cases the
datapacks got smaller as well, since they can now take advantage of history data
for delta choices.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956105
Signature: t1:4956105:1493265399:d3fa1052c207fba0045cbb92b4d833d18d48e099
Summary:
This diffs add a `getmeta` method to all content stores. The cdatapack code is
modified to pass the tests, it needs further change to support `getmeta`.
The datapack format is bumped to v1 from v0. For v1, we append a `metadata`
dict at the end of each revision. The dict is currently used to store revlog
flags and rawsize of raw revlog fulltext. In the future we can put more data
like a second hash etc, without changing API or format again.
This diff focuses on correctness. A datapack caching layer to speed up
`getmeta` will be added later.
Tests are updated since we write new v1 packfile now and the format change
leads to different content and packfile names.
`Makefile`, `ls-l.py` are added to make tests easier to maintain.
Test Plan: Updated existing tests.
Reviewers: #mercurial, rmcelroy, durham
Reviewed By: durham
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4903917
Signature: t1:4903917:1493255844:7ef5d487096cd2f78f2aaae672a68d49f33632ee
Summary:
Accessing treemanifests in revlogs is incredibly slow. This patch adds the
ability for `hg repack` to repack the revlog content into a data pack file, and
teaches the getserverpack code to look in the pack file first.
Test Plan: Need to add a test. Sent out anyway for review
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901298
Signature: t1:4901298:1492640706:2850b9f0e9bbee77952f46af3b784aa81253e626
Summary:
Future tests showed that backfilltree was not populating the fncache, which
meant the files were not streamed during a clone or read during a repack. This
fixes this by using the actual repo store (which handles fncache serialization)
instead of a creating a new store.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4916070
Signature: t1:4916070:1492647129:4deeb70c63dd414f6d5c06e4259a0bbf899f3425
Summary:
The name being passed to the store was wrong, because it had a trailing slash.
This wasn't caught before because the code was reading and writing the paths
with a slash at the end, so it matched as long as we only interacted with packs
produced by the code. The issue became more obvious when I tried to have packs
generated from revlogs interact with this code.
All the tests are affected since the entry keys changed.
Also use 'const ManifestFetcher &x' to pass the ref to avoid the copy.
Test Plan: Tests updated
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901274
Signature: t1:4901274:1492638476:ff28f8976657baec99effbd82ecd436f6282ea5b
Summary:
The treemanifest tests were setting the configs on both the client and the
server, without telling the server it was a server and without setting it up
appropriately to be a client. This caused the tests to fail in a future patch.
Let's move the treemanifest config to the clients, since that's the only bits
being tested here.
Test Plan: Ran the tests
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4864766
Signature: t1:4864766:1491903651:ab91eed5a2eb8a60db8e230c2ba5c0ee29b5adc9
Summary:
Previously, tree repacks did not take into account tree history. It would just
look at the delta base and if the base existed, it would just reuse the delta.
This would A) result in very long chains, and B) result in chains where the full
text was the oldest version, instead of the newest (recent full texts means
faster access to recent versions).
This patch threads tree history into the repacker, which already knows how to
use history for repacks.
Test Plan:
Updated the tests, and inspected the new test results to ensure tree
entries that were not deltas before the repack became reverse deltas during the
repack.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4647359
Signature: t1:4647359:1488882710:dba72cf488766ce827b7641735164fa0efc9a303
Summary:
Previously, when treemanifest would create packs of trees during pull, we
allowed trees to be delta'd against trees in other packs. This resulted in
smaller packs, but if the other pack disappeared for some reason (since it's a
cache), the chain broke.
This patch ensures that the first version of every tree added to a pack is a
full text.
This temporarily makes repacks worse, since the repacker doesn't know about
history to produce deltas when combining packs. The next patch adds history
awareness which improves the repack deltafication.
Test Plan:
Updated the tests, and inspected the new test results to ensure that
all packs only had deltas within the pack.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4647348
Signature: t1:4647348:1488882214:e850622a853a534fc60caeef604c88c30740c60d
Summary:
This adds a test for hg repack --incremental handling tree packs. It fixes a few
bugs that were caught in the process:
1. Since remotefilelog was being imported via it's file path, it was being
loaded into the process as hgext_remotefilelog. When treemanifest loaded it into
the process via 'import remotefilelog' it was being imported as 'remotefilelog'.
This meant we had the same types imported into the same process with two
different names, which meant 'isinstance()' checks could fail when they
shouldn't (which affects incremental repacks). So we just drop the filepath.
2. We need to allow repacking local tree manifest data even if the full delta
chain isn't available (since part of the delta chain may be in the cache). So we
add allowincomplete to the data pack in this case.
Test Plan: Ran it
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4262412
Signature: t1:4262412:1480706110:45bb0a1e1b031f9cfd4658a5071bbac5df2f6543
Summary:
Adds a test for making sure hg repack handles local pack files in
.hg/store/packs
Test Plan: Ran it
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4261788
Signature: t1:4261788:1480705839:efb9657654bf6d31a3ec1d1125277174cbc93276
Summary:
This adds a simple test that verifies hg repack will pack two tree manifest
packs into one.
It caught a bug where creating a treemanifest for a commit with a null parent
produced incorrect output because it constructed an empty tree and tried to use
it's node as the parent of the delta, when there should not have been any delta
in the first place. This is fixed by this diff as well.
Test Plan: Ran the new test
Reviewers: #mercurial, dsyang, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4261591
Signature: t1:4261591:1480705822:ef21fb8cebd8b89f92f58f11bb1dab59bf97664d
