Commit Graph

287 Commits

Author SHA1 Message Date
Olivier Trempe
abf6481106 windows: grp module not supported
On windows the grp module is not present, so we need to avoid importing it. This
means the shared group feature of remotefilelog is not supported on windows.
2016-05-20 08:08:57 -07:00
Olivier Trempe
a1a32e5f37 windows: use binary mode for reading and writing files
On windows, there is text mode and binary mode for reading and writing files.
Since we're dealing with files as just blob data, we always want binary mode.
2016-05-20 08:08:57 -07:00
Martin von Zweigbergk
41226c0217 debugindex: use normal handling for --dir
'debugindex --dir' is used for looking up the revlog of a directory
manifest, so it should use the normal (local file system) handling,
not remotefilelog.
2016-01-04 21:32:21 -08:00
Durham Goode
56411349f7 datapack: make fanout size dynamic
Summary:
Previously all fanout tables were 2^16 in length. For small packs, this resulted
in very sparse fanout tables that had to be linearly scanned to find the end of
the search bounds, which was slow.

With this patch, the data index now dynamically chooses what fanout table size
to use.  If the pack has over 2^16 / 10 entries, we use 2^16, otherwise we use
2^8. The reasoning is in the code.

The patch is a bit large because we had to take a bunch of constants and
duplicate them, and change their accessors to access them through member
variables.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3279000

Signature: t1:3279000:1463170357:9746d19dde14743bac9a8c40cafbc618504c420f
2016-05-16 10:59:09 -07:00
Durham Goode
c9621d3d1a repack: don't require complete history during data repack
Summary:
Previously, when repacking deltas we would require a full history of the node so
we could order the hashes optimally. In some situations though, we don't have
the full history available (like if we're only repacking a subset of packs), so
we need to be able to repack even without full history.

This patch handles the case where a given delta doesn't have history
information. We just store it as a full text.

This becomes useful in an upcoming series that will introduce incremental
packing that only packs a subset of the packs.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3278346

Signature: t1:3278346:1463086170:54c0fbefe78f9cafa7efc4b6f037887a924ab4a5
2016-05-16 10:59:09 -07:00
Durham Goode
9c3aa14c26 repack: remove copyfrom detection
Summary:
In an old version of the code, we would walk the entire history of a node during
data repack, which meant we had to keep track of when we saw a rename, and stop
walking there. Since then, we've changed the code to no longer walk the entire
history, and instead walk just the parts it was told to repack for this
particular file. This means we no longer ever walk across a copy, and therefore
don't need this copy detection logic.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3278443

Signature: t1:3278443:1463086137:c6d9eb6637bf3b8636a3df7e531f265d51cab0de
2016-05-16 10:59:09 -07:00
Durham Goode
12ec6ebcb4 store: force remote stores to always check the shared cache
Summary:
In a remotecontent/metadatastore a `get` request first runs prefetch, then reads
the resulting data from the shared cache store. Before this patch, the prefetch
would not download a value if it existed in the local data store, which means
nothing would be added to the shared cache store, and the `get` would fail. This
patch changes the remote stores to always prefetch based only on the contents of
the shared cache, so data will always be written.

This issue showed up when attempting to repack pack files that contained
references to nodes that were in the local store (which it didn't have access
to) but not the shared cache.

Test Plan:
Manually verified my issue disappeared. This isn't actually an issue
anymore, since future patches refactor the way repack works to not rely on the
remote stores, so this shouldn't be hit again. But it's a safe change
regardless.

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3278362

Signature: t1:3278362:1463086099:987d2fdd1c75e518f815c3159473e8cb22a15ba0
2016-05-16 10:59:09 -07:00
Durham Goode
e13e9ef243 historypack: fix handling of section lookup key errors
Summary:
In the old days _findsection would return None if the section wasn't found. We
have since changed it to throw a KeyError like all the other operations in the
packs. We need to update getmissing to eat that error like usual.

Test Plan: ran the repack that was failing, it succeeded.  Added a unit test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277245

Signature: t1:3277245:1463085831:0b32f852f49bd45ef2dfd3298313b9c2b87f75b6
2016-05-16 10:59:09 -07:00
Durham Goode
a5ed23b7ca fileserverclient: separate data and history in prefetching
Summary:
Previously the fileserverclient logic only checked the data store when
determining whether to prefetch a given key or not. This meant that if the file
was in the data store but not the history store, it could result in a history
lookup failing to prefetch.

The fix is to separate the notions of data and history in the prefetch logic. By
default we just check the data store like before, but an optional argument
allows us to specify checking the history store as well (and we change the
remotemetadatastore to pass that argument).

Test Plan:
Ran the tests. Also ran the repack scenario in my large repo that
reproed the issue. In another diff, I'm going to come back and add a suite of
tests around the various repack permutations that I've seen cause issues.

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277239

Signature: t1:3277239:1463085690:49ad478048cd13836b60f7ac9190e2294f5e9c64
2016-05-16 10:59:09 -07:00
Durham Goode
5bb799ed73 prefetch: add progress to pack prefetch
Summary:
Add a simple progress bar on the client when receiving a pack from the
server.

Test Plan: Ran it

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277265

Signature: t1:3277265:1463085643:aa0960092958c4f56a6d1c3a3901348dba48aa91
2016-05-16 10:59:09 -07:00
Durham Goode
5e4370b46d packs: add debug commands to view pack contents
Summary: Some simple debug commands to print the contents of each pack.

Test Plan: Ran it manually, and added a simple test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277233

Signature: t1:3277233:1463085196:c54fc875d536a96150bb1461b77247a5d7a9402c
2016-05-16 10:59:09 -07:00
Durham Goode
ad5352432e packs: change iterkeys() to __iter__
Summary:
In a future patch we'll add debug commmands to view the contents of packs. Let's
repurpose iterkeys to just be __iter__ and have it return most of the data about
each entry.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Subscribers: mitrandir, quark

Differential Revision: https://phabricator.intern.facebook.com/D3277225

Signature: t1:3277225:1463170015:c1acbc7f435f1cf8d07fea4f32bf742a22de5716
2016-05-16 10:59:09 -07:00
Durham Goode
e617c24532 datapack: add index marker for no delta base
Summary:
Previously, we assumed every delta in a pack had a complete chain in that pack,
so the index had no way to indicate a deltabase offset that didn't exist in the
pack.

Let's add a new marker to indicate that the delta base doesn't exist in this
pack.

Test Plan: Tested in my large repo repack scenario.  Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277222

Signature: t1:3277222:1463084346:1687cfa174e98c2cf3022de9e9c3808881f689cd
2016-05-16 10:59:09 -07:00
Durham Goode
05ceb8b419 store: basic wire protocol for bundle delivery
Summary:
This adds a new wire protocol command to allow clients to request a set
of file contents and histories from the server and receive them in pack format.
It's pretty simple and always returns all the history for every node requested
(which is a bit overkill), but it's labeled v1 and we can iterate on it.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277212

Signature: t1:3277212:1463421279:459cc84265502175b47df293647aab7e7a830185
2016-05-16 10:59:09 -07:00
Durham Goode
2a938a761c store: add copyfrom information to history index
Summary:
Previously we were throwing away copy information when we repacked things into
pack files. The hope was that we could store copy information somewhere else,
and keep the history pack using fixed length entries. Since storing copy
information elsewhere is a long ways off, let's just go ahead and put copy info
in the pack file.

This makes the entries non-fixed length, which means any iteration over them has
to read the length of each entry. This also affects the historypack filename
hashes since they are content based, so the tests had to change.

This matches the old remotefilelog behavior more closely (which is why no code
had to change outside the pack logic).

Test Plan: Added a test

Reviewers: #mercurial, mitrandir, ttung

Reviewed By: mitrandir

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3262185

Signature: t1:3262185:1462562602:935683692276c7fa569d381b18aa3b18656793b1
2016-05-16 10:59:09 -07:00
Durham Goode
24c9c438d3 history: remove dependency on 80 character pack entry
Summary:
In a future diff we will be changing the format of history pack entries
to have a variable length file path in them. Before doing that, let's remove the
places that depend on each entry being exactly 80 characters.

Test Plan: Ran the tests

Reviewers: #mercurial, mitrandir, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3262154

Signature: t1:3262154:1462820474:b7f5666001b3b7f8863b4de4826266204f3e87aa
2016-05-16 10:59:09 -07:00
Durham Goode
e4fb7d66bb history: remove getparents and getlinknode apis
Summary:
These APIs weren't actually used, and the questions can be answered via
the existing getancestors() api anyway.

They were originally put in place because they are the type of question that
doesn't require the full ancestor tree, so we could answer them without doing in
traversal. In an upcoming patch we add the concept of copyfrom back into the
historypack, and getparents becomes confusing since it doesn't expose knowledge
of copy information. So I just decided to delete it all until we need it.

In the future we may want a 'gethistoryinfo(filename, node)' api that just
returns (p1, p2, linknode, copyfrom), to fulfill that original need of history
information without a full ancestor traversal.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3261734

Signature: t1:3261734:1462413665:987c4703e53468a75346aa323188107a5c070fde
2016-05-16 10:59:09 -07:00
Durham Goode
e7a294e65f utils: change writefile to use custom atomic temp logic
Summary:
The mercurial atomic temp logic makes some assumptions about modes and copies
the mode bits from the existing file. If the existing file is readonly, that
means the atomic temp fails to open the file for writing after it's initially
created.

Since we only actually need like 4 lines from the atomic temp code, I just
implemented it on our end, with custom logic for mode handling.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, quark

Reviewed By: quark

Differential Revision: https://phabricator.intern.facebook.com/D3288589

Signature: t1:3288589:1462993609:653a0a63266b9129b0e18807858e5de02310ecf9
2016-05-11 13:33:06 -07:00
Durham Goode
ce493adf83 repack: add --background option
Summary:
This allows triggering a repack that can be run in the background. In the future
we will trigger this automatically under certain circumstances (like too many
pack files).

Test Plan: Added a test

Reviewers: #mercurial, ttung, quark

Reviewed By: quark

Subscribers: quark

Differential Revision: https://phabricator.intern.facebook.com/D3261161

Signature: t1:3261161:1462398568:5ae25f3e5a9acd0f4b34490b34a62be33cc69e3c
2016-05-04 14:53:23 -07:00
Durham Goode
8e290a5d4a repack: add lock to limit it to only one repack
Summary:
This adds a lock that limits us to running only one repack at a time. We also
add a simple prerepack hook to allow the tests to insert a sleep to test this
functionality.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3260428

Signature: t1:3260428:1462393311:3e1bf5dd047e7f3521679ca7640b448f5e784913
2016-05-04 14:53:19 -07:00
Durham Goode
c2cdcde2fb store: read recent packs first
Summary:
Since recent packs are likely to contain more recent data, let's put them at the
front of the pack list so they are checked first.

In a future diff I'll come back and refact the common code between datapack and
historypack into one base class.

Test Plan:
Ran the tests. Used the debugger to verify that the sort order was
correct.

Reviewers: #mercurial, ttung, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3259823

Signature: t1:3259823:1462388395:8ee48a7b02c6abc079878e53c5b336675249cb91
2016-05-04 14:53:16 -07:00
Durham Goode
1621511aa7 store: a number of performance improvements
Summary:
Some miscellaneous perf improvements:
- Get rid of unnecessary sorting
- Bail early from remotefilelog reverse filename lookup if there are no keys
- Avoid unnecessary class/enum lookups in ancestor hotpath
- Fix n^2 behavior in history repacking
- Remove unnecessary set() and tuple instantiation in hotpath
- Use __slots__ on repackentry to improve access times

Test Plan:
Ran the tests. Tested the actual perf improvements via 'hg repack' in
a large repo.  The n^2 fix caused a massive perf when, but the others shaved off
a number of seconds as well.

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3257595

Signature: t1:3257595:1462392755:80209fe66c2632d2c16a2840500b0655926e7513
2016-05-04 14:53:13 -07:00
Durham Goode
99a426f3eb store: fix fanout logic for history pack
This fix was already applied to datapack a while ago. Basically, if the pack
doesn't have a lot of revisions, it's possible that the fanout table is sparsely
populated. Therefore we need to scan forward when looking for the end bounds of
our fanout table.
2016-05-03 15:41:57 -07:00
Durham Goode
7da17af64f store: record what files were created during a repack
Summary:
Previously, if you ran repack twice in a row, it would actually delete your
packs, because the repack produced files with the same name as before, and the
cleanup then deleted them.

The fix is to have the stores record what files they produced in the ledger,
then future clean up work can avoid deleting anything that was created during
the repack.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3255819

Signature: t1:3255819:1462393814:d32155b12535990f72fbe48de045eddbb6f7fab6
2016-05-04 14:53:10 -07:00
Durham Goode
30cc85653c store: make pack files read-only
Summary:
Since pack files should never change after they are created, let's create them
with read-only permissions. It turns out that the Mercurial vfs doesn't apply
the correct permissions to files created by mkstemp (and we have to use mkstemp
since we don't know the name of the file until after we've written all the data
to it), so we have to manually call the permission fixing code.

We also need to fix our mmap calls to be readonly now, otherwise we get a
runtime permission denied exception.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3255816

Signature: t1:3255816:1462321201:dff4fb4c9301d67a77043ecc1d96262bb5d6a54a
2016-05-04 14:53:07 -07:00
Durham Goode
1d4b4dbb36 store: switch mutable packs to use openers
Summary:
Instead of passing in a path and performing joins ourselves, let's use an
opener. This will help handle all the file permission edge cases.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3255165

Signature: t1:3255165:1462393836:38a28c850a0dc06838d9c17672d3dffd9903bbd7
2016-05-04 14:53:04 -07:00
Durham Goode
0671f3d76a store: add version header to index
Summary: Pretty straight forward

Test Plan: Ran the tests

Reviewers: lcharignon, rmcelroy, ttung, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3254892

Signature: t1:3254892:1462315520:888bb27ef121c08d463f9fd4cf9eeb3c42383a96
2016-05-04 14:53:01 -07:00
Durham Goode
d71eace818 store: refactor version number and size to constants
Summary:
A future patch will add a version number to the index file. Let's move the
version size, fanout start, and index start to constants so we can more easily
change them without changing the code.

Test Plan: Ran the tests

Reviewers: lcharignon, rmcelroy, ttung, mitrandir, quark

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3254876

Signature: t1:3254876:1462315858:63fe56e8cfcdbb0209861898ce0c45c7d7b33e35
2016-05-04 14:52:58 -07:00
Durham Goode
4aa798d76e store: don't allow gc to delete pack files
Summary:
`hg gc` is very aggressive in that it deletes any files in the cache that it
determines aren't a needed key, including files it doesn't recognize. Let's
teach it to not delete pack files.

In the future we'll need to make `hg gc` able to garbage collect the contents of
pack files as well.

Test Plan: It's probably fine!

Reviewers: lcharignon, rmcelroy, ttung, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3254744

Signature: t1:3254744:1462308257:c1f932b88abf3337370f16c05c789422ea51b0e1
2016-05-04 14:52:55 -07:00
Durham Goode
312acdb24e store: implement markledger and cleanup for historypack
Summary:
Implementing these two functions allows historypacks to be repacked, either into
a new format, or by combining multiple packs into a single new one.

Test Plan: Added a test in my next patch

Reviewers: lcharignon, ttung, rmcelroy, mitrandir, quark

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251542

Signature: t1:3251542:1462392294:f95f7666a3a5df675f1351a19af7532c4742af2b
2016-05-04 14:52:49 -07:00
Durham Goode
d86ae1e525 store: implement markledger and cleanup for datapack
Summary:
Implementing these two functions will allow datapack's to be repacked (either
into other formats, or by combining multiple packs into one).

A future patch will add a test.

Test Plan: Added a test in a future patch

Reviewers: lcharignon, ttung, rmcelroy, mitrandir, quark

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251539

Signature: t1:3251539:1462393256:7caa09677fbcaaf57a47d7a833684883483c5b3a
2016-05-04 14:52:46 -07:00
Durham Goode
6b25f32192 store: add revision count to historypack filesection
Summary:
Previously, given a historypack file, we had no way of reading the contents,
since we had no way to know when to stop reading the revision entries for a
given file section.

This patch changes the format to have a revision count value after the filename
and before the revisions. The documentation already documented the format like
this, and therefore doesn't need updating.

A future patch will use this information to iterate over all the revisions in
the pack.

Test Plan: Added a test in a future patch

Reviewers: lcharignon, ttung, rmcelroy, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251538

Signature: t1:3251538:1462393282:f46b50e79237bfa8a25ff1957344588622b2699a
2016-05-04 14:52:43 -07:00
Durham Goode
548ccdeae1 store: make historypack file section writing lazier
Summary:
In a later patch we will need to add the count of revisions in a given file
section to the on-disk format. To make that easier, let's make the file section
serialization lazy, so that we will have the full list when it comes time to
count the entries.

Test Plan: added a test in a future patch

Reviewers: lcharignon, ttung, rmcelroy, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251537

Signature: t1:3251537:1462393274:60b72a47de45f5a94f4f5a8d34b3942db0aa3fda
2016-05-04 14:52:40 -07:00
Durham Goode
a54d3f1257 store: make repack only repack the shared cache
Summary:
Previously, hg repack would repack all the objects in all the store and dump the
new packs in .hg/store/packs. Initially we only want to repack the shared cache
though, so let's change repack to only operate on shared stores, and to write
out the new packs to the hgcache, under the appropriate repo name.

In a future patch I'm going to go through all this store stuff and replace all
uses of os.path and direct file reads/writes with a mercurial vfs.

Test Plan:
Ran repack in a large repo and verified packs were produced in
$HGCACHE/$REPONAME/packs

Ran hg cat on a file to verify that it read the data from the pack and did not do any remotefilelog network calls.

Reviewers: lcharignon, rmcelroy, ttung, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3250213

Signature: t1:3250213:1462315927:694661795141e2c869ba661a54cea8f4b90823df
2016-05-04 14:52:33 -07:00
Durham Goode
cfd60406ad store: add context manager to mutable pack classes
Summary:
Previously, if a repack failed, it would leave temporary pack files laying
around. By adding enter/exit functions to mutable packs, we can guarantee
cleanup happens.

Test Plan: Ran repack, verified that a failure did not leave tmp files

Reviewers: rmcelroy, quark, ttung, lcharignon, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3250201

Signature: t1:3250201:1462234552:7f20260a193ed1dd858bf6e9f489ac902d859218
2016-05-03 12:34:45 -07:00
Durham Goode
d2e7ae7519 store: make repack command use new repacker
Summary:
Now that all the repack logic is in place, let's switch the repack
command to use the new version. This also means the repack command will now
clean up the old remotefilelog blobs once it's finished.

Test Plan:
Ran hg repack in a large repo. Verified it deleted the old
remotefilelog blobs, and verified that I could still updated around the
repository without making any remotefilelog network requests.

A future diff will add standard .t mercurial tests for the repack command.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249601

Signature: t1:3249601:1462235506:03c0d95f6a82cfc04b340b139f39c02853941a17
2016-05-03 12:34:09 -07:00
Durham Goode
4ecb47b021 store: move history repack logic to repacker
Summary:
We had a naive repack implementation in historypack.py. Let's move it to the
repack module and do the minor adjustments to use the new repackerledger apis.

Test Plan:
Ran hg repack in conjunction with future diffs that make use of this
api

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249587

Signature: t1:3249587:1462232544:591cd8bec09f781370896470746eae5a4489531f
2016-05-03 12:33:54 -07:00
Durham Goode
735aa964d5 store: move data repack logic to repacker
Summary:
We had a naive repack implementation in datapack.py. Let's move it to the repack
module and do the minor adjustments to use the new repackerledger apis.

Test Plan: Ran it in conjunction with future diffs that make use of this api.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249585

Signature: t1:3249585:1462232504:a00aa65afca9562a2c1456cc4ab48c50d1ba5b68
2016-05-03 12:33:36 -07:00
Durham Goode
c902797dc9 store: implement markledger and cleanup on stores
Summary:
This implements the new markledger and cleanup apis on the existing
remotefilelog stores. These apis are used to tell the repacker what each store
has, and allows each store to cleanup if its data has been repacked.

Test Plan:
Ran repack in conjunction with the future diffs that make use of
these apis.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249584

Signature: t1:3249584:1462226133:1e8faffc9f6bf8f7c94e6e79aee8865e3c41648c
2016-05-03 12:33:00 -07:00
Durham Goode
c429030ca4 store: add class definitions and stub for repack
Summary:
This introduces the high level classes that will implement the generic repack
logic.

Test Plan: Ran the repack in conjunction with later commits that use these apis.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249577

Signature: t1:3249577:1462225435:000f9cc29ae2a3d7fdbedf546c8936ef45d1e4cf
2016-05-03 12:32:35 -07:00
Durham Goode
b049a0910a store: datapack fix perf issue
Summary:
Using range() allocates a full list, which is 2**16 entries in the fanout case.
Let's use xrange instead. This is a notable performance win when checking many
keys.

Also removed an unused variable and use index instead of self._index since this
is a hotpath.

Test Plan: Ran hg repack

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249563

Signature: t1:3249563:1462240834:c19d6cbf0b6237f15ca8d81e8da856752df0ec59
2016-05-03 12:30:44 -07:00
Durham Goode
1704e5c8fb store: add tests for historypack
Summary:
This adds a basic test suite for the historypack class, and fixes some issues it
found.

Test Plan: ./run-tests.py test-historypack.py

Reviewers: mitrandir, rmcelroy, ttung, lcharignon

Reviewed By: lcharignon

Differential Revision: https://phabricator.intern.facebook.com/D3237858

Signature: t1:3237858:1461884966:c0ec90a2735255e5ef70eade09915066a7b71ee5
2016-04-28 17:37:03 -07:00
Durham Goode
8f4d83edeb shallowbundle: fix broken fallback orig call
This was caught by tests running in an unusual configuration
2016-04-28 17:34:08 -07:00
Durham Goode
22948ce7e1 checkcode: add check code test
Summary: Adds the same check code test that upstream Mercurial uses.

Test Plan:
Ran it, and fixed all the failures. I won't land this commit until
all the failure fixes are landed.

Reviewers: #sourcecontrol, ttung, rmcelroy, wez

Reviewed By: wez

Subscribers: quark, rmcelroy, wez

Differential Revision: https://phabricator.intern.facebook.com/D3221380

Signature: t1:3221380:1461802769:19f5bdc209c05edb442faa70ae572ce31e2fbc95
2016-04-28 10:18:47 -07:00
Durham Goode
29d3dda67e checkcode: fix various store files
Summary: Fix check code for various store related files

Test Plan: Ran the tests

Reviewers: #sourcecontrol, mitrandir, ttung

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3222465

Signature: t1:3222465:1461701300:34560288be4dc921f0252d4ad8fdc9c8d9357e23
2016-04-27 16:49:33 -07:00
Durham Goode
98fd33f8cb store: add missing imports
Summary: These were missing, and only needed in exception cases.

Test Plan: nope

Reviewers: #sourcecontrol, rmcelroy, ttung

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3219749

Signature: t1:3219749:1461608742:91e3a721e78188c52431b6c5d1b3ad091e249c3a
2016-04-27 16:49:30 -07:00
Durham Goode
f92668636b store: add historypack store that reads histpack files from .hg/store/packs
Summary:
Now that we can read and write histpack files, let's add a store implementation that
can serve packed content.

My next set of commits (which haven't been written yet) will:
- add tests for all of this

Test Plan:
Ran the tests. Also repacked a repo, deleted the old cache files,
ran hg log FILE, and verified it produced results without hitting the network.

Reviewers: #sourcecontrol, ttung, mitrandir, rmcelroy

Reviewed By: mitrandir, rmcelroy

Subscribers: rmcelroy, mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3219765

Signature: t1:3219765:1461717992:9b2e8646c0555472fa00ee7059c0f283fd4c2c65
2016-04-27 16:49:27 -07:00
Durham Goode
18cde8ba89 store: add a historypack class that can read histpacks
Summary:
The previous patch added logic to repack store history and write it to
a histpack file. This patch adds a pack reader implementation that knows how to
read histpacks.

Test Plan:
Ran the tests.  Also tested this in conjunction with the next patch
which actually reads from the data structure.

Reviewers: #sourcecontrol, ttung, mitrandir, rmcelroy

Reviewed By: mitrandir, rmcelroy

Subscribers: rmcelroy, mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3219764

Signature: t1:3219764:1461718081:9d812b6aea87fe9eb48fdac9dbef282e4775c3c9
2016-04-27 16:49:24 -07:00
Durham Goode
f22bae206b store: add a historypack format and a repacker for it
Summary:
This is an initial implementation of a history pack file creator and a repacker
class that can produce it. A history pack is a pack file that contains no file
content, just history information (parents and linknodes).

A histpack is two files:

- a .histpack file consisting of a series of file sections, each of which
  contains a series of revision entries (node, p1, p2, linknode)
- a .histidx file containing a filename based index to the various file sections
  in the histpack.

See the code for documentation of the exact format.

Test Plan:
ran the tests.  A future diff will add unit tests for all the new pack
structures.

Ran `hg repack` on a large repo. Verified pack files were produced in
.hg/store/packs. In a future diff, I verified that the data could be read
correctly.

Reviewers: #sourcecontrol, mitrandir, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: mitrandir, rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3219762

Signature: t1:3219762:1461751982:e7bbc65e8f01c812fc1eb566d2d48208b0913766
2016-04-27 16:49:21 -07:00
Durham Goode
f17f6cc093 store: add revisions to datapack in alphabetical order
Summary:
This forces the revisions in the datapack to be added in alphabetical order.
This makes the algorithm more deterministic, but otherwise has little effect.

Test Plan: Ran the tests, ran repack

Reviewers: #sourcecontrol, rmcelroy, ttung

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3219760

Signature: t1:3219760:1461687720:7be5fdc1419f8214c8c83074494b33214b3684ae
2016-04-27 16:49:18 -07:00