Commit Graph

272 Commits

Author SHA1 Message Date
Durham Goode
24c9c438d3 history: remove dependency on 80 character pack entry
Summary:
In a future diff we will be changing the format of history pack entries
to have a variable length file path in them. Before doing that, let's remove the
places that depend on each entry being exactly 80 characters.

Test Plan: Ran the tests

Reviewers: #mercurial, mitrandir, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3262154

Signature: t1:3262154:1462820474:b7f5666001b3b7f8863b4de4826266204f3e87aa
2016-05-16 10:59:09 -07:00
Durham Goode
e4fb7d66bb history: remove getparents and getlinknode apis
Summary:
These APIs weren't actually used, and the questions can be answered via
the existing getancestors() api anyway.

They were originally put in place because they are the type of question that
doesn't require the full ancestor tree, so we could answer them without doing in
traversal. In an upcoming patch we add the concept of copyfrom back into the
historypack, and getparents becomes confusing since it doesn't expose knowledge
of copy information. So I just decided to delete it all until we need it.

In the future we may want a 'gethistoryinfo(filename, node)' api that just
returns (p1, p2, linknode, copyfrom), to fulfill that original need of history
information without a full ancestor traversal.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3261734

Signature: t1:3261734:1462413665:987c4703e53468a75346aa323188107a5c070fde
2016-05-16 10:59:09 -07:00
Durham Goode
e7a294e65f utils: change writefile to use custom atomic temp logic
Summary:
The mercurial atomic temp logic makes some assumptions about modes and copies
the mode bits from the existing file. If the existing file is readonly, that
means the atomic temp fails to open the file for writing after it's initially
created.

Since we only actually need like 4 lines from the atomic temp code, I just
implemented it on our end, with custom logic for mode handling.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, quark

Reviewed By: quark

Differential Revision: https://phabricator.intern.facebook.com/D3288589

Signature: t1:3288589:1462993609:653a0a63266b9129b0e18807858e5de02310ecf9
2016-05-11 13:33:06 -07:00
Durham Goode
ce493adf83 repack: add --background option
Summary:
This allows triggering a repack that can be run in the background. In the future
we will trigger this automatically under certain circumstances (like too many
pack files).

Test Plan: Added a test

Reviewers: #mercurial, ttung, quark

Reviewed By: quark

Subscribers: quark

Differential Revision: https://phabricator.intern.facebook.com/D3261161

Signature: t1:3261161:1462398568:5ae25f3e5a9acd0f4b34490b34a62be33cc69e3c
2016-05-04 14:53:23 -07:00
Durham Goode
8e290a5d4a repack: add lock to limit it to only one repack
Summary:
This adds a lock that limits us to running only one repack at a time. We also
add a simple prerepack hook to allow the tests to insert a sleep to test this
functionality.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3260428

Signature: t1:3260428:1462393311:3e1bf5dd047e7f3521679ca7640b448f5e784913
2016-05-04 14:53:19 -07:00
Durham Goode
c2cdcde2fb store: read recent packs first
Summary:
Since recent packs are likely to contain more recent data, let's put them at the
front of the pack list so they are checked first.

In a future diff I'll come back and refact the common code between datapack and
historypack into one base class.

Test Plan:
Ran the tests. Used the debugger to verify that the sort order was
correct.

Reviewers: #mercurial, ttung, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3259823

Signature: t1:3259823:1462388395:8ee48a7b02c6abc079878e53c5b336675249cb91
2016-05-04 14:53:16 -07:00
Durham Goode
1621511aa7 store: a number of performance improvements
Summary:
Some miscellaneous perf improvements:
- Get rid of unnecessary sorting
- Bail early from remotefilelog reverse filename lookup if there are no keys
- Avoid unnecessary class/enum lookups in ancestor hotpath
- Fix n^2 behavior in history repacking
- Remove unnecessary set() and tuple instantiation in hotpath
- Use __slots__ on repackentry to improve access times

Test Plan:
Ran the tests. Tested the actual perf improvements via 'hg repack' in
a large repo.  The n^2 fix caused a massive perf when, but the others shaved off
a number of seconds as well.

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3257595

Signature: t1:3257595:1462392755:80209fe66c2632d2c16a2840500b0655926e7513
2016-05-04 14:53:13 -07:00
Durham Goode
99a426f3eb store: fix fanout logic for history pack
This fix was already applied to datapack a while ago. Basically, if the pack
doesn't have a lot of revisions, it's possible that the fanout table is sparsely
populated. Therefore we need to scan forward when looking for the end bounds of
our fanout table.
2016-05-03 15:41:57 -07:00
Durham Goode
7da17af64f store: record what files were created during a repack
Summary:
Previously, if you ran repack twice in a row, it would actually delete your
packs, because the repack produced files with the same name as before, and the
cleanup then deleted them.

The fix is to have the stores record what files they produced in the ledger,
then future clean up work can avoid deleting anything that was created during
the repack.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3255819

Signature: t1:3255819:1462393814:d32155b12535990f72fbe48de045eddbb6f7fab6
2016-05-04 14:53:10 -07:00
Durham Goode
30cc85653c store: make pack files read-only
Summary:
Since pack files should never change after they are created, let's create them
with read-only permissions. It turns out that the Mercurial vfs doesn't apply
the correct permissions to files created by mkstemp (and we have to use mkstemp
since we don't know the name of the file until after we've written all the data
to it), so we have to manually call the permission fixing code.

We also need to fix our mmap calls to be readonly now, otherwise we get a
runtime permission denied exception.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3255816

Signature: t1:3255816:1462321201:dff4fb4c9301d67a77043ecc1d96262bb5d6a54a
2016-05-04 14:53:07 -07:00
Durham Goode
1d4b4dbb36 store: switch mutable packs to use openers
Summary:
Instead of passing in a path and performing joins ourselves, let's use an
opener. This will help handle all the file permission edge cases.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3255165

Signature: t1:3255165:1462393836:38a28c850a0dc06838d9c17672d3dffd9903bbd7
2016-05-04 14:53:04 -07:00
Durham Goode
0671f3d76a store: add version header to index
Summary: Pretty straight forward

Test Plan: Ran the tests

Reviewers: lcharignon, rmcelroy, ttung, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3254892

Signature: t1:3254892:1462315520:888bb27ef121c08d463f9fd4cf9eeb3c42383a96
2016-05-04 14:53:01 -07:00
Durham Goode
d71eace818 store: refactor version number and size to constants
Summary:
A future patch will add a version number to the index file. Let's move the
version size, fanout start, and index start to constants so we can more easily
change them without changing the code.

Test Plan: Ran the tests

Reviewers: lcharignon, rmcelroy, ttung, mitrandir, quark

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3254876

Signature: t1:3254876:1462315858:63fe56e8cfcdbb0209861898ce0c45c7d7b33e35
2016-05-04 14:52:58 -07:00
Durham Goode
4aa798d76e store: don't allow gc to delete pack files
Summary:
`hg gc` is very aggressive in that it deletes any files in the cache that it
determines aren't a needed key, including files it doesn't recognize. Let's
teach it to not delete pack files.

In the future we'll need to make `hg gc` able to garbage collect the contents of
pack files as well.

Test Plan: It's probably fine!

Reviewers: lcharignon, rmcelroy, ttung, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3254744

Signature: t1:3254744:1462308257:c1f932b88abf3337370f16c05c789422ea51b0e1
2016-05-04 14:52:55 -07:00
Durham Goode
312acdb24e store: implement markledger and cleanup for historypack
Summary:
Implementing these two functions allows historypacks to be repacked, either into
a new format, or by combining multiple packs into a single new one.

Test Plan: Added a test in my next patch

Reviewers: lcharignon, ttung, rmcelroy, mitrandir, quark

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251542

Signature: t1:3251542:1462392294:f95f7666a3a5df675f1351a19af7532c4742af2b
2016-05-04 14:52:49 -07:00
Durham Goode
d86ae1e525 store: implement markledger and cleanup for datapack
Summary:
Implementing these two functions will allow datapack's to be repacked (either
into other formats, or by combining multiple packs into one).

A future patch will add a test.

Test Plan: Added a test in a future patch

Reviewers: lcharignon, ttung, rmcelroy, mitrandir, quark

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251539

Signature: t1:3251539:1462393256:7caa09677fbcaaf57a47d7a833684883483c5b3a
2016-05-04 14:52:46 -07:00
Durham Goode
6b25f32192 store: add revision count to historypack filesection
Summary:
Previously, given a historypack file, we had no way of reading the contents,
since we had no way to know when to stop reading the revision entries for a
given file section.

This patch changes the format to have a revision count value after the filename
and before the revisions. The documentation already documented the format like
this, and therefore doesn't need updating.

A future patch will use this information to iterate over all the revisions in
the pack.

Test Plan: Added a test in a future patch

Reviewers: lcharignon, ttung, rmcelroy, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251538

Signature: t1:3251538:1462393282:f46b50e79237bfa8a25ff1957344588622b2699a
2016-05-04 14:52:43 -07:00
Durham Goode
548ccdeae1 store: make historypack file section writing lazier
Summary:
In a later patch we will need to add the count of revisions in a given file
section to the on-disk format. To make that easier, let's make the file section
serialization lazy, so that we will have the full list when it comes time to
count the entries.

Test Plan: added a test in a future patch

Reviewers: lcharignon, ttung, rmcelroy, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3251537

Signature: t1:3251537:1462393274:60b72a47de45f5a94f4f5a8d34b3942db0aa3fda
2016-05-04 14:52:40 -07:00
Durham Goode
a54d3f1257 store: make repack only repack the shared cache
Summary:
Previously, hg repack would repack all the objects in all the store and dump the
new packs in .hg/store/packs. Initially we only want to repack the shared cache
though, so let's change repack to only operate on shared stores, and to write
out the new packs to the hgcache, under the appropriate repo name.

In a future patch I'm going to go through all this store stuff and replace all
uses of os.path and direct file reads/writes with a mercurial vfs.

Test Plan:
Ran repack in a large repo and verified packs were produced in
$HGCACHE/$REPONAME/packs

Ran hg cat on a file to verify that it read the data from the pack and did not do any remotefilelog network calls.

Reviewers: lcharignon, rmcelroy, ttung, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3250213

Signature: t1:3250213:1462315927:694661795141e2c869ba661a54cea8f4b90823df
2016-05-04 14:52:33 -07:00
Durham Goode
cfd60406ad store: add context manager to mutable pack classes
Summary:
Previously, if a repack failed, it would leave temporary pack files laying
around. By adding enter/exit functions to mutable packs, we can guarantee
cleanup happens.

Test Plan: Ran repack, verified that a failure did not leave tmp files

Reviewers: rmcelroy, quark, ttung, lcharignon, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3250201

Signature: t1:3250201:1462234552:7f20260a193ed1dd858bf6e9f489ac902d859218
2016-05-03 12:34:45 -07:00
Durham Goode
d2e7ae7519 store: make repack command use new repacker
Summary:
Now that all the repack logic is in place, let's switch the repack
command to use the new version. This also means the repack command will now
clean up the old remotefilelog blobs once it's finished.

Test Plan:
Ran hg repack in a large repo. Verified it deleted the old
remotefilelog blobs, and verified that I could still updated around the
repository without making any remotefilelog network requests.

A future diff will add standard .t mercurial tests for the repack command.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249601

Signature: t1:3249601:1462235506:03c0d95f6a82cfc04b340b139f39c02853941a17
2016-05-03 12:34:09 -07:00
Durham Goode
4ecb47b021 store: move history repack logic to repacker
Summary:
We had a naive repack implementation in historypack.py. Let's move it to the
repack module and do the minor adjustments to use the new repackerledger apis.

Test Plan:
Ran hg repack in conjunction with future diffs that make use of this
api

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249587

Signature: t1:3249587:1462232544:591cd8bec09f781370896470746eae5a4489531f
2016-05-03 12:33:54 -07:00
Durham Goode
735aa964d5 store: move data repack logic to repacker
Summary:
We had a naive repack implementation in datapack.py. Let's move it to the repack
module and do the minor adjustments to use the new repackerledger apis.

Test Plan: Ran it in conjunction with future diffs that make use of this api.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249585

Signature: t1:3249585:1462232504:a00aa65afca9562a2c1456cc4ab48c50d1ba5b68
2016-05-03 12:33:36 -07:00
Durham Goode
c902797dc9 store: implement markledger and cleanup on stores
Summary:
This implements the new markledger and cleanup apis on the existing
remotefilelog stores. These apis are used to tell the repacker what each store
has, and allows each store to cleanup if its data has been repacked.

Test Plan:
Ran repack in conjunction with the future diffs that make use of
these apis.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249584

Signature: t1:3249584:1462226133:1e8faffc9f6bf8f7c94e6e79aee8865e3c41648c
2016-05-03 12:33:00 -07:00
Durham Goode
c429030ca4 store: add class definitions and stub for repack
Summary:
This introduces the high level classes that will implement the generic repack
logic.

Test Plan: Ran the repack in conjunction with later commits that use these apis.

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249577

Signature: t1:3249577:1462225435:000f9cc29ae2a3d7fdbedf546c8936ef45d1e4cf
2016-05-03 12:32:35 -07:00
Durham Goode
b049a0910a store: datapack fix perf issue
Summary:
Using range() allocates a full list, which is 2**16 entries in the fanout case.
Let's use xrange instead. This is a notable performance win when checking many
keys.

Also removed an unused variable and use index instead of self._index since this
is a hotpath.

Test Plan: Ran hg repack

Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3249563

Signature: t1:3249563:1462240834:c19d6cbf0b6237f15ca8d81e8da856752df0ec59
2016-05-03 12:30:44 -07:00
Durham Goode
1704e5c8fb store: add tests for historypack
Summary:
This adds a basic test suite for the historypack class, and fixes some issues it
found.

Test Plan: ./run-tests.py test-historypack.py

Reviewers: mitrandir, rmcelroy, ttung, lcharignon

Reviewed By: lcharignon

Differential Revision: https://phabricator.intern.facebook.com/D3237858

Signature: t1:3237858:1461884966:c0ec90a2735255e5ef70eade09915066a7b71ee5
2016-04-28 17:37:03 -07:00
Durham Goode
8f4d83edeb shallowbundle: fix broken fallback orig call
This was caught by tests running in an unusual configuration
2016-04-28 17:34:08 -07:00
Durham Goode
22948ce7e1 checkcode: add check code test
Summary: Adds the same check code test that upstream Mercurial uses.

Test Plan:
Ran it, and fixed all the failures. I won't land this commit until
all the failure fixes are landed.

Reviewers: #sourcecontrol, ttung, rmcelroy, wez

Reviewed By: wez

Subscribers: quark, rmcelroy, wez

Differential Revision: https://phabricator.intern.facebook.com/D3221380

Signature: t1:3221380:1461802769:19f5bdc209c05edb442faa70ae572ce31e2fbc95
2016-04-28 10:18:47 -07:00
Durham Goode
29d3dda67e checkcode: fix various store files
Summary: Fix check code for various store related files

Test Plan: Ran the tests

Reviewers: #sourcecontrol, mitrandir, ttung

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3222465

Signature: t1:3222465:1461701300:34560288be4dc921f0252d4ad8fdc9c8d9357e23
2016-04-27 16:49:33 -07:00
Durham Goode
98fd33f8cb store: add missing imports
Summary: These were missing, and only needed in exception cases.

Test Plan: nope

Reviewers: #sourcecontrol, rmcelroy, ttung

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3219749

Signature: t1:3219749:1461608742:91e3a721e78188c52431b6c5d1b3ad091e249c3a
2016-04-27 16:49:30 -07:00
Durham Goode
f92668636b store: add historypack store that reads histpack files from .hg/store/packs
Summary:
Now that we can read and write histpack files, let's add a store implementation that
can serve packed content.

My next set of commits (which haven't been written yet) will:
- add tests for all of this

Test Plan:
Ran the tests. Also repacked a repo, deleted the old cache files,
ran hg log FILE, and verified it produced results without hitting the network.

Reviewers: #sourcecontrol, ttung, mitrandir, rmcelroy

Reviewed By: mitrandir, rmcelroy

Subscribers: rmcelroy, mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3219765

Signature: t1:3219765:1461717992:9b2e8646c0555472fa00ee7059c0f283fd4c2c65
2016-04-27 16:49:27 -07:00
Durham Goode
18cde8ba89 store: add a historypack class that can read histpacks
Summary:
The previous patch added logic to repack store history and write it to
a histpack file. This patch adds a pack reader implementation that knows how to
read histpacks.

Test Plan:
Ran the tests.  Also tested this in conjunction with the next patch
which actually reads from the data structure.

Reviewers: #sourcecontrol, ttung, mitrandir, rmcelroy

Reviewed By: mitrandir, rmcelroy

Subscribers: rmcelroy, mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3219764

Signature: t1:3219764:1461718081:9d812b6aea87fe9eb48fdac9dbef282e4775c3c9
2016-04-27 16:49:24 -07:00
Durham Goode
f22bae206b store: add a historypack format and a repacker for it
Summary:
This is an initial implementation of a history pack file creator and a repacker
class that can produce it. A history pack is a pack file that contains no file
content, just history information (parents and linknodes).

A histpack is two files:

- a .histpack file consisting of a series of file sections, each of which
  contains a series of revision entries (node, p1, p2, linknode)
- a .histidx file containing a filename based index to the various file sections
  in the histpack.

See the code for documentation of the exact format.

Test Plan:
ran the tests.  A future diff will add unit tests for all the new pack
structures.

Ran `hg repack` on a large repo. Verified pack files were produced in
.hg/store/packs. In a future diff, I verified that the data could be read
correctly.

Reviewers: #sourcecontrol, mitrandir, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: mitrandir, rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3219762

Signature: t1:3219762:1461751982:e7bbc65e8f01c812fc1eb566d2d48208b0913766
2016-04-27 16:49:21 -07:00
Durham Goode
f17f6cc093 store: add revisions to datapack in alphabetical order
Summary:
This forces the revisions in the datapack to be added in alphabetical order.
This makes the algorithm more deterministic, but otherwise has little effect.

Test Plan: Ran the tests, ran repack

Reviewers: #sourcecontrol, rmcelroy, ttung

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3219760

Signature: t1:3219760:1461687720:7be5fdc1419f8214c8c83074494b33214b3684ae
2016-04-27 16:49:18 -07:00
Durham Goode
43ed70b6f1 store: add datapack store that reads pack files from .hg/store/packs
Summary:
Now that we can read and write datapack files, let's add a store implementation that
can serve packed content. With this patch, it's technically possible for someone
to prefetch and repack large portions of history for long term storage with
remotefilelog.

My next set of commits (which haven't been written yet) will:
- add tests for all of this
- add an indexpack format for packing ancestor metadata (the datapack only packs
  revision content)

Test Plan:
Ran the tests. Also repacked a repo, deleted the old cache files, ran
hg up null && hg up master, and verified it checked out master with the
right files and without fetching blobs from the server.

Reviewers: #sourcecontrol, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3205351

Signature: t1:3205351:1461751649:45a56b57d962a282aeef9478500a3b23495a0eb7
2016-04-27 16:49:15 -07:00
Durham Goode
56c83ea072 store: add a datapack class that can read datapacks
Summary:
The previous patch added logic to repack store contents and write it to a
datapack file. This patch adds a new store implementation that knows how to read
datapacks.

It's just a simple implementation without any parallelism. So there's room for
improvement.

Test Plan:
Ran the tests.  Also tested this in conjunction with the next patch
which actually reads from the data structure.

Reviewers: #sourcecontrol, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3205342

Signature: t1:3205342:1461750967:84377517cb1f285d37694a3f503d60ae85bacb66
2016-04-27 16:49:12 -07:00
Durham Goode
510ac021f3 store: add a basic repack and datapack format
Summary:
This is an initial implementation of a repack algorithm that can read data from
an arbitrary store (in this case the remotefilelog content store), and repack it
into a datapack.

A datapack is two files:

- a .datapack file consisting of a series of deltas (a delta may be a full text if the delta base is the nullid)
- a .dataidx file consisting of delta information and an index into the deltas

See the code for documentation of the exact format.

Test Plan:
ran the tests

Ran `hg repack` in a large repo. Verified that a datapack and a dataidx file
were created in .hg/store/packs. The datapack used 148MB instead of the 439MB the
old remotefilelog storage used.

Reviewers: #sourcecontrol, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3205334

Signature: t1:3205334:1461751366:ee4bf6a580ffb667071a8046fda6f0858b7f25ae
2016-04-27 16:49:09 -07:00
Durham Goode
f362c9a3a8 store: add getfiles() api to store
Summary:
This adds a api to the store contract that allows the store to return a list of
the name/node pairs that it contains. This will be used to allow a repack
algorithm to list the contents of the store so it can repack it into another
store. The old remotefilelog blob store used namehash+node keys, which is
different from the new store API's name+node keys, so the getfiles()
implementation here has to perform a reverse  namehash->name lookup so it can
satisfy the store API contract.

In the remotefilelog basestore implementation, it reads the file names from the
local data directory and the shared cache directory, and reverse resolves the
file name hashes into filenames to produce the list.

Test Plan: ran the tests

Reviewers: #sourcecontrol, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3205321

Signature: t1:3205321:1461751437:a7c44c2bbe153122a3b85b8d82907a112cf77b1a
2016-04-27 16:49:06 -07:00
Durham Goode
438db1be81 store: allow union metadatastore to combine ancestors from many stores
Summary:
The old store api required that each store be able to return the complete
ancestor history for a given name/node pair. This patch allows a store to return
only the parts of history it knows about, and the union store will combine that
history with the history from other stores to produce the full result. This is
useful for stores like bundle files, where they contain only a partial history
that needs to be annotated by the real store.

Test Plan: ran the tests

Reviewers: #sourcecontrol, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3205319

Signature: t1:3205319:1461751511:210740b82cc6767b2f0c393715ac93d8f1b96bc7
2016-04-27 16:49:04 -07:00
Durham Goode
cce75d4663 store: add concept of delta chain to content store
Summary:
The old store contracts required that every store be able to produce the full
text for a revision. This patch modifies the contract so that a store (like a
bundle file store) can serve a delta chain and the union store can combine delta
chains from multiple stores together to create the final full text.

Test Plan: ran the tests

Reviewers: #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.fb.com/D3205315

Signature: t1:3205315:1461669845:3eb8968566285f6221c7c44435b855cc65da33f4
2016-04-26 15:10:38 -07:00
Durham Goode
7e1047d11f store: change union stores to accept a list of stores
Summary:
Instead of hard coding the list of stores in each union store, let's make it a
list and just test each store in order. This will allow easily adding new stores
and reordering the priority of the existing ones.

Also fix the remote store's contains function. 'contains' is the old name, and
it now needs to be getmissing in order to fit the store contract.

Test Plan: ran the tests

Reviewers: #sourcecontrol, ttung, rmcelroy

Reviewed By: rmcelroy

Differential Revision: https://phabricator.fb.com/D3205314

Signature: t1:3205314:1461606028:3a513ac82c5de668a7e40bbf7cc88d8754e2f0bb
2016-04-26 15:10:38 -07:00
Durham Goode
9cfbf5a59e store: keep track of the writable store instead of hard coding it
Summary:
A future patch is going to change the union store to just contain an ordered
list of stores. Therefore we need a special spot to record which store is the
one that should receive writes.

Test Plan: ran the tests

Reviewers: #sourcecontrol

Differential Revision: https://phabricator.fb.com/D3205307
2016-04-26 15:10:38 -07:00
Durham Goode
c3c047f0b7 Move sortnodes into shallowutil
Summary:
This is a generic topological sort and will be useful in the upcoming repacking
code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3204124

Signature: t1:3204124:1461260520:e1cb5c9d496f11e5f44e0cdbc5ba851b1573d2e1
2016-04-26 15:10:38 -07:00
Durham Goode
84bc49f25d checkcode: fix shallowrepo, shallowutil, and setup.py
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221375

Signature: t1:3221375:1461648312:7dbdd59e6370cb32b90d864a623d8066028741e7
2016-04-26 13:00:31 -07:00
Durham Goode
3817826242 checkcode: fix remotefilelogserver and shallowbundle
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221373

Signature: t1:3221373:1461648284:23203c17f4a87e33ff4e9be17a8b99bddbcdff05
2016-04-26 13:00:31 -07:00
Durham Goode
39d350996f checkcode: fix remotefilectx and remotefilelog
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221371

Signature: t1:3221371:1461648217:e9702d761ab8fd6f85dee60a4c192cf25e784f11
2016-04-26 13:00:31 -07:00
Durham Goode
859510b65e checkcode: fix fileserverclient.py
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221369

Signature: t1:3221369:1461648197:185cbbba61a9d1a7a1beacd64153185d0d0826ed
2016-04-26 13:00:31 -07:00
Durham Goode
71bd8c2561 checkcode: fix errors in cacheclient and debugcommands
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221366

Signature: t1:3221366:1461648117:088f3a5837393499e1a383af860bd1a935e0cba7
2016-04-26 13:00:31 -07:00
Durham Goode
495a853d78 checkcode: fix __init__.py
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221365

Signature: t1:3221365:1461646159:efeb0478c66cbd49d4a0a6c02a79d530b42f8248
2016-04-26 13:00:31 -07:00