Commit Graph

39 Commits

Author SHA1 Message Date
Durham Goode
587fc95964 mutabledatapack: begin implementing DataStore trait
Summary:
A mutabledatapack also needs to be readable as a normal store. Let's
start implementing the DataStore trait, starting with get_missing

Reviewed By: quark-zju

Differential Revision: D8598657

fbshipit-source-id: 1f8bc89fae2be73fe789bc0ef1cdd922222019a2
2018-07-05 14:53:18 -07:00
Durham Goode
875758fdbe datapack: implement getdeltachain
Summary: Implements the last of the DataStore api, getdeltachain.

Reviewed By: quark-zju

Differential Revision: D8557950

fbshipit-source-id: 7f6530fe2064f0d035414b7920a126c6aab41beb
2018-07-05 14:53:18 -07:00
Durham Goode
24a4751ff0 revisionstore: change Delta.data to Rc
Summary:
In a future diff we'll be returning data read from a pack file out as a
Delta. To avoid copies, we need to be able to return an Rc from DataPack. This
seems like it will be a common pattern, so let's go ahead and make Delta contain
its data as an Rc.

Reviewed By: quark-zju

Differential Revision: D8557949

fbshipit-source-id: 276005360bfa48e9154143dedce579a21129e976
2018-07-05 14:53:18 -07:00
Durham Goode
c6af00dbc9 datapack: implement getmetadata
Summary:
Introduces the DataEntry structure which is able to parse data entries
from pack files. Uses it to implement getmetadata

Reviewed By: quark-zju

Differential Revision: D8556610

fbshipit-source-id: c25427c3c247970a879ad7d409b821f3695b97d9
2018-07-05 14:53:17 -07:00
Durham Goode
79a9dd976f datapack: implement getmissing
Summary: Adds the DataStore trait and implements the getmissing function.

Reviewed By: quark-zju

Differential Revision: D8554391

fbshipit-source-id: 41c107c07de7d6945ca7370e264c6bc0bf154754
2018-07-05 14:53:17 -07:00
Durham Goode
ebc31e8daf datapack: add initial datapack structure
Summary:
This adds the initial struct and opener for a datapack. Future diffs
will add actual functionality and tests.

Reviewed By: quark-zju

Differential Revision: D8553436

fbshipit-source-id: 3b17f995632e859019205f242a4cce389ac77407
2018-07-05 14:53:17 -07:00
Durham Goode
029666cf27 mutabledatapack: write index to disk during serialization
Summary:
Actually write the index to disk when the mutabledatapack is
serializing.

Reviewed By: quark-zju

Differential Revision: D8552276

fbshipit-source-id: 354c7fdc3fe84b91d582f0e8cde8c6ae2494c559
2018-07-05 14:53:17 -07:00
Durham Goode
e68c0ec7e0 mutabledatapack: add logic for reading DataIndex
Summary: This adds the logic for reading a DataIndex from disk.

Reviewed By: quark-zju

Differential Revision: D8552278

fbshipit-source-id: 611ff09c27716b8d8ff7424c1a27287b9fc42b78
2018-07-05 14:53:16 -07:00
Durham Goode
d22f6ce58d mutabledatapack: add logic for writing DataIndex
Summary:
Soon we will be writing the index during pack file serialization, so
let's add the logic for serializing the index.

Reviewed By: quark-zju

Differential Revision: D8552277

fbshipit-source-id: 60829631eb060f62d266c16f6016f34080311f8e
2018-07-05 14:53:16 -07:00
Durham Goode
69a59f53d9 revisionstore: add Node::from_slice
Summary: A simple helper method for producing Node's from slices in a safe way.

Reviewed By: quark-zju

Differential Revision: D8547679

fbshipit-source-id: 85ae8fcd7749c662b1459af1d84ccf9695dd5f0b
2018-07-05 14:53:16 -07:00
Durham Goode
1c3767bc11 mutabledatapack: implement data index header serialization
Summary:
We're beginning to implement the DataPack index file logic. Let's start
with header serialization/deserialization.

Reviewed By: quark-zju

Differential Revision: D8319727

fbshipit-source-id: 079aab06ececb1c5159aec2da3243268eea0cb61
2018-07-05 14:53:15 -07:00
Durham Goode
12e0e5bf16 mutabledatapack: build inmemory index as revisions are added
Summary:
Let's build an inmemory hash table of the revisions that were added. A
future diff will serialize this index into a dataidx file.

Reviewed By: quark-zju

Differential Revision: D8309730

fbshipit-source-id: 9efc7f0f34129a63c52309b4d70179f2c10840b3
2018-07-05 14:53:15 -07:00
Durham Goode
763d1e4bef datapack: implement a fanout table
Summary:
Implements a fanout table trait that history pack and data pack will
use. It basically consists of logic to build and read a quick lookup table that
uses the first few bytes of a key to determine the bounding range of a binary
search.

Reviewed By: quark-zju

Differential Revision: D8309729

fbshipit-source-id: 71e398277dc8ae041447035f044e5d47ca41cf7e
2018-07-05 14:53:15 -07:00
Durham Goode
4588cc18c8 mutabledatapack: write version number header
Summary:
The mutabledatapack format has a one byte header containing the version
number.

Reviewed By: quark-zju

Differential Revision: D8305653

fbshipit-source-id: c4a96dc48e64acd2c5849034e5d90b87363fbc8d
2018-07-05 14:53:15 -07:00
Durham Goode
99a11bbb24 mutabledatapack: use hash of contents as name
Summary:
Implements the logic that builds a hash of the contents of the pack
file and uses it as the name.

Reviewed By: quark-zju

Differential Revision: D8305654

fbshipit-source-id: d1270e7519a7718aa5427f3be5cdc0cd0dee2fe2
2018-07-05 14:53:14 -07:00
Durham Goode
3f467bd21f mutabledatapack: implement add()
Summary:
This is the start of a rust mutable datapack implementation. The first
diff adds a simple add function. Later diffs will add the logic that builds the
index, serializes the index, and computes the final hash name.

Reviewed By: quark-zju

Differential Revision: D8304036

fbshipit-source-id: db05c2b845e51a3552c039b7fc0b8f4cc0ff0852
2018-07-05 14:53:14 -07:00
Durham Goode
8057817dc1 revisionstore: add read/write functions to Metadata
Summary:
In a future diff we'll be serializing and deserializing metadata in
datapacks. Let's add the reader and writer functions for Metadata and some unit
tests.

Reviewed By: quark-zju

Differential Revision: D8303603

fbshipit-source-id: 7e7a7aa218c05179b205abf8b151b1488be674b3
2018-07-05 14:53:14 -07:00
Saurabh Singh
fa3c7b34a3 add basic tests for unionhistorystore
Summary:
This commit adds very basic tests for the Union History Store. These
tests just test for expected output of operations on bad/empty stores.

Reviewed By: quark-zju

Differential Revision: D8553821

fbshipit-source-id: a0dfa47f10083c37901535e8a810a99693a28c82
2018-06-27 19:05:31 -07:00
Saurabh Singh
40e70758b7 introduce union history store
Summary: This commit just introduces the Union History Store.

Reviewed By: DurhamG

Differential Revision: D8553822

fbshipit-source-id: 6c7ee0b5d33dae6d51b4179616d206f42eb0cd50
2018-06-27 19:05:31 -07:00
Saurabh Singh
0326fe4584 introduce history store
Summary: This commit just introduces the history store.

Reviewed By: DurhamG

Differential Revision: D8553823

fbshipit-source-id: 93af6059296d11c4fcc0dd306b4472c4f2168fa7
2018-06-27 19:05:31 -07:00
Saurabh Singh
164bf3e85a fix error messaging
Summary: This commit just fixes the messaging for the errors.

Reviewed By: DurhamG

Differential Revision: D8553820

fbshipit-source-id: 73f2cd13e7538b6870b16a0e47e657a6d08af9e3
2018-06-26 11:36:09 -07:00
Saurabh Singh
7c9227818a refactor rust datastore to a consistent naming scene
Summary: This is just a refactor to address the naming scheme.

Reviewed By: quark-zju

Differential Revision: D8269217

fbshipit-source-id: 8c52d2c67837550e0b7dc1a45b3faf9a80319b61
2018-06-04 17:39:47 -07:00
Saurabh Singh
7067e4ca1f fix nit in implementation
Summary:
Based on review for D8214151 by quark-zju, addressing the nit here as
well.

Reviewed By: quark-zju

Differential Revision: D8267140

fbshipit-source-id: 12c3355852a49859c2b0a243fa8666105c914c73
2018-06-04 16:21:38 -07:00
Saurabh Singh
8ba5a79489 adding tests for bad data store
Summary:
Adding the tests for the case when the union store has only one data
store which always returns an `Err` as `Result`. This `Err` is not of the type
`KeyError` which the union store handles differently.

Reviewed By: quark-zju

Differential Revision: D8214156

fbshipit-source-id: bd077af343086c92f46ec6a6f1551d05dd9bda09
2018-06-04 16:21:38 -07:00
Saurabh Singh
242f2b904f add tests for empty data store
Summary:
Adding tests for the case when the union store only has a single data
store which is completely empty.

Reviewed By: quark-zju

Differential Revision: D8214151

fbshipit-source-id: 9d8f329548a1b7e105a5dc6219067a6e292fe97c
2018-06-04 16:21:37 -07:00
Saurabh Singh
34ca90a6f8 rename test methods to be more specific
Summary:
This commit just renames the methods to be more specific. This is
useful for later changes.

Reviewed By: quark-zju

Differential Revision: D8214153

fbshipit-source-id: e8db9148334f7cd539aca626e3798e256b9b022f
2018-06-04 16:21:37 -07:00
Saurabh Singh
a4f5e8aefc add tests for empty union date store
Summary:
These tests just test the expected outcome when the union data store
is empty.

Reviewed By: quark-zju

Differential Revision: D8018975

fbshipit-source-id: a2cc4c87509b857dbf5f6af506f165ea62080db8
2018-05-15 18:37:47 -07:00
Saurabh Singh
cab75fbb90 add common traits for Key
Summary:
This commit derives the common traits for the Key type just as we did
for the Node type in D7872300.

Reviewed By: quark-zju

Differential Revision: D8018973

fbshipit-source-id: 566a69be16d74529c6eb5f157b84de25835f780f
2018-05-15 18:37:47 -07:00
Saurabh Singh
88a2d8ff6b implement quickcheck::Arbitrary for Key in revisionstore
Summary:
We need to implement `quickcheck::Arbitrary` for Key so that it can be
used for the quickcheck tests.

Reviewed By: quark-zju

Differential Revision: D8018977

fbshipit-source-id: dbdbb34fbd7eaeb18321eafec4604d752f496a4d
2018-05-15 18:37:47 -07:00
Saurabh Singh
0b5b994973 implement quickcheck::Arbitrary for Node type in revisionstore
Summary:
We need to implement `quickcheck::Arbitrary` for Node so that it can
be used for quickcheck tests.

Reviewed By: quark-zju

Differential Revision: D8018978

fbshipit-source-id: ceda99622370bee6e9d05b839f9856c0526f553c
2018-05-15 18:37:47 -07:00
Saurabh Singh
411736dc7c add 'quickcheck' crate to revisionstore
Summary:
I am planning to use the `quickcheck` crate for testing the union data
store. This commit just adds the crate to the revisionstore.

Reviewed By: quark-zju

Differential Revision: D8018974

fbshipit-source-id: d390deeb01aa7d1bf1e66bb5bc948d48bd3f269e
2018-05-15 18:37:47 -07:00
Saurabh Singh
8688e1cc5e union data store: union data store implementation in Rust
Summary:
This commit just introduces the `UnionDataStore` and implements the
`DataStore` trait for it.

Reviewed By: quark-zju

Differential Revision: D7801615

fbshipit-source-id: 14eabd2aa1b1e085de94aec126a7108231ec6e8d
2018-05-15 18:37:47 -07:00
Saurabh Singh
d2b9c6c6ac union store: introduce common type for the union store implmentation
Summary:
We will be implmenting multiple union stores and therefore, it makes
sense to encapsulate the common logic in its own type. This also abstracts the
usage of `RefCell` within the union store.

Reviewed By: jsgf

Differential Revision: D7884651

fbshipit-source-id: a74b6d9df5ee0d7d04359219e276fd5713b3a00b
2018-05-15 18:37:47 -07:00
Saurabh Singh
bcb7ac0b32 node: add common traits for the node type
Summary:
Based on the review comments for D7801615, this commit adds the common traits
for the `Node` type

Reviewed By: jsgf

Differential Revision: D7872300

fbshipit-source-id: 44dedfc3ec0e18ac0dee0dcfc5f7dfc4aff2511d
2018-05-15 18:37:47 -07:00
Durham Goode
18697e3fb3 hg: implement getmissing() for Rust hgstore
Summary:
Initial implementation of getmissing for a simple Rust pass through
data store. Future diffs will extend this to implement the union data store
completely in Rust.

Reviewed By: quark-zju

Differential Revision: D7632405

fbshipit-source-id: e660d33f8231410805cfaba6d77c56f27b002f8e
2018-05-14 12:05:13 -07:00
Durham Goode
f2b4d7f2e0 hg: implement getmeta() for Rust hgstore
Summary:
An initial implementation of getmeta for the Rust hg data store. Future
diffs will add more functionality.

Reviewed By: quark-zju

Differential Revision: D7632404

fbshipit-source-id: 53bd3b96b777bc3c5aef2b4d07ce1a9d9a5a52ed
2018-05-14 12:05:13 -07:00
Durham Goode
39dde8552d hg: implement getdeltachain() for Rust hgstore
Summary:
An initial implementation of getdeltachain for a simple pass through
data store. Future diffs will add additional functionality.

Reviewed By: quark-zju

Differential Revision: D7632407

fbshipit-source-id: 1a38089ba8ea70f8772af95afd871ee493082d80
2018-05-14 12:05:13 -07:00
Durham Goode
880ff5d0a9 hg: implement datastore.get() for Rust hgstore
Summary:
Implements the get function for a simple pass through rust data store
layer. Future diffs will implement more functions, and then later we will
implement the entire union data store in Rust.

Reviewed By: quark-zju

Differential Revision: D7632403

fbshipit-source-id: 3a1d0a8500e3110213d70dc1cff637cf8eadd809
2018-05-14 12:05:13 -07:00
Durham Goode
a97e97e413 hg: initial boiler plate for new hgstore crate
Summary:
This will contain all the Python centric hg store code that will let
Python call into the Rust storage layer.

Reviewed By: quark-zju

Differential Revision: D7632406

fbshipit-source-id: 6b7bcc8f47a23e9c0121e1f92de1137369bf584e
2018-05-14 12:05:12 -07:00