Commit Graph

7 Commits

Author SHA1 Message Date
Jun Wu
ba05e88179 indexedlog: add serialization for leaf and link entry
Summary: They are simpler than radix entry and similar.

Reviewed By: DurhamG

Differential Revision: D7191652

fbshipit-source-id: b516663567267a2e354748396b44c2ac8ebb691f
2018-04-13 21:51:25 -07:00
Jun Wu
dab5948078 indexedlog: add serialization for radix entry
Summary: Start serialization implementation. First, add support for the radix entry.

Reviewed By: DurhamG

Differential Revision: D7191365

fbshipit-source-id: 54a5ba5c666ba4def1e80eaa2ff7d4d77ff53f8c
2018-04-13 21:51:25 -07:00
Jun Wu
599194b15d indexedlog: define basic structures
Summary: These are Rust structures that map to the file format.

Reviewed By: DurhamG

Differential Revision: D7191366

fbshipit-source-id: 23a4431383be9713e955b74306cd68108eb80536
2018-04-13 21:51:25 -07:00
Jun Wu
6542d0ebf4 indexedlog: add comment about index file format
Summary: Document the format. Actual implementation in later diffs.

Reviewed By: DurhamG

Differential Revision: D7190575

fbshipit-source-id: 243992fd052ca7a9688d54d20694e65daebb9660
2018-04-13 21:51:25 -07:00
Jun Wu
015a4ac5d6 indexedlog: port base16 iterator from radixbuf
Summary:
The append-only index is too different so it's cleaner to cherry-pick code
from radixbuf, instead of modifying radixbuf which would break code
depending on it.

Started by picking the base16 iterator part.

`rustc-test` does not work with buck, and seems to be in an unmaintained
state, so benchmark tests are migrated to criterion.

Reviewed By: DurhamG

Differential Revision: D7189143

fbshipit-source-id: 459a79b4cf16f35d2ff86f11a5980ba1fc627951
2018-04-13 21:51:25 -07:00
Jun Wu
d2c457a6e2 indexedlog: integrity check utility on an append-only file
Summary:
Filesystem is hard. Append-only sounds like a safe way to write files, but it
only really helps with process crashes. If the OS crashes, it's possible that
other parts of the file gets corrupted. As source control, data integrity check
is important. So bytes not logically touched by appending also needs to be
checked.

Implement a `ChecksumTable` which adds integrity check ability to append-only
files. It's intended to be used by future append-only indexes.

Reviewed By: DurhamG

Differential Revision: D7108433

fbshipit-source-id: 16daf6b8d04bba464f1ee9221716beba69c1d47b
2018-04-13 21:51:24 -07:00
Jun Wu
0518016553 indexedlog: initial boilerplate
Summary:
First step of a storage-related building block that is in Rust. The goal is
to use it to replace revlog, obsstore and packfiles.

Extern crates that are likely useful are added to reduce future churns.

Reviewed By: DurhamG

Differential Revision: D7108434

fbshipit-source-id: 97ebd9ba69547d876dcecc05e604acdf9088877e
2018-04-13 21:51:24 -07:00