@@ -0,0 +1,191 @@
3+title = "Seguro Prototype"
7+grant_type = ["Proposal"]
8+grant_category = ["Runtime"]
12+description = "Implement a working FoundationDB event read/write layer that is as
13+ready as possible for interfacing with the Urbit runtime."
15+assignee = "~mastyr-bottec"
16+champion = "~master-morzod"
26+A production wrapper around FoundationDB, ready for integration with `vere`.
30+The results of Phase 1 of the Seguro project demonstrate that FoundationDB is
31+the best choice of distributed databases to utilize for the project. This is
32+Phase 2 of the Seguro project, which implements an MVP event reader/writer
33+module using FoundationDB as the underlying data store.
35+See below for additional context:
37+- [Seguro Prototype (Phase 1) Grant](https://urbit.org/grants/seguro-prototype)
38+- [Seguro Prototype (Phase 1) Repository](https://github.com/wexpertsystems/seguro)
39+- [Seguro & Armada Paper](https://gist.github.com/wexpert/0485a722185d5ee70742570036faf32f)
43+Phase 2 of the Seguro project can be considered as "all work up to actual
44+integration with `vere`". Limiting Phase 2 to this is desirable because it i.)
45+keeps the grant's scope of work at a reasonable level, and ii.) it allows more
46+time for the following three ancillary runtime projects/considerations to
49+1. Asynchronous reads in `vere`
50+2. Merging/releasing event log truncation
51+ [urbit/urbit#7501](https://github.com/urbit/urbit/pull/5701)
52+3. A decision on whether or not to add another process to the overall Urbit
55+Each of these three workstreams need to be resolved before Seguro can be
56+integrated with `vere`.
58+### Event Fragmentation Scheme
60+FoundationDB supports a maximum value size of 100KB and recommends limiting
61+value size to only 10KB for best performance. Since our data model requires
62+arbitrarily sized byte buffer values for events, an event fragmentation
63+algorithm is required to support events with larger payloads.
65+Events which exceed the size limit of a single database value must be fragmented
66+so that such events are stored across multiple database key/value pairs. To
67+achieve event fragmentation, key/value pairs will be formatted like so:
71+| | encoding | format |
72+| --------- | ----------------------------- | ------------------------------------- |
73+| **key** | ASCII string, null-terminated | `"<event id>:<fragment id>"` |
74+| **value** | raw bytes | `<fragment header><fragment payload>` |
76+`event id`s and `fragment id`s are both contiguous, non-negative integers.
80+`fragment header`s are raw byte strings with byte-widths determined by the total
81+size of the event payload. By default, fragment headers have a byte-width of 4:
83+| byte | bucket | format | example |
84+| ---- | ---------- | -------------------------------- | ----------- |
85+| 0 | byte-width | overflow toggle bit, fragment id | `0000 0000` |
86+| 1 | | ASCII `':'` character | `0011 1010` |
87+| 2 | length | total fragments | `0000 0001` |
88+| 3 | | ASCII `':'` character | `0011 1010` |
90+The overflow toggle bit is the high bit of byte 0.
92+The default fragment header byte-width of 4 supports events which can be
93+fragmented with 127 or less total fragments.
95+For events which require greater than 127 fragments to be stored, the overflow
96+toggle bit is set to 1 and the low 7 bits are used to encode the fragment header
97+byte-width instead of the `fragment id`. If those 7 bits encode a value of _x_
98+bytes, the following _x_ bytes in the fragment header will contain the
99+`fragment id`. Note that, no matter the value of the overflow toggle bit, the
100+byte-width and length buckets _always_ have the same byte-widths. Thus, the
101+total byte-width of any fragment header is _2x+2_, where _x_ is 1 when the
102+overflow toggle bit or the encoded value of the low 7 bits when the overflow
103+toggle bit is set to 1.
105+Let's see how the 255th fragment of an example event which requires 255
106+fragments to be stored would look:
108+| byte | bucket | format | example |
109+| ---- | ---------- | -------------------------------------- | ----------- |
110+| 0 | byte-width | overflow toggle bit, bucket byte-width | `1000 0010` |
111+| 1 | byte-width | fragment id | `1111 1111` |
112+| 2 | | ASCII `':'` character | `0011 1010` |
113+| 3 | length | total fragments | `0000 0000` |
114+| 4 | length | total fragments continued | `1111 1111` |
115+| 5 | | ASCII `':'` character | `0011 1010` |
117+In this case, _x_ is 2, so the total byte-width of the fragment header is _2x+2
120+#### Fragment Payload
122+The `fragment payload` is the raw bytes of the event, as encoded by `vere`,
123+which are appended to the end of the fragment header byte string.
125+The maximum size of each event's `fragment payload` is simply:
128+(maximum db value size) - (fragment header size)
131+For instance, if the `maximum db value size` is 10KB and the
132+`fragment header size` for the given event is the default of 4 bytes, each
133+fragment payload can hold up to 9,996 bytes.
135+#### Fragmentation Tests
137+Tests ensuring multi-fragment events are written and read correctly and without
138+corruption must be included in this work.
142+Single and batch reads must be supported, with the following function
143+signatures, or similar:
145+- `event read_event(int event_id);`
146+- `event[] read_event_batch(int min_event_id, int max_event_id);`
150+Single transaction batch writes and multiple, simultaneous, asynchronous writes
151+must be supported as well:
153+- `int write_event_batch(event[]);`
154+- `future write_event_async(event);`
156+Additional requirements for writes include:
158+- A benchmark determining which of the above perform best
159+- A brief analysis of pros/cons of integrating the two, based on whether the
160+ overall architecture is in- or out-of-process
164+1. Integration with `vere`
166+3. Automatic failover
170+Phase 2 - Implement a working FoundationDB event read/write layer that is as
171+ready as possible for interfacing with the Urbit runtime.
175+1. Implementation using the
176+ [FoundationDB C API](https://apple.github.io/foundationdb/api-c.html)
177+2. Event fragmentation scheme, including correctness tests
178+3. Single and batch reads
181+ 1. Single transaction batch writes
182+ 2. Multiple, simultaneous transaction writes (race a bunch of futures)
183+ 3. Benchmark determining best performance write strategy
184+ 4. Brief analysis of implementation pros/cons based on in- or out-of-process
187+5. Linux compatibility (macOS, Windows can be supported later)
191+Expected Completion: October 2022