@@ -0,0 +1,190 @@
3+title = "Seguro Prototype"
7+grant_type = ["Proposal"]
8+grant_category = ["Runtime"]
12+description = "Implement a working FoundationDB event read/write layer that is as ready as possible for interfacing with the Urbit runtime."
14+assignee = "~mastyr-bottec"
15+champion = "~master-morzod"
25+A production wrapper around FoundationDB, ready for integration with `vere`.
29+The results of Phase 1 of the Seguro project demonstrate that FoundationDB is
30+the best choice of distributed databases to utilize for the project. This is
31+Phase 2 of the Seguro project, which implements an MVP event reader/writer
32+module using FoundationDB as the underlying data store.
34+See below for additional context:
36+- [Seguro Prototype (Phase 1) Grant](https://urbit.org/grants/seguro-prototype)
37+- [Seguro Prototype (Phase 1) Repository](https://github.com/wexpertsystems/seguro)
38+- [Seguro & Armada Paper](https://gist.github.com/wexpert/0485a722185d5ee70742570036faf32f)
42+Phase 2 of the Seguro project can be considered as "all work up to actual
43+integration with `vere`". Limiting Phase 2 to this is desirable because it i.)
44+keeps the grant's scope of work at a reasonable level, and ii.) it allows more
45+time for the following three ancillary runtime projects/considerations to
48+1. Asynchronous reads in `vere`
49+2. Merging/releasing event log truncation
50+ [urbit/urbit#7501](https://github.com/urbit/urbit/pull/5701)
51+3. A decision on whether or not to add another process to the overall Urbit
54+Each of these three workstreams need to be resolved before Seguro can be
55+integrated with `vere`.
57+### Event Fragmentation Scheme
59+FoundationDB supports a maximum value size of 100KB and recommends limiting
60+value size to only 10KB for best performance. Since our data model requires
61+arbitrarily sized byte buffer values for events, an event fragmentation
62+algorithm is required to support events with larger payloads.
64+Events which exceed the size limit of a single database value must be fragmented
65+so that such events are stored across multiple database key/value pairs. To
66+achieve event fragmentation, key/value pairs will be formatted like so:
70+| | encoding | format |
71+| --------- | ----------------------------- | ------------------------------------- |
72+| **key** | ASCII string, null-terminated | `"<event id>:<fragment id>"` |
73+| **value** | raw bytes | `<fragment header><fragment payload>` |
75+`event id`s and `fragment id`s are both contiguous, non-negative integers.
79+`fragment header`s are raw byte strings with byte-widths determined by the total
80+size of the event payload. By default, fragment headers have a byte-width of 4:
82+| byte | bucket | format | example |
83+| ---- | ---------- | -------------------------------- | ----------- |
84+| 0 | byte-width | overflow toggle bit, fragment id | `0000 0000` |
85+| 1 | | ASCII `':'` character | `0011 1010` |
86+| 2 | length | total fragments | `0000 0001` |
87+| 3 | | ASCII `':'` character | `0011 1010` |
89+The overflow toggle bit is the high bit of byte 0.
91+The default fragment header byte-width of 4 supports events which can be
92+fragmented with 127 or less total fragments.
94+For events which require greater than 127 fragments to be stored, the overflow
95+toggle bit is set to 1 and the low 7 bits are used to encode the fragment header
96+byte-width instead of the `fragment id`. If those 7 bits encode a value of _x_
97+bytes, the following _x_ bytes in the fragment header will contain the
98+`fragment id`. Note that, no matter the value of the overflow toggle bit, the
99+byte-width and length buckets _always_ have the same byte-widths. Thus, the
100+total byte-width of any fragment header is _2x+2_, where _x_ is 1 when the
101+overflow toggle bit or the encoded value of the low 7 bits when the overflow
102+toggle bit is set to 1.
104+Let's see how the 255th fragment of an example event which requires 255
105+fragments to be stored would look:
107+| byte | bucket | format | example |
108+| ---- | ---------- | -------------------------------------- | ----------- |
109+| 0 | byte-width | overflow toggle bit, bucket byte-width | `1000 0010` |
110+| 1 | byte-width | fragment id | `1111 1111` |
111+| 2 | | ASCII `':'` character | `0011 1010` |
112+| 3 | length | total fragments | `0000 0000` |
113+| 4 | length | total fragments continued | `1111 1111` |
114+| 5 | | ASCII `':'` character | `0011 1010` |
116+In this case, _x_ is 2, so the total byte-width of the fragment header is _2x+2
119+#### Fragment Payload
121+The `fragment payload` is the raw bytes of the event, as encoded by `vere`,
122+which are appended to the end of the fragment header byte string.
124+The maximum size of each event's `fragment payload` is simply:
127+(maximum db value size) - (fragment header size)
130+For instance, if the `maximum db value size` is 10KB and the
131+`fragment header size` for the given event is the default of 4 bytes, each
132+fragment payload can hold up to 9,996 bytes.
134+#### Fragmentation Tests
136+Tests ensuring multi-fragment events are written and read correctly and without
137+corruption must be included in this work.
141+Single and batch reads must be supported, with the following function
142+signatures, or similar:
144+- `event read_event(int event_id);`
145+- `event[] read_event_batch(int min_event_id, int max_event_id);`
149+Single transaction batch writes and multiple, simultaneous, asynchronous writes
150+must be supported as well:
152+- `int write_event_batch(event[]);`
153+- `future write_event_async(event);`
155+Additional requirements for writes include:
157+- A benchmark determining which of the above perform best
158+- A brief analysis of pros/cons of integrating the two, based on whether the
159+ overall architecture is in- or out-of-process
163+1. Integration with `vere`
165+3. Automatic failover
169+Phase 2 - Implement a working FoundationDB event read/write layer that is as
170+ready as possible for interfacing with the Urbit runtime.
174+1. Implementation using the
175+ [FoundationDB C API](https://apple.github.io/foundationdb/api-c.html)
176+2. Event fragmentation scheme, including correctness tests
177+3. Single and batch reads
180+ 1. Single transaction batch writes
181+ 2. Multiple, simultaneous transaction writes (race a bunch of futures)
182+ 3. Benchmark determining best performance write strategy
183+ 4. Brief analysis of implementation pros/cons based on in- or out-of-process
186+5. Linux compatibility (macOS, Windows can be supported later)
190+Expected Completion: October 2022