git.iko.soy/streamly
on e304abc help sign in
streamly/docs/Developer/container-api.md
124 lines 4.5 kB history raw
1# Data Containers
3Containers are persistent containers of data e.g. files, arrays, maps. For
4consistency we use similar API for all such containers where possible. Usually
5the API names are relative to the current module e.g. toBytes in an array
6module means converting the array to bytes, it sounds intuitive if read
7qualified with the module name e.g. Array.toBytes. This doc lists some
8conventions and guidelines to be followed.
10## Bytes and Chunks
12APIs that deal with streams of Word8 may have "bytes" in their names
13while the ones dealing with "Array Word8" have "chunks" in their
14names. Usually the default is bytes when there is no explicit mention in
15the name.
17## Unfolds and Folds
19* Unfolds are named as "read" or with a "read" prefix (e.g. readChunks).
20* Folds are named as "write" or with a "write" prefix (e.g. writeChunks).
22## To and from Stream
24### From and Put
26* Immutable construction from some external source is named with a "from"
27 prefix (e.g. fromList).
28* Mutation of an existing container uses a "put" prefix instead of "from". When
29 an API uses an existing container to write to and does not return a newly
30 constructed container then use "put".
31* "from" vs "put": "from" assumes creation of a new object, it may fail if the
32 object being created already exists (e.g. the file exists), it may not take a
33 lock as it assumes immutability. "put" may create a new object or overwrite
34 an existing one, it may take a lock for writing as it assumes mutability.
36### To and Get
38* Converting the complete object to an external representation is prefixed with
39 "to" (e.g. toBytes).
40* For mutable objects "get" APIs may be used instead of "to" APIs.
41* "to" vs "get": "to" assumes immutable object so does not have to take a lock.
42 "get" assumes mutable object so may take a lock.
44### Append
46* Use "append" prefix for appending data at the end of a mutable container
48## With additional config
50Sometimes we need to modify the behavior of a combinator using some additional
51config. For example, combinators to read/write using a specified size of
52buffer. For such cases we apply the "With" suffix to standard combinator names:
54* readWith
55* readChunksWith
56* toBytesWith
57* getBytesWith
59## Random Access (Arrays)
61### Single elements
63* getIndex (for arrays)
64* putIndex (for mutable arrays)
66### Ranges
68* ...FromTo (e.g. readFromTo, readChunksFromTo, toBytesFromTo, getBytesFromTo)
69* ...FromThenTo
70* ...Indices
72FromThenTo vs FromStepN:
74We could use `fromThenTo` or `fromStepN` style APIs for stepwise
75enumeration. The first one uses absolute numbers whereas the second one
76uses relative positioning. We prefer FromThenTo style where possible for
77the following reasons:
79* It is the style used in base lists, and we are already using it in Enumerable
80 type class.
81* Both have their pros and cons. In `fromThenTo`, (1) it may be
82 non-intuitive whether `To` is inclusive, (2) specifying an empty range
83 is a bit awkward (e.g. fromThenTo 1 1 0). In `fromStepN`, if we know
84 the first and the last element then we will have to compute the offset
85 correctly. Ideally, when we know the elements then former is more
86 suitable while if we know the count then the latter is more suitable.
88Note that in case of floating point numbers `FromStep` style may have an
89advantage over `FromThen` style. Here is a quote from the documentation of
90`enumerateFromThenNum`:
92```
93Note that in the strange world of floating point numbers, using
94@enumerateFromThenNum (from, from + 1)@ is almost exactly the same as
95@enumerateFromStepNum (from, 1) but not precisely the same. Because @(from
96+ 1) - from@ is not exactly 1, it may lose some precision, the loss may
97also be aggregated in each step, if you want that precision then use
98'enumerateFromStepNum' instead.
99```
101### Appending
103* append (for mutable arrays)
105## Key-Value Store Access (Maps)
107* getKey
108* findKey (test existence)
109* createKey (insert new key for mutable maps)
110* putKey (insert or update for mutable maps)
111* updateKey (update existing or fail)
112* deleteKey (delete existing or fail)
113* destroyKey (delete existing or not)
115## Points to Consider
117The fold and unfold APIs can be used to express the to/from stream APIs. So we
118may not need both, it may just add to more APIs being proliferated.
120We may need an "append" style fold as well? What would "stream" append
121operations be called then?
123We may need locked version of "write" folds for mutable containers for
124concurrent access.