git.iko.soy/streamly
on 63aae12 help sign in
streamly/docs/Developer/module-organization.md
250 lines 9.6 kB history raw
1# Internal vs External Modules
3## Exposing Internal Modules
5We keep all modules exposed to facilitate convenient exposure of experimental
6APIs and constructors to users. It allows users of the library to experiment
7much more easily and carry a caveat that these APIs can change in future
8without notice. Since everything is exposed, maintainers do not have to think
9about what to expose as experimental and what remains completely hidden every
10time something is added to the library.
12## Internal module Namespace
14We expose the internal modules via `Streamly.Internal.*` namespace to keep all
15the internal modules together under one module tree and to have their
16documentation also separated under one head in haddock docs.
18## Exposed Modules as Wrappers
20Another decision point is about two choices:
22* Keep the implementation of all the APIs in an internal module and just
23reexport selected APIs through the external module. The disadvantages of this
24are:
26 1) users may not be able to easily figure out what unexposed APIs are available
27 other than the ones exposed through the external module. To avoid this problem
28 we can mark the unexposed APIs in the docs with a special comment.
30 2) In tests and benchmarks we will be using internal modules to test internal
31 and unexposed APIs. Since exposed APIs are exported via both internal and
32 external modules we will have to be careful in not using the internal module
33 for testing accidentally, instead we should always be using the exposed module
34 so that we are always testing exactly the way users will be using the APIs.
36* Keep the implementations of unexposed modules in the internal module file
37and exposed module in the external module file. In this approach, users can
38easily figure out the unexposed vs exposed APIs. But maintaining this would
39require us to move the APIs from internal to external module file whenever we
40expose an API.
42We choose the first approach.
44# Module Types and Naming
46## Abstract modules
48An abstract module represents an abstract interface using a type
49class. Multiple concrete modules can make use of this interface and possibly
50extend it.
52For example we have an `IsStream` type class that all stream types
53implement. The concrete stream types are `SerialT`, `AsyncT`
54etc. Assuming `Streamly.Data.Stream` is the containing name space for
55all stream modules, there are multiple ways to organize abstract and
56concrete modules. The `IsStream` type class can be placed in:
58* `Streamly.Data.Stream.Class` or `Streamly.Data.Stream.Interface`
59* `Streamly.Data.Stream.IsStream`
60* `Streamly.Data.Stream`
62Using `Streamly.Data.Stream.IsStream` is preferred over `.Class` because it
63conveys more information by the class name. We can place it in
64`Streamly.Data.Stream` as well.
66Concrete modules depend on the abstract module `IsStream` to implement
67that interface.
69Polymorphic operations utilizing the abstract interface can go in the
70parent module `Streamly.Data.Stream`.
72## Constrained Modules
74Some modules represent operations on a type which constrain a type using a type
75class or a specific instance of a general type. For example, we may have Arrays
76that operate on a `Storable` or a `Prim` type.
78One possible way to organize such module is to have a `Storable` or `Prim`
79hierarchy and all data structures using that type constraint are bundled under
80it. However, in general, a data structure may have multiple such
81constraints or may have to be organized based on some other dimension
82like an abstract interface it is implementing.
84General purpose constraints like `Prim` can be defined in their own module
85hierarchy and can be used everywhere. For example, we can have the following
86Array types, here we have organized the types under the `Array` hierarchy
87rather than putting the `PrimArray` under a `Prim` hierarchy.
89```
90Streamly.Internal.Data.Array.Boxed
91Streamly.Internal.Data.Array.Prim
92Streamly.Internal.Data.Array.Prim.Pinned
93```
95## Common Modules
97Some modules represent common types or utility functions that are shared across
98multiple higher level modules. Possible naming for such modules are:
100* `Module.Types`
101* `Module.Common`
102* `Module.Core`
103* `Module.Shared`
105## Aggregate modules
107In some cases we may want to aggregate the functionality of several small
108modules in a combined aggregate module. In many cases, the aggregate module is
109made a parent module of the constituent modules. The parent module depends on
110the child modules and exposes the functionality from the constituent modules.
112## Placeholder Modules
114In some cases a parent module is just a placeholder in the namespace and does
115not export any functionality.
117# Polymorphic vs Monomorphic Modules
119In general we can just provide a polymorphic stream API and let the user use it
120at the type he/she wants to use it. However, it has some disadvantages:
122* Some constraints e.g. "MonadAsync" are unnecessarily imposed on all APIs even
123 though they are needed by only concurrent types.
124* type errors could be harder to resolve when using polymorphic types
125* combinators like `fromAsync` can make all the combinators concurrent in one go
126 which is usually problematic. If we use monorphic combinators it encourages
127 to pick the required concurrent combinators one at a time which is
128 usually better for performance.
130Keeping this in mind our plan is to provide monomoprhic modules for each stream
131type, keep the combinators that are specific to that stream type in the
132monomorphic module and combinators that are exactly the same for all stream
133types can be kept in the polymorphic module or in both the modules. Having
134complete set of operations available in the monomorphic module has the
135advantage that if we want we can just import a `Serial` module and get
136everything if we just want to use the Serial stream.
138# Streamly Modules
140We use the "Streamly" prefix to all the module names so that they do not
141conflict with any other module on Hackage.
143We have the following module hierarchy under Streamly:
145* Data: This is a generic bucket for basic data structures a la the `base`
146 package's `Data` hierarchy.
147 * Streamly.Data.Array.Generic
149 Streams can be classified under `Data` or `Control`. Though they are
150 mostly used for processing, they can also be used to store data in
151 memory.
152 * Streamly.Data.Stream
154 The following modules could in fact be classified under `Control`
155 too as they are about processing of data rather than data itself:
156 * Streamly.Data.Unfold
157 * Streamly.Data.Fold
158 * Streamly.Data.Parser
160* Unicode: Unicode text processing:
161 * Streamly.Unicode.Char -- operations on individual chars
162 * Streamly.Unicode.Stream -- operations on streams of Char
163 * Streamly.Unicode.Array.Char -- compact strings of UTF-32 chars
164 * Streamly.Unicode.Array.Utf8 -- compact strings of UTF-8 encoded chars
166* FileSystem: This name space is for data structures that reside in files
167 provided by a file system interface on top of storage devices.
169* Network: This name space is for APIs that access data from remote computers
170 over the network.
172## Stream modules
174By default the streaming modules are effectful. The basic effectful
175stream types are:
177* `Streamly.Data.Stream`
178* `Streamly.Data.Stream.Async`
179* `Streamly.Data.Stream.Ahead`
180* `Streamly.Data.Stream.Parallel`
181* `Streamly.Data.Stream.IsStream` -- polymorphic operations
182* `Streamly.Data.Stream.Using` -- e.g. mapMUsing consMAsync
184The above streams have an append-like multi-stream combining behavior
185i.e. `concatMap` and `bind` would by default evaluate the streams one
186after another. Alternative implementations of `concatMap` and `bind` are
187possible. We can either use rebindable syntax to use a different bind or
188define newtypes with a different bind behavior, all other operations for
189these remain the same as the base type:
191* `Streamly.Data.Stream.Zip`
192* `Streamly.Data.Stream.Interleaved`
193* `Streamly.Data.Stream.RoundRobin`
194* `Streamly.Data.Stream.Async.Zip`
195* `Streamly.Data.Stream.Async.Interleaved`
196* `Streamly.Data.Stream.Async.RoundRobin`
197* ...
199Pure streams are a special case of effectful streams and have the same
200interface as lists, so we put them under `Streamly.Data.List`:
202* `Streamly.Data.List`
203* `Streamly.Data.List.Zip`
204* `Streamly.Data.List.Interleaved`
205* `Streamly.Data.List.RoundRobin`
206* ...
208We could possibly use the same type named `Stream` for all stream
209types, as the names of all stream operation are also the same and we
210distinguish only by the module name.
212## Array modules
214Similarly, the immutable Array modules would go in:
216* `Streamly.Data.Array.Generic` -- unpinned, native memory arrays
217* `Streamly.Data.Array.Storable` -- unpinned, unboxed, native memory arrays
218* `Streamly.Data.Array.Storable.Pinned` -- pinned, unboxed, native memory arrays
219* `Streamly.Data.Array.Foreign` -- pinned, unboxed, foreign capable arrays
221Unboxed arrays, based on `Prim` type class:
223* `Streamly.Data.Array.Prim`
224* `Streamly.Data.Array.Prim.Pinned`
226Mutable arrays are a generalization of immutable arrays:
228* `Streamly.Data.MutArray`
229* `Streamly.Data.Array.Storable.Mut`
230* `Streamly.Data.Array.Storable.Pinned.Mut`
231* ...
233## Stream and Fold Channels (SVar)
235* `Streamly.Data.Stream.Channel`
236* `Streamly.Data.Stream.Channel.Storable`
237* `Streamly.Data.Fold.Channel`
238* `Streamly.Data.Fold.Channel.Storable`
240## Mutable variables
242Unboxed IORef:
244* `Streamly.Data.IORef.Unboxed`
246## Strict Data
248* `Streamly.Data.Tuple.Strict`
249* `Streamly.Data.Maybe.Strict`
250* `Streamly.Data.Either.Strict`