@@ -1,5 +1,7 @@
11# Internal vs External Modules
22
3+## Exposing Internal Modules
4+
35We keep all modules exposed to facilitate convenient exposure of experimental
46APIs and constructors to users. It allows users of the library to experiment
57much more easily and carry a caveat that these APIs can change in future
@@ -7,25 +9,31 @@ without notice. Since everything is exposed, maintainers do not have to think
79about what to expose as experimental and what remains completely hidden every
810time something is added to the library.
911
10−We expose the internal modules via `Streamly.Internal` namespace to keep all
12+## Internal module Namespace
13+
14+We expose the internal modules via `Streamly.Internal.*` namespace to keep all
1115the internal modules together under one module tree and to have their
1216documentation also separated under one head in haddock docs.
1317
18+## Exposed Modules as Wrappers
19+
1420Another decision point is about two choices:
1521
16−1) Keep the implementation of all the APIs in an internal module and just
22+* Keep the implementation of all the APIs in an internal module and just
1723reexport selected APIs through the external module. The disadvantages of this
1824are:
19−a) users may not be able to easily figure out what unexposed APIs are available
20−other than the ones exposed through the external module. To avoid this problem
21−we can mark the unexposed APIs in the docs with a special comment.
22−b) In tests and benchmarks we will be using internal modules to test internal
23−and unexposed APIs. Since exposed APIs are exported via both internal and
24−external modules we will have to be careful in not using the internal module
25−for testing accidentally, instead we should always be using the exposed module
26−so that we are always testing exactly the way users will be using the APIs.
27−
28−2) Keep the implementations of unexposed modules in the internal module file
25+
26+ 1) users may not be able to easily figure out what unexposed APIs are available
27+ other than the ones exposed through the external module. To avoid this problem
28+ we can mark the unexposed APIs in the docs with a special comment.
29+
30+ 2) In tests and benchmarks we will be using internal modules to test internal
31+ and unexposed APIs. Since exposed APIs are exported via both internal and
32+ external modules we will have to be careful in not using the internal module
33+ for testing accidentally, instead we should always be using the exposed module
34+ so that we are always testing exactly the way users will be using the APIs.
35+
36+* Keep the implementations of unexposed modules in the internal module file
2937and exposed module in the external module file. In this approach, users can
3038easily figure out the unexposed vs exposed APIs. But maintaining this would
3139require us to move the APIs from internal to external module file whenever we
@@ -33,93 +41,104 @@ expose an API.
3341
3442We choose the first approach.
3543
36−# Module Name Spaces
44+# Module Types and Naming
3745
38−We use the "Streamly" prefix to all the module names so that they do not
39−conflict with any other module on Hackage.
46+## Abstract modules
4047
41−We have the following module hierarchy under Streamly:
48+An abstract module represents an abstract interface using a type
49+class. Multiple concrete modules can make use of this interface and possibly
50+extend it.
4251
43−* Data: All the data structures that make use of the unpinned GC memory to
44− store data. These data structures are suitable for stream processing but
45− may not be suitable for storing large amounts of data in memory for longer
46− durations. These are suitable for short lived and smaller structures
47− because they can be moved by the GC to defragment the heap.
52+For example we have an `IsStream` type class that all stream types
53+implement. The concrete stream types are `SerialT`, `AsyncT`
54+etc. Assuming `Streamly.Data.Stream` is the containing name space for
55+all stream modules, there are multiple ways to organize abstract and
56+concrete modules. The `IsStream` type class can be placed in:
4857
49−* Memory: This name space is for data structures that make use of the memory as
50− a persistent storage device. The memory may be allocated by foreign
51− allocators or pinned memory allocated by GHC. Because the memory is pinned it
52− can be used for interfacing with the system/kernel. These structures are
53− efficient for storing large amounts of data for longer durations because it
54− does not have to be copied by the GC. These structures may not be suitable
55− for small, short lived data because that is likely to fragment the heap.
58+* `Streamly.Data.Stream.Class` or `Streamly.Data.Stream.Interface`
59+* `Streamly.Data.Stream.IsStream`
60+* `Streamly.Data.Stream`
5661
57−* FileSystem: This name space is for data structures that reside in files
58− provided by a file system interface on top of storage devices.
62+Using `Streamly.Data.Stream.IsStream` is preferred over `.Class` because it
63+conveys more information by the class name. We can place it in
64+`Streamly.Data.Stream` as well.
5965
60−* Network: This name space is for APIs that access data from remote computers
61− over the network.
66+Concrete modules depend on the abstract module `IsStream` to implement
67+that interface.
6268
63−## Data and Memory
69+Polymorphic operations utilizing the abstract interface can go in the
70+parent module `Streamly.Data.Stream`.
6471
65−As explained above, we distinguish two types of data structures under "Data"
66−and "Memory". Alternatively, we could have used a "Memory" namespace under
67−each data structure e.g. "Streamly.Data.Array.Memory" instead of using a top
68−level "Streamly.Memory", however, we chose to distinguish such data structures
69−using a top level "Memory" name space because it enforces consistent naming by
70−fitting all such data structures under this top level hierarchy. It also makes
71−it easier to find out what all data structures fall in this category.
72+Concrete/monomorphic pure stream modules would be placed in:
7273
73−# Module Types and Naming
74+* `Streamly.Data.Stream.Serial`
75+* `Streamly.Data.Stream.WSerial`
76+* `Streamly.Data.Stream.ZipSerial`
77+* ...
7478
75−## Abstract modules
79+Monadic/effectful streams could go in:
7680
77−Abstract modules are meant to represent an abstract interface (e.g. a type
78−class). Concrete modules can make use of this interface and possibly
79−extend it to provide concrete functionality.
81+* `Streamly.Data.StreamM.Serial`
82+* `Streamly.Data.StreamM.WSerial`
83+* `Streamly.Data.StreamM.Async`
84+* ...
8085
81−The general convention in the Haskell ecosystem for naming an abstract
82−interface module is to name it as "Module.Class" (e.g. Control.Monad.IO.Class).
83−An alternative name could be "Module.Interface".
86+Pure streams are just a special case of monadic streams.
8487
85−In some other cases such modules are named after the class name (e.g. see the
86−array package for an example). This is more appropriate when there is no single
87−hierarchy where we can place the ".Class" module. For example, we have arrays
88−in Data.Array, Memory.Array, we have to choose one over the other to place the
89−".Class" module for an array abstraction. Alternatively, we can choose
90−"Data.IsArray" instead.
88+We could possibly use the same type named `Stream` for all stream types, as we
89+also have all the operation names also same and we distinguish only by the
90+module name.
9191
92−Yet another way could be to use the parent module as an interface module and
93−the child modules as concrete modules. For example, "Streamly.Data.Stream"
94−module could provide the common "Stream" type and the "IsStream" type class.
95−The submodule "Streamly.Data.Stream.Serial" can provide a concrete "Serial"
96−stream type importing the "Streamly.Data.Stream" abstract module.
92+## Constrained Modules
93+
94+Some modules represent operations on a type which constrain a type using a type
95+class or a specific instance of a general type. For example, we may have Arrays
96+that operate on a `Storable` or a `Prim` type.
97+
98+One possible way to organize such module is to have a `Storable` or `Prim`
99+hierarchy and all data structures using that type constraint are bundled under
100+it. However, in general, a data structure may have multiple such
101+constraints or may have to be organized based on some other dimension
102+like an abstract interface it is implementing.
103+
104+General purpose constraints like `Prim` can be defined in their own module
105+hierarchy and can be used everywhere. For example, we can have the following
106+Array types, here we have organized the types under the `Array` hierarchy
107+rather than putting the `PrimArray` under a `Prim` hierarchy.
108+
109+```
110+Streamly.Internal.Data.Array.Boxed
111+Streamly.Internal.Data.Array.Prim
112+Streamly.Internal.Data.Array.Prim.Pinned
113+
114+Streamly.Internal.Data.ArrayM.Boxed
115+Streamly.Internal.Data.ArrayM.Prim
116+Streamly.Internal.Data.ArrayM.Prim.Pinned
117+```
118+
119+Pure arrays are just a special case of mutable arrays.
120+
121+We use the name `ArrayM` insted of `MArray` consistent with the naming
122+of monadic operations like `mapM` and also because `Array` and `ArrayM`
123+are listed together in alphabetical listing, plus camel case naming of the
124+latter is clearer to read.
125+
126+We could use an `IsArray` type class, like `IsStream`, but it will
127+require the `Prim` constraint on all polymorphic operations, rendering
128+it of little use. So the `Array` module, unlike in streams, is just a
129+name space placeholder here. We could assign the `Array` module to the
130+Boxed array module but, having an explicit `Boxed` module along with
131+other types of arrays keeps naming explicit and clearer.
97132
98133## Common Modules
99134
100135Some modules represent common types or utility functions that are shared across
101−multiple modules. The general convention is to name such modules as
102−"Module.Types", "Module.Common", or "Module.Core".
136+multiple higher level modules. Possible naming for such modules are:
103137
104−## Constrained Modules
105−
106−Some modules represent operations on a type which constrain a type using a type
107−class or a specific instance of a general type. For example, we may have a
108−module representing operations on a stream of any type and another module that
109−specifically deals with operations on a Char stream. There are two ways to deal
110−with this.
111−
112−First is to use a submodule for the constrained type. For example,
113−`Streamly.Data.Stream` represents a general stream type whereas
114−`Streamly.Data.Stream.Char` represents operations on a stream of Char type.
115−This makes sense where the type we are constraining to is a specific type
116−rather than a type constrained using a type class.
117−
118−Second is to use a separate hierarchy for the constrained type. For example, we
119−could use `Streamly.Data.Array` for a general array and `Streamly.Prim.Array`
120−for an array that works on `Prim` types. This makes sense when the type is
121−constrained by a type class, we may have more data structures for that
122−constrained type to be bundled under that hierarchy.
138+* `Module.Types`
139+* `Module.Common`
140+* `Module.Core`
141+* `Module.Shared`
123142
124143## Aggregate modules
125144
@@ -132,3 +151,45 @@ the child modules and exposes the functionality from the constituent modules.
132151
133152In some cases a parent module is just a placeholder in the namespace and does
134153not export any functionality.
154+
155+# Polymorphic vs Monomorphic Modules
156+
157+In general we can just provide a polymorphic stream API and let the user use it
158+at the type he/she wants to use it. However, it has some disadvantages:
159+
160+* Some constraints e.g. "MonadAsync" are unnecessarily imposed on all APIs even
161+ though they are needed by only concurrent types.
162+* type errors could be harder to resolve when using polymorphic types
163+* combinators like `asyncly` can make all the combinators concurrent in one go
164+ which is usually problematic. If we use monorphic combinators it encourages
165+ to pick the required concurrent combinators one at a time which is
166+ usually better for performance.
167+
168+Keeping this in mind our plan is to provide monomoprhic modules for each stream
169+type, keep the combinators that are specific to that stream type in the
170+monomorphic module and combinators that are exactly the same for all stream
171+types can be kept in the polymorphic module or in both the modules. Having
172+complete set of operations available in the monomorphic module has the
173+advantage that if we want we can just import a `Serial` module and get
174+everything if we just want to use the Serial stream.
175+
176+# Streamly Modules
177+
178+We use the "Streamly" prefix to all the module names so that they do not
179+conflict with any other module on Hackage.
180+
181+We have the following module hierarchy under Streamly:
182+
183+* Data: This is a generic bucket for basic data structures a la the `base`
184+ package's `Data` hierarchy.
185+
186+* Unicode: Unicode text processing:
187+ * Streamly.Unicode.Char
188+ * Streamly.Unicode.Stream
189+ * Streamly.Unicode.Array
190+
191+* FileSystem: This name space is for data structures that reside in files
192+ provided by a file system interface on top of storage devices.
193+
194+* Network: This name space is for APIs that access data from remote computers
195+ over the network.