@@ -1,5 +1,7 @@
1# Internal vs External Modules
1# Internal vs External Modules
3+## Exposing Internal Modules
3We keep all modules exposed to facilitate convenient exposure of experimental
5We keep all modules exposed to facilitate convenient exposure of experimental
4APIs and constructors to users. It allows users of the library to experiment
6APIs and constructors to users. It allows users of the library to experiment
5much more easily and carry a caveat that these APIs can change in future
7much more easily and carry a caveat that these APIs can change in future
@@ -7,25 +9,31 @@ without notice. Since everything is exposed, maintainers do not have to think
7about what to expose as experimental and what remains completely hidden every
9about what to expose as experimental and what remains completely hidden every
8time something is added to the library.
10time something is added to the library.
12+## Internal module Namespace
10−We expose the internal modules via `Streamly.Internal` namespace to keep all
14+We expose the internal modules via `Streamly.Internal.*` namespace to keep all
11the internal modules together under one module tree and to have their
15the internal modules together under one module tree and to have their
12documentation also separated under one head in haddock docs.
16documentation also separated under one head in haddock docs.
18+## Exposed Modules as Wrappers
14Another decision point is about two choices:
20Another decision point is about two choices:
16−1) Keep the implementation of all the APIs in an internal module and just
22+* Keep the implementation of all the APIs in an internal module and just
17reexport selected APIs through the external module. The disadvantages of this
23reexport selected APIs through the external module. The disadvantages of this
19−a) users may not be able to easily figure out what unexposed APIs are available
26+ 1) users may not be able to easily figure out what unexposed APIs are available
20−other than the ones exposed through the external module. To avoid this problem
27+ other than the ones exposed through the external module. To avoid this problem
21−we can mark the unexposed APIs in the docs with a special comment.
28+ we can mark the unexposed APIs in the docs with a special comment.
22−b) In tests and benchmarks we will be using internal modules to test internal
30+ 2) In tests and benchmarks we will be using internal modules to test internal
23−and unexposed APIs. Since exposed APIs are exported via both internal and
31+ and unexposed APIs. Since exposed APIs are exported via both internal and
24−external modules we will have to be careful in not using the internal module
32+ external modules we will have to be careful in not using the internal module
25−for testing accidentally, instead we should always be using the exposed module
33+ for testing accidentally, instead we should always be using the exposed module
26−so that we are always testing exactly the way users will be using the APIs.
34+ so that we are always testing exactly the way users will be using the APIs.
28−2) Keep the implementations of unexposed modules in the internal module file
36+* Keep the implementations of unexposed modules in the internal module file
29and exposed module in the external module file. In this approach, users can
37and exposed module in the external module file. In this approach, users can
30easily figure out the unexposed vs exposed APIs. But maintaining this would
38easily figure out the unexposed vs exposed APIs. But maintaining this would
31require us to move the APIs from internal to external module file whenever we
39require us to move the APIs from internal to external module file whenever we
@@ -33,93 +41,104 @@ expose an API.
34We choose the first approach.
42We choose the first approach.
36−# Module Name Spaces
44+# Module Types and Naming
38−We use the "Streamly" prefix to all the module names so that they do not
46+## Abstract modules
39−conflict with any other module on Hackage.
41−We have the following module hierarchy under Streamly:
48+An abstract module represents an abstract interface using a type
49+class. Multiple concrete modules can make use of this interface and possibly
43−* Data: All the data structures that make use of the unpinned GC memory to
52+For example we have an `IsStream` type class that all stream types
44− store data. These data structures are suitable for stream processing but
53+implement. The concrete stream types are `SerialT`, `AsyncT`
45− may not be suitable for storing large amounts of data in memory for longer
54+etc. Assuming `Streamly.Data.Stream` is the containing name space for
46− durations. These are suitable for short lived and smaller structures
55+all stream modules, there are multiple ways to organize abstract and
47− because they can be moved by the GC to defragment the heap.
56+concrete modules. The `IsStream` type class can be placed in:
49−* Memory: This name space is for data structures that make use of the memory as
58+* `Streamly.Data.Stream.Class` or `Streamly.Data.Stream.Interface`
50− a persistent storage device. The memory may be allocated by foreign
59+* `Streamly.Data.Stream.IsStream`
51− allocators or pinned memory allocated by GHC. Because the memory is pinned it
60+* `Streamly.Data.Stream`
52− can be used for interfacing with the system/kernel. These structures are
53− efficient for storing large amounts of data for longer durations because it
54− does not have to be copied by the GC. These structures may not be suitable
55− for small, short lived data because that is likely to fragment the heap.
57−* FileSystem: This name space is for data structures that reside in files
62+Using `Streamly.Data.Stream.IsStream` is preferred over `.Class` because it
58− provided by a file system interface on top of storage devices.
63+conveys more information by the class name. We can place it in
64+`Streamly.Data.Stream` as well.
60−* Network: This name space is for APIs that access data from remote computers
66+Concrete modules depend on the abstract module `IsStream` to implement
61− over the network.
67+that interface.
63−## Data and Memory
69+Polymorphic operations utilizing the abstract interface can go in the
70+parent module `Streamly.Data.Stream`.
65−As explained above, we distinguish two types of data structures under "Data"
72+Concrete/monomorphic pure stream modules would be placed in:
66−and "Memory". Alternatively, we could have used a "Memory" namespace under
67−each data structure e.g. "Streamly.Data.Array.Memory" instead of using a top
68−level "Streamly.Memory", however, we chose to distinguish such data structures
69−using a top level "Memory" name space because it enforces consistent naming by
70−fitting all such data structures under this top level hierarchy. It also makes
71−it easier to find out what all data structures fall in this category.
73−# Module Types and Naming
74+* `Streamly.Data.Stream.Serial`
75+* `Streamly.Data.Stream.WSerial`
76+* `Streamly.Data.Stream.ZipSerial`
75−## Abstract modules
79+Monadic/effectful streams could go in:
77−Abstract modules are meant to represent an abstract interface (e.g. a type
81+* `Streamly.Data.StreamM.Serial`
78−class). Concrete modules can make use of this interface and possibly
82+* `Streamly.Data.StreamM.WSerial`
79−extend it to provide concrete functionality.
83+* `Streamly.Data.StreamM.Async`
81−The general convention in the Haskell ecosystem for naming an abstract
86+Pure streams are just a special case of monadic streams.
82−interface module is to name it as "Module.Class" (e.g. Control.Monad.IO.Class).
83−An alternative name could be "Module.Interface".
85−In some other cases such modules are named after the class name (e.g. see the
88+We could possibly use the same type named `Stream` for all stream types, as we
86−array package for an example). This is more appropriate when there is no single
89+also have all the operation names also same and we distinguish only by the
87−hierarchy where we can place the ".Class" module. For example, we have arrays
90+module name.
88−in Data.Array, Memory.Array, we have to choose one over the other to place the
89−".Class" module for an array abstraction. Alternatively, we can choose
90−"Data.IsArray" instead.
92−Yet another way could be to use the parent module as an interface module and
92+## Constrained Modules
93−the child modules as concrete modules. For example, "Streamly.Data.Stream"
93+
94−module could provide the common "Stream" type and the "IsStream" type class.
94+Some modules represent operations on a type which constrain a type using a type
95−The submodule "Streamly.Data.Stream.Serial" can provide a concrete "Serial"
95+class or a specific instance of a general type. For example, we may have Arrays
96−stream type importing the "Streamly.Data.Stream" abstract module.
96+that operate on a `Storable` or a `Prim` type.
98+One possible way to organize such module is to have a `Storable` or `Prim`
99+hierarchy and all data structures using that type constraint are bundled under
100+it. However, in general, a data structure may have multiple such
101+constraints or may have to be organized based on some other dimension
102+like an abstract interface it is implementing.
104+General purpose constraints like `Prim` can be defined in their own module
105+hierarchy and can be used everywhere. For example, we can have the following
106+Array types, here we have organized the types under the `Array` hierarchy
107+rather than putting the `PrimArray` under a `Prim` hierarchy.
110+Streamly.Internal.Data.Array.Boxed
111+Streamly.Internal.Data.Array.Prim
112+Streamly.Internal.Data.Array.Prim.Pinned
114+Streamly.Internal.Data.ArrayM.Boxed
115+Streamly.Internal.Data.ArrayM.Prim
116+Streamly.Internal.Data.ArrayM.Prim.Pinned
119+Pure arrays are just a special case of mutable arrays.
121+We use the name `ArrayM` insted of `MArray` consistent with the naming
122+of monadic operations like `mapM` and also because `Array` and `ArrayM`
123+are listed together in alphabetical listing, plus camel case naming of the
124+latter is clearer to read.
126+We could use an `IsArray` type class, like `IsStream`, but it will
127+require the `Prim` constraint on all polymorphic operations, rendering
128+it of little use. So the `Array` module, unlike in streams, is just a
129+name space placeholder here. We could assign the `Array` module to the
130+Boxed array module but, having an explicit `Boxed` module along with
131+other types of arrays keeps naming explicit and clearer.
98## Common Modules
133## Common Modules
100Some modules represent common types or utility functions that are shared across
135Some modules represent common types or utility functions that are shared across
101−multiple modules. The general convention is to name such modules as
136+multiple higher level modules. Possible naming for such modules are:
102−"Module.Types", "Module.Common", or "Module.Core".
104−## Constrained Modules
138+* `Module.Types`
105−
139+* `Module.Common`
106−Some modules represent operations on a type which constrain a type using a type
140+* `Module.Core`
107−class or a specific instance of a general type. For example, we may have a
141+* `Module.Shared`
108−module representing operations on a stream of any type and another module that
109−specifically deals with operations on a Char stream. There are two ways to deal
112−First is to use a submodule for the constrained type. For example,
113−`Streamly.Data.Stream` represents a general stream type whereas
114−`Streamly.Data.Stream.Char` represents operations on a stream of Char type.
115−This makes sense where the type we are constraining to is a specific type
116−rather than a type constrained using a type class.
118−Second is to use a separate hierarchy for the constrained type. For example, we
119−could use `Streamly.Data.Array` for a general array and `Streamly.Prim.Array`
120−for an array that works on `Prim` types. This makes sense when the type is
121−constrained by a type class, we may have more data structures for that
122−constrained type to be bundled under that hierarchy.
124## Aggregate modules
143## Aggregate modules
@@ -132,3 +151,45 @@ the child modules and exposes the functionality from the constituent modules.
133In some cases a parent module is just a placeholder in the namespace and does
152In some cases a parent module is just a placeholder in the namespace and does
134not export any functionality.
153not export any functionality.
155+# Polymorphic vs Monomorphic Modules
157+In general we can just provide a polymorphic stream API and let the user use it
158+at the type he/she wants to use it. However, it has some disadvantages:
160+* Some constraints e.g. "MonadAsync" are unnecessarily imposed on all APIs even
161+ though they are needed by only concurrent types.
162+* type errors could be harder to resolve when using polymorphic types
163+* combinators like `asyncly` can make all the combinators concurrent in one go
164+ which is usually problematic. If we use monorphic combinators it encourages
165+ to pick the required concurrent combinators one at a time which is
166+ usually better for performance.
168+Keeping this in mind our plan is to provide monomoprhic modules for each stream
169+type, keep the combinators that are specific to that stream type in the
170+monomorphic module and combinators that are exactly the same for all stream
171+types can be kept in the polymorphic module or in both the modules. Having
172+complete set of operations available in the monomorphic module has the
173+advantage that if we want we can just import a `Serial` module and get
174+everything if we just want to use the Serial stream.
178+We use the "Streamly" prefix to all the module names so that they do not
179+conflict with any other module on Hackage.
181+We have the following module hierarchy under Streamly:
183+* Data: This is a generic bucket for basic data structures a la the `base`
184+ package's `Data` hierarchy.
186+* Unicode: Unicode text processing:
187+ * Streamly.Unicode.Char
188+ * Streamly.Unicode.Stream
189+ * Streamly.Unicode.Array
191+* FileSystem: This name space is for data structures that reside in files
192+ provided by a file system interface on top of storage devices.
194+* Network: This name space is for APIs that access data from remote computers