@@ -1,4 +1,4 @@
1−# Internal vs External
1+# Internal vs External Modules
22
33We keep all modules exposed to facilitate convenient exposure of experimental
44APIs and constructors to users. It allows users of the library to experiment
@@ -33,27 +33,102 @@ expose an API.
3333
3434We choose the first approach.
3535
36−# Name Spaces
36+# Module Name Spaces
3737
3838We use the "Streamly" prefix to all the module names so that they do not
3939conflict with any other module on Hackage.
4040
41−We have the following division under Streamly:
41+We have the following module hierarchy under Streamly:
4242
43−* Data: All the data structures that are immutable and make use of the GC
44− memory to store data. These data structures are mainly useful for stream
45− processing and not for storing data in memory for longer durations.
43+* Data: All the data structures that make use of the unpinned GC memory to
44+ store data. These data structures are suitable for stream processing but
45+ may not be suitable for storing large amounts of data in memory for longer
46+ durations. These are suitable for short lived and smaller structures
47+ because they can be moved by the GC to defragment the heap.
4648
4749* Memory: This name space is for data structures that make use of the memory as
48− a storage device. This is not GC allocated memory but rather pinned memory.
49− The memory in this case is treated just like any other storage device e.g.
50− disk drives, but faster. The data is serialized into memory. Unlike the data
51− structure under "Data" these data structures have no impact on GC. Many of
52− the data structures in this category are parallel to the data structures
53− under "FileSystem".
50+ a persistent storage device. The memory may be allocated by foreign
51+ allocators or pinned memory allocated by GHC. Because the memory is pinned it
52+ can be used for interfacing with the system/kernel. These structures are
53+ efficient for storing large amounts of data for longer durations because it
54+ does not have to be copied by the GC. These structures may not be suitable
55+ for small, short lived data because that is likely to fragment the heap.
5456
5557* FileSystem: This name space is for data structures that reside in files
5658 provided by a file system interface on top of storage devices.
5759
5860* Network: This name space is for APIs that access data from remote computers
5961 over the network.
62+
63+## Data and Memory
64+
65+As explained above, we distinguish two types of data structures under "Data"
66+and "Memory". Alternatively, we could have used a "Memory" namespace under
67+each data structure e.g. "Streamly.Data.Array.Memory" instead of using a top
68+level "Streamly.Memory", however, we chose to distinguish such data structures
69+using a top level "Memory" name space because it enforces consistent naming by
70+fitting all such data structures under this top level hierarchy. It also makes
71+it easier to find out what all data structures fall in this category.
72+
73+# Module Types and Naming
74+
75+## Abstract modules
76+
77+Abstract modules are meant to represent an abstract interface (e.g. a type
78+class). Concrete modules can make use of this interface and possibly
79+extend it to provide concrete functionality.
80+
81+The general convention in the Haskell ecosystem for naming an abstract
82+interface module is to name it as "Module.Class" (e.g. Control.Monad.IO.Class).
83+An alternative name could be "Module.Interface".
84+
85+In some other cases such modules are named after the class name (e.g. see the
86+array package for an example). This is more appropriate when there is no single
87+hierarchy where we can place the ".Class" module. For example, we have arrays
88+in Data.Array, Memory.Array, we have to choose one over the other to place the
89+".Class" module for an array abstraction. Alternatively, we can choose
90+"Data.IsArray" instead.
91+
92+Yet another way could be to use the parent module as an interface module and
93+the child modules as concrete modules. For example, "Streamly.Data.Stream"
94+module could provide the common "Stream" type and the "IsStream" type class.
95+The submodule "Streamly.Data.Stream.Serial" can provide a concrete "Serial"
96+stream type importing the "Streamly.Data.Stream" abstract module.
97+
98+## Common Modules
99+
100+Some modules represent common types or utility functions that are shared across
101+multiple modules. The general convention is to name such modules as
102+"Module.Types", "Module.Common", or "Module.Core".
103+
104+## Constrained Modules
105+
106+Some modules represent operations on a type which constrain a type using a type
107+class or a specific instance of a general type. For example, we may have a
108+module representing operations on a stream of any type and another module that
109+specifically deals with operations on a Char stream. There are two ways to deal
110+with this.
111+
112+First is to use a submodule for the constrained type. For example,
113+`Streamly.Data.Stream` represents a general stream type whereas
114+`Streamly.Data.Stream.Char` represents operations on a stream of Char type.
115+This makes sense where the type we are constraining to is a specific type
116+rather than a type constrained using a type class.
117+
118+Second is to use a separate hierarchy for the constrained type. For example, we
119+could use `Streamly.Data.Array` for a general array and `Streamly.Prim.Array`
120+for an array that works on `Prim` types. This makes sense when the type is
121+constrained by a type class, we may have more data structures for that
122+constrained type to be bundled under that hierarchy.
123+
124+## Aggregate modules
125+
126+In some cases we may want to aggregate the functionality of several small
127+modules in a combined aggregate module. In many cases, the aggregate module is
128+made a parent module of the constituent modules. The parent module depends on
129+the child modules and exposes the functionality from the constituent modules.
130+
131+## Placeholder Modules
132+
133+In some cases a parent module is just a placeholder in the namespace and does
134+not export any functionality.