@@ -16,57 +16,40 @@
1616-- Stability : experimental
1717-- Portability : GHC
1818--
19−-- Arrays are the computing duals of streams. Streams are good at sequential
20−-- access and immutable transformations of in-transit data whereas arrays are
21−-- good at random access and in-place transformations of buffered data. Unlike
22−-- streams which are potentially infinite, arrays are necessarily /finite/.
23−-- Arrays can be used as an efficient interface between streams and external
24−-- storage systems like memory, files and network. Streams and arrays complete
25−-- each other to provide a general purpose computing system. The design of
26−-- streamly as a general purpose computing framework is centered around these
27−-- two fundamental aspects of computing and storage.
19+-- This module provides immutable arrays suitable for storage and random
20+-- access. Once created an array cannot be mutated without copying. Arrays in
21+-- this module are chunks of memory that hold a sequence of 'Storable' values
22+-- of a given type. They are designed to store serializable data, they cannot
23+-- store non-serializable data like functions, this is reflected by the
24+-- 'Storable' constraint. For efficient buffering of data, arrays use pinned
25+-- memory and therefore can hold arbitrary number of elements with a single
26+-- movable pointer visible to GC, therefore, adding no pressure to GC.
27+-- Moreover, pinned memory allows communication with foreign consumers and
28+-- producers (e.g. file or network IO) without copying the data.
29+--
30+-- By design, there are no array transformation operations provided in this
31+-- module, only conversion to and from stream is provided. Arrays can be
32+-- operated upon /efficiently/ by converting them into a stream, applying the
33+-- desired stream transformations from "Streamly.Prelude" and then converting
34+-- it back to an array.
2835--
29−-- Even though the implementation allows for in-place mutation, the current
30−-- implementation of arrays is immutable. Arrays are of fixed size. The size of
31−-- an array is fixed at creation and cannot be changed later without copying.
32−-- Arrays are chunks of memory that hold a sequence of 'Storable' values of a
33−-- given type. They are designed to store serializable data, they cannot store
34−-- non-serializable data like functions, this is reflected by the 'Storable'
35−-- constraint. For efficient buffering of data, arrays use pinned memory and
36−-- therefore can hold arbitrary number of elements with a single movable
37−-- pointer visible to GC, therefore, adding no pressure to GC. Moreover,
38−-- pinned memory allows communication with foreign consumers and producers
39−-- (e.g. file or network IO) without copying the data.
36+-- This module is designed to be imported qualified:
37+--
38+-- > import qualified Streamly.Array as A
39+
40+-- To summarize:
41+--
42+-- Arrays are finite and fixed in size
43+-- provide /O(1)/ access to elements
44+-- store only data and not functions
45+-- provide efficient IO interfacing
4046--
41−-- By design, there are no transformation operations provided in this module,
42−-- only stream conversion and IO routines are provided. An array is purely a
43−-- data store, therefore, it is not a 'Functor'. Arrays can be operated upon
44−-- /efficiently/ by converting them into a stream, applying the desired
45−-- transformations on the stream and then converting it back to an array.
4647-- 'Foldable' instance is not provided because the implementation would be much
4748-- less efficient compared to folding via streams. 'Semigroup' and 'Monoid'
4849-- instances are deliberately not provided to avoid misuse; concatenating
4950-- arrays using binary operations can be highly inefficient. Instead, use
50−-- 'spliceArrays' to concatenate N arrays at once.
51−--
52−-- To summarize:
53−--
54−-- * Arrays are finite and fixed in size
55−-- * provide /O(1)/ access to elements
56−-- * store only data and not functions
57−-- * provide efficient IO interfacing
51+-- 'Streamly.Mem.Array.Stream.toArray' to concatenate N arrays at once.
5852--
59−-- 'ByteString' data type from the 'bytestring' package and the 'Text' data
60−-- type from the 'text' package are just special cases of arrays. 'ByteString'
61−-- is equivalent to @Array Word8@ and 'Text' is equivalent to a @utf16@ encoded
62−-- @Array Word8@. All the 'bytestring' and 'text' operations can be performed
63−-- on arrays with equivalent or better performance by converting them to and
64−-- from streams.
65−--
66−-- This module is designed to be imported qualified:
67−--
68−-- > import qualified Streamly.Array as A
69−
7053-- Each array is one pointer visible to the GC. Too many small arrays (e.g.
7154-- single byte) are only as good as holding those elements in a Haskell list.
7255-- However, small arrays can be compacted into large ones to reduce the
@@ -100,9 +83,8 @@ module Streamly.Mem.Array
10083 , write
10184
10285 -- Stream Folds
103− , A.toArrayN
104− -- , toArrays
105− , toArray
86+ , A.writeNF
87+ , writeF
10688
10789 -- * Elimination
10890 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
@@ -128,12 +110,14 @@ module Streamly.Mem.Array
128110 , writeSliceRev
129111 -}
130112
113+ {-
131114 -- * Immutable Transformations
132115 , runTransform
133116
134117 -- * Folding Arrays
135118 -- , runStreamFold
136119 , runFold
120+ -}
137121 )
138122where
139123
@@ -179,6 +163,64 @@ writeN n m = do
179163 if n < 0 then error "writeN: negative write count specified" else return ()
180164 A.fromStreamDN n $ D.toStreamD m
181165
166+-- XXX The realloc based implementation needs to make one extra copy if we use
167+-- shrinkToFit. On the other hand, the stream of arrays implementation may
168+-- buffer the array chunk pointers in memory but it does not have to shrink as
169+-- we know the exact size in the end. However, memory copying does not seems to
170+-- be as expensive as the allocations. Therefore, we need to reduce the number
171+-- of allocations instead. Also, the size of allocations matters, right sizing
172+-- an allocation even at the cost of copying sems to help. Should be measured
173+-- on a big stream with heavy calls to toArray to see the effect.
174+--
175+-- XXX check if GHC's memory allocator is efficient enough. We can try the C
176+-- malloc to compare against.
177+
178+{-# INLINE_NORMAL toArrayMinChunk #-}
179+toArrayMinChunk :: forall m a. (MonadIO m, Storable a)
180+ => Int -> Fold m a (Array a)
181+-- toArrayMinChunk n = FL.mapM spliceArrays $ toArraysOf n
182+toArrayMinChunk elemCount = Fold step initial extract
183+
184+ where
185+
186+ insertElem (Array start end bound) x = do
187+ liftIO $ poke end x
188+ return $ Array start (end `plusPtr` sizeOf (undefined :: a)) bound
189+
190+ initial = do
191+ when (elemCount < 0) $ error "toArrayMinChunk: elemCount is negative"
192+ liftIO $ A.newArray elemCount
193+ step arr@(Array start end bound) x | end == bound = do
194+ let p = unsafeForeignPtrToPtr start
195+ oldSize = end `minusPtr` p
196+ newSize = max (oldSize * 2) 1
197+ arr1 <- liftIO $ A.realloc newSize arr
198+ insertElem arr1 x
199+ step arr x = insertElem arr x
200+ extract = liftIO . A.shrinkToFit
201+
202+-- | Fold the whole input to a single array.
203+--
204+-- /Caution! Do not use this on infinite streams./
205+--
206+-- @since 0.7.0
207+{-# INLINE writeF #-}
208+writeF :: forall m a. (MonadIO m, Storable a) => Fold m a (Array a)
209+writeF = toArrayMinChunk (A.bytesToCount (undefined :: a) (A.mkChunkSize 1024))
210+
211+-- | Create an 'Array' from a stream. This is useful when we want to create a
212+-- single array from a stream of unknown size. 'writeN' is at least twice
213+-- as efficient when the size is already known.
214+--
215+-- Note that if the input stream is too large memory allocation for the array
216+-- may fail. When the stream size is not known, `arraysOf` followed by
217+-- processing of indvidual arrays in the resulting stream should be preferred.
218+--
219+{-# INLINE write #-}
220+write :: (MonadIO m, Storable a) => SerialT m a -> m (Array a)
221+write = P.runFold writeF
222+-- write m = A.fromStreamD $ D.toStreamD m
223+
182224-------------------------------------------------------------------------------
183225-- Elimination
184226-------------------------------------------------------------------------------
@@ -373,93 +415,6 @@ writeSliceRev :: (IsStream t, Monad m, Storable a)
373415writeSliceRev arr i len s = undefined
374416-}
375417
376−-------------------------------------------------------------------------------
377−-- Streams of Arrays
378−-------------------------------------------------------------------------------
379−
380−-- exponentially increasing sizes of the chunks upto the max limit.
381−-- XXX this will be easier to implement with parsers/terminating folds
382−-- With this we should be able to reduce the number of chunks/allocations.
383−-- The reallocation/copy based toArray can also be implemented using this.
384−--
385−{-
386−{-# INLINE toArraysInRange #-}
387−toArraysInRange :: (IsStream t, MonadIO m, Storable a)
388− => Int -> Int -> Fold m (Array a) b -> Fold m a b
389−toArraysInRange low high (Fold step initial extract) =
390−-}
391−
392−{-
393−-- | Fold the input to a pure buffered stream (List) of arrays.
394−{-# INLINE _toArraysOf #-}
395−_toArraysOf :: (MonadIO m, Storable a)
396− => Int -> Fold m a (SerialT Identity (Array a))
397−_toArraysOf n = FL.lchunksOf n (A.toArrayN n) FL.toStream
398−-}
399−
400−-- XXX The realloc based implementation needs to make one extra copy if we use
401−-- shrinkToFit. On the other hand, the stream of arrays implementation may
402−-- buffer the array chunk pointers in memory but it does not have to shrink as
403−-- we know the exact size in the end. However, memory copying does not seems to
404−-- be as expensive as the allocations. Therefore, we need to reduce the number
405−-- of allocations instead. Also, the size of allocations matters, right sizing
406−-- an allocation even at the cost of copying sems to help. Should be measured
407−-- on a big stream with heavy calls to toArray to see the effect.
408−--
409−-- XXX check if GHC's memory allocator is efficient enough. We can try the C
410−-- malloc to compare against.
411−
412−{-# INLINE_NORMAL toArrayMinChunk #-}
413−toArrayMinChunk :: forall m a. (MonadIO m, Storable a)
414− => Int -> Fold m a (Array a)
415−-- toArrayMinChunk n = FL.mapM spliceArrays $ toArraysOf n
416−toArrayMinChunk elemCount = Fold step initial extract
417−
418− where
419−
420− insertElem (Array start end bound) x = do
421− liftIO $ poke end x
422− return $ Array start (end `plusPtr` sizeOf (undefined :: a)) bound
423−
424− initial = do
425− when (elemCount < 0) $ error "toArrayMinChunk: elemCount is negative"
426− liftIO $ A.newArray elemCount
427− step arr@(Array start end bound) x | end == bound = do
428− let p = unsafeForeignPtrToPtr start
429− oldSize = end `minusPtr` p
430− newSize = max (oldSize * 2) 1
431− arr1 <- liftIO $ A.realloc newSize arr
432− insertElem arr1 x
433− step arr x = insertElem arr x
434− extract = liftIO . A.shrinkToFit
435−
436−-- | Fold the whole input to a single array.
437−--
438−-- /Caution! Do not use this on infinite streams./
439−--
440−-- @since 0.7.0
441−{-# INLINE toArray #-}
442−toArray :: forall m a. (MonadIO m, Storable a) => Fold m a (Array a)
443−toArray =
444− toArrayMinChunk (A.bytesToCount (undefined :: a) (A.mkChunkSize 1024))
445−
446−-- | Create an 'Array' from a stream. This is useful when we want to create a
447−-- single array from a stream of unknown size. 'writeN' is at least twice
448−-- as efficient when the size is already known.
449−--
450−-- Note that if the input stream is too large memory allocation for the array
451−-- may fail. When the stream size is not known, `arraysOf` followed by
452−-- processing of indvidual arrays in the resulting stream should be preferred.
453−--
454−{-# INLINE write #-}
455−write :: (MonadIO m, Storable a) => SerialT m a -> m (Array a)
456−write = P.runFold toArray
457−-- write m = A.fromStreamD $ D.toStreamD m
458−
459−-- XXX efficiently compare two streams of arrays. Two streams can have chunks
460−-- of different sizes, we can handle that in the stream comparison abstraction.
461−-- This could be useful e.g. to fast compare whether two files differ.
462−
463418-------------------------------------------------------------------------------
464419-- Transform via stream operations
465420-------------------------------------------------------------------------------
@@ -483,17 +438,17 @@ runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
483438-- operation.
484439--
485440-- @since 0.7.0
486−{-# INLINE runTransform #-}
487−runTransform :: (MonadIO m, Storable a, Storable b)
441+{-# INLINE _runTransform #-}
442+_runTransform :: (MonadIO m, Storable a, Storable b)
488443 => (SerialT m a -> SerialT m b) -> Array a -> m (Array b)
489−runTransform f arr = P.runFold (toArrayMinChunk (length arr)) $ f (A.read arr)
444+_runTransform f arr = P.runFold (toArrayMinChunk (length arr)) $ f (A.read arr)
490445
491446-- | Fold an array using a 'Fold'.
492447--
493448-- @since 0.7.0
494−{-# INLINE runFold #-}
495−runFold :: forall m a b. (MonadIO m, Storable a) => Fold m a b -> Array a -> m b
496−runFold f arr = P.runFold f $ (A.read arr :: Serial.SerialT m a)
449+{-# INLINE _runFold #-}
450+_runFold :: forall m a b. (MonadIO m, Storable a) => Fold m a b -> Array a -> m b
451+_runFold f arr = P.runFold f $ (A.read arr :: Serial.SerialT m a)
497452
498453-- | Fold an array using a stream fold operation.
499454--