@@ -16,57 +16,40 @@
16-- Stability : experimental
16-- Stability : experimental
17-- Portability : GHC
17-- Portability : GHC
19−-- Arrays are the computing duals of streams. Streams are good at sequential
19+-- This module provides immutable arrays suitable for storage and random
20−-- access and immutable transformations of in-transit data whereas arrays are
20+-- access. Once created an array cannot be mutated without copying. Arrays in
21−-- good at random access and in-place transformations of buffered data. Unlike
21+-- this module are chunks of memory that hold a sequence of 'Storable' values
22−-- streams which are potentially infinite, arrays are necessarily /finite/.
22+-- of a given type. They are designed to store serializable data, they cannot
23−-- Arrays can be used as an efficient interface between streams and external
23+-- store non-serializable data like functions, this is reflected by the
24−-- storage systems like memory, files and network. Streams and arrays complete
24+-- 'Storable' constraint. For efficient buffering of data, arrays use pinned
25−-- each other to provide a general purpose computing system. The design of
25+-- memory and therefore can hold arbitrary number of elements with a single
26−-- streamly as a general purpose computing framework is centered around these
26+-- movable pointer visible to GC, therefore, adding no pressure to GC.
27−-- two fundamental aspects of computing and storage.
27+-- Moreover, pinned memory allows communication with foreign consumers and
28+-- producers (e.g. file or network IO) without copying the data.
30+-- By design, there are no array transformation operations provided in this
31+-- module, only conversion to and from stream is provided. Arrays can be
32+-- operated upon /efficiently/ by converting them into a stream, applying the
33+-- desired stream transformations from "Streamly.Prelude" and then converting
34+-- it back to an array.
29−-- Even though the implementation allows for in-place mutation, the current
36+-- This module is designed to be imported qualified:
30−-- implementation of arrays is immutable. Arrays are of fixed size. The size of
37+--
31−-- an array is fixed at creation and cannot be changed later without copying.
38+-- > import qualified Streamly.Array as A
32−-- Arrays are chunks of memory that hold a sequence of 'Storable' values of a
39+
33−-- given type. They are designed to store serializable data, they cannot store
40+-- To summarize:
34−-- non-serializable data like functions, this is reflected by the 'Storable'
41+--
35−-- constraint. For efficient buffering of data, arrays use pinned memory and
42+-- Arrays are finite and fixed in size
36−-- therefore can hold arbitrary number of elements with a single movable
43+-- provide /O(1)/ access to elements
37−-- pointer visible to GC, therefore, adding no pressure to GC. Moreover,
44+-- store only data and not functions
38−-- pinned memory allows communication with foreign consumers and producers
45+-- provide efficient IO interfacing
39−-- (e.g. file or network IO) without copying the data.
41−-- By design, there are no transformation operations provided in this module,
42−-- only stream conversion and IO routines are provided. An array is purely a
43−-- data store, therefore, it is not a 'Functor'. Arrays can be operated upon
44−-- /efficiently/ by converting them into a stream, applying the desired
45−-- transformations on the stream and then converting it back to an array.
46-- 'Foldable' instance is not provided because the implementation would be much
47-- 'Foldable' instance is not provided because the implementation would be much
47-- less efficient compared to folding via streams. 'Semigroup' and 'Monoid'
48-- less efficient compared to folding via streams. 'Semigroup' and 'Monoid'
48-- instances are deliberately not provided to avoid misuse; concatenating
49-- instances are deliberately not provided to avoid misuse; concatenating
49-- arrays using binary operations can be highly inefficient. Instead, use
50-- arrays using binary operations can be highly inefficient. Instead, use
50−-- 'spliceArrays' to concatenate N arrays at once.
51+-- 'Streamly.Mem.Array.Stream.toArray' to concatenate N arrays at once.
54−-- * Arrays are finite and fixed in size
55−-- * provide /O(1)/ access to elements
56−-- * store only data and not functions
57−-- * provide efficient IO interfacing
59−-- 'ByteString' data type from the 'bytestring' package and the 'Text' data
60−-- type from the 'text' package are just special cases of arrays. 'ByteString'
61−-- is equivalent to @Array Word8@ and 'Text' is equivalent to a @utf16@ encoded
62−-- @Array Word8@. All the 'bytestring' and 'text' operations can be performed
63−-- on arrays with equivalent or better performance by converting them to and
66−-- This module is designed to be imported qualified:
68−-- > import qualified Streamly.Array as A
70-- Each array is one pointer visible to the GC. Too many small arrays (e.g.
53-- Each array is one pointer visible to the GC. Too many small arrays (e.g.
71-- single byte) are only as good as holding those elements in a Haskell list.
54-- single byte) are only as good as holding those elements in a Haskell list.
72-- However, small arrays can be compacted into large ones to reduce the
55-- However, small arrays can be compacted into large ones to reduce the
@@ -100,9 +83,8 @@ module Streamly.Mem.Array
102 -- Stream Folds
85 -- Stream Folds
103− , A.toArrayN
86+ , A.writeNF
104− -- , toArrays
87+ , writeF
107 -- * Elimination
89 -- * Elimination
108 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
90 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
@@ -128,12 +110,14 @@ module Streamly.Mem.Array
128 , writeSliceRev
110 , writeSliceRev
131 -- * Immutable Transformations
114 -- * Immutable Transformations
132 , runTransform
115 , runTransform
134 -- * Folding Arrays
117 -- * Folding Arrays
135 -- , runStreamFold
118 -- , runStreamFold
136 , runFold
119 , runFold
@@ -179,6 +163,64 @@ writeN n m = do
179 if n < 0 then error "writeN: negative write count specified" else return ()
163 if n < 0 then error "writeN: negative write count specified" else return ()
180 A.fromStreamDN n $ D.toStreamD m
164 A.fromStreamDN n $ D.toStreamD m
166+-- XXX The realloc based implementation needs to make one extra copy if we use
167+-- shrinkToFit. On the other hand, the stream of arrays implementation may
168+-- buffer the array chunk pointers in memory but it does not have to shrink as
169+-- we know the exact size in the end. However, memory copying does not seems to
170+-- be as expensive as the allocations. Therefore, we need to reduce the number
171+-- of allocations instead. Also, the size of allocations matters, right sizing
172+-- an allocation even at the cost of copying sems to help. Should be measured
173+-- on a big stream with heavy calls to toArray to see the effect.
175+-- XXX check if GHC's memory allocator is efficient enough. We can try the C
176+-- malloc to compare against.
178+{-# INLINE_NORMAL toArrayMinChunk #-}
179+toArrayMinChunk :: forall m a. (MonadIO m, Storable a)
180+ => Int -> Fold m a (Array a)
181+-- toArrayMinChunk n = FL.mapM spliceArrays $ toArraysOf n
182+toArrayMinChunk elemCount = Fold step initial extract
186+ insertElem (Array start end bound) x = do
188+ return $ Array start (end `plusPtr` sizeOf (undefined :: a)) bound
191+ when (elemCount < 0) $ error "toArrayMinChunk: elemCount is negative"
192+ liftIO $ A.newArray elemCount
193+ step arr@(Array start end bound) x | end == bound = do
194+ let p = unsafeForeignPtrToPtr start
195+ oldSize = end `minusPtr` p
196+ newSize = max (oldSize * 2) 1
197+ arr1 <- liftIO $ A.realloc newSize arr
199+ step arr x = insertElem arr x
200+ extract = liftIO . A.shrinkToFit
202+-- | Fold the whole input to a single array.
204+-- /Caution! Do not use this on infinite streams./
207+{-# INLINE writeF #-}
208+writeF :: forall m a. (MonadIO m, Storable a) => Fold m a (Array a)
209+writeF = toArrayMinChunk (A.bytesToCount (undefined :: a) (A.mkChunkSize 1024))
211+-- | Create an 'Array' from a stream. This is useful when we want to create a
212+-- single array from a stream of unknown size. 'writeN' is at least twice
213+-- as efficient when the size is already known.
215+-- Note that if the input stream is too large memory allocation for the array
216+-- may fail. When the stream size is not known, `arraysOf` followed by
217+-- processing of indvidual arrays in the resulting stream should be preferred.
220+write :: (MonadIO m, Storable a) => SerialT m a -> m (Array a)
221+write = P.runFold writeF
222+-- write m = A.fromStreamD $ D.toStreamD m
182-------------------------------------------------------------------------------
224-------------------------------------------------------------------------------
183-- Elimination
225-- Elimination
184-------------------------------------------------------------------------------
226-------------------------------------------------------------------------------
@@ -373,93 +415,6 @@ writeSliceRev :: (IsStream t, Monad m, Storable a)
373writeSliceRev arr i len s = undefined
415writeSliceRev arr i len s = undefined
376−-------------------------------------------------------------------------------
378−-------------------------------------------------------------------------------
380−-- exponentially increasing sizes of the chunks upto the max limit.
381−-- XXX this will be easier to implement with parsers/terminating folds
382−-- With this we should be able to reduce the number of chunks/allocations.
383−-- The reallocation/copy based toArray can also be implemented using this.
386−{-# INLINE toArraysInRange #-}
387−toArraysInRange :: (IsStream t, MonadIO m, Storable a)
388− => Int -> Int -> Fold m (Array a) b -> Fold m a b
389−toArraysInRange low high (Fold step initial extract) =
393−-- | Fold the input to a pure buffered stream (List) of arrays.
394−{-# INLINE _toArraysOf #-}
395−_toArraysOf :: (MonadIO m, Storable a)
396− => Int -> Fold m a (SerialT Identity (Array a))
397−_toArraysOf n = FL.lchunksOf n (A.toArrayN n) FL.toStream
400−-- XXX The realloc based implementation needs to make one extra copy if we use
401−-- shrinkToFit. On the other hand, the stream of arrays implementation may
402−-- buffer the array chunk pointers in memory but it does not have to shrink as
403−-- we know the exact size in the end. However, memory copying does not seems to
404−-- be as expensive as the allocations. Therefore, we need to reduce the number
405−-- of allocations instead. Also, the size of allocations matters, right sizing
406−-- an allocation even at the cost of copying sems to help. Should be measured
407−-- on a big stream with heavy calls to toArray to see the effect.
409−-- XXX check if GHC's memory allocator is efficient enough. We can try the C
410−-- malloc to compare against.
412−{-# INLINE_NORMAL toArrayMinChunk #-}
413−toArrayMinChunk :: forall m a. (MonadIO m, Storable a)
414− => Int -> Fold m a (Array a)
415−-- toArrayMinChunk n = FL.mapM spliceArrays $ toArraysOf n
416−toArrayMinChunk elemCount = Fold step initial extract
420− insertElem (Array start end bound) x = do
422− return $ Array start (end `plusPtr` sizeOf (undefined :: a)) bound
425− when (elemCount < 0) $ error "toArrayMinChunk: elemCount is negative"
426− liftIO $ A.newArray elemCount
427− step arr@(Array start end bound) x | end == bound = do
428− let p = unsafeForeignPtrToPtr start
429− oldSize = end `minusPtr` p
430− newSize = max (oldSize * 2) 1
431− arr1 <- liftIO $ A.realloc newSize arr
433− step arr x = insertElem arr x
434− extract = liftIO . A.shrinkToFit
436−-- | Fold the whole input to a single array.
438−-- /Caution! Do not use this on infinite streams./
441−{-# INLINE toArray #-}
442−toArray :: forall m a. (MonadIO m, Storable a) => Fold m a (Array a)
444− toArrayMinChunk (A.bytesToCount (undefined :: a) (A.mkChunkSize 1024))
446−-- | Create an 'Array' from a stream. This is useful when we want to create a
447−-- single array from a stream of unknown size. 'writeN' is at least twice
448−-- as efficient when the size is already known.
450−-- Note that if the input stream is too large memory allocation for the array
451−-- may fail. When the stream size is not known, `arraysOf` followed by
452−-- processing of indvidual arrays in the resulting stream should be preferred.
455−write :: (MonadIO m, Storable a) => SerialT m a -> m (Array a)
456−write = P.runFold toArray
457−-- write m = A.fromStreamD $ D.toStreamD m
459−-- XXX efficiently compare two streams of arrays. Two streams can have chunks
460−-- of different sizes, we can handle that in the stream comparison abstraction.
461−-- This could be useful e.g. to fast compare whether two files differ.
463-------------------------------------------------------------------------------
418-------------------------------------------------------------------------------
464-- Transform via stream operations
419-- Transform via stream operations
465-------------------------------------------------------------------------------
420-------------------------------------------------------------------------------
@@ -483,17 +438,17 @@ runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
483-- operation.
438-- operation.
485-- @since 0.7.0
440-- @since 0.7.0
486−{-# INLINE runTransform #-}
441+{-# INLINE _runTransform #-}
487−runTransform :: (MonadIO m, Storable a, Storable b)
442+_runTransform :: (MonadIO m, Storable a, Storable b)
488 => (SerialT m a -> SerialT m b) -> Array a -> m (Array b)
443 => (SerialT m a -> SerialT m b) -> Array a -> m (Array b)
489−runTransform f arr = P.runFold (toArrayMinChunk (length arr)) $ f (A.read arr)
444+_runTransform f arr = P.runFold (toArrayMinChunk (length arr)) $ f (A.read arr)
491-- | Fold an array using a 'Fold'.
446-- | Fold an array using a 'Fold'.
493-- @since 0.7.0
448-- @since 0.7.0
494−{-# INLINE runFold #-}
449+{-# INLINE _runFold #-}
495−runFold :: forall m a b. (MonadIO m, Storable a) => Fold m a b -> Array a -> m b
450+_runFold :: forall m a b. (MonadIO m, Storable a) => Fold m a b -> Array a -> m b
496−runFold f arr = P.runFold f $ (A.read arr :: Serial.SerialT m a)
451+_runFold f arr = P.runFold f $ (A.read arr :: Serial.SerialT m a)
498-- | Fold an array using a stream fold operation.
453-- | Fold an array using a stream fold operation.