e218fb9 · streamly · git.iko.soy

git.iko.soy/streamly

Deprecate Streamly.Memory.Array

According to the new module structure, it should be Streamly.Data.Array.Storable.Foreign

commite218fb9d8f4263fd69d32a2ff318bac0e179c642 authoradithyaov <adi.obilisetty@gmail.com> dateFri 28 Aug 2020, 03:37 (6 years ago) parentaa1e4e1

30 files changed+115−39

compare with master

unifiedsplit

Changelog.md+1

@@ -2,6 +2,7 @@

33### Breaking changes

5+* Deprecate `Streamly.Memory.Array` in favor of `Streamly.Data.Array.Storable.Foreign`

56* Deprecate `Streamly.Data.Unicode.Stream` in favor of `Streamly.Unicode.Stream`

67* Change the signature of `foldlM'` to make the initial value of the

78 accumulator monadic.

README.md+3−3

@@ -368,7 +368,7 @@ documentation.

368368

369369## Arrays

370370

371−The `Streamly.Memory.Array` module provides immutable arrays. Arrays are the

371+The `Streamly.Data.Array.Storable.Foreign` module provides immutable arrays. Arrays are the

372372computing duals of streams. Streams are good at sequential access and immutable

373373transformations of in-transit data whereas arrays are good at random access and

374374in-place transformations of buffered data. Unlike streams which are potentially

@@ -379,7 +379,7 @@ computing system. The design of streamly as a general purpose computing

379379framework is centered around these two fundamental aspects of computing and

380380storage.

381381

382−`Streamly.Memory.Array` uses pinned memory outside GC and therefore avoid any

382+`Streamly.Data.Array.Storable.Foreign` uses pinned memory outside GC and therefore avoid any

383383GC overhead for the storage in arrays. Streamly allows efficient

384384transformations over arrays using streams. It uses arrays to transfer data to

385385and from the operating system and to store data in memory.

@@ -422,7 +422,7 @@ module Main where

422422

423423import qualified Streamly.Prelude as S

424424import qualified Streamly.Data.Fold as FL

425−import qualified Streamly.Memory.Array as A

425+import qualified Streamly.Data.Array.Storable.Foreign as A

426426import qualified Streamly.FileSystem.Handle as FH

427427import qualified System.IO as FH

428428

benchmark/NanoBenchmarks.hs+1−1

@@ -9,7 +9,7 @@

99import Streamly (SerialT)

1010import Streamly.Internal.Data.SVar (MonadAsync)

1111

12−import qualified Streamly.Memory.Array as A

12+import qualified Streamly.Data.Array.Storable.Foreign as A

1313import qualified Streamly.FileSystem.Handle as FH

1414import qualified Streamly.Internal.FileSystem.Handle as IFH

1515import qualified Streamly.Data.Fold as FL

benchmark/Streamly/Benchmark/Data/ArrayOps.hs+1−1

@@ -35,7 +35,7 @@ import qualified Streamly.Internal.Data.SmallArray as A

3535type Stream = A.SmallArray

3636#elif defined(MEMORY_ARRAY)

3737import qualified GHC.Exts as GHC

38−import qualified Streamly.Memory.Array as A

38+import qualified Streamly.Data.Array.Storable.Foreign as A

3939import qualified Streamly.Internal.Data.Array.Storable.Foreign as A

4040type Stream = A.Array

4141#elif defined(DATA_PRIM_ARRAY)

benchmark/Streamly/Benchmark/Data/Fold.hs+1−1

@@ -24,7 +24,7 @@ import qualified Streamly.Internal.Data.Pipe as Pipe

2424

2525import qualified Streamly.Internal.Data.Sink as Sink

2626

27−import qualified Streamly.Memory.Array as A

27+import qualified Streamly.Data.Array.Storable.Foreign as A

2828import qualified Streamly.Internal.Data.Array.Storable.Foreign as IA

2929import qualified Streamly.Internal.Data.Fold as IFL

3030import qualified Streamly.Internal.Data.Stream.IsStream as IP

benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+1−1

@@ -36,7 +36,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH

3636import qualified Streamly.Internal.Memory.ArrayStream as AS

3737import qualified Streamly.Internal.Unicode.Array.Char as IUA

3838import qualified Streamly.Internal.Data.Stream.IsStream as IP

39−import qualified Streamly.Memory.Array as A

39+import qualified Streamly.Data.Array.Storable.Foreign as A

4040import qualified Streamly.Prelude as S

4141

4242import Gauge hiding (env)

design/inlining.md+3−3

@@ -15,7 +15,7 @@ i.e. phase 0, 1 and 2. We have defined them as follows in `inline.h`:

1515The combinators in `Streamly.Prelude` are defined in terms of combinators in

1616`Streamly.Internal.Data.Stream.StreamD` (Direct style streams) or `Streamly.Internal.Data.Stream.StreamK`

1717(CPS style streams). We convert the stream from `StreamD` to `StreamK`

18−representation or vice versa in some cases.

18+representation or vice versa in some cases.

1919

2020In the first inlining phase (INLINE_EARLY or INLINE) we expand

2121the combinators in `Streamly.Prelude` into

@@ -82,7 +82,7 @@ consume as arguments. This means `StreamD` combinators should not be marked

8282as `INLINE` or `INLINE_EARLY`, instead they should all be marked as

8383`INLINE_NORMAL` because higher order funcitons like `concatMap`/`map`/`mapM`

8484etc are marked as `INLINE_NORMAL`. `StreamD` functions in other modules like

85−`Streamly.Memory.Array` should also follow the same rules.

85+`Streamly.Data.Array.Storable.Foreign` should also follow the same rules.

8686

8787## Stream Fusion

8888

@@ -118,6 +118,6 @@ possible. In most cases it is not necessary, but in some cases it may affect

118118fusion and make a difference of 10x performance or more. For example, using

119119non-strict fields can increase the code size for internal join points and

120120functions created during transformations, which can affect the inlining of

121−these code blocks which in turn can affect stream fusion.

121+these code blocks which in turn can affect stream fusion.

122122

123123See https://gitlab.haskell.org/ghc/ghc/issues/17075 .

examples/FileIOExamples.hs+1−1

@@ -1,6 +1,6 @@

11import qualified Streamly.Prelude as S

22import qualified Streamly.Data.Fold as FL

3−import qualified Streamly.Memory.Array as A

3+import qualified Streamly.Data.Array.Storable.Foreign as A

55import qualified Streamly.Internal.Data.Fold as FL

66import qualified Streamly.Internal.Data.Stream.IsStream as IP

examples/FileSinkServer.hs+1−1

@@ -12,7 +12,7 @@ import System.Environment (getArgs)

1212import Streamly

1313import Streamly.Unicode.Stream

1414import qualified Streamly.FileSystem.Handle as FH

15−import qualified Streamly.Memory.Array as A

15+import qualified Streamly.Data.Array.Storable.Foreign as A

1616import qualified Streamly.Network.Socket as NS

1717import qualified Streamly.Network.Inet.TCP as TCP

1818import qualified Streamly.Prelude as S

examples/HandleIO.hs+1−1

@@ -5,7 +5,7 @@ import System.IO (IOMode(..), hSeek, SeekMode(..))

55import qualified Streamly.Data.Fold as FL

66import qualified Streamly.FileSystem.Handle as FH

77import qualified System.IO as FH

8−import qualified Streamly.Memory.Array as A

8+import qualified Streamly.Data.Array.Storable.Foreign as A

99import qualified Streamly.Prelude as S

1010-- import qualified Streamly.FileSystem.FD as FH

1111

examples/WordClassifier.hs+1−1

@@ -22,7 +22,7 @@ import qualified Streamly.Data.Fold as FL

2222import qualified Streamly.Internal.Data.Fold as IFL

2323import qualified Streamly.Internal.Data.Unfold as IUF

2424import qualified Streamly.Internal.FileSystem.File as File

25−import qualified Streamly.Memory.Array as A

25+import qualified Streamly.Data.Array.Storable.Foreign as A

2626import qualified Streamly.Prelude as S

2727import System.Environment (getArgs)

2828

examples/WordCount.hs+1−1

@@ -43,7 +43,7 @@ import Streamly.Internal.Unicode.Stream

4343import qualified Streamly as S

4444import qualified Streamly.Unicode.Stream as S

4545import qualified Streamly.FileSystem.Handle as FH

46−import qualified Streamly.Memory.Array as A

46+import qualified Streamly.Data.Array.Storable.Foreign as A

4747import qualified Streamly.Prelude as S

4848import qualified Data.Vector.Storable.Mutable as V

4949

src/Streamly.hs+2−2

@@ -252,7 +252,7 @@ import qualified Streamly.Internal.Data.Stream.Async as Async

252252-- > S.length $ S.splitOnSuffix FL.drain 10 $ FH.read fh

253253--

254254-- The following example folds the lines to arrays of 'Word8' using the

255−-- 'Streamly.Memory.Array.writeF' fold and then wraps the lines in square

255+-- 'Streamly.Data.Array.Storable.Foreign.writeF' fold and then wraps the lines in square

256256-- brackets before writing them to standard output using

257257-- 'Streamly.FileSystem.Handle.write':

258258--

@@ -303,7 +303,7 @@ import qualified Streamly.Internal.Data.Stream.Async as Async

303303

304304-- $arrays

305305--

306−-- Streamly arrays (See "Streamly.Memory.Array") complement streams to provide an

306+-- Streamly arrays (See "Streamly.Data.Array.Storable.Foreign") complement streams to provide an

307307-- efficient computing paradigm. Streams are suitable for immutable

308308-- transformations of /potentially infinite/ data using /sequential access/ and

309309-- pipelined transformations whereas arrays are suitable for in-place

src/Streamly/Data/Array/Storable/Foreign.hsnew file+73

@@ -0,0 +1,73 @@

1+#include "inline.hs"

3+-- |

4+-- Module : Streamly.Data.Array.Storable.Foreign

6+--

7+-- License : BSD3

8+-- Maintainer : streamly@composewell.com

9+-- Stability : experimental

10+-- Portability : GHC

11+--

12+-- This module provides immutable arrays in pinned memory (non GC memory)

13+-- suitable for long lived data storage, random access and for interfacing with

14+-- the operating system.

15+--

16+-- Arrays in this module are chunks of pinned memory that hold a sequence of

17+-- 'Storable' values of a given type, they cannot store non-serializable data

18+-- like functions. Once created an array cannot be modified. Pinned memory

19+-- allows efficient buffering of long lived data without adding any impact to

20+-- GC. One array is just one pointer visible to GC and it does not have to be

21+-- copied across generations. Moreover, pinned memory allows communication

22+-- with foreign consumers and producers (e.g. file or network IO) without

23+-- copying the data.

24+--

25+-- = Programmer Notes

26+--

27+-- To apply a transformation to an array use 'read' to unfold the array into a

28+-- stream, apply a transformation on the stream and then use 'write' to fold it

29+-- back to an array.

30+--

31+-- This module is designed to be imported qualified:

32+--

33+-- > import qualified Streamly.Array as A

34+--

35+-- For experimental APIs see "Streamly.Internal.Data.Array.Storable.Foreign".

36+

37+module Streamly.Data.Array.Storable.Foreign

38+ (

39+ A.Array

40+

41+ -- * Arrays

42+ -- ** Construction

43+ -- | When performance matters, the fastest way to generate an array is

44+ -- 'writeN'. 'IsList' and 'IsString' instances can be

45+ -- used to conveniently construct arrays from literal values.

46+ -- 'OverloadedLists' extension or 'fromList' can be used to construct an

47+ -- array from a list literal. Similarly, 'OverloadedStrings' extension or

48+ -- 'fromList' can be used to construct an array from a string literal.

49+

50+ -- Pure List APIs

51+ , A.fromListN

52+ , A.fromList

53+

54+ -- Monadic APIs

55+ , A.writeN -- drop new

56+ , A.write -- full buffer

57+ -- , writeLastN -- drop old (ring buffer)

58+

59+ -- ** Elimination

60+ -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a

61+ -- list.

62+

63+ , A.toList

64+ , A.read

65+

66+ -- ** Random Access

67+ , A.length

68+ -- , (!!)

69+ -- , A.readIndex

70+ )

71+where

72+

73+import Streamly.Internal.Data.Array.Storable.Foreign as A

src/Streamly/Data/Unicode/Stream.hs+3−3

@@ -12,11 +12,11 @@

1212-- A 'Char' stream is the canonical representation to process Unicode strings.

1313-- It can be processed efficiently using regular stream processing operations.

1414-- A byte stream of Unicode text read from an IO device or from an

15−-- 'Streamly.Memory.Array.Array' in memory can be decoded into a 'Char' stream

15+-- 'Streamly.Data.Array.Storable.Foreign.Array' in memory can be decoded into a 'Char' stream

1616-- using the decoding routines in this module. A 'String' (@[Char]@) can be

1717-- converted into a 'Char' stream using 'Streamly.Prelude.fromList'. An @Array

1818-- Char@ can be 'Streamly.Prelude.unfold'ed into a stream using the array

19−-- 'Streamly.Memory.Array.read' unfold.

19+-- 'Streamly.Data.Array.Storable.Foreign.read' unfold.

2020--

2121-- = Storing Unicode Strings

2222--

@@ -30,7 +30,7 @@

3030-- than pinned arrays for short and short lived strings.

3131--

3232-- For longer or long lived streams you can 'Streamly.Prelude.fold' the 'Char'

33−-- stream as @Array Char@ using the array 'Streamly.Memory.Array.write' fold.

33+-- stream as @Array Char@ using the array 'Streamly.Data.Array.Storable.Foreign.write' fold.

3434-- The 'Array' type provides a more compact representation and pinned memory

3535-- reducing GC overhead. If space efficiency is a concern you can use

3636-- 'encodeUtf8' on the 'Char' stream before writing it to an 'Array' providing

src/Streamly/FileSystem/FD.hs+2−2

@@ -25,7 +25,7 @@

2525--

2626-- Reading and writing APIs are divided into two categories, sequential

2727-- streaming APIs and random or seekable access APIs. File IO APIs are quite

28−-- similar to "Streamly.Memory.Array" read write APIs. In that regard, arrays can

28+-- similar to "Streamly.Data.Array.Storable.Foreign" read write APIs. In that regard, arrays can

2929-- be considered as in-memory files or files can be considered as on-disk

3030-- arrays.

3131--

@@ -145,7 +145,7 @@ import qualified Streamly.FileSystem.FDIO as RawIO hiding (write)

145145-- import Streamly.Data.Fold (Fold)

146146-- import Streamly.String (encodeUtf8, decodeUtf8, foldLines)

147147

148−import qualified Streamly.Memory.Array as A

148+import qualified Streamly.Data.Array.Storable.Foreign as A

149149import qualified Streamly.Internal.Memory.ArrayStream as AS

150150import qualified Streamly.Prelude as S

151151import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

src/Streamly/FileSystem/Handle.hs+1−1

@@ -10,7 +10,7 @@

1010-- Portability : GHC

1111--

1212-- Read and write streams and arrays to and from file handles. File handle IO

13−-- APIs are quite similar to "Streamly.Memory.Array" read write APIs. In that

13+-- APIs are quite similar to "Streamly.Data.Array.Storable.Foreign" read write APIs. In that

1414-- regard, arrays can be considered as in-memory files or files can be

1515-- considered as on-disk arrays.

1616--

src/Streamly/Internal/Data/Fold.hs+1−1

@@ -675,7 +675,7 @@ foldMapM act = Fold step begin done

675675-- | Folds the input stream to a list.

676676--

677677-- /Warning!/ working on large lists accumulated as buffers in memory could be

678−-- very inefficient, consider using "Streamly.Memory.Array" instead.

678+-- very inefficient, consider using "Streamly.Data.Array.Storable.Foreign" instead.

679679--

680680-- @since 0.7.0

681681

src/Streamly/Internal/Data/SmallArray.hs+2−2

@@ -100,7 +100,7 @@ foldr f z arr = runIdentity $ D.foldr f z $ toStreamD arr

100100-- 'SmallArray'.

101101--

102102-- Since we are folding to a 'SmallArray' @n@ should be <= 128, for larger number

103−-- of elements use an 'Array' from either "Streamly.Data.Array" or "Streamly.Memory.Array".

103+-- of elements use an 'Array' from either "Streamly.Data.Array" or "Streamly.Data.Array.Storable.Foreign".

104104{-# INLINE_NORMAL writeN #-}

105105writeN :: MonadIO m => Int -> Fold m a (SmallArray a)

106106writeN limit = Fold step initial extract

@@ -132,7 +132,7 @@ fromStreamDN limit str = do

132132--

133133-- It is recommended to use a value of @n@ <= 128. For larger sized

134134-- arrays, use an 'Array' from "Streamly.Data.Array" or

135−-- "Streamly.Memory.Array"

135+-- "Streamly.Data.Array.Storable.Foreign"

136136{-# INLINABLE fromListN #-}

137137fromListN :: Int -> [a] -> SmallArray a

138138fromListN n xs = unsafePerformIO $ fromStreamDN n $ D.fromList xs

src/Streamly/Internal/FileSystem/Dir.hs+1−1

@@ -76,7 +76,7 @@ import Streamly.Internal.Data.Stream.StreamK.Type (IsStream)

7676import qualified Streamly.Internal.Data.Unfold as UF

7777-- import qualified Streamly.Internal.Memory.ArrayStream as AS

7878import qualified Streamly.Internal.Data.Stream.IsStream as S

79−-- import qualified Streamly.Memory.Array as A

79+-- import qualified Streamly.Data.Array.Storable.Foreign as A

8080-- import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

8181import qualified System.Directory as Dir

8282

src/Streamly/Internal/FileSystem/File.hs+1−1

@@ -108,7 +108,7 @@ import qualified Streamly.Internal.Data.Fold.Types as FL

108108import qualified Streamly.Internal.Data.Unfold as UF

109109import qualified Streamly.Internal.FileSystem.Handle as FH

110110import qualified Streamly.Internal.Memory.ArrayStream as AS

111−import qualified Streamly.Memory.Array as A

111+import qualified Streamly.Data.Array.Storable.Foreign as A

112112import qualified Streamly.Prelude as S

113113

114114-------------------------------------------------------------------------------

src/Streamly/Internal/FileSystem/Handle.hs+1−1

@@ -127,7 +127,7 @@ import qualified Streamly.Internal.Data.Unfold as UF

127127import qualified Streamly.Internal.Data.Array.Storable.Foreign as IA

128128import qualified Streamly.Internal.Memory.ArrayStream as AS

129129import qualified Streamly.Internal.Data.Stream.IsStream as S

130−import qualified Streamly.Memory.Array as A

130+import qualified Streamly.Data.Array.Storable.Foreign as A

131131import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

132132

133133-------------------------------------------------------------------------------

src/Streamly/Internal/Network/Socket.hs+1−1

@@ -100,7 +100,7 @@ import qualified Streamly.Data.Fold as FL

100100import qualified Streamly.Internal.Data.Fold.Types as FL

101101import qualified Streamly.Internal.Data.Unfold as UF

102102import qualified Streamly.Internal.Data.Array.Storable.Foreign as IA

103−import qualified Streamly.Memory.Array as A

103+import qualified Streamly.Data.Array.Storable.Foreign as A

104104import qualified Streamly.Internal.Memory.ArrayStream as AS

105105import qualified Streamly.Internal.Data.Array.Storable.Foreign.Types as A

106106import qualified Streamly.Prelude as S

src/Streamly/Internal/Unicode/Array/Char.hs+2−2

@@ -20,10 +20,10 @@ where

2020import Control.Monad.IO.Class (MonadIO)

2121import Streamly (IsStream, MonadAsync)

2222import Prelude hiding (String, lines, words, unlines, unwords)

23−import Streamly.Memory.Array (Array)

23+import Streamly.Data.Array.Storable.Foreign (Array)

2424

2525import qualified Streamly.Internal.Unicode.Stream as S

26−import qualified Streamly.Memory.Array as A

26+import qualified Streamly.Data.Array.Storable.Foreign as A

2727

2828-- | Break a string up into a stream of strings at newline characters.

2929-- The resulting strings do not contain newlines.

src/Streamly/Internal/Unicode/Stream.hs+1−1

@@ -68,7 +68,7 @@ import GHC.Ptr (Ptr (..), plusPtr)

6868import System.IO.Unsafe (unsafePerformIO)

6969import Streamly (IsStream)

7070import Streamly.Data.Fold (Fold)

71−import Streamly.Memory.Array (Array)

71+import Streamly.Data.Array.Storable.Foreign (Array)

7272import Streamly.Internal.Data.Unfold (Unfold)

7373import Streamly.Internal.Data.SVar (adaptState)

7474import Streamly.Internal.Data.Stream.StreamD (Stream(..), Step (..))

src/Streamly/Memory/Array.hs+1

@@ -35,6 +35,7 @@

3535-- For experimental APIs see "Streamly.Internal.Data.Array.Storable.Foreign".

3636

3737module Streamly.Memory.Array

38+ {-# DEPRECATED "Use Streamly.Data.Array.Storable.Foreign instead" #-}

3839 (

3940 A.Array

4041

src/Streamly/Network/Socket.hs+1−1

@@ -89,7 +89,7 @@

8989-- APIs for unconnected sockets need to explicitly specify the remote endpoint.

9090--

9191-- By design, connected socket IO APIs are similar to

92−-- "Streamly.Memory.Array" read write APIs. They are almost identical to the

92+-- "Streamly.Data.Array.Storable.Foreign" read write APIs. They are almost identical to the

9393-- sequential streaming APIs in "Streamly.Internal.FileSystem.File".

9494--

9595module Streamly.Network.Socket

src/Streamly/Unicode/Stream.hs+3−3