e218fb9 · streamly · git.iko.soy

Changelog.md+1

compare with master

unifiedsplit

@@ -2,6 +2,7 @@

2

3### Breaking changes

4

5+* Deprecate `Streamly.Memory.Array` in favor of `Streamly.Data.Array.Storable.Foreign`

5* Deprecate `Streamly.Data.Unicode.Stream` in favor of `Streamly.Unicode.Stream`

6* Deprecate `Streamly.Data.Unicode.Stream` in favor of `Streamly.Unicode.Stream`

6* Change the signature of `foldlM'` to make the initial value of the

7* Change the signature of `foldlM'` to make the initial value of the

7 accumulator monadic.

8 accumulator monadic.

README.md+3−3

@@ -368,7 +368,7 @@ documentation.

368

369## Arrays

370

371−The `Streamly.Memory.Array` module provides immutable arrays. Arrays are the

371+The `Streamly.Data.Array.Storable.Foreign` module provides immutable arrays. Arrays are the

372computing duals of streams. Streams are good at sequential access and immutable

373transformations of in-transit data whereas arrays are good at random access and

374in-place transformations of buffered data. Unlike streams which are potentially

@@ -379,7 +379,7 @@ computing system. The design of streamly as a general purpose computing

379framework is centered around these two fundamental aspects of computing and

380storage.

381

382−`Streamly.Memory.Array` uses pinned memory outside GC and therefore avoid any

382+`Streamly.Data.Array.Storable.Foreign` uses pinned memory outside GC and therefore avoid any

383GC overhead for the storage in arrays. Streamly allows efficient

384transformations over arrays using streams. It uses arrays to transfer data to

385and from the operating system and to store data in memory.

@@ -422,7 +422,7 @@ module Main where

422

423import qualified Streamly.Prelude as S

424import qualified Streamly.Data.Fold as FL

425−import qualified Streamly.Memory.Array as A

425+import qualified Streamly.Data.Array.Storable.Foreign as A

426import qualified Streamly.FileSystem.Handle as FH

427import qualified System.IO as FH

428

benchmark/NanoBenchmarks.hs+1−1

@@ -9,7 +9,7 @@

9import Streamly (SerialT)

10import Streamly.Internal.Data.SVar (MonadAsync)

11

12−import qualified Streamly.Memory.Array as A

12+import qualified Streamly.Data.Array.Storable.Foreign as A

13import qualified Streamly.FileSystem.Handle as FH

14import qualified Streamly.Internal.FileSystem.Handle as IFH

15import qualified Streamly.Data.Fold as FL

benchmark/Streamly/Benchmark/Data/ArrayOps.hs+1−1

@@ -35,7 +35,7 @@ import qualified Streamly.Internal.Data.SmallArray as A

35type Stream = A.SmallArray

36#elif defined(MEMORY_ARRAY)

37import qualified GHC.Exts as GHC

38−import qualified Streamly.Memory.Array as A

38+import qualified Streamly.Data.Array.Storable.Foreign as A

39import qualified Streamly.Internal.Data.Array.Storable.Foreign as A

40type Stream = A.Array

41#elif defined(DATA_PRIM_ARRAY)

benchmark/Streamly/Benchmark/Data/Fold.hs+1−1

@@ -24,7 +24,7 @@ import qualified Streamly.Internal.Data.Pipe as Pipe

24

25import qualified Streamly.Internal.Data.Sink as Sink

26

27−import qualified Streamly.Memory.Array as A

27+import qualified Streamly.Data.Array.Storable.Foreign as A

28import qualified Streamly.Internal.Data.Array.Storable.Foreign as IA

29import qualified Streamly.Internal.Data.Fold as IFL

30import qualified Streamly.Internal.Data.Stream.IsStream as IP

benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+1−1

@@ -36,7 +36,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH

36import qualified Streamly.Internal.Memory.ArrayStream as AS

37import qualified Streamly.Internal.Unicode.Array.Char as IUA

38import qualified Streamly.Internal.Data.Stream.IsStream as IP

39−import qualified Streamly.Memory.Array as A

39+import qualified Streamly.Data.Array.Storable.Foreign as A

40import qualified Streamly.Prelude as S

41

42import Gauge hiding (env)

design/inlining.md+3−3

@@ -15,7 +15,7 @@ i.e. phase 0, 1 and 2. We have defined them as follows in `inline.h`:

15The combinators in `Streamly.Prelude` are defined in terms of combinators in

16`Streamly.Internal.Data.Stream.StreamD` (Direct style streams) or `Streamly.Internal.Data.Stream.StreamK`

17(CPS style streams). We convert the stream from `StreamD` to `StreamK`

18−representation or vice versa in some cases.

18+representation or vice versa in some cases.

19

20In the first inlining phase (INLINE_EARLY or INLINE) we expand

21the combinators in `Streamly.Prelude` into

@@ -82,7 +82,7 @@ consume as arguments. This means `StreamD` combinators should not be marked

82as `INLINE` or `INLINE_EARLY`, instead they should all be marked as

83`INLINE_NORMAL` because higher order funcitons like `concatMap`/`map`/`mapM`

84etc are marked as `INLINE_NORMAL`. `StreamD` functions in other modules like

85−`Streamly.Memory.Array` should also follow the same rules.

85+`Streamly.Data.Array.Storable.Foreign` should also follow the same rules.

86

87## Stream Fusion

88

@@ -118,6 +118,6 @@ possible. In most cases it is not necessary, but in some cases it may affect

118fusion and make a difference of 10x performance or more. For example, using

119non-strict fields can increase the code size for internal join points and

120functions created during transformations, which can affect the inlining of

121−these code blocks which in turn can affect stream fusion.

121+these code blocks which in turn can affect stream fusion.

122

123See https://gitlab.haskell.org/ghc/ghc/issues/17075 .

examples/FileIOExamples.hs+1−1

@@ -1,6 +1,6 @@

1import qualified Streamly.Prelude as S

2import qualified Streamly.Data.Fold as FL

3−import qualified Streamly.Memory.Array as A

3+import qualified Streamly.Data.Array.Storable.Foreign as A

4

5import qualified Streamly.Internal.Data.Fold as FL

6import qualified Streamly.Internal.Data.Stream.IsStream as IP

examples/FileSinkServer.hs+1−1

@@ -12,7 +12,7 @@ import System.Environment (getArgs)

12import Streamly

13import Streamly.Unicode.Stream

14import qualified Streamly.FileSystem.Handle as FH

15−import qualified Streamly.Memory.Array as A

15+import qualified Streamly.Data.Array.Storable.Foreign as A

16import qualified Streamly.Network.Socket as NS

17import qualified Streamly.Network.Inet.TCP as TCP

18import qualified Streamly.Prelude as S

examples/HandleIO.hs+1−1

@@ -5,7 +5,7 @@ import System.IO (IOMode(..), hSeek, SeekMode(..))

5import qualified Streamly.Data.Fold as FL

6import qualified Streamly.FileSystem.Handle as FH

7import qualified System.IO as FH

8−import qualified Streamly.Memory.Array as A

8+import qualified Streamly.Data.Array.Storable.Foreign as A

9import qualified Streamly.Prelude as S

10-- import qualified Streamly.FileSystem.FD as FH

11

examples/WordClassifier.hs+1−1

@@ -22,7 +22,7 @@ import qualified Streamly.Data.Fold as FL

22import qualified Streamly.Internal.Data.Fold as IFL

23import qualified Streamly.Internal.Data.Unfold as IUF

24import qualified Streamly.Internal.FileSystem.File as File

25−import qualified Streamly.Memory.Array as A

25+import qualified Streamly.Data.Array.Storable.Foreign as A

26import qualified Streamly.Prelude as S

27import System.Environment (getArgs)

28

examples/WordCount.hs+1−1

@@ -43,7 +43,7 @@ import Streamly.Internal.Unicode.Stream

43import qualified Streamly as S

44import qualified Streamly.Unicode.Stream as S

45import qualified Streamly.FileSystem.Handle as FH

46−import qualified Streamly.Memory.Array as A

46+import qualified Streamly.Data.Array.Storable.Foreign as A

47import qualified Streamly.Prelude as S

48import qualified Data.Vector.Storable.Mutable as V

49

src/Streamly.hs+2−2

@@ -252,7 +252,7 @@ import qualified Streamly.Internal.Data.Stream.Async as Async

252-- > S.length $ S.splitOnSuffix FL.drain 10 $ FH.read fh

253--

254-- The following example folds the lines to arrays of 'Word8' using the

255−-- 'Streamly.Memory.Array.writeF' fold and then wraps the lines in square

255+-- 'Streamly.Data.Array.Storable.Foreign.writeF' fold and then wraps the lines in square

256-- brackets before writing them to standard output using

257-- 'Streamly.FileSystem.Handle.write':

258--

@@ -303,7 +303,7 @@ import qualified Streamly.Internal.Data.Stream.Async as Async

303

304-- $arrays

305--

306−-- Streamly arrays (See "Streamly.Memory.Array") complement streams to provide an

306+-- Streamly arrays (See "Streamly.Data.Array.Storable.Foreign") complement streams to provide an

307-- efficient computing paradigm. Streams are suitable for immutable

308-- transformations of /potentially infinite/ data using /sequential access/ and

309-- pipelined transformations whereas arrays are suitable for in-place

src/Streamly/Data/Array/Storable/Foreign.hsnew file+73

@@ -0,0 +1,73 @@

1+#include "inline.hs"

2+

3+-- |

4+-- Module : Streamly.Data.Array.Storable.Foreign

6+--

7+-- License : BSD3

8+-- Maintainer : streamly@composewell.com

9+-- Stability : experimental

10+-- Portability : GHC

11+--

12+-- This module provides immutable arrays in pinned memory (non GC memory)

13+-- suitable for long lived data storage, random access and for interfacing with

14+-- the operating system.

15+--

16+-- Arrays in this module are chunks of pinned memory that hold a sequence of

17+-- 'Storable' values of a given type, they cannot store non-serializable data

18+-- like functions. Once created an array cannot be modified. Pinned memory

19+-- allows efficient buffering of long lived data without adding any impact to

20+-- GC. One array is just one pointer visible to GC and it does not have to be

21+-- copied across generations. Moreover, pinned memory allows communication

22+-- with foreign consumers and producers (e.g. file or network IO) without

23+-- copying the data.

24+--

25+-- = Programmer Notes

26+--

27+-- To apply a transformation to an array use 'read' to unfold the array into a

28+-- stream, apply a transformation on the stream and then use 'write' to fold it

29+-- back to an array.

30+--

31+-- This module is designed to be imported qualified:

32+--

33+-- > import qualified Streamly.Array as A

34+--

35+-- For experimental APIs see "Streamly.Internal.Data.Array.Storable.Foreign".

36+

37+module Streamly.Data.Array.Storable.Foreign

38+ (

39+ A.Array

40+

41+ -- * Arrays

42+ -- ** Construction

43+ -- | When performance matters, the fastest way to generate an array is

44+ -- 'writeN'. 'IsList' and 'IsString' instances can be

45+ -- used to conveniently construct arrays from literal values.

46+ -- 'OverloadedLists' extension or 'fromList' can be used to construct an

47+ -- array from a list literal. Similarly, 'OverloadedStrings' extension or

48+ -- 'fromList' can be used to construct an array from a string literal.

49+

50+ -- Pure List APIs

51+ , A.fromListN

52+ , A.fromList

53+

54+ -- Monadic APIs

55+ , A.writeN -- drop new

56+ , A.write -- full buffer

57+ -- , writeLastN -- drop old (ring buffer)

58+

59+ -- ** Elimination

60+ -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a

61+ -- list.

62+

63+ , A.toList

64+ , A.read

65+

66+ -- ** Random Access

67+ , A.length

68+ -- , (!!)

69+ -- , A.readIndex

70+ )

71+where

72+

73+import Streamly.Internal.Data.Array.Storable.Foreign as A

src/Streamly/Data/Unicode/Stream.hs+3−3

@@ -12,11 +12,11 @@

12-- A 'Char' stream is the canonical representation to process Unicode strings.

13-- It can be processed efficiently using regular stream processing operations.

14-- A byte stream of Unicode text read from an IO device or from an

15−-- 'Streamly.Memory.Array.Array' in memory can be decoded into a 'Char' stream

15+-- 'Streamly.Data.Array.Storable.Foreign.Array' in memory can be decoded into a 'Char' stream

16-- using the decoding routines in this module. A 'String' (@[Char]@) can be

17-- converted into a 'Char' stream using 'Streamly.Prelude.fromList'. An @Array

18-- Char@ can be 'Streamly.Prelude.unfold'ed into a stream using the array

19−-- 'Streamly.Memory.Array.read' unfold.

19+-- 'Streamly.Data.Array.Storable.Foreign.read' unfold.

20--

21-- = Storing Unicode Strings

22--

@@ -30,7 +30,7 @@

30-- than pinned arrays for short and short lived strings.

31--

32-- For longer or long lived streams you can 'Streamly.Prelude.fold' the 'Char'

33−-- stream as @Array Char@ using the array 'Streamly.Memory.Array.write' fold.

33+-- stream as @Array Char@ using the array 'Streamly.Data.Array.Storable.Foreign.write' fold.

34-- The 'Array' type provides a more compact representation and pinned memory

35-- reducing GC overhead. If space efficiency is a concern you can use

36-- 'encodeUtf8' on the 'Char' stream before writing it to an 'Array' providing

src/Streamly/FileSystem/FD.hs+2−2

@@ -25,7 +25,7 @@

25--

26-- Reading and writing APIs are divided into two categories, sequential

27-- streaming APIs and random or seekable access APIs. File IO APIs are quite

28−-- similar to "Streamly.Memory.Array" read write APIs. In that regard, arrays can

28+-- similar to "Streamly.Data.Array.Storable.Foreign" read write APIs. In that regard, arrays can

29-- be considered as in-memory files or files can be considered as on-disk

30-- arrays.

31--

@@ -145,7 +145,7 @@ import qualified Streamly.FileSystem.FDIO as RawIO hiding (write)

145-- import Streamly.Data.Fold (Fold)

146-- import Streamly.String (encodeUtf8, decodeUtf8, foldLines)

147

148−import qualified Streamly.Memory.Array as A

148+import qualified Streamly.Data.Array.Storable.Foreign as A

149import qualified Streamly.Internal.Memory.ArrayStream as AS

150import qualified Streamly.Prelude as S

151import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

src/Streamly/FileSystem/Handle.hs+1−1

@@ -10,7 +10,7 @@

10-- Portability : GHC

11--

12-- Read and write streams and arrays to and from file handles. File handle IO

13−-- APIs are quite similar to "Streamly.Memory.Array" read write APIs. In that

13+-- APIs are quite similar to "Streamly.Data.Array.Storable.Foreign" read write APIs. In that

14-- regard, arrays can be considered as in-memory files or files can be

15-- considered as on-disk arrays.

16--

src/Streamly/Internal/Data/Fold.hs+1−1

@@ -675,7 +675,7 @@ foldMapM act = Fold step begin done

675-- | Folds the input stream to a list.

676--

677-- /Warning!/ working on large lists accumulated as buffers in memory could be

678−-- very inefficient, consider using "Streamly.Memory.Array" instead.

678+-- very inefficient, consider using "Streamly.Data.Array.Storable.Foreign" instead.

679--

680-- @since 0.7.0

681

src/Streamly/Internal/Data/SmallArray.hs+2−2

@@ -100,7 +100,7 @@ foldr f z arr = runIdentity $ D.foldr f z $ toStreamD arr

100-- 'SmallArray'.

101--

102-- Since we are folding to a 'SmallArray' @n@ should be <= 128, for larger number

103−-- of elements use an 'Array' from either "Streamly.Data.Array" or "Streamly.Memory.Array".

103+-- of elements use an 'Array' from either "Streamly.Data.Array" or "Streamly.Data.Array.Storable.Foreign".

104{-# INLINE_NORMAL writeN #-}

105writeN :: MonadIO m => Int -> Fold m a (SmallArray a)

106writeN limit = Fold step initial extract

@@ -132,7 +132,7 @@ fromStreamDN limit str = do

132--

133-- It is recommended to use a value of @n@ <= 128. For larger sized

134-- arrays, use an 'Array' from "Streamly.Data.Array" or

135−-- "Streamly.Memory.Array"

135+-- "Streamly.Data.Array.Storable.Foreign"

136{-# INLINABLE fromListN #-}

137fromListN :: Int -> [a] -> SmallArray a

138fromListN n xs = unsafePerformIO $ fromStreamDN n $ D.fromList xs

src/Streamly/Internal/FileSystem/Dir.hs+1−1

@@ -76,7 +76,7 @@ import Streamly.Internal.Data.Stream.StreamK.Type (IsStream)

76import qualified Streamly.Internal.Data.Unfold as UF

77-- import qualified Streamly.Internal.Memory.ArrayStream as AS

78import qualified Streamly.Internal.Data.Stream.IsStream as S

79−-- import qualified Streamly.Memory.Array as A

79+-- import qualified Streamly.Data.Array.Storable.Foreign as A

80-- import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

81import qualified System.Directory as Dir

82

src/Streamly/Internal/FileSystem/File.hs+1−1

@@ -108,7 +108,7 @@ import qualified Streamly.Internal.Data.Fold.Types as FL

108import qualified Streamly.Internal.Data.Unfold as UF

109import qualified Streamly.Internal.FileSystem.Handle as FH

110import qualified Streamly.Internal.Memory.ArrayStream as AS

111−import qualified Streamly.Memory.Array as A

111+import qualified Streamly.Data.Array.Storable.Foreign as A

112import qualified Streamly.Prelude as S

113

114-------------------------------------------------------------------------------

src/Streamly/Internal/FileSystem/Handle.hs+1−1

@@ -127,7 +127,7 @@ import qualified Streamly.Internal.Data.Unfold as UF

127import qualified Streamly.Internal.Data.Array.Storable.Foreign as IA

128import qualified Streamly.Internal.Memory.ArrayStream as AS

129import qualified Streamly.Internal.Data.Stream.IsStream as S

130−import qualified Streamly.Memory.Array as A

130+import qualified Streamly.Data.Array.Storable.Foreign as A

131import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

132

133-------------------------------------------------------------------------------

src/Streamly/Internal/Network/Socket.hs+1−1

@@ -100,7 +100,7 @@ import qualified Streamly.Data.Fold as FL

100import qualified Streamly.Internal.Data.Fold.Types as FL

101import qualified Streamly.Internal.Data.Unfold as UF

102import qualified Streamly.Internal.Data.Array.Storable.Foreign as IA

103−import qualified Streamly.Memory.Array as A

103+import qualified Streamly.Data.Array.Storable.Foreign as A

104import qualified Streamly.Internal.Memory.ArrayStream as AS

105import qualified Streamly.Internal.Data.Array.Storable.Foreign.Types as A

106import qualified Streamly.Prelude as S

src/Streamly/Internal/Unicode/Array/Char.hs+2−2

@@ -20,10 +20,10 @@ where

20import Control.Monad.IO.Class (MonadIO)

21import Streamly (IsStream, MonadAsync)

22import Prelude hiding (String, lines, words, unlines, unwords)

23−import Streamly.Memory.Array (Array)

23+import Streamly.Data.Array.Storable.Foreign (Array)

24

25import qualified Streamly.Internal.Unicode.Stream as S

26−import qualified Streamly.Memory.Array as A

26+import qualified Streamly.Data.Array.Storable.Foreign as A

27

28-- | Break a string up into a stream of strings at newline characters.

29-- The resulting strings do not contain newlines.

src/Streamly/Internal/Unicode/Stream.hs+1−1

@@ -68,7 +68,7 @@ import GHC.Ptr (Ptr (..), plusPtr)

68import System.IO.Unsafe (unsafePerformIO)

69import Streamly (IsStream)

70import Streamly.Data.Fold (Fold)

71−import Streamly.Memory.Array (Array)

71+import Streamly.Data.Array.Storable.Foreign (Array)

72import Streamly.Internal.Data.Unfold (Unfold)

73import Streamly.Internal.Data.SVar (adaptState)

74import Streamly.Internal.Data.Stream.StreamD (Stream(..), Step (..))

src/Streamly/Memory/Array.hs+1

@@ -35,6 +35,7 @@

35-- For experimental APIs see "Streamly.Internal.Data.Array.Storable.Foreign".

36

37module Streamly.Memory.Array

38+ {-# DEPRECATED "Use Streamly.Data.Array.Storable.Foreign instead" #-}

38 (

39 (

39 A.Array

40 A.Array

40

41

src/Streamly/Network/Socket.hs+1−1

@@ -89,7 +89,7 @@

89-- APIs for unconnected sockets need to explicitly specify the remote endpoint.

90--

91-- By design, connected socket IO APIs are similar to

92−-- "Streamly.Memory.Array" read write APIs. They are almost identical to the

92+-- "Streamly.Data.Array.Storable.Foreign" read write APIs. They are almost identical to the

93-- sequential streaming APIs in "Streamly.Internal.FileSystem.File".

94--

95module Streamly.Network.Socket

src/Streamly/Unicode/Stream.hs+3−3