352d3a2 · streamly · git.iko.soy

git.iko.soy/streamly

on master help sign in

update docs

352d3a2Harendra Kumar8 years ago

Changelog.md+4−4

compare with master

unifiedsplit

@@ -3,8 +3,8 @@

3### Breaking changes

5* Signatures of `zipWithM` and `zipAsyncWithM` have changed

6−* Some functions in prelude now require an additional Monad constraint for the

6+* Some functions in prelude now require an additional `Monad` constraint on

7− underlying type of the stream.

7+ the underlying type of the stream.

9### Deprecations

@@ -12,8 +12,8 @@

13### Enhancements

15−* Add concurrency control primitives `threads` and `buffer`.

15+* Add concurrency control primitives `maxThreads` and `maxBuffer`.

16−* Significant performance improvements

16+* Significant performance improvements utilizing stream fusion optimizations.

17* Add `yield` to construct a singleton stream from a pure value

18* Add `repeat` to generate an infinite stream by repeating a pure value

19* Add `fromList` and `fromListM` to generate streams from lists, faster than

README.md+41−24

@@ -45,7 +45,10 @@ Why use streamly?

45 * _Generality_: Unifies functionality provided by several disparate packages

46 (streaming, concurrency, list transformer, logic programming, reactive

47 programming) in a concise API.

48− * _Performance_: Streamly is designed for high performance. See

48+ * _Performance_: Streamly is designed for high performance. It employs stream

49+ fusion optimizations for best possible performance. Serial peformance is

50+ equivalent to the venerable `vector` library in most cases and even better

51+ in some cases. Concurrent performance is unbeatable. See

49 [streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)

52 [streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)

50 for a comparison of popular streaming libraries on micro-benchmarks.

53 for a comparison of popular streaming libraries on micro-benchmarks.

@@ -63,16 +66,17 @@ For more information on streamly, see:

64## Streaming Pipelines

67## Streaming Pipelines

66−Unlike `pipes` or `conduit` and like `vector` and `streaming` `streamly`

69+Unlike `pipes` or `conduit` and like `vector` and `streaming`, `streamly`

67composes stream data instead of stream processors (functions). A stream is

70composes stream data instead of stream processors (functions). A stream is

68just like a list and is explicitly passed around to functions that process the

71just like a list and is explicitly passed around to functions that process the

69stream. Therefore, no special operator is needed to join stages in a streaming

72stream. Therefore, no special operator is needed to join stages in a streaming

70−pipeline, just the standard forward (`$`) or reverse (`&`) function application

73+pipeline, just the standard function application (`$`) or reverse function

71−operator is enough. Combinators are provided in `Streamly.Prelude` to

74+application (`&`) operator is enough. Combinators are provided in

72−transform or fold streams.

75+`Streamly.Prelude` to transform or fold streams.

77+The following snippet provides a simple stream composition example that reads

74−This snippet reads numbers from stdin, prints the squares of even numbers and

78+numbers from stdin, prints the squares of even numbers and exits if an even

75−exits if an even number more than 9 is entered.

79+number more than 9 is entered.

77```haskell

81```haskell

78import Streamly

82import Streamly

@@ -92,7 +96,8 @@ main = runStream $

93Monadic construction and generation functions e.g. `consM`, `unfoldrM`,

97Monadic construction and generation functions e.g. `consM`, `unfoldrM`,

94`replicateM`, `repeatM`, `iterateM` and `fromFoldableM` etc. work concurrently

98`replicateM`, `repeatM`, `iterateM` and `fromFoldableM` etc. work concurrently

95−when used with appropriate stream type combinator.

99+when used with appropriate stream type combinator (e.g. `asyncly`, `aheadly` or

100+`parallely`).

101

97The following code finishes in 3 seconds (6 seconds when serial):

102The following code finishes in 3 seconds (6 seconds when serial):

103

@@ -113,8 +118,8 @@ runStream $ asyncly $ S.replicateM 10 $ p 10

113

118

114## Concurrent Streaming Pipelines

119## Concurrent Streaming Pipelines

115

120

116−Use `|&` or `|$` to apply stream processing functions concurrently. In the

121+Use `|&` or `|$` to apply stream processing functions concurrently. The

117−following example "hello" is printed every second, if you use `&` instead of

122+following example prints a "hello" every second; if you use `&` instead of

118`|&` you will see that the delay doubles to 2 seconds instead because of serial

123`|&` you will see that the delay doubles to 2 seconds instead because of serial

119application.

124application.

120

125

@@ -126,7 +131,7 @@ main = runStream $

126

131

127## Mapping Concurrently

132## Mapping Concurrently

128

133

129−We can use `mapM` or `sequence` concurrently on a stream.

134+We can use `mapM` or `sequence` functions concurrently on a stream.

130

135

131```

136```

132> let p n = threadDelay (n * 1000000) >> return n

137> let p n = threadDelay (n * 1000000) >> return n

@@ -136,8 +141,8 @@ We can use `mapM` or `sequence` concurrently on a stream.

136## Serial and Concurrent Merging

141## Serial and Concurrent Merging

137

142

138Semigroup and Monoid instances can be used to fold streams serially or

143Semigroup and Monoid instances can be used to fold streams serially or

139−concurrently. In the following example we are composing ten actions in the

144+concurrently. In the following example we compose ten actions in the

140−stream each with a delay of 1 to 10 seconds, respectively. Since all the

145+stream, each with a delay of 1 to 10 seconds, respectively. Since all the

141actions are concurrent we see one output printed every second:

146actions are concurrent we see one output printed every second:

142

147

143``` haskell

148``` haskell

@@ -146,11 +151,11 @@ import qualified Streamly.Prelude as S

146import Control.Concurrent (threadDelay)

151import Control.Concurrent (threadDelay)

147

152

148main = S.toList $ parallely $ foldMap delay [1..10]

153main = S.toList $ parallely $ foldMap delay [1..10]

149− where delay n = S.once $ threadDelay (n * 1000000) >> print n

154+ where delay n = S.yieldM $ threadDelay (n * 1000000) >> print n

150```

155```

151

156

152−Streams can be combined together in many ways. We are providing some examples

157+Streams can be combined together in many ways. We provide some examples

153−below, see the tutorial for more ways. We will use the following `delay`

158+below, see the tutorial for more ways. We use the following `delay`

154function in the examples to demonstrate the concurrency aspects:

159function in the examples to demonstrate the concurrency aspects:

155

160

156``` haskell

161``` haskell

@@ -158,7 +163,7 @@ import Streamly

158import qualified Streamly.Prelude as S

163import qualified Streamly.Prelude as S

159import Control.Concurrent

164import Control.Concurrent

160

165

161−delay n = S.once $ do

166+delay n = S.yieldM $ do

162 threadDelay (n * 1000000)

167 threadDelay (n * 1000000)

163 tid <- myThreadId

168 tid <- myThreadId

164 putStrLn (show tid ++ ": Delay " ++ show n)

169 putStrLn (show tid ++ ": Delay " ++ show n)

@@ -196,7 +201,7 @@ import qualified Streamly.Prelude as S

196loops = do

201loops = do

197 x <- S.fromFoldable [1,2]

202 x <- S.fromFoldable [1,2]

198 y <- S.fromFoldable [3,4]

203 y <- S.fromFoldable [3,4]

199− S.once $ putStrLn $ show (x, y)

204+ S.yieldM $ putStrLn $ show (x, y)

200

205

201main = runStream loops

206main = runStream loops

202```

207```

@@ -209,20 +214,32 @@ main = runStream loops

209

214

210## Concurrent Nested Loops

215## Concurrent Nested Loops

211

216

212−To run the above code with demand-driven depth first concurrency i.e. each

217+To run the above code with, lookahead style concurrency i.e. each iteration in

213−iteration in the loops can run concurrently depending on the consumer rate:

218+the loop can run run concurrently by but the results are presented in the same

219+order as serial execution:

220+

221+``` haskell

222+main = runStream $ aheadly $ loops

223+```

224+

225+To run it with depth first concurrency yielding results asynchronously in the

226+same order as they become available (deep async composition):

214

227

215``` haskell

228``` haskell

216main = runStream $ asyncly $ loops

229main = runStream $ asyncly $ loops

217```

230```

218

231

219−To run it with demand driven breadth first concurrency:

232+To run it with breadth first concurrency and yeilding results asynchronously

233+(wide async composition):

220

234

221``` haskell

235``` haskell

222main = runStream $ wAsyncly $ loops

236main = runStream $ wAsyncly $ loops

223```

237```

224

238

239+The above streams provide lazy/demand-driven concurrency which is automatically

240+scaled as per demand and is controlled/bounded so that it can be used on

241+infinite streams. The following combinator provides strict, unbounded

225−To run it with strict concurrency irrespective of demand:

242+concurrency irrespective of demand:

226

243

227``` haskell

244``` haskell

228main = runStream $ parallely $ loops

245main = runStream $ parallely $ loops

@@ -266,8 +283,8 @@ import qualified Streamly.Prelude as S

266

283

267main = runStream $ aheadly $ getCurrentDir >>= readdir

284main = runStream $ aheadly $ getCurrentDir >>= readdir

268 where readdir d = do

285 where readdir d = do

269− (dirs, files) <- S.once $ listDir d

286+ (dirs, files) <- S.yieldM $ listDir d

270− S.once $ mapM_ putStrLn $ map show files

287+ S.yieldM $ mapM_ putStrLn $ map show files

271 -- read the subdirs concurrently, (<>) is concurrent

288 -- read the subdirs concurrently, (<>) is concurrent

272 foldMap readdir dirs

289 foldMap readdir dirs

273```

290```

src/Streamly.hs+5−5

@@ -217,11 +217,11 @@ import Data.Semigroup (Semigroup(..))

217

218-- $concurrency

219--

220−-- These combinators can be used at any point in the composition to control

220+-- These combinators can be used at any point in a stream composition to

221−-- concurrency, and they affect only the enclosed stream. When nested

221+-- control the concurrency of the enclosed stream. When the combinators are

222−-- combinators are used, the nearest enclosing combinator overrides the outer

222+-- used in a nested manner, the nearest enclosing combinator overrides the

223−-- ones. These combinators have no effect on 'Parallel' streams, concurrency

223+-- outer ones. These combinators have no effect on 'Parallel' streams,

224−-- for 'Parallel' streams is always unbounded.

224+-- concurrency for 'Parallel' streams is always unbounded.

225-- Note that the use of these combinators does not enable concurrency, to

226-- enable concurrency you have to use one of the concurrent stream type

227-- combinators.

src/Streamly/Streams/SVar.hs+1−1

@@ -97,7 +97,7 @@ toSVar sv m = toStreamVar sv (toStream m)

97--

98-- | Specify the maximum number of threads that can be spawned concurrently

99-- when using concurrent streams. This is not the grand total number of threads

100−-- but maximum threads at each point of concurrency.

100+-- but the maximum number of threads at each point of concurrency.

101-- A value of 0 resets the thread limit to default, a negative value means

102-- there is no limit. The default value is 1500.

103--

streamly.cabal+5−1

@@ -45,7 +45,11 @@ description:

45 * /Generality/: Unifies functionality provided by several disparate packages

46 (streaming, concurrency, list transformer, logic programming, reactive

47 programming) in a concise API.

48− * /Performance/: Streamly is designed for high performance. See

48+ * /Performance/: Streamly is designed for high performance.

49+ It employs stream fusion optimizations for best possible performance.

50+ Serial peformance is equivalent to the venerable `vector` library in most

51+ cases and even better in some cases. Concurrent performance is unbeatable.

52+ See

49 <https://github.com/composewell/streaming-benchmarks streaming-benchmarks>

53 <https://github.com/composewell/streaming-benchmarks streaming-benchmarks>

50 for a comparison of popular streaming libraries on micro-benchmarks.

54 for a comparison of popular streaming libraries on micro-benchmarks.

51 .

55 .