54dd548 · streamly · git.iko.soy

git.iko.soy/streamly

on master help sign in

Move the dir listing and file-io sections down

54dd548Harendra Kumar7 years ago

README.md+90−90

compare with master

unifiedsplit

@@ -105,96 +105,6 @@ The module `Streamly` provides just the core stream types, type casting and

105concurrency control combinators. Stream construction, transformation, folding,

106merging, zipping combinators are found in `Streamly.Prelude`.

107

108−## Show me an example

109−

110−The following code snippet lists a directory tree recursively, reading multiple

111−directories concurrently:

112−

113−```haskell

114−import Control.Monad.IO.Class (liftIO)

115−import Path.IO (listDir, getCurrentDir) -- from path-io package

116−import Streamly (AsyncT, adapt)

117−import qualified Streamly.Prelude as S

118−

119−listDirRecursive :: AsyncT IO ()

120−listDirRecursive = getCurrentDir >>= readdir >>= liftIO . mapM_ putStrLn

121− where

122− readdir dir = do

123− (dirs, files) <- listDir dir

124− S.yield (map show dirs ++ map show files) <> foldMap readdir dirs

125−

126−main :: IO ()

127−main = S.drain $ adapt $ listDirRecursive

128−```

129−

130−`AsyncT` is a stream monad transformer. If you are familiar with a list

131−transformer, it is nothing but `ListT` with concurrency semantics. For example,

132−the semigroup operation `<>` is concurrent. This makes `foldMap` concurrent

133−too. You can replace `AsyncT` with `SerialT` and the above code will become

134−serial, exactly equivalent to a `ListT`.

135−

136−## File IO

137−

138−The following code snippet implements some common Unix command line utilities

139−using streamly. To get an idea about IO streaming performance, you can

140−benchmark these against the regular unix utilities using the `time` command.

141−Make sure to use a big enough input file and compile with

142−`ghc -O2 -fspec-constr-recursive=10` when benchmarking. Use `+RTS -s` flags on

143−the executable to check the space usage, look for `maximum residency` in the

144−output.

145−

146−Note that `grep -c` counts the number of lines where the pattern occurs whereas

147−the snippet below counts the total number of occurrences of the pattern,

148−therefore, the output may differ.

149−

150−``` haskell

151−import qualified Streamly.Prelude as S

152−import qualified Streamly.Fold as FL

153−import qualified Streamly.Mem.Array as A

154−import qualified Streamly.FileSystem.File as File

155−

156−import Data.Char (ord)

157−import System.Environment (getArgs)

158−import System.IO (openFile, IOMode(..), stdout)

159−

160−cat src = File.writeArrays stdout $ File.readArraysUpto (256*1024) src

161−cp src dst = File.writeArrays dst $ File.readArraysUpto (256*1024) src

162−wcl src = print =<<

163− ( S.length

164− $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.drain

165− $ File.read src)

166−grepc pat src = print . (subtract 1) =<<

167− ( S.length

168− $ FL.splitOn (A.fromList (map (fromIntegral . ord) pat)) FL.drain

169− $ File.read src)

170−avgll src = print =<<

171− ( FL.foldl' avg

172− $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length

173− $ File.read src)

174− where avg = (/) <$> toDouble FL.sum <*> toDouble FL.length

175− toDouble = fmap (fromIntegral :: Int -> Double)

176−llhisto src = print =<<

177− ( FL.foldl' (FL.classify FL.length)

178− $ S.map bucket

179− $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length

180− $ File.read src)

181− where

182− bucket n = let i = n `div` 10 in if i > 9 then (9,n) else (i,n)

183−

184−main = do

185− name <- fmap head getArgs

186− src <- openFile name ReadMode

187− -- cat src -- Unix cat program

188− -- wcl src -- Unix wc -l program

189− -- grepc "aaaa" src -- Unix grep -c program

190−

191− -- dst <- openFile "dst.txt" WriteMode

192− -- cp src dst -- Unix cp program

193−

194− -- avgll src -- get average line length

195− llhisto src -- get line length histogram

196−```

197−

198## Streaming Pipelines

108## Streaming Pipelines

199

109

200The following snippet provides a simple stream composition example that reads

110The following snippet provides a simple stream composition example that reads

@@ -401,6 +311,34 @@ The concurrency facilities provided by streamly can be compared with

401[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative

311[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative

402expression.

312expression.

403

313

314+## Example: Listing Directories Recursively/Concurrently

315+

316+The following code snippet lists a directory tree recursively, reading multiple

317+directories concurrently:

318+

319+```haskell

320+import Control.Monad.IO.Class (liftIO)

321+import Path.IO (listDir, getCurrentDir) -- from path-io package

322+import Streamly (AsyncT, adapt)

323+import qualified Streamly.Prelude as S

324+

325+listDirRecursive :: AsyncT IO ()

326+listDirRecursive = getCurrentDir >>= readdir >>= liftIO . mapM_ putStrLn

327+ where

328+ readdir dir = do

329+ (dirs, files) <- listDir dir

330+ S.yield (map show dirs ++ map show files) <> foldMap readdir dirs

331+

332+main :: IO ()

333+main = S.drain $ adapt $ listDirRecursive

334+```

335+

336+`AsyncT` is a stream monad transformer. If you are familiar with a list

337+transformer, it is nothing but `ListT` with concurrency semantics. For example,

338+the semigroup operation `<>` is concurrent. This makes `foldMap` concurrent

339+too. You can replace `AsyncT` with `SerialT` and the above code will become

340+serial, exactly equivalent to a `ListT`.

341+

404## Rate Limiting

342## Rate Limiting

405

343

406For bounded concurrent streams, stream yield rate can be specified. For

344For bounded concurrent streams, stream yield rate can be specified. For

@@ -423,6 +361,68 @@ rate. Rate control works precisely even at throughputs as high as millions of

423yields per second. For more sophisticated rate control see the haddock

361yields per second. For more sophisticated rate control see the haddock

424documentation.

362documentation.

425

363

364+## File IO

365+

366+The following code snippet implements some common Unix command line utilities

367+using streamly. To get an idea about IO streaming performance, you can

368+benchmark these against the regular unix utilities using the `time` command.

369+Make sure to use a big enough input file and compile with

370+`ghc -O2 -fspec-constr-recursive=10` when benchmarking. Use `+RTS -s` flags on

371+the executable to check the space usage, look for `maximum residency` in the

372+output.

373+

374+Note that `grep -c` counts the number of lines where the pattern occurs whereas

375+the snippet below counts the total number of occurrences of the pattern,

376+therefore, the output may differ.

377+

378+``` haskell

379+import qualified Streamly.Prelude as S

380+import qualified Streamly.Fold as FL

381+import qualified Streamly.Mem.Array as A

382+import qualified Streamly.FileSystem.File as File

383+

384+import Data.Char (ord)

385+import System.Environment (getArgs)

386+import System.IO (openFile, IOMode(..), stdout)

387+

388+cat src = File.writeArrays stdout $ File.readArraysUpto (256*1024) src

389+cp src dst = File.writeArrays dst $ File.readArraysUpto (256*1024) src

390+wcl src = print =<<

391+ ( S.length

392+ $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.drain

393+ $ File.read src)

394+grepc pat src = print . (subtract 1) =<<

395+ ( S.length

396+ $ FL.splitOn (A.fromList (map (fromIntegral . ord) pat)) FL.drain

397+ $ File.read src)

398+avgll src = print =<<

399+ ( FL.foldl' avg

400+ $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length

401+ $ File.read src)

402+ where avg = (/) <$> toDouble FL.sum <*> toDouble FL.length

403+ toDouble = fmap (fromIntegral :: Int -> Double)

404+llhisto src = print =<<

405+ ( FL.foldl' (FL.classify FL.length)

406+ $ S.map bucket

407+ $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length

408+ $ File.read src)

409+ where

410+ bucket n = let i = n `div` 10 in if i > 9 then (9,n) else (i,n)

411+

412+main = do

413+ name <- fmap head getArgs

414+ src <- openFile name ReadMode

415+ -- cat src -- Unix cat program

416+ -- wcl src -- Unix wc -l program

417+ -- grepc "aaaa" src -- Unix grep -c program

418+

419+ -- dst <- openFile "dst.txt" WriteMode

420+ -- cp src dst -- Unix cp program

421+

422+ -- avgll src -- get average line length

423+ llhisto src -- get line length histogram

424+```

425+

426## Exceptions

427

428From a library user point of view, there is nothing much to learn or talk about