@@ -105,96 +105,6 @@ The module `Streamly` provides just the core stream types, type casting and
105105concurrency control combinators. Stream construction, transformation, folding,
106106merging, zipping combinators are found in `Streamly.Prelude`.
107107
108−## Show me an example
109−
110−The following code snippet lists a directory tree recursively, reading multiple
111−directories concurrently:
112−
113−```haskell
114−import Control.Monad.IO.Class (liftIO)
115−import Path.IO (listDir, getCurrentDir) -- from path-io package
116−import Streamly (AsyncT, adapt)
117−import qualified Streamly.Prelude as S
118−
119−listDirRecursive :: AsyncT IO ()
120−listDirRecursive = getCurrentDir >>= readdir >>= liftIO . mapM_ putStrLn
121− where
122− readdir dir = do
123− (dirs, files) <- listDir dir
124− S.yield (map show dirs ++ map show files) <> foldMap readdir dirs
125−
126−main :: IO ()
127−main = S.drain $ adapt $ listDirRecursive
128−```
129−
130−`AsyncT` is a stream monad transformer. If you are familiar with a list
131−transformer, it is nothing but `ListT` with concurrency semantics. For example,
132−the semigroup operation `<>` is concurrent. This makes `foldMap` concurrent
133−too. You can replace `AsyncT` with `SerialT` and the above code will become
134−serial, exactly equivalent to a `ListT`.
135−
136−## File IO
137−
138−The following code snippet implements some common Unix command line utilities
139−using streamly. To get an idea about IO streaming performance, you can
140−benchmark these against the regular unix utilities using the `time` command.
141−Make sure to use a big enough input file and compile with
142−`ghc -O2 -fspec-constr-recursive=10` when benchmarking. Use `+RTS -s` flags on
143−the executable to check the space usage, look for `maximum residency` in the
144−output.
145−
146−Note that `grep -c` counts the number of lines where the pattern occurs whereas
147−the snippet below counts the total number of occurrences of the pattern,
148−therefore, the output may differ.
149−
150−``` haskell
151−import qualified Streamly.Prelude as S
152−import qualified Streamly.Fold as FL
153−import qualified Streamly.Mem.Array as A
154−import qualified Streamly.FileSystem.File as File
155−
156−import Data.Char (ord)
157−import System.Environment (getArgs)
158−import System.IO (openFile, IOMode(..), stdout)
159−
160−cat src = File.writeArrays stdout $ File.readArraysUpto (256*1024) src
161−cp src dst = File.writeArrays dst $ File.readArraysUpto (256*1024) src
162−wcl src = print =<<
163− ( S.length
164− $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.drain
165− $ File.read src)
166−grepc pat src = print . (subtract 1) =<<
167− ( S.length
168− $ FL.splitOn (A.fromList (map (fromIntegral . ord) pat)) FL.drain
169− $ File.read src)
170−avgll src = print =<<
171− ( FL.foldl' avg
172− $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length
173− $ File.read src)
174− where avg = (/) <$> toDouble FL.sum <*> toDouble FL.length
175− toDouble = fmap (fromIntegral :: Int -> Double)
176−llhisto src = print =<<
177− ( FL.foldl' (FL.classify FL.length)
178− $ S.map bucket
179− $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length
180− $ File.read src)
181− where
182− bucket n = let i = n `div` 10 in if i > 9 then (9,n) else (i,n)
183−
184−main = do
185− name <- fmap head getArgs
186− src <- openFile name ReadMode
187− -- cat src -- Unix cat program
188− -- wcl src -- Unix wc -l program
189− -- grepc "aaaa" src -- Unix grep -c program
190−
191− -- dst <- openFile "dst.txt" WriteMode
192− -- cp src dst -- Unix cp program
193−
194− -- avgll src -- get average line length
195− llhisto src -- get line length histogram
196−```
197−
198108## Streaming Pipelines
199109
200110The following snippet provides a simple stream composition example that reads
@@ -401,6 +311,34 @@ The concurrency facilities provided by streamly can be compared with
401311[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
402312expression.
403313
314+## Example: Listing Directories Recursively/Concurrently
315+
316+The following code snippet lists a directory tree recursively, reading multiple
317+directories concurrently:
318+
319+```haskell
320+import Control.Monad.IO.Class (liftIO)
321+import Path.IO (listDir, getCurrentDir) -- from path-io package
322+import Streamly (AsyncT, adapt)
323+import qualified Streamly.Prelude as S
324+
325+listDirRecursive :: AsyncT IO ()
326+listDirRecursive = getCurrentDir >>= readdir >>= liftIO . mapM_ putStrLn
327+ where
328+ readdir dir = do
329+ (dirs, files) <- listDir dir
330+ S.yield (map show dirs ++ map show files) <> foldMap readdir dirs
331+
332+main :: IO ()
333+main = S.drain $ adapt $ listDirRecursive
334+```
335+
336+`AsyncT` is a stream monad transformer. If you are familiar with a list
337+transformer, it is nothing but `ListT` with concurrency semantics. For example,
338+the semigroup operation `<>` is concurrent. This makes `foldMap` concurrent
339+too. You can replace `AsyncT` with `SerialT` and the above code will become
340+serial, exactly equivalent to a `ListT`.
341+
404342## Rate Limiting
405343
406344For bounded concurrent streams, stream yield rate can be specified. For
@@ -423,6 +361,68 @@ rate. Rate control works precisely even at throughputs as high as millions of
423361yields per second. For more sophisticated rate control see the haddock
424362documentation.
425363
364+## File IO
365+
366+The following code snippet implements some common Unix command line utilities
367+using streamly. To get an idea about IO streaming performance, you can
368+benchmark these against the regular unix utilities using the `time` command.
369+Make sure to use a big enough input file and compile with
370+`ghc -O2 -fspec-constr-recursive=10` when benchmarking. Use `+RTS -s` flags on
371+the executable to check the space usage, look for `maximum residency` in the
372+output.
373+
374+Note that `grep -c` counts the number of lines where the pattern occurs whereas
375+the snippet below counts the total number of occurrences of the pattern,
376+therefore, the output may differ.
377+
378+``` haskell
379+import qualified Streamly.Prelude as S
380+import qualified Streamly.Fold as FL
381+import qualified Streamly.Mem.Array as A
382+import qualified Streamly.FileSystem.File as File
383+
384+import Data.Char (ord)
385+import System.Environment (getArgs)
386+import System.IO (openFile, IOMode(..), stdout)
387+
388+cat src = File.writeArrays stdout $ File.readArraysUpto (256*1024) src
389+cp src dst = File.writeArrays dst $ File.readArraysUpto (256*1024) src
390+wcl src = print =<<
391+ ( S.length
392+ $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.drain
393+ $ File.read src)
394+grepc pat src = print . (subtract 1) =<<
395+ ( S.length
396+ $ FL.splitOn (A.fromList (map (fromIntegral . ord) pat)) FL.drain
397+ $ File.read src)
398+avgll src = print =<<
399+ ( FL.foldl' avg
400+ $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length
401+ $ File.read src)
402+ where avg = (/) <$> toDouble FL.sum <*> toDouble FL.length
403+ toDouble = fmap (fromIntegral :: Int -> Double)
404+llhisto src = print =<<
405+ ( FL.foldl' (FL.classify FL.length)
406+ $ S.map bucket
407+ $ FL.splitSuffixBy (== fromIntegral (ord '\n')) FL.length
408+ $ File.read src)
409+ where
410+ bucket n = let i = n `div` 10 in if i > 9 then (9,n) else (i,n)
411+
412+main = do
413+ name <- fmap head getArgs
414+ src <- openFile name ReadMode
415+ -- cat src -- Unix cat program
416+ -- wcl src -- Unix wc -l program
417+ -- grepc "aaaa" src -- Unix grep -c program
418+
419+ -- dst <- openFile "dst.txt" WriteMode
420+ -- cp src dst -- Unix cp program
421+
422+ -- avgll src -- get average line length
423+ llhisto src -- get line length histogram
424+```
425+
426426## Exceptions
427427
428428From a library user point of view, there is nothing much to learn or talk about