The performance is similar to flattenArrays, the code fuses perfectly. But in
the linecount benchmark Unfold case performs 50% slower even though this is not
due to fusion, just less optimal code being generated.
Each container type (e.g. Handle/Socket/File) may have similar nested/stream
level operations. We need a standardized way of naming the combinators related
to streams of containers. Also, we cannot have a separate module for such
combinators for each container type. Therefore it makes sense to put them in
the same module.
rename toArray/toArrayN to writeF/writeNF to keep the API name consistent with
their regular counterparts. Now the only difference in fold APIs is a suffix F.
Measure space usage in interesting cases.
The motivation for these benchmarks is parallel stream consuming a lot of stack
space in concatMap case. These benchmarks will hopefully catch any such
problems in future.
These benchmarks may take a lot of time to allow memory consumption to slowly
buildup to noticeable amount in case there is a problem. Therefore they are
enabled only in dev builds. We can use `--include-first-iter`, `--min-duration
0` options in gauge to run them in reasonable time. They need to be run with
`--measure-with` option to allow isolated runs, otherwise memory measurement
won't be correct.
The implementation of fromStreamDArraysOf is now 3x more efficient compared to
the earlier implementation. This makes byte stream level operations almost as
efficient as array level operations.
Other than this the following changes are added in this commit:
* Add insertAfterEach
* Add writeArraysPackedUpto to Handle IO
* Implement `wc -l` example more efficiently using arrays
* Add benchmark for lines/unlines using arrays
* Add tests for splitArraysOn
* Rename some array/file/handle APIs
* Error handling when the group size in grouping operations is 0 or negative
* Streamly.FileSystem.FD module for unbuffered IO. Buffering can be controlled
from the stream itself.
* Support IO using writev as well
Array APIs include:
* coalesceChunksOf to coalesce smaller arrays in a stream into bigger ones
* unlinesArraysBy to join a stream of arrays using a line separator
* splitArraysOn to split a stream of arrays on a separator byte
* Implement a stream flavor specific (concurrent/wSerial etc.) concatMap
(concatMapBy). Now concatMap has the same capabilities that the monad bind of
different streams has. In fact the monad bind is implemented using concatMap
* Use foldr/build fusion for concatMap (currently disabled) and for map/fmap
and mapM (enabled)
* Add benchmarks for bind/concatMap performance
unsafe use of unsafeInlineIO caused each array allocated in the toArrayN fold
to share the same memory.
This fix uses the IO monad to make sure that the code is not pure and therefore
we always allocate new memory. All such usage of unsafePerformIo have been
fixed. The remaining ones are reviewed to be safe.
After perf measurement these seems to perform the same as a scan followed by
map therefore we have not exposed these but kept for perf comparison and just
in case use.
* Deprecate `scanx`, `foldx`, `foldxM`
* Remove deprecated APIs `scan`, `foldl`, `foldlM`
* Fix the signature of foldrM
* Implement some custom folds in terms of foldr
* Document folds and scans better
* Reorganize the documentation in Prelude
* Add foldrS and foldrT for transforming folds
* add toRevList
* Add benchmarks and tests for the new folds
