15209b3 · streamly · git.iko.soy

git.iko.soy/streamly

Update docs, expose getChunk/putChunk

commit15209b38c9fa01a3a610f7ab62edd60669e24ab0 authorHarendra Kumar <harendra@composewell.com> dateSun 8 Aug 2021, 03:54 (5 years ago) parentd51f8c0

2 files changed+106−62

compare with master

unifiedsplit

src/Streamly/FileSystem/Handle.hs+44−34 src/Streamly/Internal/FileSystem/Handle.hs+62−28

src/Streamly/FileSystem/Handle.hs+44−34

@@ -9,49 +9,59 @@

99-- Stability : released

1010-- Portability : GHC

1111--

12−-- Read and write streams and arrays to and from file handles. File handle IO

13−-- APIs are quite similar to "Streamly.Data.Array.Foreign" read write APIs. In that

14−-- regard, arrays can be considered as in-memory files or files can be

15−-- considered as on-disk arrays.

12+-- >>> import qualified Streamly.FileSystem.Handle as Handle

1613--

17−-- Control over file reading and writing behavior in terms of buffering,

18−-- encoding, decoding is in the hands of the programmer, the 'TextEncoding',

19−-- 'NewLineMode', and 'Buffering' options of the underlying handle provided by

20−-- GHC are not needed and ignored.

14+-- Read and write byte streams and array streams to and from file handles

15+-- ('Handle').

2116--

22−-- = Programmer Notes

17+-- The 'TextEncoding', 'NewLineMode', and 'Buffering' options of the underlying

18+-- GHC 'Handle' are ignored by these APIs. Please use @Streamly.Unicode.*@

19+-- modules for encoding and decoding a byte stream, use stream splitting

20+-- operations in "Streamly.Prelude" to create a stream of lines or to split the

21+-- input stream on any other type of boundaries.

2322--

24−-- > import qualified Streamly.FileSystem.Handle as Handle

23+-- To set the read or write start position use 'hSeek' on the 'Handle', the

24+-- 'Streamly.Prelude.before' combinator may be used to do that on a streaming

25+-- combinator. To restrict the length of read or write use the stream trimming

26+-- operations like 'Streamly.Prelude.take'.

27+--

28+-- Note that a 'Handle' is inherently stateful, therefore, we cannot use these

29+-- APIs from multiple threads without serialization; reading or writing in one

30+-- thread would affect the file position for other threads.

2531--

2632-- For additional, experimental APIs take a look at

2733-- "Streamly.Internal.FileSystem.Handle" module.

34+

35+-- Design notes:

2836--

29−-- = Performance Notes

37+-- By design, file handle IO APIs are quite similar to

38+-- "Streamly.Data.Array.Foreign" read write APIs. In that regard, arrays can be

39+-- considered as in-memory files or files can be considered as on-disk arrays.

3040--

31−-- In some cases the stream type based APIs in the

32−-- "Streamly.Internal.FileSystem.Handle" module may be more efficient compared

33−-- to the unfold/fold based APIs exposed from this module because of better

34−-- fusion by GHC. However, with the streamly fusion GHC plugin (upcoming) these

35−-- APIs would perform as well as the stream based APIs in all cases.

36−

37−-- IO APIs are divided into two categories, sequential streaming IO APIs and

38−-- random access IO APIs.

39−

4041module Streamly.FileSystem.Handle

4142 (

42− -- * Sequential/Streaming IO

43− -- | Stream data to or from a file or device sequentially. When reading,

44− -- the stream is lazy and generated on-demand as the consumer consumes it.

45− -- Read IO requests to the IO device are performed in chunks limited to a

46− -- maximum size of 32KiB, this is referred to as

47− -- 'Streamly.Internal.Data.Array.Foreign.Type.defaultChunkSize' in the

48− -- documentation. One IO request may or may not read the full

49− -- chunk. If the whole stream is not consumed, it is possible that we may

50− -- read slightly more from the IO device than what the consumer needed.

43+ -- * Singleton IO

44+ -- | Read or write a single buffer.

45+ getChunk

46+ , putChunk

47+

48+ -- * Streaming IO

49+ -- | Read or write a stream of data to or from a file or device

50+ -- sequentially.

51+ --

52+ -- Read requests to the IO device are performed in chunks limited to a

53+ -- maximum size of 'Streamly.Internal.System.IO.defaultChunkSize'. Note

54+ -- that the size of the actual chunks in the resulting stream may be less

55+ -- than the @defaultChunkSize@ but it can never exceed it. If the whole

56+ -- stream is not consumed, it is possible that we may have read slightly

57+ -- more from the IO device than what the consumer needed.

58+ --

5159 -- Unless specified otherwise in the API, writes are collected into chunks

52− -- of 'Streamly.Internal.Data.Array.Foreign.Type.defaultChunkSize' before they

53− -- are written to the IO device.

60+ -- of 'Streamly.Internal.System.IO.defaultChunkSize' before they are

61+ -- written to the IO device.

5462

63+ -- Internal notes:

64+ --

5565 -- Streaming APIs work for all kind of devices, seekable or non-seekable;

5666 -- including disks, files, memory devices, terminals, pipes, sockets and

5767 -- fifos. While random access APIs work only for files or devices that have

@@ -59,18 +69,18 @@ module Streamly.FileSystem.Handle

5969 -- Devices like terminals, pipes, sockets and fifos do not have random

6070 -- access capability.

6171

62− -- ** Read From Handle

72+ -- ** Reading

6373 -- | 'TextEncoding', 'NewLineMode', and 'Buffering' options of the

6474 -- underlying handle are ignored. The read occurs from the current seek

6575 -- position of the file handle. The stream ends as soon as EOF is

6676 -- encountered.

6777

68− read

78+ , read

6979 , readWithBufferOf

7080 , readChunks

7181 , readChunksWithBufferOf

7282

73− -- ** Write to Handle

83+ -- ** Writing

7484 -- | 'TextEncoding', 'NewLineMode', and 'Buffering' options of the

7585 -- underlying handle are ignored. The write occurs from the current seek

7686 -- position of the file handle. The write behavior depends on the 'IOMode'

src/Streamly/Internal/FileSystem/Handle.hs+62−28

@@ -9,15 +9,24 @@

99-- Stability : experimental

1010-- Portability : GHC

1111--

12−-- The fundamental singleton APIs are 'getChunk' and 'putChunk' and the

13−-- fundamental stream APIs built on top of those are 'readChunksWithBufferOf'

14−-- and 'writeChunks'. Rest of this module is just combinatorial programming

15−-- using these.

12+-- The fundamental singleton IO APIs are 'getChunk' and 'putChunk' and the

13+-- fundamental stream IO APIs built on top of those are

14+-- 'readChunksWithBufferOf' and 'writeChunks'. Rest of this module is just

15+-- combinatorial programming using these.

1616

17+-- TODO: Need a separate module for pread/pwrite based reading writing for

18+-- seekable devices. Stateless read/write can be helpful in multithreaded

19+-- applications.

20+--

1721module Streamly.Internal.FileSystem.Handle

1822 (

19− -- ** Read from Handle

20− read

23+ -- * Singleton APIs

24+ getChunk

25+ , getChunkOf

26+ , putChunk

27+

28+ -- * Byte Stream Read

29+ , read

2130 -- , readUtf8

2231 -- , readLines

2332 -- , readFrames

@@ -26,16 +35,14 @@ module Streamly.Internal.FileSystem.Handle

2635 , toBytes

2736 , toBytesWithBufferOf

2837

29− -- -- * Array Read

30− , getChunk

31− -- , readArrayOf

38+ -- * Chunked Stream Read

3239 , readChunks

3340 , readChunksWithBufferOf

3441

3542 , toChunksWithBufferOf

3643 , toChunks

3744

38− -- ** Write to Handle

45+ -- * Byte Stream Write

3946 -- Byte stream write (Folds)

4047 , write

4148 , write2

@@ -45,25 +52,22 @@ module Streamly.Internal.FileSystem.Handle

4552 -- , writeLines

4653 , writeWithBufferOf

4754

48− -- Byte stream write (Streams)

4955 , putBytes

5056 , putBytesWithBufferOf

5157

52− -- -- * Array Write

53− , putChunk

58+ -- * Chunked Stream Write

5459 , writeChunks

5560 , writeChunksWithBufferOf

5661

57− -- -- * Array stream Write

5862 , putChunksWithBufferOf

5963 , putChunks

6064

61− -- -- * Random Access (Seek)

62− -- -- | Unlike the streaming APIs listed above, these APIs apply to devices or

65+ -- * Random Access (Seek)

66+ -- | Unlike the streaming APIs listed above, these APIs apply to devices or

6367 -- files that have random access or seek capability. This type of devices

6468 -- include disks, files, memory devices and exclude terminals, pipes,

6569 -- sockets and fifos.

66− --

70+

6771 -- We can also generate the request pattern using a funciton.

6872 --

6973 -- , readIndex

@@ -76,7 +80,7 @@ module Streamly.Internal.FileSystem.Handle

7680

7781 -- , readChunksFrom

7882 -- , readChunksFromTo

79− -- , readChunksFromToWith

83+ , readChunksFromToWith

8084 -- , readChunksFromThenToWith

8185

8286 -- , writeIndex

@@ -154,13 +158,16 @@ import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

154158-- Array IO (Input)

155159-------------------------------------------------------------------------------

156160

157−-- | Read a 'ByteArray' from a file handle. If no data is available on the

158−-- handle it blocks until some data becomes available. If data is available

159−-- then it immediately returns that data without blocking. It reads a maximum

160−-- of up to the size requested.

161+-- | Read a 'ByteArray' consisting of one or more bytes from a file handle. If

162+-- no data is available on the handle it blocks until at least one byte becomes

163+-- available. If any data is available then it immediately returns that data

164+-- without blocking. As a result of this behavior, it may read less than or

165+-- equal to the size requested.

166+--

167+-- @since 0.8.1

161168{-# INLINABLE getChunk #-}

162−getChunk :: Int -> Handle -> IO (Array Word8)

163−getChunk size h = do

169+getChunk :: MonadIO m => Int -> Handle -> m (Array Word8)

170+getChunk size h = liftIO $ do

164171 ptr <- mallocPlainForeignPtrBytes size

165172 -- ptr <- mallocPlainForeignPtrAlignedBytes size (alignment (undefined :: Word8))

166173 withForeignPtr ptr $ \p -> do

@@ -170,6 +177,18 @@ getChunk size h = do

170177 unsafeFreezeWithShrink $

171178 mutableArray ptr (p `plusPtr` n) (p `plusPtr` size)

172179

180+-- This could be useful in implementing the "reverse" read APIs or if you want

181+-- to read arrays of exact size instead of compacting them later. Compacting

182+-- later requires more copying.

183+--

184+-- | Read a 'ByteArray' consisting of exactly the specified number of bytes

185+-- from a file handle.

186+--

187+-- /Unimplemented/

188+{-# INLINABLE getChunkOf #-}

189+getChunkOf :: Int -> Handle -> IO (Array Word8)

190+getChunkOf = undefined

191+

173192-------------------------------------------------------------------------------

174193-- Stream of Arrays IO

175194-------------------------------------------------------------------------------

@@ -184,7 +203,7 @@ _toChunksWithBufferOf size h = go

184203 where

185204 -- XXX use cons/nil instead

186205 go = mkStream $ \_ yld _ stp -> do

187− arr <- liftIO $ getChunk size h

206+ arr <- getChunk size h

188207 if A.length arr == 0

189208 then stp

190209 else yld arr go

@@ -203,7 +222,7 @@ toChunksWithBufferOf size h = D.fromStreamD (D.Stream step ())

203222 where

204223 {-# INLINE_LATE step #-}

205224 step _ _ = do

206− arr <- liftIO $ getChunk size h

225+ arr <- getChunk size h

207226 return $

208227 case A.length arr of

209228 0 -> D.Stop

@@ -221,12 +240,27 @@ readChunksWithBufferOf = Unfold step return

221240 where

222241 {-# INLINE_LATE step #-}

223242 step (size, h) = do

224− arr <- liftIO $ getChunk size h

243+ arr <- getChunk size h

225244 return $

226245 case A.length arr of

227246 0 -> D.Stop

228247 _ -> D.Yield arr (size, h)

229248

249+-- There are two ways to implement this.

250+--

251+-- 1. Idiomatic: use a scan on the output of readChunksWithBufferOf to total

252+-- the array lengths and trim the last array to correct size.

253+-- 2. Simply implement it from scratch like readChunksWithBufferOf.

254+--

255+-- | The input to the unfold is @(from, to, bufferSize, handle)@. It starts

256+-- reading from the offset `from` in the file and reads up to the offset `to`.

257+--

258+--

259+{-# INLINE_NORMAL readChunksFromToWith #-}

260+readChunksFromToWith :: -- MonadIO m =>

261+ Unfold m (Int, Int, Int, Handle) (Array Word8)

262+readChunksFromToWith = undefined

263+

230264-- XXX read 'Array a' instead of Word8

231265--

232266-- | @toChunks handle@ reads a stream of arrays from the specified file

@@ -316,7 +350,7 @@ toBytes = AS.concat . toChunks

316350

317351-- | Write an 'Array' to a file handle.

318352--

319−-- @since 0.7.0

353+-- @since 0.8.1

320354{-# INLINABLE putChunk #-}

321355putChunk :: (MonadIO m, Storable a) => Handle -> Array a -> m ()

322356putChunk _ arr | A.length arr == 0 = return ()