15209b3 · streamly · git.iko.soy

git.iko.soy/streamly

on master help sign in

Update docs, expose getChunk/putChunk

15209b3Harendra Kumar5 years ago

src/Streamly/FileSystem/Handle.hs+44−34

compare with master

unifiedsplit

@@ -9,49 +9,59 @@

9-- Stability : released

10-- Portability : GHC

11--

12−-- Read and write streams and arrays to and from file handles. File handle IO

12+-- >>> import qualified Streamly.FileSystem.Handle as Handle

13−-- APIs are quite similar to "Streamly.Data.Array.Foreign" read write APIs. In that

14−-- regard, arrays can be considered as in-memory files or files can be

15−-- considered as on-disk arrays.

16--

13--

17−-- Control over file reading and writing behavior in terms of buffering,

14+-- Read and write byte streams and array streams to and from file handles

18−-- encoding, decoding is in the hands of the programmer, the 'TextEncoding',

15+-- ('Handle').

19−-- 'NewLineMode', and 'Buffering' options of the underlying handle provided by

20−-- GHC are not needed and ignored.

21--

16--

22−-- = Programmer Notes

17+-- The 'TextEncoding', 'NewLineMode', and 'Buffering' options of the underlying

18+-- GHC 'Handle' are ignored by these APIs. Please use @Streamly.Unicode.*@

19+-- modules for encoding and decoding a byte stream, use stream splitting

20+-- operations in "Streamly.Prelude" to create a stream of lines or to split the

21+-- input stream on any other type of boundaries.

23--

22--

24−-- > import qualified Streamly.FileSystem.Handle as Handle

23+-- To set the read or write start position use 'hSeek' on the 'Handle', the

24+-- 'Streamly.Prelude.before' combinator may be used to do that on a streaming

25+-- combinator. To restrict the length of read or write use the stream trimming

26+-- operations like 'Streamly.Prelude.take'.

27+--

28+-- Note that a 'Handle' is inherently stateful, therefore, we cannot use these

29+-- APIs from multiple threads without serialization; reading or writing in one

30+-- thread would affect the file position for other threads.

25--

31--

26-- For additional, experimental APIs take a look at

32-- For additional, experimental APIs take a look at

27-- "Streamly.Internal.FileSystem.Handle" module.

33-- "Streamly.Internal.FileSystem.Handle" module.

34+

35+-- Design notes:

28--

36--

29−-- = Performance Notes

37+-- By design, file handle IO APIs are quite similar to

38+-- "Streamly.Data.Array.Foreign" read write APIs. In that regard, arrays can be

39+-- considered as in-memory files or files can be considered as on-disk arrays.

30--

40--

31−-- In some cases the stream type based APIs in the

32−-- "Streamly.Internal.FileSystem.Handle" module may be more efficient compared

33−-- to the unfold/fold based APIs exposed from this module because of better

34−-- fusion by GHC. However, with the streamly fusion GHC plugin (upcoming) these

35−-- APIs would perform as well as the stream based APIs in all cases.

36−

37−-- IO APIs are divided into two categories, sequential streaming IO APIs and

38−-- random access IO APIs.

39−

40module Streamly.FileSystem.Handle

41module Streamly.FileSystem.Handle

41 (

42 (

43+ -- * Singleton IO

44+ -- | Read or write a single buffer.

45+ getChunk

46+ , putChunk

47+

42− -- * Sequential/Streaming IO

48+ -- * Streaming IO

43− -- | Stream data to or from a file or device sequentially. When reading,

49+ -- | Read or write a stream of data to or from a file or device

44− -- the stream is lazy and generated on-demand as the consumer consumes it.

50+ -- sequentially.

51+ --

45− -- Read IO requests to the IO device are performed in chunks limited to a

52+ -- Read requests to the IO device are performed in chunks limited to a

46− -- maximum size of 32KiB, this is referred to as

47− -- 'Streamly.Internal.Data.Array.Foreign.Type.defaultChunkSize' in the

53+ -- maximum size of 'Streamly.Internal.System.IO.defaultChunkSize'. Note

48− -- documentation. One IO request may or may not read the full

54+ -- that the size of the actual chunks in the resulting stream may be less

55+ -- than the @defaultChunkSize@ but it can never exceed it. If the whole

49− -- chunk. If the whole stream is not consumed, it is possible that we may

56+ -- stream is not consumed, it is possible that we may have read slightly

50− -- read slightly more from the IO device than what the consumer needed.

57+ -- more from the IO device than what the consumer needed.

58+ --

51 -- Unless specified otherwise in the API, writes are collected into chunks

59 -- Unless specified otherwise in the API, writes are collected into chunks

52− -- of 'Streamly.Internal.Data.Array.Foreign.Type.defaultChunkSize' before they

60+ -- of 'Streamly.Internal.System.IO.defaultChunkSize' before they are

53− -- are written to the IO device.

61+ -- written to the IO device.

63+ -- Internal notes:

64+ --

55 -- Streaming APIs work for all kind of devices, seekable or non-seekable;

65 -- Streaming APIs work for all kind of devices, seekable or non-seekable;

56 -- including disks, files, memory devices, terminals, pipes, sockets and

66 -- including disks, files, memory devices, terminals, pipes, sockets and

57 -- fifos. While random access APIs work only for files or devices that have

67 -- fifos. While random access APIs work only for files or devices that have

@@ -59,18 +69,18 @@ module Streamly.FileSystem.Handle

59 -- Devices like terminals, pipes, sockets and fifos do not have random

69 -- Devices like terminals, pipes, sockets and fifos do not have random

60 -- access capability.

70 -- access capability.

62− -- ** Read From Handle

72+ -- ** Reading

63 -- | 'TextEncoding', 'NewLineMode', and 'Buffering' options of the

73 -- | 'TextEncoding', 'NewLineMode', and 'Buffering' options of the

64 -- underlying handle are ignored. The read occurs from the current seek

74 -- underlying handle are ignored. The read occurs from the current seek

65 -- position of the file handle. The stream ends as soon as EOF is

75 -- position of the file handle. The stream ends as soon as EOF is

66 -- encountered.

76 -- encountered.

68− read

78+ , read

69 , readWithBufferOf

79 , readWithBufferOf

70 , readChunks

80 , readChunks

71 , readChunksWithBufferOf

81 , readChunksWithBufferOf

73− -- ** Write to Handle

83+ -- ** Writing

74 -- | 'TextEncoding', 'NewLineMode', and 'Buffering' options of the

84 -- | 'TextEncoding', 'NewLineMode', and 'Buffering' options of the

75 -- underlying handle are ignored. The write occurs from the current seek

85 -- underlying handle are ignored. The write occurs from the current seek

76 -- position of the file handle. The write behavior depends on the 'IOMode'

86 -- position of the file handle. The write behavior depends on the 'IOMode'

src/Streamly/Internal/FileSystem/Handle.hs+62−28

@@ -9,15 +9,24 @@

9-- Stability : experimental

10-- Portability : GHC

11--

12−-- The fundamental singleton APIs are 'getChunk' and 'putChunk' and the

12+-- The fundamental singleton IO APIs are 'getChunk' and 'putChunk' and the

13−-- fundamental stream APIs built on top of those are 'readChunksWithBufferOf'

13+-- fundamental stream IO APIs built on top of those are

14−-- and 'writeChunks'. Rest of this module is just combinatorial programming

14+-- 'readChunksWithBufferOf' and 'writeChunks'. Rest of this module is just

15−-- using these.

15+-- combinatorial programming using these.

17+-- TODO: Need a separate module for pread/pwrite based reading writing for

18+-- seekable devices. Stateless read/write can be helpful in multithreaded

19+-- applications.

20+--

17module Streamly.Internal.FileSystem.Handle

21module Streamly.Internal.FileSystem.Handle

18 (

22 (

19− -- ** Read from Handle

23+ -- * Singleton APIs

24+ getChunk

25+ , getChunkOf

26+ , putChunk

27+

28+ -- * Byte Stream Read

20− read

29+ , read

21 -- , readUtf8

30 -- , readUtf8

22 -- , readLines

31 -- , readLines

23 -- , readFrames

32 -- , readFrames

@@ -26,16 +35,14 @@ module Streamly.Internal.FileSystem.Handle

26 , toBytes

35 , toBytes

27 , toBytesWithBufferOf

36 , toBytesWithBufferOf

29− -- -- * Array Read

38+ -- * Chunked Stream Read

30− , getChunk

31− -- , readArrayOf

32 , readChunks

39 , readChunks

33 , readChunksWithBufferOf

40 , readChunksWithBufferOf

35 , toChunksWithBufferOf

42 , toChunksWithBufferOf

36 , toChunks

43 , toChunks

38− -- ** Write to Handle

45+ -- * Byte Stream Write

39 -- Byte stream write (Folds)

46 -- Byte stream write (Folds)

40 , write

47 , write

41 , write2

48 , write2

@@ -45,25 +52,22 @@ module Streamly.Internal.FileSystem.Handle

45 -- , writeLines

52 -- , writeLines

46 , writeWithBufferOf

53 , writeWithBufferOf

48− -- Byte stream write (Streams)

49 , putBytes

55 , putBytes

50 , putBytesWithBufferOf

56 , putBytesWithBufferOf

52− -- -- * Array Write

58+ -- * Chunked Stream Write

53− , putChunk

54 , writeChunks

59 , writeChunks

55 , writeChunksWithBufferOf

60 , writeChunksWithBufferOf

57− -- -- * Array stream Write

58 , putChunksWithBufferOf

62 , putChunksWithBufferOf

59 , putChunks

63 , putChunks

61− -- -- * Random Access (Seek)

65+ -- * Random Access (Seek)

62− -- -- | Unlike the streaming APIs listed above, these APIs apply to devices or

66+ -- | Unlike the streaming APIs listed above, these APIs apply to devices or

63 -- files that have random access or seek capability. This type of devices

67 -- files that have random access or seek capability. This type of devices

64 -- include disks, files, memory devices and exclude terminals, pipes,

68 -- include disks, files, memory devices and exclude terminals, pipes,

65 -- sockets and fifos.

69 -- sockets and fifos.

66− --

70+

67 -- We can also generate the request pattern using a funciton.

71 -- We can also generate the request pattern using a funciton.

68 --

72 --

69 -- , readIndex

73 -- , readIndex

@@ -76,7 +80,7 @@ module Streamly.Internal.FileSystem.Handle

77 -- , readChunksFrom

81 -- , readChunksFrom

78 -- , readChunksFromTo

82 -- , readChunksFromTo

79− -- , readChunksFromToWith

83+ , readChunksFromToWith

80 -- , readChunksFromThenToWith

84 -- , readChunksFromThenToWith

82 -- , writeIndex

86 -- , writeIndex

@@ -154,13 +158,16 @@ import qualified Streamly.Internal.Data.Stream.StreamD.Type as D

154-- Array IO (Input)

158-- Array IO (Input)

155-------------------------------------------------------------------------------

159-------------------------------------------------------------------------------

156

160

157−-- | Read a 'ByteArray' from a file handle. If no data is available on the

161+-- | Read a 'ByteArray' consisting of one or more bytes from a file handle. If

158−-- handle it blocks until some data becomes available. If data is available

162+-- no data is available on the handle it blocks until at least one byte becomes

159−-- then it immediately returns that data without blocking. It reads a maximum

163+-- available. If any data is available then it immediately returns that data

164+-- without blocking. As a result of this behavior, it may read less than or

160−-- of up to the size requested.

165+-- equal to the size requested.

166+--

167+-- @since 0.8.1

161{-# INLINABLE getChunk #-}

168{-# INLINABLE getChunk #-}

162−getChunk :: Int -> Handle -> IO (Array Word8)

169+getChunk :: MonadIO m => Int -> Handle -> m (Array Word8)

163−getChunk size h = do

170+getChunk size h = liftIO $ do

164 ptr <- mallocPlainForeignPtrBytes size

171 ptr <- mallocPlainForeignPtrBytes size

165 -- ptr <- mallocPlainForeignPtrAlignedBytes size (alignment (undefined :: Word8))

172 -- ptr <- mallocPlainForeignPtrAlignedBytes size (alignment (undefined :: Word8))

166 withForeignPtr ptr $ \p -> do

173 withForeignPtr ptr $ \p -> do

@@ -170,6 +177,18 @@ getChunk size h = do

170 unsafeFreezeWithShrink $

177 unsafeFreezeWithShrink $

171 mutableArray ptr (p `plusPtr` n) (p `plusPtr` size)

178 mutableArray ptr (p `plusPtr` n) (p `plusPtr` size)

172

179

180+-- This could be useful in implementing the "reverse" read APIs or if you want

181+-- to read arrays of exact size instead of compacting them later. Compacting

182+-- later requires more copying.

183+--

184+-- | Read a 'ByteArray' consisting of exactly the specified number of bytes

185+-- from a file handle.

186+--

187+-- /Unimplemented/

188+{-# INLINABLE getChunkOf #-}

189+getChunkOf :: Int -> Handle -> IO (Array Word8)

190+getChunkOf = undefined

191+

173-------------------------------------------------------------------------------

192-------------------------------------------------------------------------------

174-- Stream of Arrays IO

193-- Stream of Arrays IO

175-------------------------------------------------------------------------------

194-------------------------------------------------------------------------------

@@ -184,7 +203,7 @@ _toChunksWithBufferOf size h = go

184 where

203 where

185 -- XXX use cons/nil instead

204 -- XXX use cons/nil instead

186 go = mkStream $ \_ yld _ stp -> do

205 go = mkStream $ \_ yld _ stp -> do

187− arr <- liftIO $ getChunk size h

206+ arr <- getChunk size h

188 if A.length arr == 0

207 if A.length arr == 0

189 then stp

208 then stp

190 else yld arr go

209 else yld arr go

@@ -203,7 +222,7 @@ toChunksWithBufferOf size h = D.fromStreamD (D.Stream step ())

203 where

222 where

204 {-# INLINE_LATE step #-}

223 {-# INLINE_LATE step #-}

205 step _ _ = do

224 step _ _ = do

206− arr <- liftIO $ getChunk size h

225+ arr <- getChunk size h

207 return $

226 return $

208 case A.length arr of

227 case A.length arr of

209 0 -> D.Stop

228 0 -> D.Stop

@@ -221,12 +240,27 @@ readChunksWithBufferOf = Unfold step return

221 where

240 where

222 {-# INLINE_LATE step #-}

241 {-# INLINE_LATE step #-}

223 step (size, h) = do

242 step (size, h) = do

224− arr <- liftIO $ getChunk size h

243+ arr <- getChunk size h

225 return $

244 return $

226 case A.length arr of

245 case A.length arr of

227 0 -> D.Stop

246 0 -> D.Stop

228 _ -> D.Yield arr (size, h)

247 _ -> D.Yield arr (size, h)

229

248

249+-- There are two ways to implement this.

250+--

251+-- 1. Idiomatic: use a scan on the output of readChunksWithBufferOf to total

252+-- the array lengths and trim the last array to correct size.

253+-- 2. Simply implement it from scratch like readChunksWithBufferOf.

254+--

255+-- | The input to the unfold is @(from, to, bufferSize, handle)@. It starts

256+-- reading from the offset `from` in the file and reads up to the offset `to`.

257+--

258+--

259+{-# INLINE_NORMAL readChunksFromToWith #-}

260+readChunksFromToWith :: -- MonadIO m =>

261+ Unfold m (Int, Int, Int, Handle) (Array Word8)

262+readChunksFromToWith = undefined

263+

230-- XXX read 'Array a' instead of Word8

264-- XXX read 'Array a' instead of Word8

231--

265--

232-- | @toChunks handle@ reads a stream of arrays from the specified file

266-- | @toChunks handle@ reads a stream of arrays from the specified file

@@ -316,7 +350,7 @@ toBytes = AS.concat . toChunks

316

350

317-- | Write an 'Array' to a file handle.

351-- | Write an 'Array' to a file handle.

318--

352--

319−-- @since 0.7.0

353+-- @since 0.8.1

320{-# INLINABLE putChunk #-}

354{-# INLINABLE putChunk #-}

321putChunk :: (MonadIO m, Storable a) => Handle -> Array a -> m ()

355putChunk :: (MonadIO m, Storable a) => Handle -> Array a -> m ()

322putChunk _ arr | A.length arr == 0 = return ()

356putChunk _ arr | A.length arr == 0 = return ()