@@ -8,23 +8,43 @@
8-- Stability : experimental
8-- Stability : experimental
9-- Portability : GHC
9-- Portability : GHC
11−-- A byte stream of Unicode text can be decoded into a stream of 'Char' and
11+-- = Processing Unicode Strings
12−-- processed efficiently using regular stream processing operations. A stream
13−-- of 'Char' can be encoded into a byte stream using UTF formats and written to
16−-- If you have to store a unicode string in memory you can either write it as a
13+-- A 'Char' stream is the canonical representation to process Unicode strings.
17−-- pure stream "SerialT Identity Char" or write the 'Char' stream as "Array
14+-- It can be processed efficiently using regular stream processing operations.
18−-- Char". If space efficiency is a concern you can UTF8 encode the 'Char'
15+-- A byte stream of Unicode text read from an IO device or from an
19−-- stream and then write it as "Array Word8". "SerialT Identity Char" and
16+-- 'Streamly.Memory.Array.Array' in memory can be decoded into a 'Char' stream
20−-- "Array Char" are instances of 'IsString' and 'IsList', therefore,
17+-- using the decoding routines in this module. A 'String' (@[Char]@) can be
21−-- 'OverloadedStrings' and 'OverloadedLists' extensions can be used for
18+-- converted into a 'Char' stream using 'Streamly.Prelude.fromList'. An @Array
22−-- convenience.
19+-- Char@ can be 'Streamly.Prelude.unfold'ed into a stream using the array
20+-- 'Streamly.Memory.Array.read' unfold.
24−-- Some experimental APIs to conveniently process text using "Array Char"
22+-- = Storing Unicode Strings
25−-- represenation can be found in "Streamly.Internal.Unicode.Array".
27−-- Please note the following:
24+-- A stream of 'Char' can be encoded into a byte stream using the encoding
25+-- routines in this module and then written to IO devices or to arrays in
28+-- If you have to store a 'Char' stream in memory you can convert it into a
29+-- 'String' using 'Streamly.Prelude.toList' or using the
30+-- 'Streamly.Data.Fold.toList' fold. The 'String' type can be more efficient
31+-- than pinned arrays for short and short lived strings.
33+-- For longer or long lived streams you can 'Streamly.Prelude.fold' the 'Char'
34+-- stream as @Array Char@ using the array 'Streamly.Memory.Array.write' fold.
35+-- The 'Array' type provides a more compact representation and pinned memory
36+-- reducing GC overhead. If space efficiency is a concern you can use
37+-- 'encodeUtf8' on the 'Char' stream before writing it to an 'Array' providing
38+-- an even more compact representation.
42+-- @SerialT Identity Char@ and @Array Char@ are instances of 'IsString' and
43+-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
44+-- can be used for convenience when specifying unicode strings literals using
29-- * Case conversion: Some unicode characters translate to more than one code
49-- * Case conversion: Some unicode characters translate to more than one code
30-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
50-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
@@ -35,7 +55,15 @@
35-- different unicode representations, therefore, a character stream or
55-- different unicode representations, therefore, a character stream or
36-- character array cannot be directly compared. A normalized comparison may be
56-- character array cannot be directly compared. A normalized comparison may be
37-- needed to check string equivalence correctly.
57-- needed to check string equivalence correctly.
59+-- = Experimental APIs
61+-- Some experimental APIs to conveniently process text using the @Array Char@
62+-- represenation directly can be found in "Streamly.Internal.Unicode.Array".
64+-- XXX an unpinned array representation can be useful to store short and short
65+-- lived strings in memory.
39module Streamly.Data.Unicode.Stream
67module Streamly.Data.Unicode.Stream
41 -- * Construction (Decoding)
69 -- * Construction (Decoding)