@@ -8,23 +8,43 @@
88-- Stability : experimental
99-- Portability : GHC
1010--
11−-- A byte stream of Unicode text can be decoded into a stream of 'Char' and
12−-- processed efficiently using regular stream processing operations. A stream
13−-- of 'Char' can be encoded into a byte stream using UTF formats and written to
14−-- IO devices.
11+-- = Processing Unicode Strings
1512--
16−-- If you have to store a unicode string in memory you can either write it as a
17−-- pure stream "SerialT Identity Char" or write the 'Char' stream as "Array
18−-- Char". If space efficiency is a concern you can UTF8 encode the 'Char'
19−-- stream and then write it as "Array Word8". "SerialT Identity Char" and
20−-- "Array Char" are instances of 'IsString' and 'IsList', therefore,
21−-- 'OverloadedStrings' and 'OverloadedLists' extensions can be used for
22−-- convenience.
13+-- A 'Char' stream is the canonical representation to process Unicode strings.
14+-- It can be processed efficiently using regular stream processing operations.
15+-- A byte stream of Unicode text read from an IO device or from an
16+-- 'Streamly.Memory.Array.Array' in memory can be decoded into a 'Char' stream
17+-- using the decoding routines in this module. A 'String' (@[Char]@) can be
18+-- converted into a 'Char' stream using 'Streamly.Prelude.fromList'. An @Array
19+-- Char@ can be 'Streamly.Prelude.unfold'ed into a stream using the array
20+-- 'Streamly.Memory.Array.read' unfold.
2321--
24−-- Some experimental APIs to conveniently process text using "Array Char"
25−-- represenation can be found in "Streamly.Internal.Unicode.Array".
22+-- = Storing Unicode Strings
2623--
27−-- Please note the following:
24+-- A stream of 'Char' can be encoded into a byte stream using the encoding
25+-- routines in this module and then written to IO devices or to arrays in
26+-- memory.
27+--
28+-- If you have to store a 'Char' stream in memory you can convert it into a
29+-- 'String' using 'Streamly.Prelude.toList' or using the
30+-- 'Streamly.Data.Fold.toList' fold. The 'String' type can be more efficient
31+-- than pinned arrays for short and short lived strings.
32+--
33+-- For longer or long lived streams you can 'Streamly.Prelude.fold' the 'Char'
34+-- stream as @Array Char@ using the array 'Streamly.Memory.Array.write' fold.
35+-- The 'Array' type provides a more compact representation and pinned memory
36+-- reducing GC overhead. If space efficiency is a concern you can use
37+-- 'encodeUtf8' on the 'Char' stream before writing it to an 'Array' providing
38+-- an even more compact representation.
39+--
40+-- = String Literals
41+--
42+-- @SerialT Identity Char@ and @Array Char@ are instances of 'IsString' and
43+-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
44+-- can be used for convenience when specifying unicode strings literals using
45+-- these types.
46+--
47+-- = Pitfalls
2848--
2949-- * Case conversion: Some unicode characters translate to more than one code
3050-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
@@ -35,7 +55,15 @@
3555-- different unicode representations, therefore, a character stream or
3656-- character array cannot be directly compared. A normalized comparison may be
3757-- needed to check string equivalence correctly.
58+--
59+-- = Experimental APIs
60+--
61+-- Some experimental APIs to conveniently process text using the @Array Char@
62+-- represenation directly can be found in "Streamly.Internal.Unicode.Array".
3863
64+-- XXX an unpinned array representation can be useful to store short and short
65+-- lived strings in memory.
66+--
3967module Streamly.Data.Unicode.Stream
4068 (
4169 -- * Construction (Decoding)