@@ -0,0 +1,96 @@
1+-- |
2+-- Module : Streamly.Unicode.Stream
3+-- Copyright : (c) 2020 Composewell Technologies
4+--
5+-- License : BSD-3-Clause
6+-- Maintainer : streamly@composewell.com
7+-- Stability : experimental
8+-- Portability : GHC
9+--
10+-- = Processing Unicode Strings
11+--
12+-- A 'Char' stream is the canonical representation to process Unicode strings.
13+-- It can be processed efficiently using regular stream processing operations.
14+-- A byte stream of Unicode text read from an IO device or from an
15+-- 'Streamly.Memory.Array.Array' in memory can be decoded into a 'Char' stream
16+-- using the decoding routines in this module. A 'String' (@[Char]@) can be
17+-- converted into a 'Char' stream using 'Streamly.Prelude.fromList'. An @Array
18+-- Char@ can be 'Streamly.Prelude.unfold'ed into a stream using the array
19+-- 'Streamly.Memory.Array.read' unfold.
20+--
21+-- = Storing Unicode Strings
22+--
23+-- A stream of 'Char' can be encoded into a byte stream using the encoding
24+-- routines in this module and then written to IO devices or to arrays in
25+-- memory.
26+--
27+-- If you have to store a 'Char' stream in memory you can convert it into a
28+-- 'String' using 'Streamly.Prelude.toList' or using the
29+-- 'Streamly.Data.Fold.toList' fold. The 'String' type can be more efficient
30+-- than pinned arrays for short and short lived strings.
31+--
32+-- For longer or long lived streams you can 'Streamly.Prelude.fold' the 'Char'
33+-- stream as @Array Char@ using the array 'Streamly.Memory.Array.write' fold.
34+-- The 'Array' type provides a more compact representation and pinned memory
35+-- reducing GC overhead. If space efficiency is a concern you can use
36+-- 'encodeUtf8' on the 'Char' stream before writing it to an 'Array' providing
37+-- an even more compact representation.
38+--
39+-- = String Literals
40+--
41+-- @SerialT Identity Char@ and @Array Char@ are instances of 'IsString' and
42+-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
43+-- can be used for convenience when specifying unicode strings literals using
44+-- these types.
45+--
46+-- = Pitfalls
47+--
48+-- * Case conversion: Some unicode characters translate to more than one code
49+-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
50+-- package do not handle such characters. Therefore, operations like @map
51+-- toUpper@ on a character stream or character array may not always perform
52+-- correct conversion.
53+-- * String comparison: In some cases, visually identical strings may have
54+-- different unicode representations, therefore, a character stream or
55+-- character array cannot be directly compared. A normalized comparison may be
56+-- needed to check string equivalence correctly.
57+--
58+-- = Experimental APIs
59+--
60+-- Some experimental APIs to conveniently process text using the
61+-- @Array Char@ represenation directly can be found in
62+-- "Streamly.Internal.Memory.Unicode.Array".
63+
64+-- XXX an unpinned array representation can be useful to store short and short
65+-- lived strings in memory.
66+--
67+module Streamly.Unicode.Stream
68+ (
69+ -- * Construction (Decoding)
70+ decodeLatin1
71+ , decodeUtf8
72+ , decodeUtf8Lax
73+
74+ -- * Elimination (Encoding)
75+ , encodeLatin1
76+ , encodeLatin1Lax
77+ , encodeUtf8
78+ , encodeUtf8Lax
79+ {-
80+ -- * Operations on character strings
81+ , strip -- (dropAround isSpace)
82+ , stripEnd
83+ , stripStart
84+ -}
85+ -- Not exposing these yet as we have consider these with respect to Unicode
86+ -- segmentation routines which are yet to be implemented.
87+ -- -- * Transformation
88+ -- , lines
89+ -- , words
90+ -- , unlines
91+ -- , unwords
92+ )
93+where
94+
95+import Streamly.Internal.Unicode.Stream
96+import Prelude hiding (lines, words, unlines, unwords)