git.iko.soy/streamly
on c8d1ca3 help sign in
streamly/src/Streamly/Unicode/Stream.hs
107 lines 4.0 kB history raw
1-- |
2-- Module : Streamly.Unicode.Stream
3-- Copyright : (c) 2020 Composewell Technologies
4--
5-- License : BSD-3-Clause
6-- Maintainer : streamly@composewell.com
7-- Stability : released
8-- Portability : GHC
9--
10-- = Processing Unicode Strings
11--
12-- A 'Char' stream is the canonical representation to process Unicode strings.
13-- It can be processed efficiently using regular stream processing operations.
14-- A byte stream of Unicode text read from an IO device or from an
15-- 'Streamly.Data.Array.Foreign.Array' in memory can be decoded into a 'Char' stream
16-- using the decoding routines in this module. A 'String' (@[Char]@) can be
17-- converted into a 'Char' stream using 'Streamly.Prelude.fromList'. An @Array
18-- Char@ can be 'Streamly.Prelude.unfold'ed into a stream using the array
19-- 'Streamly.Data.Array.Foreign.read' unfold.
20--
21-- = Storing Unicode Strings
22--
23-- A stream of 'Char' can be encoded into a byte stream using the encoding
24-- routines in this module and then written to IO devices or to arrays in
25-- memory.
26--
27-- If you have to store a 'Char' stream in memory you can convert it into a
28-- 'String' using 'Streamly.Prelude.toList' or using the
29-- 'Streamly.Data.Fold.toList' fold. The 'String' type can be more efficient
30-- than pinned arrays for short and short lived strings.
31--
32-- For longer or long lived streams you can 'Streamly.Prelude.fold' the 'Char'
33-- stream as @Array Char@ using the array 'Streamly.Data.Array.Foreign.write' fold.
34-- The 'Array' type provides a more compact representation and pinned memory
35-- reducing GC overhead. If space efficiency is a concern you can use
36-- 'encodeUtf8'' on the 'Char' stream before writing it to an 'Array' providing
37-- an even more compact representation.
38--
39-- = String Literals
40--
41-- @SerialT Identity Char@ and @Array Char@ are instances of 'IsString' and
42-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
43-- can be used for convenience when specifying unicode strings literals using
44-- these types.
45--
46-- = Idioms
47--
48-- Some simple text processing operations can be represented simply as
49-- operations on Char streams. Follow the links for the following idioms:
50--
51-- * 'Streamly.Internal.Unicode.Stream.lines'
52-- * 'Streamly.Internal.Unicode.Stream.words'
53-- * 'Streamly.Internal.Unicode.Stream.unlines'
54-- * 'Streamly.Internal.Unicode.Stream.unwords'
55--
56-- = Pitfalls
57--
58-- * Case conversion: Some unicode characters translate to more than one code
59-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
60-- package do not handle such characters. Therefore, operations like @map
61-- toUpper@ on a character stream or character array may not always perform
62-- correct conversion.
63-- * String comparison: In some cases, visually identical strings may have
64-- different unicode representations, therefore, a character stream or
65-- character array cannot be directly compared. A normalized comparison may be
66-- needed to check string equivalence correctly.
67--
68-- = Experimental APIs
69--
70-- Some experimental APIs to conveniently process text using the
71-- @Array Char@ represenation directly can be found in
72-- "Streamly.Internal.Unicode.Array.Char".
74-- XXX an unpinned array representation can be useful to store short and short
75-- lived strings in memory.
76--
77module Streamly.Unicode.Stream
78 (
79 -- * Construction (Decoding)
80 decodeLatin1
81 , decodeUtf8
82 , decodeUtf8'
84 -- * Elimination (Encoding)
85 , encodeLatin1
86 , encodeLatin1'
87 , encodeUtf8
88 , encodeUtf8'
89 , encodeStrings
90 {-
91 -- * Operations on character strings
92 , strip -- (dropAround isSpace)
93 , stripEnd
94 , stripStart
95 -}
96 -- Not exposing these yet as we have consider these with respect to Unicode
97 -- segmentation routines which are yet to be implemented.
98 -- -- * Transformation
99 -- , lines
100 -- , words
101 -- , unlines
102 -- , unwords
103 )
104where
106import Streamly.Internal.Unicode.Stream
107import Prelude hiding (lines, words, unlines, unwords)