git.iko.soy/streamly
on 01fa8dd help sign in
streamly/core/src/Streamly/Unicode/Stream.hs
107 lines 4.0 kB history raw
1-- |
2-- Module : Streamly.Unicode.Stream
3-- Copyright : (c) 2020 Composewell Technologies
4--
5-- License : BSD-3-Clause
6-- Maintainer : streamly@composewell.com
7-- Stability : released
8-- Portability : GHC
9--
10-- = Processing Unicode Strings
11--
12-- A 'Char' stream is the canonical representation to process Unicode strings.
13-- It can be processed efficiently using regular stream processing operations.
14-- A byte stream of Unicode text read from an IO device or from an
15-- 'Streamly.Data.Array.Array' in memory can be decoded into a 'Char' stream
16-- using the decoding routines in this module. A 'String' (@[Char]@) can be
17-- converted into a 'Char' stream using 'Streamly.Data.Stream.fromList'. An
18-- @Array Char@ can be 'Streamly.Data.Stream.unfold'ed into a stream using the
19-- array 'Streamly.Data.Array.read' unfold.
20--
21-- = Storing Unicode Strings
22--
23-- A stream of 'Char' can be encoded into a byte stream using the encoding
24-- routines in this module and then written to IO devices or to arrays in
25-- memory.
26--
27-- If you have to store a 'Char' stream in memory you can convert it into a
28-- 'String' using 'Streamly.Data.Fold.toList' fold. The 'String' type can be
29-- more efficient than pinned arrays for short and short lived strings.
30--
31-- For longer or long lived streams you can 'Streamly.Data.Stream.fold' the
32-- 'Char' stream as @Array Char@ using the array 'Streamly.Data.Array.write'
33-- fold. The 'Array' type provides a more compact representation and pinned
34-- memory reducing GC overhead. If space efficiency is a concern you can use
35-- 'encodeUtf8'' on the 'Char' stream before writing it to an 'Array' providing
36-- an even more compact representation.
37--
38-- = String Literals
39--
40-- @Stream Identity Char@ and @Array Char@ are instances of 'IsString' and
41-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
42-- can be used for convenience when specifying unicode strings literals using
43-- these types.
44--
45-- = Idioms
46--
47-- Some simple text processing operations can be represented simply as
48-- operations on Char streams. Follow the links for the following idioms:
49--
50-- * 'Streamly.Internal.Unicode.Stream.lines'
51-- * 'Streamly.Internal.Unicode.Stream.words'
52-- * 'Streamly.Internal.Unicode.Stream.unlines'
53-- * 'Streamly.Internal.Unicode.Stream.unwords'
54--
55-- = Pitfalls
56--
57-- * Case conversion: Some unicode characters translate to more than one code
58-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
59-- package do not handle such characters. Therefore, operations like @map
60-- toUpper@ on a character stream or character array may not always perform
61-- correct conversion.
62-- * String comparison: In some cases, visually identical strings may have
63-- different unicode representations, therefore, a character stream or
64-- character array cannot be directly compared. A normalized comparison may be
65-- needed to check string equivalence correctly.
66--
67-- = Experimental APIs
68--
69-- Some experimental APIs to conveniently process text using the
70-- @Array Char@ represenation directly can be found in
71-- "Streamly.Internal.Unicode.Array".
73-- XXX an unpinned array representation can be useful to store short and short
74-- lived strings in memory.
75--
76module Streamly.Unicode.Stream
77 (
78 -- * Construction (Decoding)
79 decodeLatin1
80 , decodeUtf8
81 , decodeUtf8'
82 , decodeUtf8Chunks
84 -- * Elimination (Encoding)
85 , encodeLatin1
86 , encodeLatin1'
87 , encodeUtf8
88 , encodeUtf8'
89 , encodeStrings
90 {-
91 -- * Operations on character strings
92 , strip -- (dropAround isSpace)
93 , stripEnd
94 , stripStart
95 -}
96 -- Not exposing these yet as we have consider these with respect to Unicode
97 -- segmentation routines which are yet to be implemented.
98 -- -- * Transformation
99 -- , lines
100 -- , words
101 -- , unlines
102 -- , unwords
103 )
104where
106import Streamly.Internal.Unicode.Stream
107import Prelude hiding (lines, words, unlines, unwords)