git.iko.soy/streamly
on d29030f help sign in
streamly/core/src/Streamly/Unicode/Stream.hs
101 lines 3.6 kB history raw
1-- |
2-- Module : Streamly.Unicode.Stream
3-- Copyright : (c) 2020 Composewell Technologies
4--
5-- License : BSD-3-Clause
6-- Maintainer : streamly@composewell.com
7-- Stability : released
8-- Portability : GHC
9--
10-- = Processing Unicode Strings
11--
12-- A 'Char' stream is the canonical representation to process Unicode strings.
13-- It can be processed efficiently using regular stream processing operations.
14-- A byte stream of Unicode text read from an IO device or from an
15-- 'Streamly.Data.Array.Array' in memory can be decoded into a 'Char' stream
16-- using the decoding routines in this module. A 'String' (@[Char]@) can be
17-- converted into a 'Char' stream using 'Streamly.Data.Stream.fromList'. An
18-- @Array Char@ can be 'Streamly.Data.Stream.unfold'ed into a stream using the
19-- array 'Streamly.Data.Array.read' unfold.
20--
21-- = Storing Unicode Strings
22--
23-- A stream of 'Char' can be encoded into a byte stream using the encoding
24-- routines in this module and then written to IO devices or to arrays in
25-- memory.
26--
27-- If you have to store a 'Char' stream in memory you can
28-- 'Streamly.Data.Stream.fold' the
29-- 'Char' stream as @Array Char@ using the array 'Streamly.Data.Array.write'
30-- fold. The 'Array' type provides a more compact representation
31-- reducing GC overhead. If space efficiency is a concern you can use
32-- 'encodeUtf8'' on the 'Char' stream before writing it to an 'Array' providing
33-- an even more compact representation.
34--
35-- = String Literals
36--
37-- @Stream Identity Char@ and @Array Char@ are instances of 'IsString' and
38-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
39-- can be used for convenience when specifying unicode strings literals using
40-- these types.
41--
42-- = Idioms
43--
44-- Some simple text processing operations can be represented simply as
45-- operations on Char streams. Follow the links for the following idioms:
46--
47-- * 'Streamly.Internal.Unicode.Stream.lines'
48-- * 'Streamly.Internal.Unicode.Stream.words'
49-- * 'Streamly.Internal.Unicode.Stream.unlines'
50-- * 'Streamly.Internal.Unicode.Stream.unwords'
51--
52-- = Pitfalls
53--
54-- * Case conversion: Some unicode characters translate to more than one code
55-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
56-- package do not handle such characters. Therefore, operations like @map
57-- toUpper@ on a character stream or character array may not always perform
58-- correct conversion.
59-- * String comparison: In some cases, visually identical strings may have
60-- different unicode representations, therefore, a character stream or
61-- character array cannot be directly compared. A normalized comparison may be
62-- needed to check string equivalence correctly.
63--
64-- = Experimental APIs
65--
66-- Some experimental APIs to conveniently process text using the
67-- @Array Char@ represenation directly can be found in
68-- "Streamly.Internal.Unicode.Array".
69--
70module Streamly.Unicode.Stream
71 (
72 -- * Construction (Decoding)
73 decodeLatin1
74 , decodeUtf8
75 , decodeUtf8'
76 , decodeUtf8Chunks
78 -- * Elimination (Encoding)
79 , encodeLatin1
80 , encodeLatin1'
81 , encodeUtf8
82 , encodeUtf8'
83 , encodeStrings
84 {-
85 -- * Operations on character strings
86 , strip -- (dropAround isSpace)
87 , stripEnd
88 , stripStart
89 -}
90 -- Not exposing these yet as we have consider these with respect to Unicode
91 -- segmentation routines which are yet to be implemented.
92 -- -- * Transformation
93 -- , lines
94 -- , words
95 -- , unlines
96 -- , unwords
97 )
98where
100import Streamly.Internal.Unicode.Stream
101import Prelude hiding (lines, words, unlines, unwords)