git.iko.soy/streamly
on 89b95bd help sign in
streamly/src/Streamly/Data/Unicode/Stream.hs
99 lines 3.7 kB history raw
1-- |
2-- Module : Streamly.Data.Unicode.Stream
3-- Copyright : (c) 2018 Composewell Technologies
4--
5-- License : BSD3
6-- Maintainer : streamly@composewell.com
7-- Stability : experimental
8-- Portability : GHC
9--
10-- = Processing Unicode Strings
11--
12-- A 'Char' stream is the canonical representation to process Unicode strings.
13-- It can be processed efficiently using regular stream processing operations.
14-- A byte stream of Unicode text read from an IO device or from an
15-- 'Streamly.Data.Array.Foreign.Array' in memory can be decoded into a 'Char' stream
16-- using the decoding routines in this module. A 'String' (@[Char]@) can be
17-- converted into a 'Char' stream using 'Streamly.Prelude.fromList'. An @Array
18-- Char@ can be 'Streamly.Prelude.unfold'ed into a stream using the array
19-- 'Streamly.Data.Array.Foreign.read' unfold.
20--
21-- = Storing Unicode Strings
22--
23-- A stream of 'Char' can be encoded into a byte stream using the encoding
24-- routines in this module and then written to IO devices or to arrays in
25-- memory.
26--
27-- If you have to store a 'Char' stream in memory you can convert it into a
28-- 'String' using 'Streamly.Prelude.toList' or using the
29-- 'Streamly.Data.Fold.toList' fold. The 'String' type can be more efficient
30-- than pinned arrays for short and short lived strings.
31--
32-- For longer or long lived streams you can 'Streamly.Prelude.fold' the 'Char'
33-- stream as @Array Char@ using the array 'Streamly.Data.Array.Foreign.write' fold.
34-- The 'Array' type provides a more compact representation and pinned memory
35-- reducing GC overhead. If space efficiency is a concern you can use
36-- 'encodeUtf8'' on the 'Char' stream before writing it to an 'Array' providing
37-- an even more compact representation.
38--
39-- = String Literals
40--
41-- @Stream Identity Char@ and @Array Char@ are instances of 'IsString' and
42-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
43-- can be used for convenience when specifying unicode strings literals using
44-- these types.
45--
46-- = Pitfalls
47--
48-- * Case conversion: Some unicode characters translate to more than one code
49-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
50-- package do not handle such characters. Therefore, operations like @map
51-- toUpper@ on a character stream or character array may not always perform
52-- correct conversion.
53-- * String comparison: In some cases, visually identical strings may have
54-- different unicode representations, therefore, a character stream or
55-- character array cannot be directly compared. A normalized comparison may be
56-- needed to check string equivalence correctly.
57--
58-- = Experimental APIs
59--
60-- Some experimental APIs to conveniently process text using the
61-- @Array Char@ represenation directly can be found in
62-- "Streamly.Internal.Memory.Unicode.Array".
64-- XXX an unpinned array representation can be useful to store short and short
65-- lived strings in memory.
66--
67module Streamly.Data.Unicode.Stream
68 {-# DEPRECATED "Use \"Streamly.Unicode.Stream\" instead" #-}
69 (
70 -- * Construction (Decoding)
71 decodeLatin1
72 , decodeUtf8
74 -- * Elimination (Encoding)
75 , encodeLatin1
76 , encodeUtf8
77 {-
78 -- * Operations on character strings
79 , strip -- (dropAround isSpace)
80 , stripEnd
81 , stripStart
82 -}
83 -- Not exposing these yet as we have consider these with respect to Unicode
84 -- segmentation routines which are yet to be implemented.
85 -- -- * Transformation
86 -- , lines
87 -- , words
88 -- , unlines
89 -- , unwords
91 -- * Deprecations
92 , decodeUtf8Lax
93 , encodeLatin1Lax
94 , encodeUtf8Lax
95 )
96where
98import Streamly.Internal.Unicode.Stream
99import Prelude hiding (lines, words, unlines, unwords)