git.iko.soy/streamly
on 79df417 help sign in
streamly/core/src/Streamly/Unicode/Stream.hs
110 lines 3.9 kB history raw
1{-# LANGUAGE CPP #-}
2-- |
3-- Module : Streamly.Unicode.Stream
4-- Copyright : (c) 2020 Composewell Technologies
5--
6-- License : BSD-3-Clause
7-- Maintainer : streamly@composewell.com
8-- Stability : released
9-- Portability : GHC
10--
11-- = Processing Unicode Strings
12--
13-- A 'Char' stream is the canonical representation to process Unicode strings.
14-- It can be processed efficiently using regular stream processing operations.
15-- A byte stream of Unicode text read from an IO device or from an
16-- 'Streamly.Data.Array.Array' in memory can be decoded into a 'Char' stream
17-- using the decoding routines in this module. A 'String' (@[Char]@) can be
18-- converted into a 'Char' stream using 'Streamly.Data.Stream.fromList'. An
19-- @Array Char@ can be 'Streamly.Data.Stream.unfold'ed into a stream using the
20-- array 'Streamly.Data.Array.read' unfold.
21--
22-- = Storing Unicode Strings
23--
24-- A stream of 'Char' can be encoded into a byte stream using the encoding
25-- routines in this module and then written to IO devices or to arrays in
26-- memory.
27--
28-- If you have to store a 'Char' stream in memory you can
29-- 'Streamly.Data.Stream.fold' the
30-- 'Char' stream as @Array Char@ using the array 'Streamly.Data.Array.write'
31-- fold. The 'Array' type provides a more compact representation
32-- reducing GC overhead. If space efficiency is a concern you can use
33-- 'encodeUtf8'' on the 'Char' stream before writing it to an 'Array' providing
34-- an even more compact representation.
35--
36-- = String Literals
37--
38-- @Stream Identity Char@ and @Array Char@ are instances of 'IsString' and
39-- 'IsList', therefore, 'OverloadedStrings' and 'OverloadedLists' extensions
40-- can be used for convenience when specifying unicode strings literals using
41-- these types.
42--
43-- = Idioms
44--
45-- Some simple text processing operations can be represented simply as
46-- operations on Char streams. Follow the links for the following idioms:
47--
48-- * 'Streamly.Internal.Unicode.Stream.lines'
49-- * 'Streamly.Internal.Unicode.Stream.words'
50-- * 'Streamly.Internal.Unicode.Stream.unlines'
51-- * 'Streamly.Internal.Unicode.Stream.unwords'
52--
53-- = Pitfalls
54--
55-- * Case conversion: Some unicode characters translate to more than one code
56-- point on case conversion. The 'toUpper' and 'toLower' functions in @base@
57-- package do not handle such characters. Therefore, operations like @map
58-- toUpper@ on a character stream or character array may not always perform
59-- correct conversion.
60-- * String comparison: In some cases, visually identical strings may have
61-- different unicode representations, therefore, a character stream or
62-- character array cannot be directly compared. A normalized comparison may be
63-- needed to check string equivalence correctly.
64--
65-- = Experimental APIs
66--
67-- Some experimental APIs to conveniently process text using the
68-- @Array Char@ represenation directly can be found in
69-- "Streamly.Internal.Unicode.Array".
70--
71module Streamly.Unicode.Stream
72 (
73 -- * Setup
74 -- | To execute the code examples provided in this module in ghci, please
75 -- run the following commands first.
76 --
77 -- $setup
79 -- * Construction (Decoding)
80 decodeLatin1
81 , decodeUtf8
82 , decodeUtf8'
83 , decodeUtf8Chunks
85 -- * Elimination (Encoding)
86 , encodeLatin1
87 , encodeLatin1'
88 , encodeUtf8
89 , encodeUtf8'
90 , encodeStrings
91 {-
92 -- * Operations on character strings
93 , strip -- (dropAround isSpace)
94 , stripEnd
95 , stripStart
96 -}
97 -- Not exposing these yet as we have consider these with respect to Unicode
98 -- segmentation routines which are yet to be implemented.
99 -- -- * Transformation
100 -- , lines
101 -- , words
102 -- , unlines
103 -- , unwords
104 )
105where
107import Streamly.Internal.Unicode.Stream
108import Prelude hiding (lines, words, unlines, unwords)
110#include "DocTestUnicodeStream.hs"