@@ -0,0 +1,224 @@
1+## Overview
2+
3+Unicode is a standard which consists of a `char set`, `encodings`, attributes
4+and properties of characters, processing of strings, paragraphs, processing of
5+text in `locale` specific manner.
6+
7+### Char Set
8+
9+Unicode `char set` represents characters from all languages in the world. Each
10+character is assigned a, code point, a unique number identifying the character,
11+and written as `U+0076` where the four hex digits `0076` represent the unique
12+number assigned to the character.
13+
14+### Encodings
15+
16+A unicode character can be encoded as a:
17+
18+* fixed length encoding with a 32-bit value directly representing the code
19+ point in little endian (UTF32LE) or big endian (UTF32BE) byte ordering. See
20+ https://en.wikipedia.org/wiki/UTF-32.
21+* variable length encoding with one or two 16-bit values depending on the code
22+ point, UTF16LE and UTF16BE. See https://en.wikipedia.org/wiki/UTF-16.
23+* variable length encoding with one, two or three 8-bit values depending on the
24+ code point, UTF8. See https://en.wikipedia.org/wiki/UTF-8.
25+
26+### i18n and L10n
27+
28+Internationalization (i18n) is being able to represent and process all
29+languages in the world. Unicode performs i18n by representing all languages and
30+their common processing rules.
31+
32+Localization (L10n) is being able to customize the common internationalized
33+processing to a country or region (`locale`). Unicode specifies various
34+standard locales which includes customization of the attributes and processing
35+rules for each locale. Custom locales can be created with custom text
36+processing rules.
37+
38+* https://en.wikipedia.org/wiki/Internationalization_and_localization .
39+* http://userguide.icu-project.org/i18n
40+
41+## POSIX Locales
42+
43+On Debian Linux, the default system wide locale can be administered using
44+`localectl` or `sudo dpkg-reconfigure locales`.
45+
46+In a shell, the `locale` command shows the current locale settings. When you
47+start a program from the shell it inherits these settings via the process
48+environment and the C library loads and uses the appropriate locale. Even some
49+GUI programs if started from the shell can use the values from the
50+environment. Other GUI programs may have there own locale settings that can be
51+configured from their menu.
52+
53+The following environment variables can override the system wide locale and
54+determine how a process (handled by libc) performs unicode text processing for
55+different locale aspects:
56+
57+```
58+LC_CTYPE Defines character classification and case conversion.
59+LC_COLLATE Defines collation rules.
60+LC_MONETARY Defines the format and symbols used in formatting of monetary information.
61+LC_NUMERIC Defines the decimal delimiter, grouping, and grouping symbol for non-monetary numeric editing.
62+LC_TIME Defines the format and content of date and time information.
63+LC_MESSAGES Defines the format and values of affirmative and negative responses.
64+```
65+
66+Each environment variable above can be set to available `locale` settings. For
67+example `LC_CTYPE=en_US.UTF-8` (locale.charmap) specifies a locale `en_US` for
68+the language `en` (English) and the territory `US` (USA) to be used for
69+character classifications (e.g. `iswalpha`) and case conversions (e.g.
70+`toupper`). `UTF-8` is the charmap used by the locale.
71+
72+`C` and `POSIX` locales are the same and are used by default. glibc also
73+provides a generic `i18n` locale.
74+
75+Use `locale -a` command to see all available locales on a system and `locale
76+-m` for charmaps. See `man locale` as well. Also see
77+`/usr/share/i18n/SUPPORTED` on Linux (Debian).
78+https://www.gnu.org/software/libc/manual/html_node/Locale-Names.html for the
79+format in which these environment variables can be specified.
80+
81+The environment variables are preferred in the following order:
82+
83+```
84+LC_ALL Overrides everything else
85+LC_* Individual aspect customization
86+LANG Used as a substitute for any unset LC_* variable
87+```
88+
89+Normally only `LANG` should be used. If a particular aspect needs to be
90+cutsomized then `LC_*` variables can be used. `LC_ALL` overrides everything.
91+On GNU/Linux `LANGUAGE` can also be used as a list of preferred languages
92+separated by ":". LANGUAGE is effective only if LANG has been set to something
93+other than default and has higher priority than anything else.
94+
95+* https://www.gnu.org/software/gettext/manual/gettext.html#Users has a good
96+ overview of locale settings.
97+* https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html
98+ POSIX Locales standard
99+
100+### Localizing Messages
101+
102+GNU https://www.gnu.org/software/gettext/manual/gettext.html can be used by
103+programs to translate/localize the user interfacing text to multiple languages.
104+Catalogs of localized program error, alert, notification messages are installed
105+at e.g. `/usr/share/locale/en_US/LC_MESSAGES/`.
106+
107+* https://www.gnu.org/software/gettext/manual/gettext.html
108+
109+### Creating Locales
110+
111+From the `debian` manpage of `localedef` POSIX command:
112+
113+The localedef program reads the indicated charmap and input files,
114+compiles them to a binary form quickly usable by the locale functions in the C
115+library (setlocale(3), localeconv(3), etc.), and places the output in
116+outputpath.
117+
118+See `locale-gen` for a more high level program to generate locales.
119+/usr/lib/locale/locale-archive contains the generated binary files. On Debian
120+you can use `sudo dpkg-reconfigure locales` to select locales and set system
121+locale.
122+
123+On Linux/glibc (Debian), installed `charmaps` can be found at
124+`/usr/share/i18n/charmaps/` and locale definition input files at
125+`/usr/share/i18n/locales/`.
126+
127+## ICU Locales
128+
129+An ICU locale is frequently confused with a POSIX locale ID. An ICU locale ID
130+is not a POSIX locale ID. ICU locales do not specify the encoding and specify
131+variant locales differently.
132+
133+* http://userguide.icu-project.org/locale
134+
135+### Localizing Messages
136+
137+* http://userguide.icu-project.org/locale/localizing
138+
139+## Unicode Text processing
140+
141+* http://userguide.icu-project.org/posix
142+* http://userguide.icu-project.org/strings/properties
143+
144+### Locale Independent
145+
146+1) Char Properties
147+2) Normalization
148+3) Regex matching
149+
150+### Locale Specific
151+
152+1) Case Mapping:
153+ * https://unicode.org/faq/casemap_charprop.html
154+ * http://www.unicode.org/versions/latest/ch05.pdf#G21180
155+ * ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt
156+ * ftp://ftp.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt
157+2) Breaking
158+3) Collation
159+4) Charset Conversion
160+
161+## Haskell
162+
163+### Haskell Unicode Text Processing
164+
165+Related packages on hackage:
166+
167+* [base](https://www.stackage.org/lts/package/base) Data.Char module, uses libc
168+* [text](https://www.stackage.org/lts/package/text)
169+* [text-icu](https://stackage.org/lts/package/text-icu) C bindings to icu
170+
171+
172+* https://github.com/composewell/unicode-transforms
173+* https://github.com/llelf/prose
174+* [unicode-properties](https://hackage.haskell.org/package/unicode-properties) Unicode 3.2.0 character properties
175+* [hxt-charproperties](http://www.stackage.org/lts/package/hxt-charproperties) Character properties and classes for XML and Unicode
176+* [unicode-names](http://hackage.haskell.org/package/unicode-names) Unicode 3.2.0 character names
177+* [unicode](https://hackage.haskell.org/package/unicode) Construct and transform unicode characters
178+
179+
180+* [charset](https://www.stackage.org/lts/package/charset) Fast unicode character sets
181+
182+None of the existing Haskell packages provide comprehensive and fast access to
183+properties. `unicode-transforms` provides composition/decomposition data and
184+the script to extract data from unicode database.
185+
186+### Haskell Localization
187+
188+* http://wiki.haskell.org/Internationalization_of_Haskell_programs
189+* https://hackage.haskell.org/package/localize
190+* http://hackage.haskell.org/package/localization
191+* http://hackage.haskell.org/package/hgettext
192+* http://hackage.haskell.org/package/i18n
193+* https://hackage.haskell.org/package/setlocale
194+* http://hackage.haskell.org/package/system-locale
195+* https://github.com/llelf/numerals
196+
197+## TODO
198+
199+Factor out a `unicode-data` package from `unicode-transforms`.
200+`unicode-transforms` package will depend on unicode-data and can continue to be
201+used as is. Other packages can take advantage of the `unicode-data` to provide
202+unicode text processing services.
203+
204+To begin with, this package will contain:
205+
206+* char properties data
207+* case mapping data
208+* unicode normalization data
209+
210+This package can be used in `Streamly.Internal.Data.Unicode.*` to provide:
211+
212+* Fast access to char properties, we will no longer depend on libc for that and
213+ no FFI will be required to do iswspace, iswalpha etc.
214+* Correct case mappings from a single char to multi-char
215+* Stream based unicode normalization
216+* Breaking (locale independent for now)
217+
218+## Later
219+
220+* add locale data from CLDR to `unicode-data` to provide locale specific
221+ services as well.
222+* support locale specific breaking, regex, collation and charset conversion
223+* facility to add customized locale data
224+* Support providing application level resource data for localization