git.iko.soy/streamly
on master help sign in
streamly/docs/Developer/unicode.md
224 lines 8.9 kB history raw
1## Overview
3Unicode is a standard which consists of a `char set`, `encodings`, attributes
4and properties of characters, processing of strings, paragraphs, processing of
5text in `locale` specific manner.
7### Char Set
9Unicode `char set` represents characters from all languages in the world. Each
10character is assigned a, code point, a unique number identifying the character,
11and written as `U+0076` where the four hex digits `0076` represent the unique
12number assigned to the character.
14### Encodings
16A unicode character can be encoded as a:
18* fixed length encoding with a 32-bit value directly representing the code
19 point in little endian (UTF32LE) or big endian (UTF32BE) byte ordering. See
20 https://en.wikipedia.org/wiki/UTF-32.
21* variable length encoding with one or two 16-bit values depending on the code
22 point, UTF16LE and UTF16BE. See https://en.wikipedia.org/wiki/UTF-16.
23* variable length encoding with one, two or three 8-bit values depending on the
24 code point, UTF8. See https://en.wikipedia.org/wiki/UTF-8.
26### i18n and L10n
28Internationalization (i18n) is being able to represent and process all
29languages in the world. Unicode performs i18n by representing all languages and
30their common processing rules.
32Localization (L10n) is being able to customize the common internationalized
33processing to a country or region (`locale`). Unicode specifies various
34standard locales which includes customization of the attributes and processing
35rules for each locale. Custom locales can be created with custom text
36processing rules.
38* https://en.wikipedia.org/wiki/Internationalization_and_localization .
39* http://userguide.icu-project.org/i18n
41## POSIX Locales
43On Debian Linux, the default system wide locale can be administered using
44`localectl` or `sudo dpkg-reconfigure locales`.
46In a shell, the `locale` command shows the current locale settings. When you
47start a program from the shell it inherits these settings via the process
48environment and the C library loads and uses the appropriate locale. Even some
49GUI programs if started from the shell can use the values from the
50environment. Other GUI programs may have there own locale settings that can be
51configured from their menu.
53The following environment variables can override the system wide locale and
54determine how a process (handled by libc) performs unicode text processing for
55different locale aspects:
57```
58LC_CTYPE Defines character classification and case conversion.
59LC_COLLATE Defines collation rules.
60LC_MONETARY Defines the format and symbols used in formatting of monetary information.
61LC_NUMERIC Defines the decimal delimiter, grouping, and grouping symbol for non-monetary numeric editing.
62LC_TIME Defines the format and content of date and time information.
63LC_MESSAGES Defines the format and values of affirmative and negative responses.
64```
66Each environment variable above can be set to available `locale` settings. For
67example `LC_CTYPE=en_US.UTF-8` (locale.charmap) specifies a locale `en_US` for
68the language `en` (English) and the territory `US` (USA) to be used for
69character classifications (e.g. `iswalpha`) and case conversions (e.g.
70`toupper`). `UTF-8` is the charmap used by the locale.
72`C` and `POSIX` locales are the same and are used by default. glibc also
73provides a generic `i18n` locale.
75Use `locale -a` command to see all available locales on a system and `locale
76-m` for charmaps. See `man locale` as well. Also see
77`/usr/share/i18n/SUPPORTED` on Linux (Debian).
78https://www.gnu.org/software/libc/manual/html_node/Locale-Names.html for the
79format in which these environment variables can be specified.
81The environment variables are preferred in the following order:
83```
84LC_ALL Overrides everything else
85LC_* Individual aspect customization
86LANG Used as a substitute for any unset LC_* variable
87```
89Normally only `LANG` should be used. If a particular aspect needs to be
90cutsomized then `LC_*` variables can be used. `LC_ALL` overrides everything.
91On GNU/Linux `LANGUAGE` can also be used as a list of preferred languages
92separated by ":". LANGUAGE is effective only if LANG has been set to something
93other than default and has higher priority than anything else.
95* https://www.gnu.org/software/gettext/manual/gettext.html#Users has a good
96 overview of locale settings.
97* https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html
98 POSIX Locales standard
100### Localizing Messages
102GNU https://www.gnu.org/software/gettext/manual/gettext.html can be used by
103programs to translate/localize the user interfacing text to multiple languages.
104Catalogs of localized program error, alert, notification messages are installed
105at e.g. `/usr/share/locale/en_US/LC_MESSAGES/`.
107* https://www.gnu.org/software/gettext/manual/gettext.html
109### Creating Locales
111From the `debian` manpage of `localedef` POSIX command:
113The localedef program reads the indicated charmap and input files,
114compiles them to a binary form quickly usable by the locale functions in the C
115library (setlocale(3), localeconv(3), etc.), and places the output in
116outputpath.
118See `locale-gen` for a more high level program to generate locales.
119/usr/lib/locale/locale-archive contains the generated binary files. On Debian
120you can use `sudo dpkg-reconfigure locales` to select locales and set system
121locale.
123On Linux/glibc (Debian), installed `charmaps` can be found at
124`/usr/share/i18n/charmaps/` and locale definition input files at
125`/usr/share/i18n/locales/`.
127## ICU Locales
129An ICU locale is frequently confused with a POSIX locale ID. An ICU locale ID
130is not a POSIX locale ID. ICU locales do not specify the encoding and specify
131variant locales differently.
133* http://userguide.icu-project.org/locale
135### Localizing Messages
137* http://userguide.icu-project.org/locale/localizing
139## Unicode Text processing
141* http://userguide.icu-project.org/posix
142* http://userguide.icu-project.org/strings/properties
144### Locale Independent
1461) Char Properties
1472) Normalization
1483) Regex matching
150### Locale Specific
1521) Case Mapping:
153 * https://unicode.org/faq/casemap_charprop.html
154 * http://www.unicode.org/versions/latest/ch05.pdf#G21180
155 * ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt
156 * ftp://ftp.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt
1572) Breaking
1583) Collation
1594) Charset Conversion
161## Haskell
163### Haskell Unicode Text Processing
165Related packages on hackage:
167* [base](https://www.stackage.org/lts/package/base) Data.Char module, uses libc
168* [text](https://www.stackage.org/lts/package/text)
169* [text-icu](https://stackage.org/lts/package/text-icu) C bindings to icu
172* https://github.com/composewell/unicode-transforms
173* https://github.com/llelf/prose
174* [unicode-properties](https://hackage.haskell.org/package/unicode-properties) Unicode 3.2.0 character properties
175* [hxt-charproperties](http://www.stackage.org/lts/package/hxt-charproperties) Character properties and classes for XML and Unicode
176* [unicode-names](http://hackage.haskell.org/package/unicode-names) Unicode 3.2.0 character names
177* [unicode](https://hackage.haskell.org/package/unicode) Construct and transform unicode characters
180* [charset](https://www.stackage.org/lts/package/charset) Fast unicode character sets
182None of the existing Haskell packages provide comprehensive and fast access to
183properties. `unicode-transforms` provides composition/decomposition data and
184the script to extract data from unicode database.
186### Haskell Localization
188* http://wiki.haskell.org/Internationalization_of_Haskell_programs
189* https://hackage.haskell.org/package/localize
190* http://hackage.haskell.org/package/localization
191* http://hackage.haskell.org/package/hgettext
192* http://hackage.haskell.org/package/i18n
193* https://hackage.haskell.org/package/setlocale
194* http://hackage.haskell.org/package/system-locale
195* https://github.com/llelf/numerals
197## TODO
199Factor out a `unicode-data` package from `unicode-transforms`.
200`unicode-transforms` package will depend on unicode-data and can continue to be
201used as is. Other packages can take advantage of the `unicode-data` to provide
202unicode text processing services.
204To begin with, this package will contain:
206* char properties data
207* case mapping data
208* unicode normalization data
210This package can be used in `Streamly.Internal.Data.Unicode.*` to provide:
212* Fast access to char properties, we will no longer depend on libc for that and
213 no FFI will be required to do iswspace, iswalpha etc.
214* Correct case mappings from a single char to multi-char
215* Stream based unicode normalization
216* Breaking (locale independent for now)
218## Later
220* add locale data from CLDR to `unicode-data` to provide locale specific
221 services as well.
222* support locale specific breaking, regex, collation and charset conversion
223* facility to add customized locale data
224* Support providing application level resource data for localization