Nominatim/settings/icu_tokenizer.yaml

normalization:
    - ":: lower ()"
    - ":: Hans-Hant"
    - !include icu-rules/unicode-digits-to-decimal.yaml
    - "'№' > 'no'"
    - "'n°' > 'no'"
    - "'nº' > 'no'"
    - "ª > a"
    - "º > o"
    - "[[:Punctuation:][:Symbol:]]  > ' '"
    - "ß > 'ss'" # German szet is unimbigiously equal to double ss
    - "[^[:Letter:] [:Number:] [:Space:]] >"
    - "[:Lm:] >"
    - ":: [[:Number:]] Latin ()"
    - ":: [[:Number:]] Ascii ();"
    - ":: [[:Number:]] NFD ();"
    - "[[:Nonspacing Mark:] [:Cf:]] >;"
    - "[:Space:]+ > ' '"
transliteration:
    - ":: Latin ()"
    - !include icu-rules/extended-unicode-to-asccii.yaml
    - ":: Ascii ()"
    - ":: NFD ()"
    - "[^[:Ascii:]] >"
    - ":: lower ()"
    - ":: NFC ()"
variants:
    - !include icu-rules/variants-bg.yaml
    - !include icu-rules/variants-ca.yaml
    - !include icu-rules/variants-cs.yaml
    - !include icu-rules/variants-da.yaml
    - !include icu-rules/variants-de.yaml
    - !include icu-rules/variants-el.yaml
    - !include icu-rules/variants-en.yaml
    - !include icu-rules/variants-es.yaml
    - !include icu-rules/variants-et.yaml
    - !include icu-rules/variants-eu.yaml
    - !include icu-rules/variants-fi.yaml
    - !include icu-rules/variants-fr.yaml
    - !include icu-rules/variants-gl.yaml
    - !include icu-rules/variants-hu.yaml
    - !include icu-rules/variants-it.yaml
    - !include icu-rules/variants-ja.yaml
    - !include icu-rules/variants-mg.yaml
    - !include icu-rules/variants-ms.yaml
    - !include icu-rules/variants-nl.yaml
    - !include icu-rules/variants-no.yaml
    - !include icu-rules/variants-pl.yaml
    - !include icu-rules/variants-pt.yaml
    - !include icu-rules/variants-ro.yaml
    - !include icu-rules/variants-ru.yaml
    - !include icu-rules/variants-sk.yaml
    - !include icu-rules/variants-sl.yaml
    - !include icu-rules/variants-sv.yaml
    - !include icu-rules/variants-tr.yaml
    - !include icu-rules/variants-uk.yaml
    - !include icu-rules/variants-vi.yaml
move abbreviation computation into import phase This adds precomputation of abbreviated terms for names and removes abbreviation of terms in the query. Basic import works but still needs some thorough testing as well as speed improvements during import. New dependency for python library datrie. 2021-05-28 23:06:13 +03:00			`normalization:`
			`- ":: lower ()"`
icu: normalise simplified to traditional chinese The conversion is unambigious in most cases, so that the information loss is minimal. 2021-08-31 12:18:34 +03:00			`- ":: Hans-Hant"`
improve normalization Make sure all special symbols are removed during normalization already. Those won't be interpreted in any way because they are unlikely to be searched for. 2021-06-26 20:38:08 +03:00			`- !include icu-rules/unicode-digits-to-decimal.yaml`
			`- "'№' > 'no'"`
			`- "'n°' > 'no'"`
			`- "'nº' > 'no'"`
			`- "ª > a"`
			`- "º > o"`
			`- "[[:Punctuation:][:Symbol:]] > ' '"`
move abbreviation computation into import phase This adds precomputation of abbreviated terms for names and removes abbreviation of terms in the query. Basic import works but still needs some thorough testing as well as speed improvements during import. New dependency for python library datrie. 2021-05-28 23:06:13 +03:00			`- "ß > 'ss'" # German szet is unimbigiously equal to double ss`
improve normalization Make sure all special symbols are removed during normalization already. Those won't be interpreted in any way because they are unlikely to be searched for. 2021-06-26 20:38:08 +03:00			`- "[^[:Letter:] [:Number:] [:Space:]] >"`
			`- "[:Lm:] >"`
			`- ":: [[:Number:]] Latin ()"`
			`- ":: [[:Number:]] Ascii ();"`
			`- ":: [[:Number:]] NFD ();"`
			`- "[[:Nonspacing Mark:] [:Cf:]] >;"`
			`- "[:Space:]+ > ' '"`
use yaml tag syntax to mark include files 2021-06-21 00:45:33 +03:00			`transliteration:`
improve normalization Make sure all special symbols are removed during normalization already. Those won't be interpreted in any way because they are unlikely to be searched for. 2021-06-26 20:38:08 +03:00			`- ":: Latin ()"`
use yaml tag syntax to mark include files 2021-06-21 00:45:33 +03:00			`- !include icu-rules/extended-unicode-to-asccii.yaml`
			`- ":: Ascii ()"`
			`- ":: NFD ()"`
			`- "[^[:Ascii:]] >"`
			`- ":: lower ()"`
			`- ":: NFC ()"`
switch to a more flexible variant description format The new format combines compound splitting and abbreviation. It also allows to restrict rules to additional conditions (like language or region). This latter ability is not used yet. 2021-06-24 21:02:07 +03:00			`variants:`
import abbreviations from OSM Wiki Replaces the variant rules with a slightly cleaned-up version of the abbreviation lists at https://wiki.openstreetmap.org/wiki/Name_finder:Abbreviations 2021-06-30 22:37:29 +03:00			`- !include icu-rules/variants-bg.yaml`
			`- !include icu-rules/variants-ca.yaml`
			`- !include icu-rules/variants-cs.yaml`
			`- !include icu-rules/variants-da.yaml`
			`- !include icu-rules/variants-de.yaml`
			`- !include icu-rules/variants-el.yaml`
			`- !include icu-rules/variants-en.yaml`
			`- !include icu-rules/variants-es.yaml`
			`- !include icu-rules/variants-et.yaml`
			`- !include icu-rules/variants-eu.yaml`
			`- !include icu-rules/variants-fi.yaml`
			`- !include icu-rules/variants-fr.yaml`
			`- !include icu-rules/variants-gl.yaml`
			`- !include icu-rules/variants-hu.yaml`
			`- !include icu-rules/variants-it.yaml`
			`- !include icu-rules/variants-ja.yaml`
			`- !include icu-rules/variants-mg.yaml`
			`- !include icu-rules/variants-ms.yaml`
			`- !include icu-rules/variants-nl.yaml`
			`- !include icu-rules/variants-no.yaml`
			`- !include icu-rules/variants-pl.yaml`
			`- !include icu-rules/variants-pt.yaml`
			`- !include icu-rules/variants-ro.yaml`
			`- !include icu-rules/variants-ru.yaml`
			`- !include icu-rules/variants-sk.yaml`
			`- !include icu-rules/variants-sl.yaml`
			`- !include icu-rules/variants-sv.yaml`
			`- !include icu-rules/variants-tr.yaml`
			`- !include icu-rules/variants-uk.yaml`
			`- !include icu-rules/variants-vi.yaml`