osm-search/Nominatim
1
1
Fork 0
mirror of https://github.com/osm-search/Nominatim.git synced 2024-11-22 21:28:10 +03:00
Code Issues Packages Projects Releases Wiki Activity
de828b723e
Nominatim/settings/icu_tokenizer.yaml

203 lines
5.2 KiB
YAML
Raw Normal View History

move abbreviation computation into import phase This adds precomputation of abbreviated terms for names and removes abbreviation of terms in the query. Basic import works but still needs some thorough testing as well as speed improvements during import. New dependency for python library datrie.
2021-05-28 23:06:13 +03:00
normalization:
- ":: lower ()"
icu: normalise simplified to traditional chinese The conversion is unambigious in most cases, so that the information loss is minimal.
2021-08-31 12:18:34 +03:00
- ":: Hans-Hant"
improve normalization Make sure all special symbols are removed during normalization already. Those won't be interpreted in any way because they are unlikely to be searched for.
2021-06-26 20:38:08 +03:00
- !include icu-rules/unicode-digits-to-decimal.yaml
- "'№' > 'no'"
- "'n°' > 'no'"
- "'nº' > 'no'"
- "ª > a"
- "º > o"
consider "modifier letter apostrophe" to be punctuation While technically being a letter, the apostrophe is often replaced with a normal apostrophe in writing which is a punctuation mark. This makes sure that the modifier letter apostrophe yields the same normalization results and thus is really interchangable. Only has an effect after the next reimport. Fixes #2569.
2022-01-10 19:40:03 +03:00
- "[[:Punctuation:][:Symbol:]\u02bc] > ' '"
move abbreviation computation into import phase This adds precomputation of abbreviated terms for names and removes abbreviation of terms in the query. Basic import works but still needs some thorough testing as well as speed improvements during import. New dependency for python library datrie.
2021-05-28 23:06:13 +03:00
- "ß > 'ss'" # German szet is unimbigiously equal to double ss
improve normalization Make sure all special symbols are removed during normalization already. Those won't be interpreted in any way because they are unlikely to be searched for.
2021-06-26 20:38:08 +03:00
- "[^[:Letter:] [:Number:] [:Space:]] >"
- "[:Lm:] >"
- ":: [[:Number:]] Latin ()"
- ":: [[:Number:]] Ascii ();"
- ":: [[:Number:]] NFD ();"
- "[[:Nonspacing Mark:] [:Cf:]] >;"
- "[:Space:]+ > ' '"
use yaml tag syntax to mark include files
2021-06-21 00:45:33 +03:00
transliteration:
improve normalization Make sure all special symbols are removed during normalization already. Those won't be interpreted in any way because they are unlikely to be searched for.
2021-06-26 20:38:08 +03:00
- ":: Latin ()"
use yaml tag syntax to mark include files
2021-06-21 00:45:33 +03:00
- !include icu-rules/extended-unicode-to-asccii.yaml
- ":: Ascii ()"
- ":: NFD ()"
- ":: lower ()"
avoid special characters in word tokens Transliteration should only consist of ASCII letters and numbers. Avoid any other characters.
2021-11-10 19:14:13 +03:00
- "[^a-z0-9[:Space:]] >"
use yaml tag syntax to mark include files
2021-06-21 00:45:33 +03:00
- ":: NFC ()"
introduce sanitizer step before token analysis Sanatizer functions allow to transform name and address tags before they are handed to the tokenizer. Theses transformations are visible only for the tokenizer and thus only have an influence on the search terms and address match terms for a place. Currently two sanitizers are implemented which are responsible for splitting names with multiple values and removing bracket additions. Both was previously hard-coded in the tokenizer.
2021-09-30 22:30:13 +03:00
sanitizers:
factor out housenumber splitting into sanitizer
2022-01-08 00:41:09 +03:00
- step: clean-housenumbers
complete documentation for new clean-houseunubmers sanatizer
2022-01-20 17:49:32 +03:00
filter-kind:
- housenumber
- conscriptionnumber
- streetnumber
sanitizer: move helpers into a configuration class
2022-02-07 12:48:00 +03:00
convert-to-name:
implement is-a-name option for housenumbers
2022-02-05 17:13:45 +03:00
- (\A|.*,)[^\d,]{3,}(,.*|\Z)
- step: split-name-list
- step: strip-brace-terms
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- step: tag-analyzer-by-language
filter-kind: [".*name.*"]
whitelist: [bg,ca,cs,da,de,el,en,es,et,eu,fi,fr,gl,hu,it,ja,mg,ms,nl,no,pl,pt,ro,ru,sk,sl,sv,tr,uk,vi]
use-defaults: all
mode: append
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
token-analysis:
make token analyzers configurable modules Adds a mandatory section 'analyzer' to the token-analysis entries which define, which analyser to use. Currently there is exactly one, generic, which implements the former ICUNameProcessor.
2021-10-04 18:34:30 +03:00
- analyzer: generic
add new analyser for houenumbers This analyser makes spaces optional.
2022-02-16 19:18:23 +03:00
- id: "@housenumber"
analyzer: housenumbers
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: bg
analyzer: generic
mode: variant-only
make token analyzers configurable modules Adds a mandatory section 'analyzer' to the token-analysis entries which define, which analyser to use. Currently there is exactly one, generic, which implements the former ICUNameProcessor.
2021-10-04 18:34:30 +03:00
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-bg.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: ca
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-ca.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: cs
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-cs.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: da
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-da.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: de
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-de.yaml
introduce mutation variants to generic token analyser Mutations are regular-expression-based replacements that are applied after variants have been computed. They are meant to be used for variations on character level. Add spelling variations for German umlauts.
2022-01-12 18:25:47 +03:00
mutations:
- pattern: ä
replacements: ["ä", "ae"]
- pattern: ö
replacements: ["ö", "oe"]
- pattern: ü
replacements: ["ü", "ue"]
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: el
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-el.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: en
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-en.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: es
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-es.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: et
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-et.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: eu
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-eu.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: fi
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-fi.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: fr
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-fr.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: gl
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-gl.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: hu
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-hu.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: it
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-it.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: ja
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-ja.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: mg
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-mg.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: ms
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-ms.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: nl
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-nl.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: no
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-no.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: pl
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-pl.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: pt
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-pt.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: ro
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-ro.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: ru
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-ru.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: sk
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-sk.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: sl
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-sl.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: sv
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-sv.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: tr
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-tr.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: uk
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-uk.yaml
apply variants by languages Adds a tagger for names by language so that the analyzer of that language is used. Thus variants are now only applied to names in the specific language and only tag name tags, no longer to reference-like tags.
2021-10-05 18:18:10 +03:00
- id: vi
analyzer: generic
mode: variant-only
variants:
extend ICU config to accomodate multiple analysers Adds parsing of multiple variant lists from the configuration. Every entry except one must have a unique 'id' paramter to distinguish the entries. The entry without id is considered the default. Currently only the list without an id is used for analysis.
2021-10-04 17:40:28 +03:00
- !include icu-rules/variants-vi.yaml
Reference in New Issue Copy Permalink