google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2025-01-06 09:19:12 +03:00
Code Issues Packages Projects Releases Wiki Activity

Removed codes where Zero Width Joiner replaced with whitespace.

Browse Source
This commit is contained in:
Sarubi 2021-02-23 20:47:25 +05:30
parent ba7e11a17f
commit c970dedd8f
3 changed files with 2 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
data/nmt_nfkc.tsv
View File

@ -57263,8 +57263,7 @@ FB9 F90 FB5 # ྐྵ => ྐྵ
200A 20 #   =>
200B 20 # ​ =>
200C 20 # ‌ =>
200D 20 # ‍ =>
200E 20 # ‎ =>
200E 20 # ‎ =>
200F 20 # ‏ =>
2011 2010 # ‑ => ‐
2017 20 333 # ‗ => ̳

Can't render this file because it is too large.

3
data/nmt_nfkc_cf.tsv
View File

@ -57980,8 +57980,7 @@ FB9 F90 FB5 # ྐྵ => ྐྵ
200A 20 #   =>
200B 20 # ​ =>
200C 20 # ‌ =>
200D 20 # ‍ =>
200E 20 # ‎ =>
200E 20 # ‎ =>
200F 20 # ‏ =>
2011 2010 # ‑ => ‐
2017 20 333 # ‗ => ̳

Can't render this file because it is too large.

1
src/builder.cc
View File

@ -368,7 +368,6 @@ util::Status Builder::BuildNmtNFKCMap(CharsMap *chars_map) {
nfkc_map[{0xFEFF}] = {0x20}; // ZERO WIDTH NO-BREAK
nfkc_map[{0xFFFD}] = {0x20}; // REPLACEMENT CHARACTER
nfkc_map[{0x200C}] = {0x20}; // ZERO WIDTH NON-JOINER
nfkc_map[{0x200D}] = {0x20}; // ZERO WIDTH JOINER
// Ascii Control characters
nfkc_map[{0x0001}] = {};