google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2024-09-11 10:55:42 +03:00
Code Issues Packages Projects Releases Wiki Activity

Fix SIL symbol in code snippet: _ -> _

Browse Source
This commit is contained in:
Dmitrii Kravchenko 2020-10-15 14:51:35 +03:00
parent 0b5bd12205
commit d796421cba
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -84,7 +84,7 @@ Then, this text is segmented into small pieces, for example:
Since the whitespace is preserved in the segmented text, we can detokenize the text without any ambiguities.
```
detokenized = ''.join(pieces).replace('_', ' ')
detokenized = ''.join(pieces).replace('▁', ' ')
```
This feature makes it possible to perform detokenization without relying on language-specific resources.