mirror of
https://github.com/google/sentencepiece.git
synced 2024-09-11 10:55:42 +03:00
Fix SIL symbol in code snippet: _ -> _
This commit is contained in:
parent
0b5bd12205
commit
d796421cba
@ -84,7 +84,7 @@ Then, this text is segmented into small pieces, for example:
|
||||
Since the whitespace is preserved in the segmented text, we can detokenize the text without any ambiguities.
|
||||
|
||||
```
|
||||
detokenized = ''.join(pieces).replace('_', ' ')
|
||||
detokenized = ''.join(pieces).replace('▁', ' ')
|
||||
```
|
||||
|
||||
This feature makes it possible to perform detokenization without relying on language-specific resources.
|
||||
|
Loading…
Reference in New Issue
Block a user