mirror of
https://github.com/google/sentencepiece.git
synced 2024-09-11 10:55:42 +03:00
A minor README tweak.
This commit is contained in:
parent
93b6f93bfa
commit
a4d5f42eb2
@ -33,7 +33,7 @@ vocabulary. Unlike most unsupervised word segmentation algorithms, which
|
||||
assume an infinite vocabulary, SentencePiece trains the segmentation model such
|
||||
that the final vocabulary size is fixed, e.g., 8k, 16k, or 32k.
|
||||
|
||||
#### Whitespace is considered as as a basic symbol
|
||||
#### Whitespace is treated as a basic symbol
|
||||
The first step of Natural Language processing is text tokenization. For
|
||||
example, a standard English tokenizer would segment the text "Hello world." into the
|
||||
following three tokens.
|
||||
|
Loading…
Reference in New Issue
Block a user