Helsinki-NLP/OPUS-MT-train
1
1
Fork 0
mirror of https://github.com/Helsinki-NLP/OPUS-MT-train.git synced 2024-10-26 21:19:02 +03:00
Code Issues Packages Projects Releases Wiki Activity
d1ffe9106c
OPUS-MT-train/tatoeba/SentencePieceModels.md
Joerg Tiedemann 4f5bc5780f example added for using sp-models
2022-07-12 23:08:00 +03:00

459 KiB
Raw Blame History

Tatoeba-MT Sentence Piece Models

import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file='opusTC.eng.16k.spm')
print(sp.encode(['Hello world', 'This is a tokenization-test'], out_type=str))