mirror of
https://github.com/moses-smt/mosesdecoder.git
synced 2024-09-11 11:25:40 +03:00
Separate comma after a number end sentence
Separate "," after a number if it's the end of a sentence. Example: He is tall, He was born in 1800, He wants to go there in 2000. He is tall , He was born in 1800 , He wants to go there in 2000 .
This commit is contained in:
parent
76469df5aa
commit
d04bdc7440
@ -284,6 +284,9 @@ sub tokenize
|
||||
# will also space digit,letter or letter,digit forms (redundant with next section)
|
||||
$text =~ s/([^\p{IsN}])[,]/$1 , /g;
|
||||
$text =~ s/[,]([^\p{IsN}])/ , $1/g;
|
||||
|
||||
# separate "," after a number if it's the end of a sentence
|
||||
$text =~ s/([\p{IsN}])[,]$/$1 ,/g;
|
||||
|
||||
# separate , pre and post number
|
||||
#$text =~ s/([\p{IsN}])[,]([^\p{IsN}])/$1 , $2/g;
|
||||
|
Loading…
Reference in New Issue
Block a user