moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-08-16 23:10:31 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #144 from jimregan/patch-2

Browse Source 
.' at end of sentence is missed
This commit is contained in:
Hieu Hoang 2016-03-22 08:07:17 +00:00
commit 000de1c1dd
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
scripts/tokenizer/tokenizer.perl
View File

@ -348,6 +348,9 @@ sub tokenize
$text =~ s/^ //g;
$text =~ s/ $//g;
# .' at end of sentence is missed
$text =~ s/\.\' ?$/ . ' /;
# restore protected
for (my $i = 0; $i < scalar(@protected); ++$i) {
my $subst = sprintf("THISISPROTECTED%.3d", $i);