moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-08-16 23:10:31 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #215 from moses-smt/alvations-patch-normalization

Browse Source 
Single quotes should be escaped as single quotes.
This commit is contained in:
Hieu Hoang 2019-11-24 18:38:05 -08:00 committed by GitHub
commit 41b31167fd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
scripts/tokenizer/normalize-punctuation.perl
View File

@ -42,8 +42,8 @@ while(<STDIN>) {
s/´/\'/g;
s/([a-z])‘([a-z])/$1\'$2/gi;
s/([a-z])’([a-z])/$1\'$2/gi;
s/‘/\"/g;
s/‚/\"/g;
s/‘/\'/g;
s/‚/\'/g;
s/’/\"/g;
s/''/\"/g;
s/´´/\"/g;