moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-11 11:25:40 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,298 Commits 136 Branches 26 Tags 147 MiB
b1d9fb6d75
Commit Graph

2322 Commits

Author SHA1 Message Date
Barry Haddow
b1d9fb6d75 full cjk test 2019-10-28 09:53:45 +00:00
Barry Haddow
8ebebbc680 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2019-10-28 09:48:40 +00:00
Kevin Canwen Xu
5d3331b922
Update replace-unicode-punctuation.perl 2019-10-14 16:33:58 +08:00
alvations
555829a771
Undoing 0578892581 
Causes abbreviations to not split when ending with a fullstop. E.g. 

> The restructuring of IBM was essential to enable it organisationally to take up the responsibilities entrusted in the role with the recent changes in the policy and legislations, revised charter of function of IBM and the new activities and initiatives undertaken by IBM. IBM is also engaged in handholding the States for auction of mineral blocks for greater transparency in allocation of mineral concessions.
 2019-10-01 05:27:06 +08:00
Barry Haddow
486dce270f debug 2019-09-30 16:58:21 +01:00
Barry Haddow
9bffde57ba revert 05788925 2019-09-30 16:53:06 +01:00
Barry Haddow
257d7e5e66 enable custom non breaking prefixes 2019-09-30 16:52:24 +01:00
Barry Haddow
01a8ec41e8 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2019-09-30 15:33:33 +01:00
Barry Haddow
768944d851 do not add spaces in cjk 2019-09-30 15:33:26 +01:00
titsuki
490dc3996a Enable use strict pragma 2019-09-23 15:40:13 +09:00
alvations
0578892581
The dot before an acronym should be optional. 2019-09-04 14:16:41 +08:00
Achim Ruopp
7ad5ffa0c0 Support for Urdu in sentence splitter 2019-07-10 10:48:32 -04:00
Matt Post
63c450b401
escape angle brackets 
The script doesn't escape angle brackets which can result in bad SGML / XML output. This fixes that, although ideally, this should be implemented with a proper parser and dumper.
 2019-04-26 14:24:07 -04:00
Joel Barry
fdb7384d3d Fix non-ASCII lowercasing 2019-02-27 10:17:29 -05:00
Hieu Hoang
26940e714a Revert "use ucfirst instead of defined uppercase function" 
This reverts commit dfbb17e549.
 2019-01-04 14:55:55 +00:00
Hieu Hoang
7bc56b66f2
Merge pull request #207 from alvations/patch-truecaser 
Reverting split_xml()
 2019-01-03 16:56:56 +00:00
alvations
8fdbc74bbf
Reverting split_xml() 2019-01-03 20:51:27 +08:00
Hieu Hoang
db1894ad24 consistent output 2018-12-30 12:05:57 +00:00
alvations
dfbb17e549 use ucfirst instead of defined uppercase function 2018-12-20 11:57:48 +08:00
alvations
40748e528d split_xml should be consistent for training and using 2018-12-20 11:53:02 +08:00
Hieu Hoang
413ba6b583 increase cores to 16. For bitextor azure pipeline 2018-12-10 16:17:16 +00:00
Hieu Hoang
c753350641 ems config for moses2 2018-12-08 19:47:10 +00:00
Hieu Hoang
3d4bf99367 sacre bleu 2018-12-04 15:40:00 +00:00
Hieu Hoang
dbbc47292f sacre bleu 2018-12-04 15:27:09 +00:00
Hieu Hoang
345dabcde6 use --discount_fallback 2018-12-04 14:34:47 +00:00
Hieu Hoang
1591cf3676 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2018-11-12 14:03:54 +00:00
Hieu Hoang
13e48bc8b4 removing python port. Sacremoses is newer 2018-11-12 14:03:38 +00:00
Loïc Vial
4133726ef9 Add option "-b" (unbuffer output) to tokenizer scripts 2018-11-09 22:53:33 +01:00
Hieu Hoang
a2315ffd3a rename directory to work with python import 2018-11-09 13:01:17 +00:00
Hieu Hoang
a70086c1e6 python wrapper works 2018-11-09 12:58:22 +00:00
Hieu Hoang
2451c46960 start borging Luis Gomes code 2018-11-07 17:12:05 +00:00
Ozan Caglayan
9fc964da7f tokenizer.perl: split final dots unconditionally 
Allow tokenization of non-breaking prefixes at end of sentences. This should
be a fair compromise in many cases to construct a cleaner vocabulary.

EN-old: So am I.
EN-new: So am I .

DE-old: ... schwer wie ein iPhone 5.
DE-new: ... schwer wie ein iPhone 5 .

FR-old: Des gens admirent une œuvre d' art.
FR-new: Des gens admirent une œuvre d' art .

CS-old: Dvě děti, které běží bez bot.
CS-new: Dvě děti, které běží bez bot .
 2018-11-07 10:59:54 +01:00
Barry Haddow
d2b558728f basic support for Gujarati and Hindi, backported from one of the many upstreams 2018-10-30 14:16:16 +00:00
Rico Sennrich
411f45f249 multi-bleu-detok should take raw reference 2018-09-26 12:24:07 +01:00
Hieu Hoang
48fa6e92a9 grammar 2018-09-16 14:58:39 +01:00
Hieu Hoang
fd1758ba74 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2018-09-10 18:30:46 +01:00
Hieu Hoang
e760db2d17 unused script 2018-09-10 18:30:36 +01:00
Barry Haddow
06f519d4e2 Handle glottal stops in Somalian 2018-09-06 16:09:36 +01:00
Louis MARTIN
53da5f4dbe Fix multi-bleu.perl bug when file does not end with newline 
When reading hypothesis and reference files, multi-bleu.perl uses the
chop function to remove the trailing newline character.
If one of these files happens to not end with a newline, then chop will
remove the last character of the last line (instead of the newline).
This causes the BLEU score to be slightly off from its theoretical
value.
Using the safest chomp function solves this problem, i.e. it only
removes newlines when present.
 2018-07-03 04:06:09 -06:00
Joachim Wagner
5bbd5ca160
fix syntax error; credit https://www.mail-archive.com/moses-support@mit.edu/msg15226.html 2018-06-23 08:19:36 +01:00
Joachim Wagner
2aa5cd2152
fix syntax error in regular expression 2018-06-22 18:16:11 +01:00
Tomas Fulajtar
3a2a63b9dc * Added missing step for the "TRAINING:build-generation-custom". 
* Fixed the $cmd parameter - should be "-corpus" instead of "-generation-corpus".
 2018-05-18 14:18:11 +02:00
Hieu Hoang
999e83d128
Merge pull request #196 from astronautguo/master 
fix bug when copying to cache
 2018-05-04 14:42:35 +01:00
Kenneth Heafield
ae47469919 Don't drop last character if file does not end with newline 2018-05-03 10:28:11 +01:00
astro
f47e670f20 fix bug when copying to cache 2018-04-27 19:52:20 -04:00
alvations
686034488a
Contributing MosesTokenizer from NLTK to Moses 2018-04-11 00:27:37 +08:00
Scherrer Yves
4a7f16b366 add fi/sv-specific colon handling in tokenizer.perl 2018-02-14 10:27:46 +02:00
alvations
194964c017
Korean words has spaces =) 2018-01-19 13:29:53 +08:00
Hieu Hoang
3a0631a05b better default 2017-12-12 15:30:56 +00:00
Tomas Fulajtar
5b9a6da9a4 The .gz extension should be also added for 'On Disk' and 'Probing' Phrase tables. 2017-11-28 10:29:58 +01:00