phikoehn
79a2c98ff7
better ems support for different binarizers and reordering models
2013-08-25 20:30:37 +01:00
Jeremy Gwinnup
a5fb4d1550
Fixed bug in tokenizer.perl where comma separated lists of single
...
characters aren't handled correctly
input> A,B,C,D,E,F
yielded> A, B,C , D,E , F
now yields> A, B, C, D, E, F
Updated Russian nonbreaking prefixes list with capital letters
2013-08-16 14:39:50 -04:00
phikoehn
b368085609
xml constraint
2013-08-15 11:46:45 +01:00
Hieu Hoang
249aefc0c3
Merge github.com:moses-smt/mosesdecoder into hieu_opt_input
2013-08-12 13:04:29 +01:00
Hieu Hoang
02c7af3fb8
Mira changes. Manually applied Eva's patch
2013-08-12 13:03:26 +01:00
Hieu Hoang
ac50d9f349
Merge /home/hieu/workspace/github/mosesdecoder into hieu_opt_input
2013-08-01 16:55:46 +01:00
Hieu Hoang
f234aa203f
number recognizer treats each word as atomic, replace all of the word or nothing at all. Recognizer is designed to be run after the text has been tokenized, not before
2013-08-01 16:55:11 +01:00
Hieu Hoang
bb1f49e10c
Merge /home/hieu/workspace/github/mosesdecoder into hieu_opt_input
2013-07-31 14:29:18 +01:00
Rico Sennrich
d0e2c43011
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2013-07-30 17:18:32 +02:00
Rico Sennrich
29cde2a204
allow overriding table filtering in config (required for multimodelcounts)
2013-07-30 16:46:23 +02:00
Hieu Hoang
2792291849
change default for PhrasePenalty to match what is was before
2013-07-30 15:44:30 +01:00
Hieu Hoang
b05a443f36
correct arguments to substitute-filtered-tables-and-weights.perl
2013-07-30 11:14:17 +01:00
Hieu Hoang
66349474d4
correct arguments to substitute-filtered-tables-and-weights.perl
2013-07-30 11:12:49 +01:00
Hieu Hoang
52ff87a6ce
phrase penalty now has it's own ff. No longer in the phrase table
2013-07-29 17:51:20 +01:00
Hieu Hoang
9cdcf713a6
phrase penalty now has it's own ff. No longer in the phrase table
2013-07-29 12:55:44 +01:00
Hieu Hoang
2980dde163
add script to strip xml
2013-07-29 11:44:00 +01:00
Hieu Hoang
4bc7ce99ed
add script to strip xml
2013-07-29 11:27:13 +01:00
Hieu Hoang
ff4c4339bd
if using CreateOnDiskPt to binarize pt, use processLexicalTable to binarize reordering table. HACK
2013-07-28 12:00:35 +01:00
Hieu Hoang
abe90b5af7
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2013-07-27 04:19:16 +01:00
Hieu Hoang
9dab7950fa
move closing of filtered file before binarizing. Otherwise file not flushed, causes error in binarizing
2013-07-27 04:18:50 +01:00
Hieu Hoang
4ea975e108
move closing of filtered file before binarizing. Otherwise file not flushed, causes error in binarizing
2013-07-27 03:58:27 +01:00
Hieu Hoang
90651a9b22
move closing of filtered file before binarizing. Otherwise file not flushed, causes error in binarizing
2013-07-27 03:57:05 +01:00
Hieu Hoang
6be88d3f44
update filtering script to correct binarize pb models with CreateOnDiskPt
2013-07-26 15:12:15 +01:00
Hieu Hoang
efd9789577
update filtering script to correct binarize pb models with CreateOnDiskPt
2013-07-26 15:06:07 +01:00
Hieu Hoang
a3e3289b08
In corpus mode, replace number with number symbol
2013-07-25 15:54:47 +01:00
Hieu Hoang
e2c2bc59f1
beautify
2013-07-25 15:23:05 +01:00
Hieu Hoang
78381d0213
@NUM@ --> @num@. In case using recaser
2013-07-25 15:16:15 +01:00
Hieu Hoang
d0172ed5cd
create script to convert phrase-table with alignment in Moses' dead-end format to standard format
2013-07-25 12:56:20 +01:00
Hieu Hoang
018998247a
create script to convert phrase-table with alignment in Moses' dead-end format to standard format
2013-07-25 12:52:05 +01:00
Barry Haddow
f79746b3c2
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2013-07-24 20:49:59 +01:00
Hieu Hoang
6fc21a32fc
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2013-07-24 19:01:57 +01:00
Hieu Hoang
c104dee3b2
merge glue grammars, rather than writing them all to the same file. Required by Phil Williams & others when doing syntax extraction
2013-07-24 19:01:46 +01:00
Achim Ruopp
1813f9784b
Additional factoring to allow more NE recognizers; bug fixes
2013-07-24 12:44:53 -04:00
Barry Haddow
46ee1ca42d
More lattice fixes squashed by merge
2013-07-24 16:09:32 +01:00
Barry Haddow
0ce50a4c70
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2013-07-24 15:58:08 +01:00
Phil Williams
1238041f98
Add option to do Penn Treebank style tokenization
...
tokenizer.perl and detokenizer.perl now have an option called -penn
which does Penn Treebank-like tokenization (English only). This is
useful if your pipeline involves processing the corpus with tools
trained on PTB-tokenized text.
Unlike PTB, the tokenizer splits on slashes (e.g. "Monday/Tuesday"
becomes "Monday", "@/@", "Tuesday"). If using parse-de-berkeley.perl,
the option -split-slash re-joins tokens that are separated by slashes
for parsing then splits them afterwards.
2013-07-24 13:41:21 +01:00
Barry Haddow
d5e40a5b08
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2013-07-24 11:38:23 +01:00
Nadir Durrani
30544ae17e
Sample Config File
2013-07-23 12:29:23 +01:00
Nadir Durrani
61e56ecdcd
Sample Config File
2013-07-23 12:18:57 +01:00
Barry Haddow
50de0e06d1
Generate correct ini file for lattices
2013-07-23 11:46:37 +01:00
Barry Haddow
8ed8bcafc2
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2013-07-23 11:21:47 +01:00
Barry Haddow
887d5dad62
Restore EMS lattice fixes, squashed by merge.
2013-07-23 10:38:11 +01:00
Phil Williams
91cc7c329e
parse-de-berkeley.perl: escape @ characters in input
2013-07-23 10:22:56 +01:00
Barry Haddow
ecc6c7177c
Reinstate lattice fixes squashed by merge
2013-07-22 17:25:01 +01:00
unknown
54eb50523b
Converted into modulino; added support for French numbers
2013-07-19 14:41:01 -04:00
Hieu Hoang
17366b9d7e
add mada wrapper from Phi's collection
2013-07-13 22:42:28 +01:00
Nadir Durrani
418abf42fa
Merge branch 'nadir_osm'
2013-07-09 11:44:14 +01:00
Hieu Hoang
2203bb3284
beautify
2013-07-08 19:08:31 +01:00
Nadir Durrani
deae3ac7b9
OSM entries in train-model.perl, experiment.*
2013-07-07 13:05:09 +01:00
Nadir Durrani
d2bc6a2584
In EMS
2013-07-04 19:58:19 +01:00
