hieuhoang1972
7fa74c1eb2
roll back kenlm tests. Binarizy files are OS-dependent
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4287 1f5c12ca-751b-0410-a591-d2e778427230
2011-10-01 12:22:24 +00:00
hieuhoang1972
e1c808ad9a
roll back kenlm tests. Binarizy files are OS-dependent
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4286 1f5c12ca-751b-0410-a591-d2e778427230
2011-10-01 11:33:19 +00:00
hieuhoang1972
6faf20707c
roll back kenlm tests. Binarizy files are OS-dependent
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4285 1f5c12ca-751b-0410-a591-d2e778427230
2011-10-01 11:09:19 +00:00
hieuhoang1972
f51239cf68
kenlm regression tests
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4284 1f5c12ca-751b-0410-a591-d2e778427230
2011-09-30 14:12:18 +00:00
hieuhoang1972
358720c24a
don't score with good-turing. phi has moved this calc to another file
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4220 1f5c12ca-751b-0410-a591-d2e778427230
2011-09-13 18:14:14 +00:00
bhaddow
d53350dd95
Regression test for lm oov feature
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4218 1f5c12ca-751b-0410-a591-d2e778427230
2011-09-12 20:53:22 +00:00
bhaddow
8d558d2e04
Fix test script
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4194 1f5c12ca-751b-0410-a591-d2e778427230
2011-09-08 18:03:30 +00:00
bhaddow
ed8d85acc1
moving mert tests
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4193 1f5c12ca-751b-0410-a591-d2e778427230
2011-09-08 17:51:09 +00:00
Ales Tamchyna
94fa872b6d
return '2' if a test failed
2011-09-07 11:16:44 +02:00
bojar
ca1912961d
first draft of cruise control for Moses
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4166 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-29 06:20:25 +00:00
hieuhoang1972
3763b2466b
run scorer regression test from any directory
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4155 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-22 07:56:46 +00:00
hieuhoang1972
1873030d24
forgot to add these files for regression tests
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4154 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-22 07:40:14 +00:00
bgottesman
436a285f18
stop using 'subtest' because it doesn't work for everyone, e.g. Hieu reports it doesn't work on a Mac even with an up-to-date Test::Simple module
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4144 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-16 10:51:43 +00:00
hieuhoang1972
87216f55be
rename & make executable
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4136 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-09 09:07:34 +00:00
bgottesman
0fe1c629da
if we fail to make the output directory for a test, just abort the test, don't exit the whole script
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4135 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-08 18:26:57 +00:00
bgottesman
24f5bf6723
when detokenizing, remove whitespace between a pair of CJK (Chinese/Japanese/Korean) words
...
This gets the Chinese and Japanese tests working, so remove the failure expectation.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4134 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-08 15:30:54 +00:00
bgottesman
14587cdafc
fix a detokenization bug that was preventing the removal of the whitespace following a contracted French or Italian article/pronoun (e.g. "l' immigration") when the contraction was the second-last word in the segment
...
remove the expectation of failure on the corresponding unit test
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4133 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-08 15:02:56 +00:00
bgottesman
9d9977bc6f
add TODO tests for detokenization of Chinese and Japanese
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4131 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-08 13:21:05 +00:00
bgottesman
c030dae094
Allow a test case to have an undefined language, since the detokenizer doesn't require a language to be passed in and, indeed, errors if a language is passed in for which there are no special rules (which seems dubious to me ...). Add test case TEST_GERMAN_NONASCII with an undefined language.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4130 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 19:14:01 +00:00
bgottesman
024bbe0bcc
- factor out class DetokenizerTestCase
...
- create an array of all of the test cases before running any of them
- in the case of an expected failure, move the TODO block deeper, just around the validation of the results
I'm not 100% I like this change, I think it makes the code slightly more elegant but it also makes it longer.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4128 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 18:48:37 +00:00
bgottesman
d521287a3f
move commas to after here-docs, to hopefully make test cases more readable; and remove unused import
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4125 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 16:37:49 +00:00
bgottesman
76c3ef4dba
a few more detokenization tests, including a TODO one that exposes a bug
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4124 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 16:23:47 +00:00
bgottesman
eda0f4e370
An initial test suite for detokenizer.perl.
...
I realize this doesn't quite fit the paradigm if the existing moses test suite. On the other hand, it's self-contained, easy to run, easy to add tests to (just follow the pattern in the section titled 'Definitions of individual test cases'), and uses an established Perl testing framework. I don't think it will be infeasible to incorporate it into the existing test suite.
Usage:
run-test-detokenizer.t --results-dir <RESULTS-DIRECTORY>
where <RESULTS-DIRECTORY> is an empty existing directory where the output can be written
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4121 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 14:32:39 +00:00
hieuhoang1972
30ca534b86
faster scorer
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4119 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 10:27:15 +00:00
hieuhoang1972
b4c79f721e
regression test for scorer
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4118 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 10:18:07 +00:00
hieuhoang1972
b618aadf8d
regression test for scorer
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4117 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 09:23:48 +00:00
hieuhoang1972
b8a0b09206
regression test for scorer
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4116 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-05 02:48:30 +00:00
hieuhoang1972
fc176801d6
regression test for score
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4112 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-04 09:15:43 +00:00
hieuhoang1972
e988361d62
regression test for score
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4111 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-04 08:11:40 +00:00
nicolabertoldi
579d8b0760
added few regression tests explicitly working with IRSTLM; modified few regression tests wrongly working with IRSTLM/SRILM; modified the required data archive (now version is 6);
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4105 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-03 09:19:48 +00:00
hieuhoang1972
49e56f35bb
regression test for score
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4102 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-02 10:28:36 +00:00
hieuhoang1972
d45a29d9c7
data for score regression test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4101 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-02 10:10:09 +00:00
hieuhoang1972
ed4367ceb0
data for score regression test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4100 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-02 10:04:53 +00:00
hieuhoang1972
69fe991923
data for score regression test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4099 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-02 09:53:25 +00:00
hieuhoang1972
1ae8c53a08
executable perl script
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4098 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-02 09:31:01 +00:00
hieuhoang1972
acb7e984de
starting regression test for score program
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4097 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-02 09:27:59 +00:00
hieuhoang1972
e389e9fec7
default decoders if none specified
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4094 1f5c12ca-751b-0410-a591-d2e778427230
2011-07-27 10:14:33 +00:00
hieuhoang1972
6f8f1adf3b
remove unnecessary parameters from ini files
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4010 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-11 04:52:05 +00:00
hieuhoang1972
b8e517d167
remove unnecessary parameters from ini files
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4009 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-11 04:48:22 +00:00
hieuhoang1972
265b4451ad
remove unnecessary parameters from ini files
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4008 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-11 04:41:55 +00:00
hieuhoang1972
840f3915ce
add xml markup back to regression testing
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3862 1f5c12ca-751b-0410-a591-d2e778427230
2011-02-01 18:30:01 +00:00
hieuhoang1972
db404d0fc0
pass regression. Not sure why is passed before, wasn't beam threshold implemented?
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3861 1f5c12ca-751b-0410-a591-d2e778427230
2011-02-01 18:17:44 +00:00
hieuhoang1972
c6e0391b21
spans must be in consistent format start-end, not start,end
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3860 1f5c12ca-751b-0410-a591-d2e778427230
2011-02-01 15:17:13 +00:00
bhaddow
0e5fbcdb4a
Add show-weights for moses_chart
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3745 1f5c12ca-751b-0410-a591-d2e778427230
2010-11-29 17:05:16 +00:00
bhaddow
50f0e6c07d
Add a show-weights option. It prints out the moses features and exits. May
...
load tables as a side-effect.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3744 1f5c12ca-751b-0410-a591-d2e778427230
2010-11-29 16:44:28 +00:00
phkoehn
2a594c0e2a
fixed xml regression test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3646 1f5c12ca-751b-0410-a591-d2e778427230
2010-10-23 21:27:34 +00:00
hieuhoang1972
a875ef8361
regression test for ondisk
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3619 1f5c12ca-751b-0410-a591-d2e778427230
2010-10-12 15:07:43 +00:00
hieuhoang1972
2d06b12a2c
regression test for ondisk
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3618 1f5c12ca-751b-0410-a591-d2e778427230
2010-10-12 15:04:31 +00:00
hieuhoang1972
ac875ffc81
mem leak
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3617 1f5c12ca-751b-0410-a591-d2e778427230
2010-10-12 14:47:58 +00:00
hieuhoang1972
bdfa2beab8
delete eclipse build
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3422 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-10 09:16:50 +00:00
bhaddow
904133fcb7
Merge in the multiple models branch. These changes allow the moses server
...
to support multiple translation, language and generation models within the
same process. The main design change is the introduction of a TranslationSystem
object to manage the models, which have been moved out of StaticData.
The changes should have no effect on existing systems.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3394 1f5c12ca-751b-0410-a591-d2e778427230
2010-08-10 13:12:00 +00:00
rafpayen
1896cc7fff
better messages
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3383 1f5c12ca-751b-0410-a591-d2e778427230
2010-08-03 15:34:34 +00:00
hieuhoang1972
7221bf2dd4
alignment info, for chart decoding, updated regression
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3368 1f5c12ca-751b-0410-a591-d2e778427230
2010-07-28 09:53:21 +00:00
hieuhoang1972
e8ab51f594
regression test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3314 1f5c12ca-751b-0410-a591-d2e778427230
2010-06-09 13:48:44 +00:00
hieuhoang1972
65194ba505
drop unknown option for chart decoding, xcode, minor adjustments to regression test formatting
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3312 1f5c12ca-751b-0410-a591-d2e778427230
2010-06-09 12:56:31 +00:00
hieuhoang1972
f028e1d9bd
add space @ end of line after each trans
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3251 1f5c12ca-751b-0410-a591-d2e778427230
2010-05-11 23:07:15 +00:00
bojar
0118733a45
fixing traditional distortion penalty with -continue-partial-translation
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3222 1f5c12ca-751b-0410-a591-d2e778427230
2010-05-03 21:41:28 +00:00
bojar
b7a48d3962
adding a simple regression test for continue-partial-translation
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3220 1f5c12ca-751b-0410-a591-d2e778427230
2010-05-03 21:40:43 +00:00
hieuhoang1972
e9cd6d0fde
regression
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3209 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-26 18:41:53 +00:00
hieuhoang1972
b73293f3a8
regression
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3208 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-26 18:39:05 +00:00
hieuhoang1972
a8f75cb789
regression
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3207 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-26 17:48:15 +00:00
hieuhoang1972
87c3becbf6
regression
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3206 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-26 16:21:19 +00:00
hieuhoang1972
d5a0eb2423
regression
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3205 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-26 16:16:44 +00:00
hieuhoang1972
430446f404
regression
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3204 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-26 16:03:35 +00:00
chardmeier
ca353d3e68
- word lattice distortion calculation and distortion limit checks revisited
...
- crash in a controlled way if a jump in a word lattice exceeds max-phrase-length
- check if a sufficient number of ttable-limits is specified in the configuration
Thanks to Arianna Bisazza for help with sorting out lattice distortion and
to Suzy Howlett for help with debugging the other issues.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3193 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-22 08:42:32 +00:00
bhaddow
a76b7e5bce
Test for consensus decoding
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3128 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-12 17:01:51 +00:00
hieuhoang1972
c117ef7c17
Copy in changes from the chart_merge branch (doing it manually because the
...
server doesn't seem to support subversion's --reintegrate option).
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3078 1f5c12ca-751b-0410-a591-d2e778427230
2010-04-08 17:16:10 +00:00
bhaddow
407dd68aec
Update lexicalised reordering test truths to take account of new, improved scores.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@3001 1f5c12ca-751b-0410-a591-d2e778427230
2010-03-25 11:04:17 +00:00
hieuhoang1972
cbb329b754
minor
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2955 1f5c12ca-751b-0410-a591-d2e778427230
2010-03-07 19:39:07 +00:00
chardmeier
917737d1eb
Fixed regression test for lattice distortion because implementation is now right and regression test isn't.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2652 1f5c12ca-751b-0410-a591-d2e778427230
2010-01-18 11:13:39 +00:00
phkoehn
8833098925
generalized n-best list reporting for feature functions, added experimental version of global lexical model
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2343 1f5c12ca-751b-0410-a591-d2e778427230
2009-05-26 19:30:35 +00:00
redpony
558c723c7c
fix for one round of problems- this moses will work with g++ > 4.3
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2236 1f5c12ca-751b-0410-a591-d2e778427230
2009-03-12 16:52:27 +00:00
redpony
63effe85b5
Feature function overhaul. Each feature function is computed in one of three ways:
...
1) Stateless feature functions from the phrase table/generation table: these are computed when
the TranslationOption is created. They become part of the ScoreBreakdown object contained in
the TranslationOption and are added to the feature value vector when a hypothesis is extended.
2) Stateless feature functions that are computed during state exploration. Currently, only
WordPenalty falls into this category, but these functions implement a method Evaluate which
do does not receive a Hypothesis or any contextual information.
3) Stateful feature functions: these features receive the arc information (translation option),
compute some value and then return some context information. The context information created
by a particular feature function is passed back to it as the previous context when a hypothesis
originating at the node where the previous edge terminates is created. States in the search
space may be recombined if the context information is identical. The context information must
be stored in an object implementing the FFState interface.
TODO:
1) the command line interface / MERT interface needs to go to named parameters that are otherwise opaque
2) StatefulFeatureFunction's Evaluate method should just take a TranslationOption and a context object. It is not good that it takes a hypothesis, because then people may be tempted to access information about the "previous" hypothesis without "declaring" this dependency.
3) Future cost estimates should be handled using feature functions. All stateful feature functions need some kind of future cost estimate.
4) Philipp's poor-man's cube pruning is broken.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2087 1f5c12ca-751b-0410-a591-d2e778427230
2009-02-06 15:43:06 +00:00
redpony
02067b2be0
fix some nasty edge cases in lattice decoding that arise when decoding lattices with complex topologies. also fix lattice regression tests. work done jointly with j schroeder.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1853 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-17 17:32:19 +00:00
hieuhoang1972
4f84e3ccfe
don't run 2 tests which don't pass
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1850 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-11 13:42:23 +00:00
redpony
aed1758623
fix bug in word lattice decoding that prevented some viable hypotheses from being considered.
...
also fix XML-markup test failures.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1563 1f5c12ca-751b-0410-a591-d2e778427230
2008-02-19 22:03:05 +00:00
redpony
9521e78fbf
add functionalty to support adding direct translation (ie, passthrouh) even when there are other translation options available for the span.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1510 1f5c12ca-751b-0410-a591-d2e778427230
2007-11-10 23:19:41 +00:00
redpony
180d9bac5d
add support for unary XML options. refer to regression-testing/tests/xml-markup/to-translate.txt for examples.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1508 1f5c12ca-751b-0410-a591-d2e778427230
2007-11-10 04:40:18 +00:00
redpony
3b008f67a9
add support for multiple xml options for the same span, start refactoring to add support for overlapping xml options.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1507 1f5c12ca-751b-0410-a591-d2e778427230
2007-11-09 23:56:57 +00:00
redpony
7dbcef925d
Implemented -r (-recover-input-path) to recover the actual path through the CN/word lattice that was used in the best translation. Also cleaned up handling of source Phrases in the context of a particular hypothesis, which may improve performance translating with lexicalized reordering models.
...
Removed PhraseReference since the concept was broken for CN/lattice inputs.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1468 1f5c12ca-751b-0410-a591-d2e778427230
2007-09-28 16:43:33 +00:00
redpony
17884106d4
is this the last of the word lattice distortion errors? maybe. i hope so. the nbest list is at least in the right order now.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1466 1f5c12ca-751b-0410-a591-d2e778427230
2007-09-27 23:22:58 +00:00
redpony
541a486c0c
stricter checks on word lattice distortion
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1464 1f5c12ca-751b-0410-a591-d2e778427230
2007-09-27 12:14:42 +00:00
redpony
251aa600fa
fixed a subtle bug in the word lattice decoder. it now acts sane. This experience also gave me more evidence for thinking that numbering the words in a phrase is not good-- we should be numbering the spaces between them. this would fix oh so many nasty things we've got (like the fact that we represent word drops with [n,n-1] etc).
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1463 1f5c12ca-751b-0410-a591-d2e778427230
2007-09-27 05:03:06 +00:00
redpony
81c0642e36
fix distortion limiting functionality for WordLattices. Now, the shortest path to a another node is used to determine whether the threshold has been exceeded.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1460 1f5c12ca-751b-0410-a591-d2e778427230
2007-09-17 02:49:08 +00:00
redpony
63da638077
The primary change here is that the distortion distance in a word lattice is now defined to be the number of words on the shortest path between the previous phrase's ending span and the next phrase's starting span. Originally, the distance was defined to be the difference in node number, which could be extremely large even for words that are, in some sense, adjacent. Only the distortion scoring has been fixed for word lattices, but the distortion limit is still broken.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1458 1f5c12ca-751b-0410-a591-d2e778427230
2007-09-14 07:31:56 +00:00
redpony
d8536d15b7
add lattice test, not just CN test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1422 1f5c12ca-751b-0410-a591-d2e778427230
2007-07-03 15:40:26 +00:00
redpony
bf756e12fb
add test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1363 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-18 17:03:38 +00:00
redpony
945c77e3f2
fix bug with unknown words/CNs/lexical reordering
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1362 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-18 16:49:51 +00:00
redpony
c80d8b8d47
Support for the decoding of arbitrary word lattices. Must be given in the form of a "plf" file, which is a little tricky. I'll add documentation at some point; for now, refer to the example plf file in the "lattice-surface" regression test.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1359 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-18 14:08:46 +00:00
hieuhoang1972
f4a02f840d
changed lex reordering regression test to fit with philipp's change to staticData
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1346 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-08 22:15:14 +00:00
hieuhoang1972
d48e71cb57
cleanup regression files - consistent file extension, delete old ini files
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1344 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-06 17:07:54 +00:00
hieuhoang1972
b6945f91bc
change user message for downloading data
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1343 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-06 14:06:39 +00:00
hieuhoang1972
10ee6bedf5
update regression data
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1342 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-06 13:55:45 +00:00
hieuhoang1972
fa0231aea9
add lexicalized reordering test
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1341 1f5c12ca-751b-0410-a591-d2e778427230
2007-04-06 13:25:20 +00:00
hieuhoang1972
3c07c5df4d
add svn id comments to start of file
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1307 1f5c12ca-751b-0410-a591-d2e778427230
2007-03-14 22:22:36 +00:00
hieuhoang1972
5f22fb13d3
make output of decimal places consistent by not format anywhere but in Main.cpp
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1266 1f5c12ca-751b-0410-a591-d2e778427230
2007-03-04 00:43:45 +00:00
hieuhoang1972
6b611279d5
minor gcc compile error.
...
also, no longer use IRSTLM as a subsitutute for SRILM, and vice versa. They don't give identitcal results - avoids confusion.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1229 1f5c12ca-751b-0410-a591-d2e778427230
2007-02-21 20:05:27 +00:00
hieuhoang1972
b62dda41ed
change unknown word processing to be closer to the way pharaoh does it - create unknown word whenever single word is not in translation table but penalise hypothesis for using it.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1228 1f5c12ca-751b-0410-a591-d2e778427230
2007-02-21 19:51:17 +00:00
hieuhoang1972
7ecb0ce66e
change unknown word processing to be closer to the way pharaoh does it - create unknown word whenever single word is not in translation table but penalise hypothesis for using it.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1227 1f5c12ca-751b-0410-a591-d2e778427230
2007-02-21 19:48:53 +00:00
hieuhoang1972
e247f1da6f
fixed regression test failing. Number of features for generation models MUST be specified in ini file, no backward compatability hack
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1209 1f5c12ca-751b-0410-a591-d2e778427230
2007-02-13 19:15:34 +00:00