mosesdecoder/mert/TODO

25 lines
1.1 KiB
Plaintext

- check that mert-moses.pl emits devset score after every iteration
- correctly for whichever metric we are optimizing
- even when using --pairwise-ranked (PRO)
- this may make use of 'evaluator', soon to be added by Matous Machacek
- check that --pairwise-ranked is compatible with all optimization metrics
- Replace the standard rand() currently used in MERT and PRO with better
random generators such as Boost's random generators (e.g., boost::mt19937).
- create a Random class to hide the details, i.e., how to generate
random numbers, which allows us to use custom random generators more
easily.
Pros:
- In MERT, you might want to use the random restarting technique to avoid
local optima.
- PRO uses a sampling technique to choose candidate translation pairs
from N-best lists, which means the choice of random generators seems to
be important.
Cons:
- This change will require us to re-create the truth results for regression
testing related to MERT and PRO because the new random generator will
generate different numbers from the current generator does.