mosesdecoder/sigtest-filter
2010-04-20 11:06:51 +00:00
..
check-install Implementation of idea in "Improving Translation Quality by Discarding Most of the Phrasetable". Johnson et al. 2007. EMNLP. Requires Joy Zhang's SALM toolkit. 2007-07-26 20:26:32 +00:00
filter-pt.cpp win32 compile 2007-11-10 09:32:07 +00:00
Makefile allow changing FLAVOR using env. var. 2010-04-20 11:06:51 +00:00
README.txt more recommendations 2009-02-25 17:28:38 +00:00
sigtest-filter.sln make filter-pt compile under windows 2007-08-16 18:13:04 +00:00
sigtest-filter.vcproj win32 compile 2007-11-10 09:32:07 +00:00
WIN32_functions.cpp win32 compile 2007-11-10 09:32:07 +00:00
WIN32_functions.h win32 compile 2007-11-10 09:32:07 +00:00

Re-implementation of Johnson et al. (2007)'s phrasetable filtering strategy.

This implementation relies on Joy Zhang's SALM Suffix Array toolkit. It is
available here:

  http://projectile.sv.cmu.edu/research/public/tools/salm/salm.htm
  
--Chris Dyer <redpony@umd.edu>

BUILD INSTRUCTIONS
---------------------------------

1. Download and build SALM.

2. make SALMDIR=/path/to/SALM


USAGE INSTRUCTIONS
---------------------------------

1. Using the SALM/Bin/Linux/Index/IndexSA.O32, create a suffix array index
   of the source and target sides of your training bitext.

2. cat phrase-table.txt | ./filter-pt -e TARG.suffix -f SOURCE.suffix \
    -l <FILTER-VALUE>

   FILTER-VALUE is the -log prob threshold described in Johnson et al.
     (2007)'s paper.  It may be either 'a+e', 'a-e', or a positive real
     value. 'a+e' is a good setting- it filters out <1,1,1> phrase pairs.
     I also recommend using -n 30, which filteres out all but the top
     30 phrase pairs, sorted by P(e|f).  This was used in the paper.

3. Run with no options to see more use-cases.


REFERENCES
---------------------------------

H. Johnson, J. Martin, G. Foster and R. Kuhn. (2007) Improving Translation
  Quality by Discarding Most of the Phrasetable. In Proceedings of the 2007
  Joint Conference on Empirical Methods in Natural Language Processing and
  Computational Natural Language Learning (EMNLP-CoNLL), pp. 967-975.