moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-25 04:43:03 +03:00
Code Issues Packages Projects Releases Wiki Activity
14,171 Commits 136 Branches 26 Tags 147 MiB
9d46c5efa1
Commit Graph

770 Commits

Author SHA1 Message Date
Jeroen Vermeulen
ebc0930500 Replace use of tmpnam with boost::filesystem. 
Silences a few annoying warnings from gcc: "tmpnam is dangerous" (and
the suggestion to use mkstemp instead).
 2015-04-02 10:42:06 +07:00
XapaJIaMnu
29a729c99b Remove old obsolete probingPT tests 2015-04-01 16:58:21 +01:00
Ulrich Germann
a9dbced81d Bug fix. 2015-03-30 02:56:49 +01:00
Ulrich Germann
fcbfc5a535 Feature functions and the constructors of TranslationOptionCollections 
now have access to the current translation task.

This was done to allow context-sensitive processing (if provided by the FF).
 2015-03-30 01:20:17 +01:00
Ulrich Germann
79cd40d2c4 Disabled temporarily. Needs to be adapted to API changes in Mmsapt. 2015-03-29 23:58:17 +01:00
Ulrich Germann
2899645992 Cleanup. 2015-03-29 23:57:14 +01:00
Ulrich Germann
3541838a46 Included TargetPhraseCollectionCache.* in fakelib mmsapt. 2015-03-29 23:55:47 +01:00
Ulrich Germann
1525f1ea62 Cleanup. 2015-03-29 23:44:06 +01:00
Ulrich Germann
529a766da7 Initial check-in. 2015-03-29 23:43:50 +01:00
Jeroen Vermeulen
b124d99330 Use boost::filesystem for "rm -rf". 
Replaces a system() call (which was a portability problem) and fixes,
en passant, a warning about its return value being ignored.
 2015-03-29 18:33:58 +07:00
Jeroen Vermeulen
789a2e2bc3 Fix some compile warnings (gcc 4.9.2). 
Mostly signed/unsigned comparisons and reordered member
initializations; also a few unused variables.

There are more, but if I chip away at them for a while, who knows, it
may catch on and warnings may eventually become socially stigmatizing.
:)
 2015-03-29 18:10:51 +07:00
Ulrich Germann
1b23edf62f Cache for the N most recently used TargetPhraseCollections. Refactored out of mmsapt.h. 2015-03-28 14:41:08 +00:00
Jeroen Vermeulen
a9c8f44896 Modernize "C" includes in moses. 
This is one of those little chores in managing a long-lived C++
project: standard C headers like stdio.h and math.h now have their own
place in the C++ standard as resp. cstdio, cmath, and so on.  In this
branch the #include names are updated for the moses/ subdirectory; more
branches to follow.

C++11 adds cstdint, but to support compilation with the previous
standard, that change is left for later.
 2015-03-28 20:09:03 +07:00
Hieu Hoang
1064aaacbe delete typedefs for UINT32 and UINT64. MSVC now has uint32_t and uint64_t /Ken 2015-03-25 00:55:39 +00:00
Ulrich Germann
8ca11d941d 1. Lifetime of tasks in ThreadPool is now managed via shared pointers. 
2. Code cleanup in IOWrapper and a bit elsewhere.
 2015-03-21 16:12:52 +00:00
Ulrich Germann
ee4e396a4d Removed pointer to TranslationTask in InputTypes again. Not the right place to store this information. 2015-03-21 15:29:37 +00:00
Ulrich Germann
dcffbb5f4d Made LRModel::ReorderingType an enumerated type. 2015-03-16 00:24:11 +00:00
Ulrich Germann
085c88cc7b Eliminated sources of some compiler warnings (unused variables; signed/usigned comparisons). 2015-03-15 22:45:01 +00:00
Ulrich Germann
ad805c133b Instances of InputType (and derived classes) now know which TranslationTask (if any) created them. 
This is a first step towards providing phrase tables etc. access to context information etc.
associated with specific translation tasks.
 2015-03-15 20:38:31 +00:00
Ulrich Germann
2a66a55c85 Added document map (maps from sentences to document ids) to Bitext class. 
Minor overhaul to the bias regime, which allows to specify bias by document
name (as provided in the document map) rather than by sentence in the static
parallel corpus.
 2015-03-15 13:32:09 +00:00
Ulrich Germann
51824355f9 Sampling now keeps track of counts for hierarchical lexicalized reordering. 2015-03-10 10:41:41 +00:00
Ulrich Germann
524376fad4 Code cleanup. 2015-03-09 00:34:47 +00:00
Hieu Hoang
32de075022 beautify 2015-02-19 12:27:23 +00:00
Ulrich Germann
ccf44f39fb Code cleanup and reorganization. A few classes have been renamed to shorter names. 2015-02-15 01:45:22 +00:00
Hieu Hoang
755bd609f5 Using boost for prefix/suffix checks /Jeroen Vermeulen 2015-02-06 15:52:25 +00:00
Hieu Hoang
70e8eb54ce Using boost for prefix/suffix checks /Jeroen Vermeulen 2015-02-05 16:23:47 +00:00
Marcin Junczys-Dowmunt
4140756fdf Add missing chck for empty range while flushing 2015-01-22 22:18:19 +01:00
Marcin Junczys-Dowmunt
7d9013a85b Work-around for temporary translation option collection size during phrase table binarization 2015-01-19 23:15:08 +01:00
Marcin Junczys-Dowmunt
fbcf2dcb56 Fixed thread-safety 2015-01-19 21:56:04 +01:00
Marcin Junczys-Dowmunt
82c603213a Thread-safety and constness 2015-01-18 23:58:28 +01:00
Marcin Junczys-Dowmunt
16ffc2c978 Added new VW feature and execption to Simple9 2015-01-18 23:26:32 +01:00
Hieu Hoang
6d61db28fa use astyle 2.01. It's on Edinburgh server and doesn't screw up enum 2015-01-14 19:21:11 +00:00
Hieu Hoang
05ead45e71 beautify 2015-01-14 11:07:42 +00:00
Phil Williams
e5ebf30664 Fix a few warnings. 2015-01-13 21:13:55 +00:00
Hieu Hoang
be0ab92d16 delete oov pt 2015-01-09 22:32:08 +00:00
Hieu Hoang
e195bdf6d9 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-01-08 02:37:01 +04:00
Rico Sennrich
7123d1cc80 eliminate spurious copy / memory leak 2015-01-07 18:42:20 +00:00
Hieu Hoang
ff7fbd55ee add oovpt 2015-01-07 15:33:42 +04:00
Hieu Hoang
99b4b63c0c change signature of GetChartRuleCollection() 2015-01-07 12:59:08 +04:00
Hieu Hoang
b9bef2fc44 add oovpt 2015-01-07 12:18:09 +04:00
Hieu Hoang
3b3f11365d delete UserMessage. Too difficult to police 2015-01-07 10:01:10 +04:00
Hieu Hoang
1e0a2835bf add oovpt 2015-01-04 19:10:48 +05:30
XapaJIaMnu
d0807c45f2 Fixed crash in probingPT when probability is precisely 0 2014-12-23 15:21:06 +00:00
Nicola Bertoldi
d0cddf0f2d Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-12-16 17:35:47 +01:00
Nikolay Bogoychev
d0f4402e86 Fix incorrect hashing in ProbingPT 2014-12-16 11:15:12 +00:00
Nicola Bertoldi
4e77665d30 better handling of cache-based models with inconsistent parameters 2014-12-15 17:42:41 +01:00
Nicola Bertoldi
e4eb201c52 merged master into dynamic-models and solved conflicts 2014-12-13 12:52:47 +01:00
Nicola Bertoldi
cea2d9d8bb beautify 2014-12-09 12:39:37 +01:00
Hieu Hoang
8c6310bf4c Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-12-05 23:26:24 +00:00
Matthias Huck
bfeb7d641f log output 2014-12-05 22:31:54 +00:00
Hieu Hoang
4b10c59bea add OutputSearchGraphHypergraph() to API framework. Move m_source to BaseManager 2014-12-05 21:33:59 +00:00
Rico Sennrich
56921cae3b small simplification of recursive CYK+ 
(following Chris Dyer's suggestion and Phil's refactoring in S2T decoder)
 2014-12-01 11:05:17 +00:00
Ulrich Germann
7aa4d5d8d5 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 
Conflicts:
	moses-cmd/simulate-pe.cc
 2014-11-20 17:55:51 +00:00
XapaJIaMnu
52c520c042 Resolve merge conflicts 2014-11-20 15:50:32 +00:00
Ulrich Germann
bda7ace530 Minor changes due to changes in the Moses API. Removed from list of standard programs to be compiled and installed. May need some work to get it working again. 2014-11-16 16:31:12 +00:00
XapaJIaMnu
4bea830188 doesn't work 2014-11-13 15:50:05 +00:00
Hieu Hoang
e1092c0dad merge 2014-11-07 14:35:36 +00:00
Laura Kieras
ecae85e9a8 mm2dTable now opens its data file read-only, using mapped_file_source, so that we don't need write permissions on the file 2014-11-04 16:30:46 -05:00
Ulrich Germann
07202c544c Added ptable-describe-features to list features used by PhraseDictionaryBitextSampling. 2014-10-25 12:06:38 -07:00
Ulrich Germann
44215b79c0 Added ptable-describe-features to list features used by PhraseDictionaryBitextSampling. 2014-10-25 12:06:24 -07:00
Ulrich Germann
53ef6c5c38 Added demo program for use of suffix arrays. 2014-10-23 11:11:28 -07:00
Barry Haddow
562cf7e007 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-10-21 15:11:22 +01:00
Hieu Hoang
cce818015d Merge ../mosesdecoder into merge-cmd 2014-10-10 15:50:12 +01:00
Phil Williams
05ecc914c2 Fix a few more compiler warnings (from Clang mostly). 2014-10-10 15:47:53 +01:00
Phil Williams
ee57e59f2b Fix a few compiler warnings (from Clang mostly). 2014-10-10 14:22:53 +01:00
Hieu Hoang
1743f7eeb2 Merge ../mosesdecoder into merge-cmd 2014-10-08 17:55:07 +01:00
Ulrich Germann
576931b088 Mmsapt now adds word alignment info to target phrases. 2014-10-07 18:08:31 +01:00
Hieu Hoang
33ed15ef19 move misc common functions into moses/ 2014-09-30 14:22:38 +01:00
Barry Haddow
091948bff0 Improved debug 2014-09-18 17:03:19 +01:00
Ulrich Germann
1d834e2b48 Fixed bug with respect to adding check option to Mmsapt::Load(). 2014-09-10 18:51:20 +02:00
Ulrich Germann
a58c7ceb18 Fixed issues with ambiguity in typedef of uint64_t (conflict between boost typedef and stdint typedef). 2014-09-10 12:07:57 +02:00
Ulrich Germann
31578d4915 Finished code for bias loading from Mmsapt config file. 2014-09-09 18:07:26 +01:00
Ulrich Germann
cda94c7d85 Fix in biased sampling. Started code on loading and using bias in Mmsapt. 2014-09-09 17:45:48 +01:00
Ulrich Germann
f86fa65a6f Added utility count-ptable-features to count features in Mmsapt given a moses.ini config line. 2014-09-08 16:56:45 +01:00
Ulrich Germann
db6e5de641 Added initial code for utility to count features of PhraseDictionaryBitextSampling. 2014-09-08 11:03:05 +01:00
Ulrich Germann
5571ec91c6 Code cleanup. 2014-09-08 09:26:09 +01:00
Ulrich Germann
a86d49fc88 Added bias to bitext sampling. 2014-09-08 09:26:08 +01:00
Ulrich Germann
cef6460981 Initial check-in. 2014-09-08 09:26:08 +01:00
Ulrich Germann
a87a9ff207 Moved class PhrasePair back to ug_bitext. 
Moved function expand() from mmsapt.cc to ug_bitext.h.
Added new lookup function to class Bitext.
Bug fixes related to inverse lookup in class Bitext.
 2014-09-08 09:26:08 +01:00
Ulrich Germann
b588df77f0 Bug fix related to threading. 2014-09-08 09:26:08 +01:00
Ulrich Germann
2405293aaa Fiddling around with the code. Not for production. 2014-09-08 09:26:08 +01:00
Ulrich Germann
90c91ae9bb Added fakelib stringdist. 2014-09-08 09:26:08 +01:00
Ulrich Germann
9af3a61678 Added try-align2. 2014-09-08 09:26:08 +01:00
Ulrich Germann
a028fec7af Work in progress. 2014-09-08 09:26:08 +01:00
Michael Denkowski
3304030a4e Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-09-04 11:19:32 -04:00
Michael Denkowski
6c33bc99dc Option to add TM-specific word and phrase counts 2014-09-04 11:17:42 -04:00
Michael Denkowski
756bcf0f15 Option to add TM-specific word and phrase counts 2014-09-04 01:49:26 -04:00
Rico Sennrich
2a46e8ccea parse chart compression for faster CYK+ parsing with syntax systems. 2014-09-01 18:16:22 +01:00
Michael Denkowski
1c45d780d4 all-restrict mode for MultiModel (restrict to phrases in first model) 2014-08-26 13:43:23 -04:00
Hieu Hoang
97e5a30d3a compiles with clang on osx 2014-08-25 18:07:42 +01:00
Michael Denkowski
da0ed4df81 tunable=false option for mmsapt 2014-08-18 19:22:50 -04:00
Michael Denkowski
93e99be108 Mode to pass through "all" scores in MultiModel 2014-08-18 17:57:05 -04:00
Nicola Bertoldi
77e9e91b08 minor fixes 2014-08-18 19:13:51 +02:00
Hieu Hoang
00a338d576 clang only function 2014-08-14 16:44:20 +01:00
Hieu Hoang
303387f9ac compiles with clang on osx 2014-08-14 16:17:21 +01:00
Hieu Hoang
fcbd64b3ac eclipse 2014-08-14 14:04:25 +01:00
Hieu Hoang
2bbaf69409 Merge branch 'master' into bo-safe 2014-08-13 18:52:14 +01:00
Hieu Hoang
94c44c03d5 merge 2014-08-13 18:03:05 +01:00
Hieu Hoang
18c1c4a132 method rename 2014-08-08 18:11:30 +01:00
Hieu Hoang
efa5befb16 method rename 2014-08-08 15:59:34 +01:00
Ulrich Germann
95b04d2558 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-08-05 21:28:06 +01:00
Ulrich Germann
5480499309 Fixed (?) problem with multiple identical extractable target phrases per source phrase occurrence. 2014-08-05 21:26:29 +01:00
Michael Denkowski
13942b77ab Add alias PhraseDictionaryBitextSampling 2014-08-05 14:47:07 -04:00
Ulrich Germann
f32a313a05 Mmsapt now uses timespec on linux, timeval om MacOS for time stamps. 2014-08-05 02:22:20 +01:00
Hieu Hoang
11471de9b8 mac osx 2014-08-04 18:50:10 +01:00
Ulrich Germann
c269abb083 Added num_read_write.cc to fakelib mm. 2014-08-04 17:52:08 +01:00
Ulrich Germann
9fad5d3eb0 Eliminated dependence on endian.h and related byte swapping on big-endian machines. 2014-08-04 17:52:08 +01:00
Hieu Hoang
3f29ed10f1 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-08-05 11:00:01 +01:00
Hieu Hoang
84d6b25802 TargetPhrase to have pointer to the phrase table that creates it 2014-08-05 10:59:48 +01:00
Hieu Hoang
f447a23067 TargetPhrase to have pointer to the phrase table that creates it 2014-08-05 10:26:42 +01:00
Hieu Hoang
e863592f40 TargetPhrase to have pointer to the phrase table that creates it 2014-08-04 19:28:04 +01:00
Hieu Hoang
abe68be588 initialise m_container 2014-08-04 15:59:32 +01:00
Hieu Hoang
3f3912772d initialise m_container 2014-08-04 15:46:40 +01:00
Hieu Hoang
5f90ccdb13 initialise m_container 2014-08-04 15:20:22 +01:00
Marcin Junczys-Dowmunt
5c9017c632 Forgot to add SetFeaturesToApply 2014-08-03 19:44:43 +02:00
Marcin Junczys-Dowmunt
ff6ed8cd21 Fixed segfault for features depending on factors not in phrase table (i.e. added by generation models) 2014-08-03 18:03:42 +02:00
Hieu Hoang
688bf4c061 each target phrase knows what decode graph created it 2014-08-02 17:15:01 +01:00
hieu
5741ef2635 compile error in gcc 4.4 2014-07-30 18:01:51 +01:00
Ulrich Germann
f9d167345a Changed feature and parameter names for Mmsapt / PhraseDictionaryBitextSampling as requested by PK. 2014-07-29 13:57:00 +01:00
Ulrich Germann
6a1beb770d Cleanup work to get rid of compiler warnings. 2014-07-29 13:51:44 +01:00
Nicola Bertoldi
1063012892 added a flag do disable the decaying in the cache 2014-07-22 11:25:03 +02:00
Nicola Bertoldi
02bf6d5d5e fixings about file loading and precomputation of ascores 2014-07-22 09:45:41 +02:00
Hieu Hoang
b10760f428 delete PhraseTableImplementation. Old enum 2014-07-18 20:36:53 +01:00
Hieu Hoang
1347b153ee compiles with c++11. Used by oxlm 2014-07-17 23:13:06 +01:00
Ulrich Germann
f06b145735 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-07-10 17:24:42 +01:00
Ulrich Germann
4d41211c2c Major overhaul of Mmsapt. Reorganization of old and addition of new features in phrase tables. Many critical bug fixes. 2014-07-09 02:41:28 +01:00
Ulrich Germann
e1c9405b23 Utilities to check gain in phrase coverage by dynamic augmentation of the phrase table in a post-editing scenario. 2014-07-09 02:39:28 +01:00
Rico Sennrich
bdf8d1a405 CreateFromString no longer requires factorDelimiter 2014-07-04 10:28:24 +01:00
Ulrich Germann
556e112366 Major bug fix in Mmsapt.combine_pstats. 2014-06-30 00:32:11 +01:00
Ulrich Germann
5c77f17833 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-06-27 01:12:22 +01:00
Ulrich Germann
9eaf506ace Renamed lookup_mmsapt.cc to ptable-lookup.cc. 2014-06-27 01:10:14 +01:00
Ulrich Germann
1e50ac587d Replaced TRACE_ERR(..) with VERBOSE(1,...) in PhraseDictionaryTree.cpp 2014-06-27 01:09:50 +01:00
Ulrich Germann
9dfb354041 Phrase probability smoothing can now be turned off in ug_bitext.cc by setting the confidence threshold to 0. 2014-06-27 01:07:23 +01:00
Ulrich Germann
40ab68b3d2 Adapted lookup_mmsapt to binary phrase tables; renamed the utility to ptable-lookup 2014-06-27 01:06:00 +01:00
Hieu Hoang
f2d3f57d07 Merge branch 'hieu' of ../mosesdecoder.hieu 2014-06-16 10:57:02 +01:00
Ulrich Germann
2ac93eca5c PScoreBwd can now scales backward marginals according as specified by user. 2014-06-15 21:11:33 +01:00
Ulrich Germann
5ff14f9798 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-06-14 13:29:52 +01:00
Ulrich Germann
1a3d7d3266 Reorganization of phrase scorers in Mmsapt. 2014-06-14 13:03:31 +01:00
Ulrich Germann
c3db1a3a67 Better labeling of feature values in lookup_mmsapt. 2014-06-14 13:02:55 +01:00
Ulrich Germann
135ce20509 Smoothing parameter in LexicalPhraseScorer2 is now a configurable paramter. 2014-06-14 13:02:08 +01:00
Ulrich Germann
450c8c5b80 Moved phrase scorers to a separate file. 2014-06-14 12:58:33 +01:00
Hieu Hoang
eb78782c5d merge with master 2014-06-13 10:35:35 +01:00
Hieu Hoang
9e43830c6e bjam for ProbingPT 2014-06-13 10:20:30 +01:00
Hieu Hoang
2c6cdb5889 bjam for ProbingPT 2014-06-13 09:48:32 +01:00
Hieu Hoang
4ebcfc2142 bjam for ProbingPT 2014-06-12 18:38:04 +01:00
Ulrich Germann
ead3fc565e Initial check-in. 2014-06-12 18:30:41 +01:00
Hieu Hoang
ad4825ad44 integrated probing pt into decoder 2014-06-11 14:25:18 +01:00
Hieu Hoang
253f8e35fd integrated probing pt into decoder 2014-06-11 13:52:20 +01:00
XapaJIaMnu
095877742c Add ProbingPT files to Moses 2014-06-11 11:47:00 +01:00
Hieu Hoang
341969e61d Merge ../mosesdecoder into hieu 2014-06-10 18:04:55 +01:00