Commit Graph

4933 Commits

Author SHA1 Message Date
Nikolay Bogoychev
ab6b826083
Add GCC 11 support (#888)
* Add GCC 11 support

Some C++ Standard Library headers have been changed to no longer include other headers that they do need to depend on. As such, C++ programs that used standard library components without including the right headers will no longer compile.
The following headers are used less widely in libstdc++ and may need to be included explicitly when compiled with GCC 11:

<limits> (for std::numeric_limits)
<memory> (for std::unique_ptr, std::shared_ptr etc.)
<utility> (for std::pair, std::tuple_size, std::index_sequence etc.)
<thread> (for members of namespace std::this_thread.)

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
2021-11-23 10:13:29 +00:00
Nikolay Bogoychev
1adf80b7c9
Task alias validation during training mode (#886)
* Attempt to validate task alias
* Validate allowed options for --task alias
* Update comment in aliases.cpp
* Show allowed values for alias

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
2021-11-22 19:19:58 +00:00
Roman Grundkiewicz
3d15cd3d20 Update submodule regression-tests 2021-11-22 06:41:16 -08:00
David Meikle
3b4e943cda
Added pragma to ignore unused-private-field error on elementType_ which failed in macOS (#872)
Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
2021-11-22 12:22:06 +00:00
Marcin Junczys-Dowmunt
c85d060848 Merged PR 20729: Add top-k sampling
This adds Top-K sampling to Marian and extends the --output-sampling option to take arguments
2021-11-22 03:32:54 +00:00
Roman Grundkiewicz
2bdfbd3f02
Update badges in README.md 2021-11-21 17:06:01 +00:00
Marcin Junczys-Dowmunt
1404201926 Merged PR 21151: Cleaning up fp16 behavior
This PR improves clipping and pruning behavior of NaNs and Infs during fp16 training, ultimately avoiding the underflow problems that we were facing so far.
2021-10-26 20:25:39 +00:00
Roman Grundkiewicz
7f06f3c5d2 Merged PR 21166: Keep building on macOS-10.15
Marian does not compile on macOS 11.6, so the build has stopped working due to an upgrade from macOS-10.15 to macOS 11.6 in Azure Pipelines: https://github.com/actions/virtual-environments/issues/4060
This PR explicitly set macOS 10.15 in the workflow.
2021-10-26 11:20:41 +00:00
Hieu Hoang
2d79ad02bb Merged PR 20933: beam & batch works for n on-factored models 2021-10-13 20:20:14 +00:00
Roman Grundkiewicz
12a1bfaf6f
Remove Ubuntu 16.04 from GitHub workflows (#879)
* Add --allow-unauthenticated when installing CUDA
* Remove workflow with Ubuntu 16.04
2021-10-11 16:59:52 +01:00
Marcin Junczys-Dowmunt
03fe175876 Merged PR 20879: Adjustable ffn width and depth in transformer decoder 2021-09-28 17:19:07 +00:00
Marcin Junczys-Dowmunt
d796a3c3b7 Merged PR 20839: Do not ignore ignoreEOS for spm decoding
With final space this eliminates trailing whitespace caused by appending EOS
2021-09-28 17:17:12 +00:00
Roman Grundkiewicz
aa58ba8e23 Merged PR 20593: Fix and update Azure pipelines
- Add `--allow-unauthenticated` to `apt` when installing CUDA on Ubuntu
- Removing `ubuntu-16.04` image from Azure pipelines, which will become unavailable after September 20
2021-09-20 13:14:24 +00:00
Roman Grundkiewicz
8470c16bdd Merged PR 20230: Add option for running regression tests only in Azure Pipelines
This PR adds a checkbox which can be unchecked to skip running compilation checks when triggering them manually. It is useful for generating expected outputs on different CPUs for tests using 8-bit models.
2021-09-16 02:00:00 +00:00
Kenneth Heafield
4dd30b5065
Factor concatenation improvements and documentation (#748)
* concatenation combining option added when embeding using factors
* crossMask not used by default
* added an option to better clarify when choosing factor predictor options
* fixed bug when choosing re-embedding option and not setting embedding size
* avoid uncessary string copy
* Check in factors documentation
* Fix duplication in merge
* Self-referential repository
* change --factors-predictor to --lemma-dependency. Default behaviour changed.
* factor related options are now stored with the model
* Update doc/factors.md
* add backward compability for the target factors
* Move backward compatibility checks for factors to happen after the model.npz config is loaded
* Add explicit error msg if using concat on target
* Update func comments. Fix spaces
* Add Marian version requirement
* delete experimental code

Co-authored-by: Pedro Coelho <pedrodiascoelho97@gmail.com>
Co-authored-by: Pedro Coelho <pedro.coelho@unbabel.com>
Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
2021-09-08 14:02:21 +01:00
Roman Grundkiewicz
8d0a3c0c27
Add --allow-unauthenticated when installing CUDA (#878) 2021-09-07 11:11:58 +01:00
Rohit Jain
6652b310b1 Merged PR 20560: Update SPM in Marian
Update SPM in Marian
2021-09-02 05:20:56 +00:00
Roman Grundkiewicz
e025bfb07c Merged PR 20070: Run regression tests in Azure Pipelines
The changes proposed in this pull request:
* Added regression testing with internal models into Azure Pipelines on both Windows and Ubuntu
* Created https://machinetranslation.visualstudio.com/Marian/_git/marian-prod-tests (more tests will be added over time)
* Made regression test outputs (all `.log`, `.out`, `.diff` files) available for inspection as a downloadable artifact.
* Made `--build-info` option available in CMake-based Windows builds

Warning: I tried to handle multiple cases, but some regression tests may occasionally fail, especially tests using avx2 or avx512 models, because the outputs are system/CPU dependent. I think it's better to merge this already, monitoring the stability of tests, and adding expected outputs variations if necessary, improving the coverage and stability of regression tests over time.
2021-08-06 08:02:18 +00:00
Marcin Junczys-Dowmunt
d124ca9f5b allow float32 conversion in QS interface 2021-08-03 23:00:06 -07:00
Rohit Jain
4ff2ef189e Merged PR 19761: Expose SPM Interface from Marian
This PR adds interfaces in Marian to allow it to handle segmentation duties.

Related work items: #121418
2021-07-30 03:28:00 +00:00
Martin Junczys-Dowmunt
b653db0a9b Merged PR 19910: Fix training/scoring error with FSM
Fixes a dimension mismatch during training and scoring introduced in the decoding-only shortlist changes.

Related work items: #122643
2021-07-22 21:00:44 +00:00
Roman Grundkiewicz
6b568f4afa Merged PR 19904: Update instructions for building on Windows
Changes proposed in this pull request:
1. Clarified instructions how to build Marian on Windows in `vs/README.md`.
2. `vs/CheckOrInstallDeps.bat` does not stop if CUDA or MKL are not installed.
2021-07-22 16:44:35 +00:00
Roman Grundkiewicz
b83b06fb73 Merged PR 19914: Fix Windows Azure Pipelines
Updating vcpkg seems to fix the recent issues with Windows builds in Azure Pipelines.
2021-07-22 16:36:43 +00:00
Hieu Hoang
f6cb1b5c6a Merged PR 19864: add bias if it exists
Fixes backcompat with shortlist and bias.
2021-07-21 00:12:02 +00:00
Rohit Jain
056c4bef5b Merged PR 19860: Case augmented data, if not using factored vocab must not set guided alignments
This change allows marking SentenceTuples as 'altered', if they were generated or modified by data augmentation internally in such a way so as to impact processing. In particular, for such sentence tuples, we do not want to try setting guided alignments if the externally provided guided alignments might no longer be correct after that alteration.
2021-07-17 23:03:16 +00:00
Martin Junczys-Dowmunt
8e88071ae8 Merged PR 19842: Adapt LSH to work with Leaf
Small changes to make the LSH work with Leaf server and QuickSand.
2021-07-16 20:04:16 +00:00
Qianqian Zhu
42f0b8b74b
Binary shortlist (#856)
Co-authored-by: Kenneth Heafield <github@kheafield.com>
2021-07-10 22:56:58 -07:00
Marcin Junczys-Dowmunt
7e6ea51841 silence unreferenced formal parameter warning on windows 2021-07-10 08:36:18 -07:00
Marcin Junczys-Dowmunt
3a478fc47d update version and changelog 2021-07-09 13:46:18 -07:00
Martin Junczys-Dowmunt
35c822eb4e Merged PR 19685: Marianize LSH as operators for mmapping and use in Quicksand
This PR turns the LSH index and search into a set of operators that live in the expression graph. This makes creation etc. thread-safe (one index per graph) and allows to later implement GPU versions.

This allows to mmap the LSH as a Marian parameter since now we only need to turn the index into something that can be saved to disk using the existing tensors. This happens in marian_conv or the equivalent interface function in the Quicksand interface.
2021-07-09 20:35:09 +00:00
Hieu Hoang
d6c09b24de Merged PR 19409: Unify LSH and short list interface
This PR unifies the shortlist and LSH interface achieving significant speed-up for the LSH.
2021-07-03 20:19:39 +00:00
Marcin Junczys-Dowmunt
9772aa293f remaining comments 2021-07-03 12:13:26 -07:00
Marcin Junczys-Dowmunt
8bfa6a44e3 Merge branch 'hihoan/lsh7' of vs-ssh.visualstudio.com:v3/machinetranslation/Marian/marian-dev into hihoan/lsh7 2021-07-03 12:01:22 -07:00
Hieu Hoang
4ace42f35a paper 2021-07-02 20:30:50 -07:00
Hieu Hoang
5ad0edf6df remove todo 2021-07-02 19:07:42 -07:00
Hieu Hoang
9acf27d6bc credit SLIDE 2021-07-02 12:16:57 -07:00
Hieu Hoang
bd1f1ee9cb marcin's review changes 2021-07-02 12:06:03 -07:00
Hieu Hoang
ff8af52624 lock index before creation 2021-07-01 15:39:18 -07:00
Marcin Junczys-Dowmunt
64e787afce Merge branch 'master' into hihoan/lsh7 2021-06-29 10:42:13 -07:00
Hieu Hoang
7e2fce8f09 Merge branch 'hihoan/lsh7' of vs-ssh.visualstudio.com:v3/machinetranslation/Marian/marian-dev into hihoan/lsh7 2021-06-28 21:26:27 -07:00
Hieu Hoang
24c644bae0 pass shortlist regression tests 2021-06-28 21:26:02 -07:00
Marcin Junczys-Dowmunt
8daa0a4255 fix compilation errors due to narrow conversion 2021-06-28 19:26:40 -07:00
Martin Junczys-Dowmunt
fc0f41f24a Merged PR 19597: Enable mpi wrapper to use size larger than MAX_INT
Enable mpi wrapper to use size larger than MAX_INT.
2021-06-28 23:15:23 +00:00
Hieu Hoang
cd292d3b32 changes for review 2021-06-18 10:18:31 -07:00
Hieu Hoang
a332e550a5 debug 2021-06-16 12:56:36 -07:00
Marcin Junczys-Dowmunt
85eb6adce0 update sentencepiece pointer to version with case-awareness 2021-06-16 12:40:38 -07:00
Hieu Hoang
9b4a845cc7 clean up bias 2021-06-16 11:19:23 -07:00
Hieu Hoang
892554129e lemma Et is optional 2021-06-16 10:19:24 -07:00
Hieu Hoang
395a4f94d0 init vector 2021-06-15 18:08:45 -07:00
Hieu Hoang
6981b21f4e Merge branch 'hihoan/lsh7' of vs-ssh.visualstudio.com:v3/machinetranslation/Marian/marian-dev into hihoan/lsh7 2021-06-15 16:54:57 -07:00