Marcin Junczys-Dowmunt
65bf82ffce
version 1.12.0 ( #980 )
2023-02-21 17:56:29 +00:00
Marcin Junczys-Dowmunt
4ffd292881
Merge branch 'master' into pmaster
2023-02-20 12:15:33 -08:00
Marcin Junczys-Dowmunt
031dbb3266
Merged PR 27804: Fallback to old LSH code for MSVC due to bad loop unrolling
...
The Visual Studio compiler has inferior optimization and loop unrolling to gcc which results in much slower LSH code that was written to explicitly take advantage of loop unrolling at compile time.
Added an #ifdef to fall back to old LSH code on MSVC.
2023-02-13 15:44:19 +00:00
Marcin Junczys-Dowmunt
9ad5203ca2
Merged PR 26476: Sanitize guided-alignment with case-augmentation (still somewhat wonky)
...
This fixes the blow-ups of using case-augmentation with guided-alignment. However, it's still not recommended to use this particular combination, results will be unreliable.
2023-02-11 16:35:29 +00:00
Varun Mathur
4f145c450f
Merged PR 26311: [FSM] make model loading lock non-static
...
make lock non-static
2023-02-10 16:34:37 +00:00
Roman Grundkiewicz
ee50d4aaea
Merged PR 27051: Add an option for completely resetting validation metrics
...
Added `--valid-reset-all` that works as `--valid-reset-stalled` but it also resets last best saved validation metrics, which is useful for when the validation sets change for continued training.
Added new regression test: https://github.com/marian-nmt/marian-regression-tests/pull/89
2022-12-20 17:56:10 +00:00
Alex Muzio
b7205fc0b0
Merged PR 25220: Add extra model information to model_info.py script
...
Adding model shapes flag to model_info.py script: dtype and total number of model parameters.
Example: `python model_info.py -m ~/model.npz -mi`
2022-11-30 12:23:38 +00:00
dependabot[bot]
3c2a432995
Bump examples from 25e8438 to 58f48a0 ( #975 )
...
Bumps [examples](https://github.com/marian-nmt/marian-examples ) from `25e8438` to `58f48a0`.
- [Release notes](https://github.com/marian-nmt/marian-examples/releases )
- [Commits](25e8438322...58f48a0675
2022-11-29 18:22:22 +00:00
dependabot[bot]
d5569cee60
Bump regression-tests from 494d6de to 488d454 ( #974 )
...
Bumps [regression-tests](https://github.com/marian-nmt/marian-regression-tests ) from `494d6de` to `488d454`.
- [Release notes](https://github.com/marian-nmt/marian-regression-tests/releases )
- [Commits](494d6de2bd...488d454a01
2022-11-29 18:22:00 +00:00
Roman Grundkiewicz
b6581c4c44
Merged PR 26667: Update examples submodule to fix vulnerability issues
...
Updating examples submodule using [protobuf 3.20.2](https://github.com/marian-nmt/marian-examples/pull/29 ) to fix recent [vulnerability issues](https://machinetranslation.visualstudio.com/MachineTranslation/_componentGovernance/mtmain/alert/8035094?typeId=14698327&pipelinesTrackingFilter=0 ).
Related work items: #134319
2022-11-23 19:16:44 +00:00
Roman Grundkiewicz
c79dc80a2f
Merged PR 26617: Update regression-tests & fix CI pipelines
...
Update regression-tests & fix CI pipelines
2022-11-20 13:31:10 +00:00
dependabot[bot]
36349645b8
Bump src/3rd_party/sentencepiece from 31ac8e8 to 8dc9172 ( #970 )
...
Bumps [src/3rd_party/sentencepiece](https://github.com/marian-nmt/sentencepiece ) from `31ac8e8` to `8dc9172`.
- [Release notes](https://github.com/marian-nmt/sentencepiece/releases )
- [Commits](31ac8e8876...8dc9172f88
2022-11-19 08:36:06 +00:00
dependabot[bot]
4187aab9ba
Bump regression-tests from 92e116e to 494d6de ( #973 )
...
Bumps [regression-tests](https://github.com/marian-nmt/marian-regression-tests ) from `92e116e` to `494d6de`.
- [Release notes](https://github.com/marian-nmt/marian-regression-tests/releases )
- [Commits](92e116efa3...494d6de2bd
2022-11-19 08:35:45 +00:00
Nikolay Bogoychev
07a2ac8126
best-deep alias broken ( #968 )
...
The best-deep alias in marian is currently broken, because it doesn't set the model type and the default is `amum` which is incompatible with multiple layers. This commit just adds the type to the best-deep alias entry.
2022-11-02 11:16:14 +00:00
Roman Grundkiewicz
cda2f2112f
Temporarily download MKL tarball from a mirror server ( #972 )
2022-11-02 11:09:43 +00:00
Roman Grundkiewicz
be1ee3fa94
Merged PR 26318: Fix incorrect envvar name in Azure Pipeline
...
Fix incorrect environment variable name for SAS token in Windows tests
2022-11-01 10:07:40 +00:00
Roman Grundkiewicz
a6de1b781c
Merged PR 26271: Update CI pipeline triggers
...
Updates to the CI triggers:
- Stop running parallel CI runs, i.e. if a pipeline is running, it must finish before new runs are started.
- Exclude paths to files, which are not related to/critical the codebase
- Downloading MKL from a mirror hosting server
2022-11-01 06:26:56 +00:00
Marcin Junczys-Dowmunt
4d3702c4ec
Merged PR 25950: Add missing defaults for concatenated factors
...
This PR adds missing default values for concatenated factors.
2022-10-06 05:53:16 +00:00
Marcin Junczys-Dowmunt
da6e30bfe3
merge with internal master
2022-10-03 17:51:03 -07:00
Marcin Junczys-Dowmunt
1e92cff93d
Merged PR 25919: Sync with public master - no review required
...
Sync with public master, checking compilation, regression tests etc.
2022-10-04 00:42:52 +00:00
Marcin Junczys-Dowmunt
2c55cdb3c0
Merged PR 25889: Fixes bad memory access problem in hashing
...
Fix bad memory access problem in hashing by using the graph allocator
2022-09-29 19:01:49 +00:00
Marcin Junczys-Dowmunt
2cd3055d76
Merged PR 25836: Check via hashing if re-syncing in local mode is required
...
* This adds GPU-side hashing to tensors (a hash based on mumurhash3)
* The hash is used to check if parameters across nodes have diverged, if yes, resync all parameters and optimizer shards. Before it would resync every N (100 or 200) updates. Now this can be skipped if nothing diverged.
2022-09-27 18:40:53 +00:00
Marcin Junczys-Dowmunt
1f2929d528
Merged PR 25733: Fused inplace ReLU and Dropout in transformer FFN layer
...
* First attempt at fused inplace ReLU and Dropout in transformer FFN layer
* Adds optional output projection to SSRU.
For large FFN blocks and dropout about 20-25% speed improvement during training.
2022-09-26 20:17:33 +00:00
Marcin Junczys-Dowmunt
cfc33f5498
only use tcmalloc_minimal
2022-09-22 15:11:33 -07:00
Marcin Junczys-Dowmunt
7d2045a907
Merged PR 25686: Loading checkpoints from main node only via MPI
...
Enables loading of model checkpoints from main node only via MPI.
Until now the checkpoint needed to present in the same location on all nodes. That could be done either via writing to a shared filesystem (problematic due to bad syncing) or by manual copying to the same local location, e.g. /tmp on each node (while writing only happened to one main location).
Now, marian can resume training from only one location on the main node. The remaining nodes do not need to have access. E.g. local /tmp on the main node can be used, or race conditons on shared storage are avoided.
Also avoids creating files for logging on more than one node. This is a bit wonky, done via environment variable lookup.
2022-09-21 20:39:54 +00:00
Marcin Junczys-Dowmunt
76964791ad
Merged PR 23767: More principled sampling and force-decoding
...
This PR adds correct force-decoding and more principled sampling, both should now work for ensembles, batches and with beam search.
2022-09-16 22:53:08 +00:00
Roman Grundkiewicz
e13053a6f2
Merged PR 25698: Install Python 3.8 on GPU pool
...
Python >= 3.8 is required for numpy >= 1.22, which is the minimum version without vulnerability issues.
2022-09-16 09:30:10 +00:00
Roman Grundkiewicz
6f7766f837
Merged PR 25465: Choose top checkpoints from train.log for averaging
...
Added `--from-log logfile N metric asc|desc` option to `average.py`, which selects top N checkpoint paths from the provided train.log file according to the selected metric. Last 3 arguments to this option are optional. If the last argument is omitted, "asc" is assumed for perplexity and "desc" for other metrics.
2022-09-15 06:19:18 +00:00
Roman Grundkiewicz
a47912d9f1
Merged PR 25518: Upgrade Azure Pipelines to macos-12
...
macos-10.15 will become unsupported in December 2022. Changes:
* Upgrade Azure DevOps to macos-12
* Pull https://github.com/marian-nmt/sentencepiece/pull/14
* Fix clang 13 errors as in https://github.com/marian-nmt/marian-dev/pull/939
2022-09-15 06:18:42 +00:00
Roman Grundkiewicz
6b41df2a44
Version 1.11.8
2022-09-05 16:53:55 +01:00
dependabot[bot]
a5223e28ae
Bump examples from 29f4f7c to 25e8438 ( #963 )
...
Bumps [examples](https://github.com/marian-nmt/marian-examples ) from `29f4f7c` to `25e8438`.
- [Release notes](https://github.com/marian-nmt/marian-examples/releases )
- [Commits](29f4f7c380...25e8438322
2022-09-05 16:52:36 +01:00
dependabot[bot]
b6d066794e
Bump regression-tests from 4fa9ff5 to 92e116e ( #964 )
...
Bumps [regression-tests](https://github.com/marian-nmt/marian-regression-tests ) from `4fa9ff5` to `92e116e`.
- [Release notes](https://github.com/marian-nmt/marian-regression-tests/releases )
- [Commits](4fa9ff55af...92e116efa3
2022-09-05 16:52:16 +01:00
Roman Grundkiewicz
347ab4d54a
Upgrade dependencies in the documentation framework ( #965 )
...
* Upgrade lxml to 4.9.1; set docutils no higher than 0.17
* Remove mistune<2.0.3 due to potential vulnerability
* Fix badges in docs
* Update CHANGELOG
2022-09-05 16:51:48 +01:00
Roman Grundkiewicz
0afe2478aa
Upgrade workflows to ubuntu-20.04 and macos-12 ( #962 )
...
* Upgrade workflows to ubuntu-20.04 and macos-12
* Update sentencepiece module
* Install libunwind-dev on ubuntu-22.04 images
* Update simple-websocket-server module
2022-09-02 16:30:45 +01:00
Jelmer
7d654603bf
Fix guaranteed YAML::InvalidNode when compiled with COMPILE_CPU=Off ( #944 )
2022-09-02 12:25:42 +01:00
dependabot[bot]
bf5eafab0c
Bump src/3rd_party/intgemm from a05a2e5 to 0eda93a ( #933 )
...
Bumps [src/3rd_party/intgemm](https://github.com/marian-nmt/intgemm ) from `a05a2e5` to `0eda93a`.
- [Release notes](https://github.com/marian-nmt/intgemm/releases )
- [Commits](a05a2e51ab...0eda93a95a
2022-09-02 12:22:54 +01:00
Graeme Nail
650cf19e5a
Update Catch2 from 2.10.1 to 2.13.9 ( #941 )
...
* Update Catch2 from 2.10.1 to 2.13.9
* Update CHANGELOG
2022-09-02 10:36:43 +01:00
Nikolay Bogoychev
3bd281c6c9
Fix clang 13.0.1 ( #939 )
...
Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
2022-09-02 10:36:15 +01:00
KOLANICH
6250cd88bb
Fixed some warnings on clang 15 that are promoted into errors ( #936 )
2022-09-02 09:04:23 +01:00
Roman Grundkiewicz
f9a1ed10ce
Add a workflow compiling Marian using clang-14 ( #940 )
...
* Add a cpu-only compilation using clang-14
* Always install gcc/g++
* Use ubuntu-20.04 image
* Define clang variable in all jobs
2022-09-02 07:19:33 +01:00
Roman Grundkiewicz
5d466bc367
Merged PR 25507: Upgrade Azure Pipelines to ubuntu-20.04
...
Ubuntu-18.04 will not be supported after October 2022.
2022-09-02 05:55:20 +00:00
Alex Muzio
a90950ea25
Merged PR 25154: Add model shapes flag to model_info.py script
...
Add model shapes flag to model_info.py script through `--matrix-shapes` flag
This will print something like:
```
...
encoder_l6_ffn_W1 (1024, 4096)
encoder_l6_ffn_W2 (4096, 1024)
encoder_l6_ffn_b1 (1, 4096)
encoder_l6_ffn_b2 (1, 1024)
encoder_l6_ffn_ffn_ln_bias (1, 1024)
encoder_l6_ffn_ffn_ln_scale (1, 1024)
encoder_l6_self_Wk (1024, 1024)
encoder_l6_self_Wo (1024, 1024)
encoder_l6_self_Wo_ln_bias (1, 1024)
encoder_l6_self_Wo_ln_scale (1, 1024)
encoder_l6_self_Wq (1024, 1024)
encoder_l6_self_Wv (1024, 1024)
encoder_l6_self_bk (1, 1024)
encoder_l6_self_bo (1, 1024)
encoder_l6_self_bq (1, 1024)
encoder_l6_self_bv (1, 1024)
special:model.yml (1264,)
```
2022-08-10 22:23:47 +00:00
Graeme Nail
e27da62393
Directory listing in Ubuntu and macOS workflows ( #938 )
2022-06-06 13:32:58 +01:00
Roman Grundkiewicz
c5081df93f
Merged PR 24111: Remove external reference to Docker images
...
The reference to docker.io triggers a security warning (https://eng.ms/docs/more/containers-secure-supply-chain ) making our pipelines flashing orange, which cover the real status of regression testing. This PR simply replaced the external reference to an internal mirror (https://eng.ms/docs/more/containers-secure-supply-chain/approved-images ).
2022-05-31 15:31:39 +00:00
Roman Grundkiewicz
5df240f534
Update status badges ( #935 )
2022-05-31 12:38:47 +01:00
Marcin Junczys-Dowmunt
042ed8f2e2
Merged PR 24072: Revert changes to transformer caching
...
This PR reverts changes to transformer caching (public PR https://github.com/marian-nmt/marian-dev/pull/881 )
It seems to cause catastrophic memory leaks or incorrect de-allocation during decoding.
2022-05-30 07:27:15 +00:00
Marcin Junczys-Dowmunt
f3e1efe731
merge with internal master
2022-05-26 06:28:06 -07:00
Graeme Nail
95720ae19f
Update NVIDIA CUDA signing key for CI; fix for building docs ( #932 )
...
* Update NVIDIA CUDA signing key for CI
* Constrain Jinja2 to build docs
2022-05-18 11:11:28 +01:00
Roman Grundkiewicz
704a323142
Merged PR 22799: Running regression tests on Azure Pipelines
...
This PR adds an Azure Pipeline for running regression tests on an Azure Hosted GPU Pool.
It currently run on Ubuntu 18.04, GCC 8, CUDA 11.1, a single Nvidia M60 GPU device (Maxwell).
The pipeline needs to be started manually: go to "Pipelines", then "Marian GPU Pool", click "Run pipeline", select the branch, click "Run".
2022-05-13 07:30:36 +00:00
Roman Grundkiewicz
e0e3287a3b
Merged PR 23840: Update CUDA installation script for Ubuntu
...
Updates CUDA deb/key fetching
https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/
2022-05-12 16:23:58 +00:00
