marian-nmt/marian
1
1
Fork 0
mirror of https://github.com/marian-nmt/marian.git synced 2024-08-15 01:00:23 +03:00
Code Issues Packages Projects Releases Wiki Activity
4,894 Commits 41 Branches 10 Tags 44 MiB
76964791ad
Commit Graph

4894 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Marcin Junczys-Dowmunt
76964791ad Merged PR 23767: More principled sampling and force-decoding 
This PR adds correct force-decoding and more principled sampling, both should now work for ensembles, batches and with beam search.
 2022-09-16 22:53:08 +00:00
Roman Grundkiewicz
e13053a6f2 Merged PR 25698: Install Python 3.8 on GPU pool 
Python >= 3.8 is required for numpy >= 1.22, which is the minimum version without vulnerability issues.
 2022-09-16 09:30:10 +00:00
Roman Grundkiewicz
6f7766f837 Merged PR 25465: Choose top checkpoints from train.log for averaging 
Added `--from-log logfile N metric asc|desc` option to `average.py`, which selects top N checkpoint paths from the provided train.log file according to the selected metric. Last 3 arguments to this option are optional. If the last argument is omitted, "asc" is assumed for perplexity and "desc" for other metrics.
 2022-09-15 06:19:18 +00:00
Roman Grundkiewicz
a47912d9f1 Merged PR 25518: Upgrade Azure Pipelines to macos-12 
macos-10.15 will become unsupported in December 2022. Changes:
* Upgrade Azure DevOps to macos-12
* Pull https://github.com/marian-nmt/sentencepiece/pull/14
* Fix clang 13 errors as in https://github.com/marian-nmt/marian-dev/pull/939
 2022-09-15 06:18:42 +00:00
Roman Grundkiewicz
5d466bc367 Merged PR 25507: Upgrade Azure Pipelines to ubuntu-20.04 
Ubuntu-18.04 will not be supported after October 2022.
 2022-09-02 05:55:20 +00:00
Alex Muzio
a90950ea25 Merged PR 25154: Add model shapes flag to model_info.py script 
Add model shapes flag to model_info.py script through `--matrix-shapes` flag

This will print something like:
```
...
encoder_l6_ffn_W1 (1024, 4096)
encoder_l6_ffn_W2 (4096, 1024)
encoder_l6_ffn_b1 (1, 4096)
encoder_l6_ffn_b2 (1, 1024)
encoder_l6_ffn_ffn_ln_bias (1, 1024)
encoder_l6_ffn_ffn_ln_scale (1, 1024)
encoder_l6_self_Wk (1024, 1024)
encoder_l6_self_Wo (1024, 1024)
encoder_l6_self_Wo_ln_bias (1, 1024)
encoder_l6_self_Wo_ln_scale (1, 1024)
encoder_l6_self_Wq (1024, 1024)
encoder_l6_self_Wv (1024, 1024)
encoder_l6_self_bk (1, 1024)
encoder_l6_self_bo (1, 1024)
encoder_l6_self_bq (1, 1024)
encoder_l6_self_bv (1, 1024)
special:model.yml (1264,)
```
 2022-08-10 22:23:47 +00:00
Roman Grundkiewicz
c5081df93f Merged PR 24111: Remove external reference to Docker images 
The reference to docker.io triggers a security warning (https://eng.ms/docs/more/containers-secure-supply-chain) making our pipelines flashing orange, which cover the real status of regression testing. This PR simply replaced the external reference to an internal mirror (https://eng.ms/docs/more/containers-secure-supply-chain/approved-images).
 2022-05-31 15:31:39 +00:00
Marcin Junczys-Dowmunt
042ed8f2e2 Merged PR 24072: Revert changes to transformer caching 
This PR reverts changes to transformer caching (public PR https://github.com/marian-nmt/marian-dev/pull/881)

It seems to cause catastrophic memory leaks or incorrect de-allocation during decoding.
 2022-05-30 07:27:15 +00:00
Marcin Junczys-Dowmunt
f3e1efe731 merge with internal master 2022-05-26 06:28:06 -07:00
Graeme Nail
95720ae19f
Update NVIDIA CUDA signing key for CI; fix for building docs (#932) 
* Update NVIDIA CUDA signing key for CI
* Constrain Jinja2 to build docs
 2022-05-18 11:11:28 +01:00
Roman Grundkiewicz
704a323142 Merged PR 22799: Running regression tests on Azure Pipelines 
This PR adds an Azure Pipeline for running regression tests on an Azure Hosted GPU Pool.
It currently run on Ubuntu 18.04, GCC 8, CUDA 11.1, a single Nvidia M60 GPU device (Maxwell).

The pipeline needs to be started manually: go to "Pipelines", then "Marian GPU Pool", click "Run pipeline", select the branch, click "Run".
 2022-05-13 07:30:36 +00:00
Roman Grundkiewicz
e0e3287a3b Merged PR 23840: Update CUDA installation script for Ubuntu 
Updates CUDA deb/key fetching
https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/
 2022-05-12 16:23:58 +00:00
Marcin Junczys-Dowmunt
e4f3d0f740 add fallback option for sampling, for back-compat 2022-05-09 13:28:28 -07:00
Marcin Junczys-Dowmunt
1a74358277 Merged PR 23429: Small fixes around fp16 training and batch fitting 
This PR introduces small fixes around fp16 training and batch fitting:
* Multi-loss casts type to first loss-type before accumulation (aborted before due to missing cast)
* Throw `ShapeSizeException` if total expanded shape size exceeds numeric capacity of the maximum int value (2^31-1)
* During mini-batch-fitting, catch `ShapeSizeException` and use another sizing hint. Aborts outside mini-batch-fitting.
* Negative `--workspace -N` value allocates workspace as total available GPU memory minus N megabytes.
 2022-04-11 20:19:58 +00:00
Roman Grundkiewicz
1e4e1014ed Merged PR 23415: Set Windows image back to windows-2019 
This should resolve latest issues with Windows checks.
 2022-04-08 17:15:56 +00:00
Marcin Junczys-Dowmunt
d5c7372a67 Merged PR 23407: Fix incorrect/missing gradient accumulation for affine biases 
This PR fixes incorrect/missing gradient accumulation with delay > 1 or large effective batch size of biases of affine operations.
 2022-04-08 16:00:04 +00:00
Artur Nowakowski
23c36ec1a3
Fixed fp16 training/inference with factors-combine concat (#926) 2022-03-22 10:07:41 +00:00
dependabot[bot]
78bef7aeba
Bump src/3rd_party/sentencepiece from c307b87 to 5312a30 (#927) 
Bumps [src/3rd_party/sentencepiece](https://github.com/marian-nmt/sentencepiece) from `c307b87` to `5312a30`.
- [Release notes](https://github.com/marian-nmt/sentencepiece/releases)
- [Commits](c307b874de...5312a306c4)

---
updated-dependencies:
- dependency-name: src/3rd_party/sentencepiece
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-22 10:06:11 +00:00
dependabot[bot]
75a7a1dfd2
Bump regression-tests from 88e6382 to 4fa9ff5 (#929) 
Bumps [regression-tests](https://github.com/marian-nmt/marian-regression-tests) from `88e6382` to `4fa9ff5`.
- [Release notes](https://github.com/marian-nmt/marian-regression-tests/releases)
- [Commits](88e6382241...4fa9ff55af)

---
updated-dependencies:
- dependency-name: regression-tests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-22 08:40:11 +00:00
dependabot[bot]
c809843f14
Bump examples from 6d5921c to 29f4f7c (#928) 
Bumps [examples](https://github.com/marian-nmt/marian-examples) from `6d5921c` to `29f4f7c`.
- [Release notes](https://github.com/marian-nmt/marian-examples/releases)
- [Commits](6d5921cc7d...29f4f7c380)

---
updated-dependencies:
- dependency-name: examples
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-22 08:38:30 +00:00
Marcin Junczys-Dowmunt
16bfa0c913 Merged PR 23094: Adapt --cost-scaling to more stable setting 
This PR sets default parameters for cost-scaling to 8.f 10000 1.f 8.f, i.e. when scaling scale by 8 and do not try to automatically scale up or down. This seems most stable than variable cost-scaling with larger numbers that was the default before.
 2022-03-16 14:44:17 +00:00
Marcin Junczys-Dowmunt
310d2f42f6 Merged PR 22939: Fix case augmentation with multi-threaded reading 
This PR fixes case augmentation with multi-threaded reading. The solution is to not look at iterator::pos_ in lazy processing, rather pass it as an argument to the lazy function.
 2022-03-07 16:57:32 +00:00
Marcin Junczys-Dowmunt
adaaf087e4 better error message 2022-02-16 13:20:48 -08:00
Graeme Nail
601c9ac980
Detect fortran_order in npz (#911) 
* Fix fortran_order parsing
* Abort on non row-major NPZ entries
* Update CHANGELOG
* Update VERSION

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
 2022-02-15 13:22:49 +00:00
dependabot[bot]
58c4576e5d
Bump regression-tests from da95717 to 88e6382 (#923) 
Bumps [regression-tests](https://github.com/marian-nmt/marian-regression-tests) from `da95717` to `88e6382`.
- [Release notes](https://github.com/marian-nmt/marian-regression-tests/releases)
- [Commits](da95717d41...88e6382241)

---
updated-dependencies:
- dependency-name: regression-tests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-15 11:21:14 +00:00
Nikolay Bogoychev
8a9580b329
update the intgemm version to upstream (#924) 
Some data types got upper cased, that's why there is a larger diff than expected

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
 2022-02-15 11:18:29 +00:00
Marcin Junczys-Dowmunt
b8bf086b10 move regression-tests pointer 2022-02-11 06:04:38 -08:00
Marcin Junczys-Dowmunt
b0275e7754 merge with internal master 2022-02-11 06:03:16 -08:00
Marcin Junczys-Dowmunt
4b51dcbd06 Merged PR 22524: Optimize guided alignment training speed via sparse alignments - part 1 
This replaces dense alignment storage and training with a sparse representation. Training speed with guided alignment matches now nearly normal training speed, regaining about 25% speed.

This is no. 1 of 2 PRs. The next one will introduce a new guided-alignment training scheme with better alignment accuracy.
 2022-02-11 13:50:47 +00:00
Marcin Junczys-Dowmunt
3b21ff39c5 update VERSION and CHANGELOG 2022-02-10 08:35:49 -08:00
Marcin Junczys-Dowmunt
b3feecc82b Merged PR 22483: Make C++17 the official standard for Marian 
Make C++17 the official standard for Marian
 2022-02-10 16:34:23 +00:00
Marcin Junczys-Dowmunt
e6dbacb310 Merged PR 22490: Faster LSH top-k for CPU 
This PR replaces the top-k search from FAISS on the CPU with a more specialized version for discrete distances in sub-linear time.
 2022-02-10 16:30:21 +00:00
dependabot[bot]
8fd553e582
Bump examples from 6d5921c to 0ca966e (#919) 
Bumps [examples](https://github.com/marian-nmt/marian-examples) from `6d5921c` to `0ca966e`.
- [Release notes](https://github.com/marian-nmt/marian-examples/releases)
- [Commits](6d5921cc7d...0ca966eadd)

---
updated-dependencies:
- dependency-name: examples
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-10 14:03:37 +00:00
Roman Grundkiewicz
17e55f5a7d
Update VERSION 2022-02-10 11:20:47 +00:00
Graeme Nail
4d44627f26
PyYaml safe_load instead of load (#913) 
* pyyaml safe_load instead of load
* Update CHANGELOG
 2022-02-10 11:20:27 +00:00
dependabot[bot]
a492bc57d2
Bump regression-tests from 0716f4e to f7971b7 (#918) 
Bumps [regression-tests](https://github.com/marian-nmt/marian-regression-tests) from `0716f4e` to `f7971b7`.
- [Release notes](https://github.com/marian-nmt/marian-regression-tests/releases)
- [Commits](0716f4e012...f7971b790a)

---
updated-dependencies:
- dependency-name: regression-tests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-10 10:28:04 +00:00
Roman Grundkiewicz
73f1899307
Add dependabot for git submodules (#916) 2022-02-10 10:25:08 +00:00
Roman Grundkiewicz
b97645846a
Update release workflow (#915) 
* Add CUDA 11.x to Windows installation script
* Update release.yml workflow
 2022-02-09 18:56:56 +00:00
Graeme Nail
bcf29b8cd2
Update acknowledgements (#914) 2022-02-09 17:05:48 +00:00
Marcin Junczys-Dowmunt
f00d062189 update VERSION and CHANGELOG - Release 1.11.0 2022-02-08 08:40:33 -08:00
Graeme Nail
8e659bb5c0
Document Structure (#910) 
* Add architectural outline
* Update index
 2022-02-08 10:58:09 +00:00
Marcin Junczys-Dowmunt
05ba9e4c31
add -DDETERMINISTIC=ON/OFF flag (#912) 
* Add -DDETERMINISTIC=ON/OFF flag to CMake
* Use -DDETERMINISTIC=on in GitHub/Azure workflows

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
 2022-02-08 10:57:20 +00:00
Marcin Junczys-Dowmunt
a365bb5ce9 fix server behaviour 2022-02-07 08:09:54 -08:00
Marcin Junczys-Dowmunt
aafe8fb5ca update regression tests pointer 2022-02-07 02:36:20 -08:00
Marcin Junczys-Dowmunt
3cf9e83bac resolve conflicts 2022-02-06 12:33:58 -08:00
Marcin Junczys-Dowmunt
8da539e835 merged with master 2022-02-06 12:00:48 -08:00
Roman Grundkiewicz
266b931daa
Update list of contributors (#906) 2022-01-30 20:11:38 +00:00
Roman Grundkiewicz
07c39c7d76
Cherry picked cleaning/refeactoring patches (#905) 
Cherry-picked updates from pull request #457

Co-authored-by: Mateusz Chudyk <mateuszchudyk@gmail.com>
 2022-01-28 14:16:41 +00:00
Qianqian Zhu
71b5454b9e
Layer documentation (#892) 
* More examples for MLP layers and docs about RNN layers
* Docs about embedding layer and more doxygen code docs
* Add layer and factors docs into index.rst
* Update layer documentation
* Fix typos

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
Co-authored-by: Graeme Nail <graemenail.work@gmail.com>
 2022-01-26 15:17:38 +00:00
Roman Grundkiewicz
3b458b044e
Update VERSION 2022-01-24 15:28:37 +00:00