Commit Graph

37 Commits

Author SHA1 Message Date
Roman Grundkiewicz
ee50d4aaea Merged PR 27051: Add an option for completely resetting validation metrics
Added `--valid-reset-all` that works as `--valid-reset-stalled` but it also resets last best saved validation metrics, which is useful for when the validation sets change for continued training.

Added new regression test: https://github.com/marian-nmt/marian-regression-tests/pull/89
2022-12-20 17:56:10 +00:00
Roman Grundkiewicz
c79dc80a2f Merged PR 26617: Update regression-tests & fix CI pipelines
Update regression-tests & fix CI pipelines
2022-11-20 13:31:10 +00:00
Roman Grundkiewicz
be1ee3fa94 Merged PR 26318: Fix incorrect envvar name in Azure Pipeline
Fix incorrect environment variable name for SAS token in Windows tests
2022-11-01 10:07:40 +00:00
Roman Grundkiewicz
a6de1b781c Merged PR 26271: Update CI pipeline triggers
Updates to the CI triggers:
- Stop running parallel CI runs, i.e. if a pipeline is running, it must finish before new runs are started.
- Exclude paths to files, which are not related to/critical the codebase
- Downloading MKL from a mirror hosting server
2022-11-01 06:26:56 +00:00
Roman Grundkiewicz
a47912d9f1 Merged PR 25518: Upgrade Azure Pipelines to macos-12
macos-10.15 will become unsupported in December 2022. Changes:
* Upgrade Azure DevOps to macos-12
* Pull https://github.com/marian-nmt/sentencepiece/pull/14
* Fix clang 13 errors as in https://github.com/marian-nmt/marian-dev/pull/939
2022-09-15 06:18:42 +00:00
Roman Grundkiewicz
5d466bc367 Merged PR 25507: Upgrade Azure Pipelines to ubuntu-20.04
Ubuntu-18.04 will not be supported after October 2022.
2022-09-02 05:55:20 +00:00
Roman Grundkiewicz
1e4e1014ed Merged PR 23415: Set Windows image back to windows-2019
This should resolve latest issues with Windows checks.
2022-04-08 17:15:56 +00:00
Marcin Junczys-Dowmunt
b3feecc82b Merged PR 22483: Make C++17 the official standard for Marian
Make C++17 the official standard for Marian
2022-02-10 16:34:23 +00:00
Marcin Junczys-Dowmunt
05ba9e4c31
add -DDETERMINISTIC=ON/OFF flag (#912)
* Add -DDETERMINISTIC=ON/OFF flag to CMake
* Use -DDETERMINISTIC=on in GitHub/Azure workflows

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
2022-02-08 10:57:20 +00:00
Marcin Junczys-Dowmunt
8da539e835 merged with master 2022-02-06 12:00:48 -08:00
Roman Grundkiewicz
e025bfb07c Merged PR 20070: Run regression tests in Azure Pipelines
The changes proposed in this pull request:
* Added regression testing with internal models into Azure Pipelines on both Windows and Ubuntu
* Created https://machinetranslation.visualstudio.com/Marian/_git/marian-prod-tests (more tests will be added over time)
* Made regression test outputs (all `.log`, `.out`, `.diff` files) available for inspection as a downloadable artifact.
* Made `--build-info` option available in CMake-based Windows builds

Warning: I tried to handle multiple cases, but some regression tests may occasionally fail, especially tests using avx2 or avx512 models, because the outputs are system/CPU dependent. I think it's better to merge this already, monitoring the stability of tests, and adding expected outputs variations if necessary, improving the coverage and stability of regression tests over time.
2021-08-06 08:02:18 +00:00
Roman Grundkiewicz
b83b06fb73 Merged PR 19914: Fix Windows Azure Pipelines
Updating vcpkg seems to fix the recent issues with Windows builds in Azure Pipelines.
2021-07-22 16:36:43 +00:00
Roman Grundkiewicz
a05124176d Merged PR 18531: Install GCC in Azure pipelines
This fixes Azure pipelines after recent changes in Azure-hosted runners removing GCC 8 and older on some Ubuntu images. GCC is now installed explicitly via `apt-get`.
2021-04-09 18:44:11 +00:00
Roman Grundkiewicz
a1aaa32c6a Merged PR 18201: Install Boost in Azure pipelines
Installing Boost manually in all workflows, because it has been recently removed from Azure/GitHub hosted runners. This should fix recent failures of Marian CI builds.
2021-03-17 17:34:09 +00:00
Roman Grundkiewicz
80b74b8441 Merged PR 17572: Add Azure pipeline with make install
A new Azure pipeline testing `make install`.
2021-02-08 16:09:28 +00:00
Roman Grundkiewicz
bd9feacf49 Merged PR 17548: Fix 'no disk space left' in Azure Pipelines jobs
Removing `build/src` after compilation seems to fix an issue with 'no disk space left' in Ubuntu 18.04 CUDA 10.2 gcc-8 pipeline that we see recently.
2021-02-05 19:15:50 +00:00
Roman Grundkiewicz
e407d49bda Merged PR 17467: Enable pipeline with CUDA 11.2
Enabling Azure job with Ubuntu 20.04, CUDA 11.1, gcc-9, static, GPU-only.
2021-02-01 19:40:46 +00:00
Roman Grundkiewicz
a6436668cc Merged PR 16429: Use Boost 1.72 in Azure pipelines
Update to Boost 1.72 and explicitly use the ubuntu-18.04 image.
2020-11-13 19:17:25 +00:00
Roman Grundkiewicz
b32144c055 Clean up Ubuntu builds 2020-09-14 14:36:04 +01:00
Roman Grundkiewicz
621be16954 Build marian-server on Windows with Boost 1.72 2020-09-14 14:23:10 +01:00
Roman Grundkiewicz
26175e8286 Set timeout for Ubuntu jobs 2020-09-14 09:59:05 +01:00
Roman Grundkiewicz
fbdff66ee0 Remove call to marian-server.exe 2020-09-11 17:53:27 +01:00
Roman Grundkiewicz
3348390f27 Use CUDA 10 in UbuntuBuildMinimal 2020-09-11 17:36:55 +01:00
Roman Grundkiewicz
9e0d6e5f3a Set CUDA_TOOLKIT_ROOT_DIR 2020-09-11 16:58:11 +01:00
Roman Grundkiewicz
20338fd0d5 Fix path to custom cmake 2020-09-11 16:50:41 +01:00
Roman Grundkiewicz
1931bbaa8c Add Ubuntu build with minimum GCC and CMake supported 2020-09-11 14:28:29 +01:00
Roman Grundkiewicz
c192be1660 Disable building marian-server in Windows CI 2020-09-11 13:43:23 +01:00
Roman Grundkiewicz
f9a069c0eb Add matrix.static to Ubuntu builds 2020-09-11 13:31:02 +01:00
Roman Grundkiewicz
976f892917 Add comment on Boost on Azure-hosted runners 2020-09-11 13:25:32 +01:00
Roman Grundkiewicz
02d15861a6 Windows builds with marian-server 2020-09-11 13:03:26 +01:00
Roman Grundkiewicz
29d805a101 Try variables.var syntax for matrix variables 2020-09-11 12:20:31 +01:00
Roman Grundkiewicz
b71c66b8a3 Add matrix.gpu to Ubuntu builds 2020-09-11 11:49:55 +01:00
Roman Grundkiewicz
96291157d1 Move 'variables' to job level 2020-09-11 11:30:27 +01:00
Roman Grundkiewicz
0f1cab6c03 Add CPU-only and GPU-only builds to Azure DevOps 2020-09-11 11:11:34 +01:00
Roman Grundkiewicz
4f8c2ed6a6 Merged PR 15232: Update Azure Pipelines
A few improvements to Azure Pipelines:
- Disabling build on Ubuntu 20.04 due to [issues with FBGEMM and GCC 9+](https://github.com/marian-nmt/marian-dev/issues/709)
- Replacing Invoke-WebRequest with wget.exe
- Cleaning environmental variables
2020-09-08 16:58:03 +00:00
Roman Grundkiewicz
428703b939 Merged PR 14978: Add Azure DevOps on Windows and with CUDA 11
Updating Azure pipelines with two builds on Windows (CMake, CPU-only and CPU+CUDA), and a build on Ubuntu 20.04 with CUDA 11.
2020-08-26 19:56:14 +00:00
Roman Grundkiewicz
e41c3dbb86 Merged PR 14928: Initial Azure Pipelines
This PR adds initial Azure Pipelines with builds on Ubuntu (with CUDA) and macOS (CPU-only).

The scripts installing CUDA are already in the public master.
2020-08-25 06:59:07 +00:00