This fixes Azure pipelines after recent changes in Azure-hosted runners removing GCC 8 and older on some Ubuntu images. GCC is now installed explicitly via `apt-get`.
* allow to choose fine-grained CPU intrinsics on as CMake options
* inform user that e.g. -DCOMPILE_AVX2=off will be ignored with -march=native if there is compiler support
This PR extends the --allow-special feature to default vocabulary items as well. If the default vocabulary is provided with symbols ostensibly generated from the SentencePiece Byte Fallback mechanism, we suppress the control characters from that list.
This PR adds cleaner suppression of unwanted output words. We identified a situation where SPM with byte-fallback can generate random bytes with output-sampling.
That is particularly harmful when that random bytes happens to be a newline symbol. Here we suppress newline in output unless explicitly wanted.
* unit tests for binary file operations
* adjust changelog
* Set file_ in TemporaryFile for MSVC
Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
This PR updates Windows build using Visual Studio CMake compilation with Ninja. It does not affect standard VS compilation or Windows builds on Azure/GitHub CI.
List of changes:
- Fixed syntax in the script installing dependencies via vcpkg.
- Removed installation of Protobuf (already included as a submodule) and Boost 1.72 (the previous solution no longer works with new vcpkg).
- Disabled compilation of marian-server in the default setting due to Boost issues.
- Disabled compilation of NCCL in the default setting due to an error (see comment in the code).
- Updated vs/README.
Installing Boost manually in all workflows, because it has been recently removed from Azure/GitHub hosted runners. This should fix recent failures of Marian CI builds.
If --train-sets or --valid-sets are not provided, the fallback to the
hard-coded paths does not occur. This commit requires that these
entities have a non-empty value.
