Changes proposed in this pull request:
1. Clarified instructions how to build Marian on Windows in `vs/README.md`.
2. `vs/CheckOrInstallDeps.bat` does not stop if CUDA or MKL are not installed.
This PR updates Windows build using Visual Studio CMake compilation with Ninja. It does not affect standard VS compilation or Windows builds on Azure/GitHub CI.
List of changes:
- Fixed syntax in the script installing dependencies via vcpkg.
- Removed installation of Protobuf (already included as a submodule) and Boost 1.72 (the previous solution no longer works with new vcpkg).
- Disabled compilation of marian-server in the default setting due to Boost issues.
- Disabled compilation of NCCL in the default setting due to an error (see comment in the code).
- Updated vs/README.
This PR updates Windows build via CMake and build instructions. With https://github.com/marian-nmt/marian-dev/pull/676, this should be fully workable, including CUDA, FBGEMM, SentencePiece, unit tests, marian-server.
List of changes:
- Fixing compilation of marian-server on Windows via CMake
- Updating vs/CheckDeps.bat
- zlib no longer needs to be installed as it is included in 3rd_party
- Installing Boost 1.72 since newer is not supported
- Installing minimal required Boost components in CheckDeps.bat
- Installing protobuf in CheckDeps.bat
- Updating CMakeSettings.json
- Updating vs/README.md
- Development notes extracted to vs/NOTES.md
I did not update and test with CUDA, because I do not have a machine for that, but AFAIK it works properly.
Repeated memory allocation/deallocation in a multi-threaded environment caused heap contention problem which all the threads are waiting each other to synchronously allocate/deallocate memory. To resolve this, several unnecessary memory allocations are removed.
1. Remove repeated memory allocation and deallocation for the fbgemm's scratch space.
2. Remove unnecessary memory allocation in fbgemm submodule (https://github.com/marian-nmt/FBGEMM/pull/3/files)
3. Make "USE_ONNX" false as a default option in vsproj windows build
4. Improve the variable naming in int8 fbgemm code for a better readability
* adds the correct definitions for FINTEGER for Linux and Windows
* moves things around a bit in the CMakeLists.txt files to keep things more local
* fixes a few more warnings in 3rd party code
1. Change the weight matrix quantization to use 7-bit min/max quantization
-> This resolves all the overflow issue, because weight and activations are quantized by min/max range.
2. Clip fp16 quantization to avoid overflow
3. Fix windows build errors (cmake options, vcproj file)
4. int8 pack model (encoder -> fp16)
FBGEMM based Int8 model - working with the master
1. Added int8 implementation into packed_gemm.h/cpp with FBGEMM
2. Update FBGEMM library to make it work on windows
3. Split 'ispacked' into packed8 and packed16
4. Change all names for PackFp32 to PackFp16 which is more accurate
This PR enables loading and serving of offline packed models.
* Enable support for FBGEMM offline-packed models
* Added different parameters sets for different parameter element types (currently inference only)
* New types (packed*) are used for the packed memory format.
* Added CPU instruction query into the quicksand API.
Related work items: #102572
