FBGEMM based Int8 model - working with the master
1. Added int8 implementation into packed_gemm.h/cpp with FBGEMM
2. Update FBGEMM library to make it work on windows
3. Split 'ispacked' into packed8 and packed16
4. Change all names for PackFp32 to PackFp16 which is more accurate
This fixes a number of bugs in our GPU reduce-kernels that would manifest mainly for larger matrices and during back-prop. We also drop support for CUDA 8.0 to be able to take advantage of new GPU primitives introduced by NVidia in CUDA 9.0.
* Const diligence and thread safety with respect to Vocab and ShortlistGenerator
- Use pointers to const Vocab instead of Vocab in corpus_base.h and shortlist.h
- Declare some member functions of ShortlistGenerator const
- Use pointers to const ShortlistGenerator instead of ShortlistGenerator in
numerous places.
- Use a thread-local random generator in shortlist.h
* Bug fixes (missing const; instantiate *gen_) in SampledShortlistGenerator.
The guided-alignment loss has two bugs related to two different types of normalization:
* The returned `RationalLoss` should not have its denominator weighted by `guided-alignment-weight`, except in `sum` mode.
* The CE loss cannot directly use the alignment labels as ground truth in the CE loss function, since CE requires one-hot labels, while the actual labels those are multi-hot. They must be renormalized.
* Return exit code 15 (SIGTERM) after SIGTERM.
When marian receives signal SIGTERM and exits gracefully (save model & exit),
it should then exit with a non-zero exit code, to signal to any parent process
that it did not exit "naturally".
* Added explanatory comment about exiting marian_train with non-zero status after SIGTERM.
- Use pointers to const Vocab instead of Vocab in corpus_base.h and shortlist.h
- Declare some member functions of ShortlistGenerator const
- Use pointers to const ShortlistGenerator instead of ShortlistGenerator in
numerous places.
- Use a thread-local random generator in shortlist.h
This PR introduces batch-purging in Marian, i.e. whenever a virtual beam becomes inactive (empty) the entire batch entry that corresponds to that beam can be removed from the encoder and decoder neural states. The CPU-side beam search keeps tracks of the hypotheses as before, but needs to perform mappings between original and shifted batch indices.
In FastOpt we do not want to use locking during access, but that makes reference counting not thread-safe. We now use std::unique_ptr to const objects or const references everywhere. This fixes random segfaults with multi-GPU training. @TODO: clean-up option merging to make option generally immutable.
This fixes the memory increase while doing memory-mapping (we need regression tests for that in Marian, although that particular error would be hard to monitor. Maybe the current way by observing runtime behavior was just correct).
**This PR does:**
* Fixes memory-mapping for default parameter type. The buggy code was not replacing the in-memory parameter set with mmapping for the new default parameter. Now the parameter object is destroyed and replaced with a mmapable version if it is not mmapable.
* Add small cross-system library (mio) for memory-mapping, currently only to enable easy diagnostics of mmap on Linux. to be extended to a proper feature.
**Comments:**
* It seems we need to re-binarize the models for correct mmap behavior. Since we have identified a smaller bug in the binary file (incorrect byte-boundary alignment) this might be the right moment to fix this issue. This might help to stabilize the format for the future.
* The incorrect alignment needs to be addressed anyway, and currently is causing small losses in speed (about 3%).
While moving to the explicit matmul functions like `cublasGemmEx` the specific algorithm needs to be set. This seems to override math mode from `cublasSetMathMode`. Now we query explicitly and choose the algorithm accordingly.
Thie PR adds the FastOpt object and extends the Options object by
* Using YAML only as a background option container that is used for lazily updating the FastOpt object.
* The FastOpt object gets rebuilt whenever an access is about to occur and the doLazyRebuild_ flag is set.
* This will be the case whenever the underlying YAML object has been modified through the public interface.
* The YAML object itself it not accessible by the outside anymore (or only through cloning) to avoid untrackable modifications.
This seems to result in a >20% speed-up for decoding.
