mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-09-11 17:25:31 +03:00
Summary: FUNCTIONALITY: This diff provides two core pieces of functionality - Adds training with quantization noise from "Training with Quantization Noise for Extreme Model Compression" - controlled by the "quant_noise" and "quant_noise_block_size" parameters. Added in embeddings, attention, FFN for BERT and Transformer LM training - Adds quantization with product quantization based on code from "And the bit goes down: Revisiting the quantization of neural networks" (Stock et al, 2019). This is applied to a fairseq trained model to quantize after training. TODO: -> Pierre, look at quantization code -> int4 and int8 quantization will be added soon. EVALUATED TEST CASES: 0. Training of LM and BERT models starts from scratch with no errors -> yes 1. Retrain LM from scratch with code, no quantization, reproduces Wikitext-103 LM results -> yes, see /checkpoint/angelafan/qn_open_source_noise 2. Reload previously trained LM from scratch, not trained with quant noise, reproduces Wikitext-103 LM results -> yes 3. Train LM from scratch with code, no trained with quant noise, reproduces Wikitext-103 LM results -> yes, see /checkpoint/angelafan/qn_open_source_baseline 4. Train BERT model from scratch with code, no quantization, training curve looks the same as before -> yes 5. Check wps during training and wps during inference, no large change from before -> yes 6. Check structured dropout isn't being applied at eval time -> yes 7. Works in combination with LayerDrop -> yes Pull Request resolved: https://github.com/pytorch/fairseq/pull/1896 Reviewed By: myleott Differential Revision: D20609420 Pulled By: huihuifan fbshipit-source-id: 94468dd811c4caaaef46a9fab2b8d381f9d2b955 |
||
---|---|---|
.. | ||
speech_recognition | ||
__init__.py | ||
test_average_checkpoints.py | ||
test_backtranslation_dataset.py | ||
test_binaries.py | ||
test_bmuf.py | ||
test_character_token_embedder.py | ||
test_concat_dataset.py | ||
test_convtbc.py | ||
test_dictionary.py | ||
test_export.py | ||
test_file_io.py | ||
test_iterators.py | ||
test_label_smoothing.py | ||
test_lstm_jitable.py | ||
test_memory_efficient_fp16.py | ||
test_metrics.py | ||
test_multi_corpus_sampled_dataset.py | ||
test_multihead_attention.py | ||
test_noising.py | ||
test_reproducibility.py | ||
test_resampling_dataset.py | ||
test_sequence_generator.py | ||
test_sequence_scorer.py | ||
test_sparse_multihead_attention.py | ||
test_token_block_dataset.py | ||
test_train.py | ||
test_utils.py | ||
transformer_quantization_config.yaml | ||
utils.py |