mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-09-11 17:25:31 +03:00
1c8ab79ca5
Summary: FUNCTIONALITY: This diff provides two core pieces of functionality - Adds training with quantization noise from "Training with Quantization Noise for Extreme Model Compression" - controlled by the "quant_noise" and "quant_noise_block_size" parameters. Added in embeddings, attention, FFN for BERT and Transformer LM training - Adds quantization with product quantization based on code from "And the bit goes down: Revisiting the quantization of neural networks" (Stock et al, 2019). This is applied to a fairseq trained model to quantize after training. TODO: -> Pierre, look at quantization code -> int4 and int8 quantization will be added soon. EVALUATED TEST CASES: 0. Training of LM and BERT models starts from scratch with no errors -> yes 1. Retrain LM from scratch with code, no quantization, reproduces Wikitext-103 LM results -> yes, see /checkpoint/angelafan/qn_open_source_noise 2. Reload previously trained LM from scratch, not trained with quant noise, reproduces Wikitext-103 LM results -> yes 3. Train LM from scratch with code, no trained with quant noise, reproduces Wikitext-103 LM results -> yes, see /checkpoint/angelafan/qn_open_source_baseline 4. Train BERT model from scratch with code, no quantization, training curve looks the same as before -> yes 5. Check wps during training and wps during inference, no large change from before -> yes 6. Check structured dropout isn't being applied at eval time -> yes 7. Works in combination with LayerDrop -> yes Pull Request resolved: https://github.com/pytorch/fairseq/pull/1896 Reviewed By: myleott Differential Revision: D20609420 Pulled By: huihuifan fbshipit-source-id: 94468dd811c4caaaef46a9fab2b8d381f9d2b955 |
||
|---|---|---|
| .. | ||
| speech_recognition | ||
| __init__.py | ||
| test_average_checkpoints.py | ||
| test_backtranslation_dataset.py | ||
| test_binaries.py | ||
| test_bmuf.py | ||
| test_character_token_embedder.py | ||
| test_concat_dataset.py | ||
| test_convtbc.py | ||
| test_dictionary.py | ||
| test_export.py | ||
| test_file_io.py | ||
| test_iterators.py | ||
| test_label_smoothing.py | ||
| test_lstm_jitable.py | ||
| test_memory_efficient_fp16.py | ||
| test_metrics.py | ||
| test_multi_corpus_sampled_dataset.py | ||
| test_multihead_attention.py | ||
| test_noising.py | ||
| test_reproducibility.py | ||
| test_resampling_dataset.py | ||
| test_sequence_generator.py | ||
| test_sequence_scorer.py | ||
| test_sparse_multihead_attention.py | ||
| test_token_block_dataset.py | ||
| test_train.py | ||
| test_utils.py | ||
| transformer_quantization_config.yaml | ||
| utils.py | ||