mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-08-16 20:10:40 +03:00
8feccf9441
Summary: Adds Exponential moving average (EMA) model for Kaizen semi-supervised training https://arxiv.org/abs/2106.07759 1. Add `ema.store_ema` to enable storing EMA. EMA will be written to extra_state in the state dict while saving checkpoint. 2. `ema.ema_start_update` to control when the EMA starts accumulating 3. Tasks can use `uses_ema` property to decide if the EMA should be passed to the task. (Default is False) 4. `load_ema_from_checkpoint` can be used to load EMA model in place of the model to be used for evalutation. Pyspeech has eval-ema option for this. ``` This module has the EMA class used to store a copy of the exponentially decayed model params. Typical usage of EMA class involves initializing an object using an existing model (random or from a seed model) and setting the config like ema_decay, ema_start_update which determine how the EMA model is updated. After every update of the model i.e. at the end of the train_step, the EMA should be updated by passing the new model to the EMA.step function. The EMA model state dict can be stored in the extra state under the key of "ema" and dumped into a checkpoint and loaded. The EMA object can be passed to tasks by setting task.uses_ema property. EMA is a smoothed/ensemble model which might have better performance when used for inference or further fine-tuning. EMA class has a reverse function to load the EMA params into a model and use it like a regular model. ``` Reviewed By: cruvadom Differential Revision: D24238379 fbshipit-source-id: 879d3ba5070a614b7d365f9503af357001e875b2 |
||
|---|---|---|
| .. | ||
| distributed | ||
| gpu | ||
| speech_recognition | ||
| __init__.py | ||
| test_activation_checkpointing.py | ||
| test_amp_optimizer.py | ||
| test_average_checkpoints.py | ||
| test_backtranslation_dataset.py | ||
| test_binaries.py | ||
| test_character_token_embedder.py | ||
| test_checkpoint_utils.py | ||
| test_concat_dataset.py | ||
| test_constraints.py | ||
| test_convtbc.py | ||
| test_data_utils.py | ||
| test_dataclass_utils.py | ||
| test_dataset.py | ||
| test_dictionary.py | ||
| test_ema.py | ||
| test_export.py | ||
| test_file_chunker_utils.py | ||
| test_file_io.py | ||
| test_fp16_optimizer.py | ||
| test_huffman.py | ||
| test_inference_dropout.py | ||
| test_iopath.py | ||
| test_iterators.py | ||
| test_label_smoothing.py | ||
| test_lm_context_window.py | ||
| test_lstm_jitable.py | ||
| test_memory_efficient_fp16.py | ||
| test_metrics.py | ||
| test_multi_corpus_dataset.py | ||
| test_multi_corpus_sampled_dataset.py | ||
| test_multihead_attention.py | ||
| test_noising.py | ||
| test_online_backtranslation.py | ||
| test_plasma_utils.py | ||
| test_reproducibility.py | ||
| test_resampling_dataset.py | ||
| test_roberta.py | ||
| test_sequence_generator.py | ||
| test_sequence_scorer.py | ||
| test_sparse_multihead_attention.py | ||
| test_token_block_dataset.py | ||
| test_train.py | ||
| test_transformer.py | ||
| test_utils.py | ||
| test_valid_subset_checks.py | ||
| utils.py | ||