mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-08-16 20:10:40 +03:00
b3fa5100c6
Summary: Support multihead attention prune for Fairseq. For example, user can apply pruning on top of Roberta base model by specify the argument "--mha-heads-to-keep 8". Also, user needs to provide a ckpt which is already pruned so that the pruned ckpt can be loaded correctly. The idea of prune can be summarized as 1. Fine tune model (e.g. roberta encoder) on a certain datasets with regularization 2. After the model is trained. User could use get_reserve_head_index and _adaptive_prune_heads functions to get the top X heads with most importance. Then user uses the rank to prune a new roberta encoder and save the pruned ckpt manually. 3. User will fine tune the the new roberta encoder via the ckpt saved above To get rid of registering different pruned version of Roberta, I use the argument --mha-heads-to-keep to prune the Roberta model into a pruned version which matches the pruned ckpt. Reviewed By: dianaml0 Differential Revision: D32449003 fbshipit-source-id: a952fd9ad723a6dbc5c2af574c42f2e9a1fa27dc |
||
|---|---|---|
| .. | ||
| distributed | ||
| gpu | ||
| speech | ||
| speech_recognition | ||
| __init__.py | ||
| test_activation_checkpointing.py | ||
| test_amp_optimizer.py | ||
| test_average_checkpoints.py | ||
| test_backtranslation_dataset.py | ||
| test_binaries.py | ||
| test_character_token_embedder.py | ||
| test_checkpoint_utils.py | ||
| test_concat_dataset.py | ||
| test_constraints.py | ||
| test_convtbc.py | ||
| test_data_utils.py | ||
| test_dataclass_utils.py | ||
| test_dataset.py | ||
| test_dictionary.py | ||
| test_ema.py | ||
| test_espnet_multihead_attention.py | ||
| test_export.py | ||
| test_file_chunker_utils.py | ||
| test_file_io.py | ||
| test_fp16_optimizer.py | ||
| test_hf_hub.py | ||
| test_huffman.py | ||
| test_inference_dropout.py | ||
| test_iopath.py | ||
| test_iterators.py | ||
| test_label_smoothing.py | ||
| test_lm_context_window.py | ||
| test_lstm_jitable.py | ||
| test_memory_efficient_fp16.py | ||
| test_metrics.py | ||
| test_multi_corpus_dataset.py | ||
| test_multi_corpus_sampled_dataset.py | ||
| test_multihead_attention.py | ||
| test_noising.py | ||
| test_online_backtranslation.py | ||
| test_plasma_utils.py | ||
| test_positional_encoding.py | ||
| test_reproducibility.py | ||
| test_resampling_dataset.py | ||
| test_roberta.py | ||
| test_rotary_positional_embedding.py | ||
| test_sequence_generator.py | ||
| test_sequence_scorer.py | ||
| test_sparse_multihead_attention.py | ||
| test_token_block_dataset.py | ||
| test_train.py | ||
| test_transformer.py | ||
| test_utils.py | ||
| test_valid_subset_checks.py | ||
| utils.py | ||