mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-09-11 17:25:31 +03:00
Summary: Pull Request resolved: https://github.com/facebookresearch/pytext/pull/804 Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/746 Pull Request resolved: https://github.com/pytorch/fairseq/pull/894 Adding an implementation of the sparse transformer to multi-head attention using the fixed attention pattern specified https://arxiv.org/pdf/1904.10509.pdf. The sparse_mask masks out words using -inf; after softmax, -inf becomes 0. Thus, a mask does not need to be re-calculated and re-applied when multiplying attn_weights and values. Four inputs are added to the config: sparse, is_bidirectional, stride, expressivity. If we are using the sparse transformer, is_bidirectional, stride, and expressivity must be specified (there are defaults). If is_bidirectional is False, the mask values using the fixed attention pattern described in the paper. If is_bidirectional is True, subset one includes all values in the current stride window and a summary from every stride window--all other values are masked. Stride (L in the paper) controls the window size and expressivity (c in the paper) controls the size of the summary. Reviewed By: borguz Differential Revision: D16042988 fbshipit-source-id: c59166dc7cfe89187a256e4076000c2458842fd5 |
||
---|---|---|
.. | ||
__init__.py | ||
test_average_checkpoints.py | ||
test_backtranslation_dataset.py | ||
test_binaries.py | ||
test_character_token_embedder.py | ||
test_concat_dataset.py | ||
test_convtbc.py | ||
test_dictionary.py | ||
test_iterators.py | ||
test_label_smoothing.py | ||
test_memory_efficient_fp16.py | ||
test_multi_corpus_sampled_dataset.py | ||
test_noising.py | ||
test_reproducibility.py | ||
test_sequence_generator.py | ||
test_sequence_scorer.py | ||
test_sparse_multihead_attention.py | ||
test_token_block_dataset.py | ||
test_train.py | ||
test_utils.py | ||
utils.py |