mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-09-11 17:25:31 +03:00
573c2f4b60
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? Opensource code for Deep Transformer with Latent Depth (https://arxiv.org/pdf/2009.13102.pdf). New features and design choices made: - New feature: allow non-residual block to be weighted by sample z (generated per batch) instead of `x = residual + x`. - Design choice: move `x = residual + x` in transformer_layer.py into a function where the subclass (with latent depth) could overwrite it to `x = residual + z*x`. - New feature: allow TransformerEncoder or TransformerDecoder to have additional logits parameters which will generate the samples z. - Design choice: added subclass LatentTransformerEncoder and LatentTransformerDecoder, which has additional attributes for the logits parameters, and instantiate the corresponding LatentTransformerEncoderLayer and LatentTransformerDecoderLayer. - New feature: allow multilingual_translation task to train with latent depth (results in the paper). - Design choice: - added additional arguments in the multilingual_translation task. - added option for multilingual_transformer to use LatentTransformerEncoder and LatentTransformerDecoder besides standard TransformerEncoder. - added option in multilingual_translation task's `train_step` to generate the samples z and compute the KL (and sparsity) loss per batch. ## PR review Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged. ## Did you have fun? Make sure you had fun coding � Pull Request resolved: https://github.com/pytorch/fairseq/pull/2703 Reviewed By: myleott Differential Revision: D24155059 Pulled By: xianxl fbshipit-source-id: f3e41639429f9664ec5565839709aa857a643668 |
||
|---|---|---|
| .. | ||
| gpu | ||
| speech_recognition | ||
| __init__.py | ||
| test_average_checkpoints.py | ||
| test_backtranslation_dataset.py | ||
| test_binaries.py | ||
| test_bmuf.py | ||
| test_character_token_embedder.py | ||
| test_concat_dataset.py | ||
| test_constraints.py | ||
| test_convtbc.py | ||
| test_dictionary.py | ||
| test_export.py | ||
| test_file_io.py | ||
| test_fp16_optimizer.py | ||
| test_inference_dropout.py | ||
| test_iterators.py | ||
| test_label_smoothing.py | ||
| test_lstm_jitable.py | ||
| test_memory_efficient_fp16.py | ||
| test_metrics.py | ||
| test_multi_corpus_sampled_dataset.py | ||
| test_multihead_attention.py | ||
| test_noising.py | ||
| test_reproducibility.py | ||
| test_resampling_dataset.py | ||
| test_sequence_generator.py | ||
| test_sequence_scorer.py | ||
| test_sparse_multihead_attention.py | ||
| test_token_block_dataset.py | ||
| test_train.py | ||
| test_utils.py | ||
| utils.py | ||