Summary:
# Before submitting
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
- [ ] Did you make sure to update the docs?
- [x] Did you write any new necessary tests?
## What does this PR do?
Fixes https://github.com/pytorch/fairseq/issues/2027 .
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/pytorch/fairseq/pull/2028
Reviewed By: ngoyal2707
Differential Revision: D21134466
Pulled By: myleott
fbshipit-source-id: 070d7f971bc8d88ec1ca43d52797e2f0b07fb6af
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/2016
It is to update Fairseq LSTM to jitable version
Reviewed By: cndn
Differential Revision: D20937370
fbshipit-source-id: 26f677fcb58bbeaa507d303e9a81060ff78f0502
Summary:
With 1 GPU, BMUF is no longer required. Instead, it just works like a simple model training.
Add unit test too for Single GPU BMUF
Reviewed By: jay-mahadeokar
Differential Revision: D21033060
fbshipit-source-id: 9030187c05d49548222c8d1e2fe9534a6c6c4389
Summary: Moving ``_test_save_and_load()` up top top-level for possible reuse across classes.
Reviewed By: cndn
Differential Revision: D20971566
fbshipit-source-id: b9d9c554d03f26cd43eee9f209e1c1367679af72
Summary: The fix in MHA is suggested by driazati, to avoid JIT compilation for if branch in MHA forward when in scripting. Without this quantization wouldn't work. Details in https://fb.workplace.com/groups/2240361332735959/permalink/626166461295703/
Reviewed By: jhcross
Differential Revision: D20881076
fbshipit-source-id: b50347b45cd7dbdef02ac7b71316ba734019f57e
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1127
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1953
Script the `reorder_incremental_states` in the base FairseqModel
Remove the overwrite scriptable `reorder_incremental_states` in the TransformerModel
Change the decoder_len, since len(Tuple) is supported in Script
Relanded reverted diff D20797390
Reviewed By: myleott
Differential Revision: D20896200
fbshipit-source-id: cc4ae34f89f16007656cce6ec6f7e01b13899278
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1120
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1940
Deprecate the SequenceGenerator in Fairseq with the Scripted vision.
Pass all integration unit tests
- Copy ScriptSequenceGenerator to SequenceGenerator:
- Modified the forward_decoder to fix bug when using adaptive_softmax in `get_prob_normalize` (marked with the inline comment)
- Add support for other EnsembleModels as input arg (marked with the inline comment)
- Add `FBEnsembleModelWithFork` to support folk/join in ensemblemodel
- Add `test_fb_ensemble_model` to test folk/join feature
- Still have bugs in folk/join feature when running in the Fairseq interface (like generation and interactive). Need further investigation P128130029. cc cndn, jhcross
- Modified SequenceGenerator initialization the interface
- Clear up the codes: delete unused functions `get_normalized_probs` and `_decode`
Reland reverted diff D20685075
Reviewed By: cndn
Differential Revision: D20895977
fbshipit-source-id: 424ee318e67d5d6ffed3edb92c7fa78485ba34af
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1127
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1953
Script the `reorder_incremental_states` in the base FairseqModel
Remove the overwrite scriptable `reorder_incremental_states` in the TransformerModel
Change the decoder_len, since len(Tuple) is supported in Script
Reviewed By: myleott
Differential Revision: D20797390
fbshipit-source-id: ab29874973adc5dbd556c591942a0e071c81fc52
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1120
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1940
Deprecate the SequenceGenerator in Fairseq with the Scripted vision.
Pass all integration unit tests
- Copy ScriptSequenceGenerator to SequenceGenerator:
- Modified the forward_decoder to fix bug when using adaptive_softmax in `get_prob_normalize` (marked with the inline comment)
- Add support for other EnsembleModels as input arg (marked with the inline comment)
- Add `FBEnsembleModelWithFork` to support folk/join in ensemblemodel
- Add `test_fb_ensemble_model` to test folk/join feature
- Still have bugs in folk/join feature when running in the Fairseq interface (like generation and interactive). Need further investigation P128130029. cc cndn, jhcross
- Modified SequenceGenerator initialization the interface
- Clear up the codes: delete unused functions `get_normalized_probs` and `_decode`
Reviewed By: myleott
Differential Revision: D20685075
fbshipit-source-id: 046b76874465a70d8118a97ad670311c6ce1d1c8
Summary:
# Before submitting
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
- [ ] Did you make sure to update the docs?
- [ ] Did you write any new necessary tests?
## What does this PR do?
Fixes validation happening twice at the end of epoch after refactor. Spotted by freewym
here: b5dad3b7e0 (r38103577)
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1934
Reviewed By: myleott
Differential Revision: D20724205
Pulled By: louismartin
fbshipit-source-id: 8c26c39b9904508780e8542813797c8e1306ca80
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1125
Pull Request resolved: https://github.com/pytorch/translate/pull/695
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1927
- Switches the model to the scripted sequence generator recently implemented in fairseq. Involved making the input/ouput format of this model to conform to that in Fairseq TransformerEncoder/Decoder
- Modify the `EncoderOut` format for fairseq transformer and added optional fields needed for copy ptr decoder
- Switches to using WordEmbedding directly instead of the non scriptable EmbeddingList for src/trg embedding layer
- Small assorted syntactic changes to make it jit scriptable
- Adds a torchscriptify method for this model. Preliminarily latency seems similar to the unexported model. Also verified that the outputs match
- Currently the Roberta decoupled model is not scriptable because the base TransformerSentenceEncoder it is based on is not scriptable. We can look at adding that later
Reviewed By: einolghozati
Differential Revision: D20687247
fbshipit-source-id: 8232972bba2f1b2df4100f3c1776b6bad08a71db
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1894
Having a uniform return type for `FairseqEncoder` makes these test models function more similarly to real models.
Reviewed By: myleott, cndn
Differential Revision: D20596971
fbshipit-source-id: a744614c015af9b150f2b0ae8381b1368556f738
Summary:
# Before submitting
- [x] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
- [x] Did you make sure to update the docs?
- [x] Did you write any new necessary tests?
## What does this PR do?
Fixes https://github.com/pytorch/fairseq/issues/1830
Adds tests for RoBERTa (masked_lm, classification, single regression, multiple regression)
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1831
Reviewed By: ngoyal2707
Differential Revision: D20446010
Pulled By: myleott
fbshipit-source-id: 9f37bcedf0910d85446245d71bc234bc74c62da5
Summary:
# Before submitting
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
- [ ] Did you make sure to update the docs?
- [x] Did you write any new necessary tests?
## What does this PR do?
Fixes https://github.com/pytorch/fairseq/issues/1791.
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1792
Reviewed By: jmp84
Differential Revision: D20322704
Pulled By: myleott
fbshipit-source-id: 3cfa1bddda06b966e9dc9bc8ff183009d844b23c
Summary:
[This commit](dd1298e15f) made it so that duplicate entries in a dictionary are ignored. Unfortunately the Camembert model depends on overwriting `<unk>`, `<s>` and `</s>`.
The proposed solution here is to allow the dictionary to have entries like:
```
<unk> 999 #fairseq:overwrite
<s> 999 #fairseq:overwrite
</s> 999 #fairseq:overwrite
, 999
▁de 999
. 999
(...)
```
These will preserve the old overwriting behavior. Thus we can release a new `camembert.v0.tar.gz` with a dictionary like above and it works.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1073
Reviewed By: kahne
Differential Revision: D20284569
Pulled By: myleott
fbshipit-source-id: bf78fbff13c94bf8a6485cbdda62305ddc30c056
Summary:
# Before submitting
- [x] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
- [x] Did you make sure to update the docs?
- [x] Did you write any new necessary tests?
## What does this PR do?
Fixes https://github.com/pytorch/fairseq/issues/1672 in part (part 1: [context](https://github.com/pytorch/fairseq/pull/1714#issuecomment-587507040))
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1729
Differential Revision: D20049353
Pulled By: myleott
fbshipit-source-id: 732077a1cc339c9f7ebe26dae42a7e8d7b5a07b4
Summary:
We are somewhat inconsistent in whether we're using 0-based or 1-based indexing for epochs. This should fix things to be 0-based internally, with logging and checkpoint naming still using 1-based indexing.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1053
Reviewed By: spencerp
Differential Revision: D20160715
Pulled By: myleott
fbshipit-source-id: 4ed94f9c371e1bfe29bcfa087fa6756507d6e627
Summary:
sanitized vq-wav2vec implementation. i will also add docs to this. i have a fixed-up checkpoint that this code can load and verified that it produces same results as what we used in paper
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1029
Differential Revision: D20129246
Pulled By: alexeib
fbshipit-source-id: f72f455e0c309168e644ab86ec18c768c308da98
Summary:
1. Overwrite the base class function `get_normalized_probs` in scriptable TransformerModel
2. Change the unit test setup to match the Transformer decoder output format
3. Initialze the buffer in the simple sequence generator [WIP]
1. It is the initial step to script the sequence generator from simple scriptable version.
4. Refactor the unit test of simple sequence generator.
5. Change the input format of simple sequence generator and unit test.
Reviewed By: myleott
Differential Revision: D20017859
fbshipit-source-id: a3e93b57c22e49840e460469fa2b1c530346886d
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1653
Earlier we had some issues at pickling. Type information gets lost. Fixed in https://github.com/pytorch/pytorch/pull/32569.
These save_and_load tests are added for protection in the future.
Reviewed By: myleott
Differential Revision: D19435988
fbshipit-source-id: 560ea65ed3493bebcf394327818364b3fcd6fc92
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1011
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1620
Make Fairseq transformer scriptable. Discussion points on possible code refactoring:
(1) Original decoder output is a tuple (x, {"attn": attn, "inner_states": inner_states}). TorchScript does not support dictionary with values of different types (attn: Tensor, inner_states: List[Tensor]). Current workaround is to use [attn] for attention field and access via output["attn"][0] in downstream. This is currently used in fairspeq custom transformer code. Another (maybe) cleaner alternative is to use namedtuple for decoder output but involves tons of downstream changes too.
(2) Currently TorchScript doesn't support **kwargs. Some unused arguments might get passed in due to polymorphism. Now the only workaround I can think of is to add possible unused arguments, (e.g. line 666 in transformer.py)
Reviewed By: myleott
Differential Revision: D19234599
fbshipit-source-id: db3dd364ecf3ae14fb7ac8c0928bd0ebe250f19d
Summary:
When training with `--fp16` we usually flatten the grads since it's faster. But flat grads are not semantically equivalent for certain optimizers (e.g., Adafactor, LAMB), thus the user needed to be aware of this and set `--fp16-no-flatten-grads`. Let's raise a RuntimeError in this case instead.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1010
Differential Revision: D19575773
Pulled By: myleott
fbshipit-source-id: bac99c3026f9870e6127e0fa55f70e8a3e4507dc
Summary:
* Now that we have `FairseqIncrementalState`, we can move `get_incremental_state` and `set_incremental_state` as methods in that class, instead of having the helper functions in `utils.py`. I think this will eventually help with type checking too.
* The incremental ID logic was overly complicated, we can just use `uuid` to generate a unique ID for every instance.
* Add missing `with_incremental_state` to light/dynamic conv modules.
* Add additional unit test: `test_incremental_state_multihead_attention`
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1005
Test Plan:
* unit tests
Also confirmed this matches master:
```
$ python generate.py ~/data/data-bin/wmt16_en_de_bpe32k --path /checkpoint/myleott/s3/models/wmt16.en-de.joined-dict.transformer/model.pt --beam 4 --lenpen 0.6 --remove-bpe --quiet
(...)
2020-01-22 09:53:38 | INFO | fairseq_cli.generate | Generate test with beam=4: BLEU4 = 29.28, 60.8/35.1/22.8/15.3 (BP=0.997, ratio=0.997, syslen=62859, reflen=63078)
```
Reviewed By: cndn
Differential Revision: D19517908
Pulled By: myleott
fbshipit-source-id: a406490e342d0d30a9231bf823d3350999bda4c0
Summary:
Currently, the LSTM models in Fairseq master can only be used in an encoder/decoder setting, for example, in `class LSTMModel(FairseqEncoderDecoderModel)`. This PR adds a standalone LSTM decoder language model.
Changes:
- adds support for `LSTMDecoder` in cases where an encoder is not present, for instance, where `encoder_output_units=0`.
- fixes bugs in `LSTMDecoder` that only become apparent when using it in a standalone fashion, for example, not handling `src_lengths` as an optional argument.
- adds `class LSTMLanguageModel(FairseqLanguageModel)` for training LSTM language models.
- tests for the `LSTMLanguageModel`. Changes to the `LSTMDecoder` are handled by existing test cases.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/934
Reviewed By: myleott
Differential Revision: D18816310
Pulled By: joshim5
fbshipit-source-id: 4773695a7f5d36aa773da8a45db2e02f76c968a9
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1007
# Before submitting
- [x] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
- [x] Did you make sure to update the docs?
- [ ] Did you write any new necessary tests?
## What does this PR do?
Fixes https://github.com/pytorch/fairseq/issues/1622
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1631
Differential Revision: D19555401
Pulled By: myleott
fbshipit-source-id: c62dfc109e09a7d732a9fc73ac6feef63a8dd341
Summary:
Pull Request resolved: https://github.com/pytorch/translate/pull/683
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1612
Make SinusoidalPositionalEmbedding scriptable. Mostly adding types. The only change that affects lots of downstream code is to have max_positions as member variable instead of method.
Reviewed By: myleott
Differential Revision: D18924939
fbshipit-source-id: 2b6486563e9ec5cc34bcf11acdff9054658f4674
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1002
Pull Request resolved: https://github.com/pytorch/translate/pull/681
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1524
Make fairseq MultiheadAttention scriptable. Looking for feedbacks.
1. Add types
2. Move incremental state management logic from util functions to initializers. TorchScript in general doesn't support global dict. As a result modules with multihead attention in it would assign itself fairseq_instance_id in the initializer.
3. There might be opportunities to make assertions and annotations cleaner.
Reviewed By: myleott
Differential Revision: D18772594
fbshipit-source-id: 377aef4bbb7ef51da5b6bac9a87a6f7b03b16fe1
Summary:
* fix: mid-epoch validation metrics were previously polluting training metrics
* fix: mid-epoch metrics were not properly saved/restored in checkpoints
* added tests, both for metrics and for mid-epoch reproducibility
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1634
Differential Revision: D19470714
Pulled By: myleott
fbshipit-source-id: 491fa8d830b653cdd6a86095645aabcac758d214
Summary: This is needed to support other build environments (e.g., Windows)
Reviewed By: ngoyal2707
Differential Revision: D19409984
fbshipit-source-id: e970510781abf92f1b02d0961bc30e1210b524dd
Summary:
This PR implements a new generation strategy that we experimented with in project Pinocchio (https://github.com/fairinternal/Pinocchio), see the paper submission in: https://fburl.com/hduj2me7.
Specifically in this PR:
- added a Diverse Beam Search variant as described in https://arxiv.org/abs/1611.08562
- moved the Search object generation out of `sequence_generation.py`, which allows for limiting the number of kwargs passes around
- made sure the above changes are backward compatible based on grep - P124083926
- added test cases covering these scenarios
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/953
Test Plan:
- `python -m unittest tests.test_binaries -v`- including added test cases, see issues below for some details
- `python -m unittest tests.test_sequence_generator -v` - including added test cases
- tested locally in conjunction with the Pinocchio repo
- grepped for all instantiations of `SequenceGeneration`, made sure they're backward compatible
# Issues
- when I try to run all tests with `python -m unittest tests.test_binaries -v` command, the execution gets stuck on `test_binaries.TestTranslation.test_generation` - the test otherwise passes without problems when ran individually. Is this a known problem?
- discovered T59235948 - assigned to fairseq oncall
Reviewed By: myleott, fabiopetroni
Differential Revision: D19142394
Pulled By: ola13
fbshipit-source-id: d24543424c14a9537e7b6485951d9f841da62b07
Summary:
- Fix github issue [1393](https://github.com/pytorch/fairseq/issues/1393), [1315](https://github.com/pytorch/fairseq/issues/1315).
- Add unit test to cover training, validation and generation for multilingual model to make sure they can run without problem. (didn't test the correctness)
Reviewed By: lematt1991
Differential Revision: D19149575
fbshipit-source-id: 9ec9000d037cc5c3bd8457feb527f2305375a442
Summary: Added unit test for PathManager file io (with or without fvcore).
Reviewed By: theweiho
Differential Revision: D18880067
fbshipit-source-id: 969c2be90415d22041b8276b7a5ff264571561d0
Summary:
This diff mainly first contains the implementation for NAT-CRF model:
- Fast Structured Decoding for Sequence Models (NAT-CRF, Sun et al., 2019)
We implemented a dynamic CRF module and incorporate it into the implementation of vanilla NAT model. In order to reproduce the performance on paper.
We implemented the length beam as well as reranking from a learned autoregressive model in the iterative-refinement-generator;
We also implemented a new ensemble code which enables to do ensemble for all NAT models, not only Levenshtein Transformer itself. We refactor all the codes and move the models into ``fairseq/models/nat``.
Finally, we updated the README.md for NAT models.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/925
Differential Revision: D18738085
Pulled By: MultiPath
fbshipit-source-id: 4e421c5d52d2456fbe99e7863d715c756b1fd49b
Summary:
https://github.com/pytorch/fairseq/pull/1097 added key padding mask history in TransformerDecoderLayer, but during an edge case where only the current or only the previous key_padding_mask exists, the resulting key_padding_mask is the wrong size.
This diff adds empty columns in such a case to ensure key_padding_mask is a usable size.
Reviewed By: myleott
Differential Revision: D18224313
fbshipit-source-id: c9fb7266baf0a2d79a66704e00a5ea8bd2987ff6
Summary:
This unit test guards the bmuf code.
change:
1. distributed_init assumes we are always using cuda device which is not the case if you are using "gloo" backend on CPU machine.
Reviewed By: jay-mahadeokar
Differential Revision: D17821391
fbshipit-source-id: 28e1bb39f7a4889b1dc6bd636b7c499e55bfc69a
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/877
This PR implements guided alignment training described in "Jointly Learning to Align and Translate with Transformer Models (https://arxiv.org/abs/1909.02074)".
In summary, it allows for training selected heads of the Transformer Model with external alignments computed by Statistical Alignment Toolkits. During inference, attention probabilities from the trained heads can be used to extract reliable alignments. In our work, we did not see any regressions in the translation performance because of guided alignment training.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1095
Differential Revision: D17170337
Pulled By: myleott
fbshipit-source-id: daa418bef70324d7088dbb30aa2adf9f95774859
Summary:
This PR implements a new attention module which combines cross-attention (encoder-decoder attention) and the decoder self-attention. This work was accepted as an abstract at WeCNLP 2019 (https://www.wecnlp.ai/wecnlp-2019).
Cross+Self-Attention reduces the amount of parameter and increases the inference speed without any degradation in translation quality.
More details can be found in the attached [abstract](https://github.com/pytorch/fairseq/files/3561282/paper.pdf)
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1097
Differential Revision: D17653168
Pulled By: myleott
fbshipit-source-id: deb834c2c78a229d7418ffbfea20ba3ce252991c
Summary:
Code for our NeurIPS paper [Levenshtein Transformer](https://arxiv.org/abs/1905.11006)
* Added Levenshtein Transformer model, task and criterion class
* Added iterative NAT Transformer, insertion Transformer and CMLM Transformer model class for baselines
* Add an option for prepending BOS to dictionary class and translation task class
Reviewed By: myleott
Differential Revision: D17297372
fbshipit-source-id: 54eca60831ae95dc721c2c34e882e1810ee575c7
Summary:
As discussed with Naman earlier today. Weighted sampling with
replacement can be done on a per-epoch basis using `set_epoch()`
functionality, which generates the samples as a function of random seed
and epoch.
Additionally, `FairseqTask` needs to set the starting epoch for the
dataset at the very beginning of iterator construction.
Not yet implemented is the per-epoch iterator construction, which
is necessary to actually regenerate the batches for each epoch.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/861
Differential Revision: D17460687
Pulled By: jma127
fbshipit-source-id: 1c2a54f04ac96b3561c100a6fd66a9fccbe3c658
Summary:
Initial code for speech recognition task.
Right now only one ASR model added - https://arxiv.org/abs/1904.11660
unit test testing:
python -m unittest discover tests
also run model training with this code and obtained
5.0 test_clean | 13.4 test_other
on librispeech with pytorch/audio features
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/810
Reviewed By: cpuhrsch
Differential Revision: D16706659
Pulled By: okhonko
fbshipit-source-id: 89a5f9883e50bc0e548234287aa0ea73f7402514
Summary:
The previous BSD+PATENTS license was controversial. We have been
approved to relicense fairseq under the MIT license.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/786
Differential Revision: D16560654
Pulled By: myleott
fbshipit-source-id: f78b1beb4f2895dd7b9bfc79f5f952a2bfb94034
Summary:
Pull Request resolved: https://github.com/facebookresearch/pytext/pull/804
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/746
Pull Request resolved: https://github.com/pytorch/fairseq/pull/894
Adding an implementation of the sparse transformer to multi-head attention using the fixed attention pattern specified https://arxiv.org/pdf/1904.10509.pdf. The sparse_mask masks out words using -inf; after softmax, -inf becomes 0. Thus, a mask does not need to be re-calculated and re-applied when multiplying attn_weights and values.
Four inputs are added to the config: sparse, is_bidirectional, stride, expressivity. If we are using the sparse transformer, is_bidirectional, stride, and expressivity must be specified (there are defaults). If is_bidirectional is False, the mask values using the fixed attention pattern described in the paper. If is_bidirectional is True, subset one includes all values in the current stride window and a summary from every stride window--all other values are masked. Stride (L in the paper) controls the window size and expressivity (c in the paper) controls the size of the summary.
Reviewed By: borguz
Differential Revision: D16042988
fbshipit-source-id: c59166dc7cfe89187a256e4076000c2458842fd5
Summary:
See #467. Ping myleott to review.
This is a work-related contribution. Ping lark to review.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/794
Differential Revision: D15756816
Pulled By: myleott
fbshipit-source-id: 6dce3ff3a713bf5f60e5782bc260b2ca9d2c0a9b
Summary: We never actually load the model parameters from an XLM model when using tranformer_from_pretrained_xlm. Also, change encoder_learned_pos from True -> False
Reviewed By: liezl200
Differential Revision: D15629061
fbshipit-source-id: 759eadc88041eae94505477960de57dd78a99dcb
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/747
In https://github.com/pytorch/fairseq/pull/647, checkpoint averaging
is not Implemented correctly when it comes to shared parameters. This diff
has the right Implementation and a test case to guard future change.
Reviewed By: myleott
Differential Revision: D15402943
fbshipit-source-id: 8004836d5c2571814ea54844650618008a9ee522
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/730
Pull Request resolved: https://github.com/pytorch/translate/pull/528
Add/modify necessary functions for ConcatDataset to work in PytorchTranslateTask and replace MultiCorpusSampledDataset which doesn't support mixed batch.
Any idea on how to implement collater here for mixed batch? Now I'm just using the collater of the first dataset.
Reviewed By: liezl200
Differential Revision: D15260872
fbshipit-source-id: 14b148c506e9f8ebf4fe60a49f95444d4123d76f
Summary:
Move `load_checkpoint`, `save_checkpoint` and `reload_train` from train.py to checkpoint_utils.py
Move `get_perplexity` from train.py to utils.py.
This will make train.py lighter and allow us to reuse all this utils functionality when fairseq is used as external library.
Reviewed By: myleott
Differential Revision: D15289607
fbshipit-source-id: 4b7c95225ac22e402bcda3497811361809110df1
Summary: the old no_bias_kv argument for masked_lm models are not used. Split it into 2 arguments and expose them.
Reviewed By: myleott
Differential Revision: D15266154
fbshipit-source-id: 60b041f8370ca1d8869ed3402fb9a67d1cd8e0e8
Summary:
Following discussion in https://github.com/pytorch/fairseq/issues/574:
- Implemented MMapIndexedDataset and MMapIndexedDatasetBuilder compatible with IndexedDataset/IndexedDatasetBuilder
- Update scripts/read_binarized.py to support new MMapIndexedDataset
- Option '--raw-text' and '--lazy-load' replaced with '--dataset-impl' and moved the option definition custom task args to more high-level options.add_dataset_args() (more appropriate)
- Implemented also utils functions in indexed_dataset: make_dataset(), dataset_exists()
Pull Request resolved: https://github.com/pytorch/fairseq/pull/589
Differential Revision: D14597128
Pulled By: myleott
fbshipit-source-id: 4e92d99920cbaa52cfe5a0f1f5d9ae5c92d4268e
Summary:
Co-authored-by: myleott <myleott@fb.com>
Changing `data` to be `str` with colon separated list for loading sharded datasets. This change is useful for loading large datasets that cannot fit into, memory. The large dataset can be sharded and then each shard is loaded in one epoch in roudrobin manner.
For example, if there are `5` shards of data and `10` epochs then the shards will be iterated upon `[0, 1, 2, 3, 4, 0, 1, 2, 3, 4]`.
myleott We need to look into `translation.py` as it currently already expects a list and then concats the datasets.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/696
Differential Revision: D15214049
fbshipit-source-id: 03e43a7b69c7aefada2ca668abf1eac1969fe013
Summary:
Pull Request resolved: https://github.com/pytorch/translate/pull/508
The previous version applied the temperature after the softmax. Fix that, and
also generalize so it works with other search approaches.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/694
Differential Revision: D15175160
Pulled By: myleott
fbshipit-source-id: cc87ff0e97a8a1dd37f9983163f58a8641155ab0
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/666
Option to load the XLM weights into only the encoder or the decoder
Reviewed By: pipibjc
Differential Revision: D14881004
fbshipit-source-id: 6d0d598ea9c445ec468f71b8e855712de89a5dac
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/639
Add argument sampling_func in the constructor to enable custom sampling over a list of dataset keys. The default strategy is to sample uniformly as it did previously.
Reviewed By: liezl200
Differential Revision: D14965774
fbshipit-source-id: f3285688a9ae3729c0ba12c22254c1144d0eea9e
Summary: sequence_generator assumes that model input is 2d tensor of longs. But it can be something like 3d tensor of floats and we should be able to handle this as long as first dimension is batch size followed by source lengths.
Reviewed By: myleott
Differential Revision: D14420044
fbshipit-source-id: bf8b1e42ad1873f7b803c1a377b0af21648db015
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/541
Just a combo of a stacked pair D14057943 & D14176011,
Made this as a separete diff cause there seems to be some issue with porting a stacked change into github repo
Differential Revision: D14251048
fbshipit-source-id: 0a47f534a69d6ab2ebe035fba40fd51748cccfb8
Summary:
The `preprocess.py` script has been refactored in order to:
1. Use the `options` module for command line arguments parsing. This will give to `preprocess.py` the ability to load custom modules with `--user-dir` flag (already implemented to all other binaries)
2. Dictionary loading and building code has moved to Task implementation. This allows custom Dictionary classes to be used during the data generation step.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/448
Differential Revision: D13674819
Pulled By: myleott
fbshipit-source-id: b40648a98ed6c08284577e5ec25876e018d8c822
Summary:
Changelog:
- `4889802`: can now remove detokenize sentencepiece output with `--remove-bpe=sentencepiece` (fixes#331). Also added `--sacrebleu` for computing detokenized BLEU.
- `0d76427`: fix assertion error when training language model with dataset containing empty sentences
- minor bug and style fixes
Pull Request resolved: https://github.com/pytorch/fairseq/pull/483
Differential Revision: D13867899
Pulled By: myleott
fbshipit-source-id: 25c940b847fe270262ac8f5ac838407b3977fdda
Summary:
There are 2 ways to implement BPE:
1. use a continuation marker suffix to indicate that there is at least one more subtoken left in the word
2. use a end of word marker suffix to indicate that there is no more subtokens left in the word
This adds some logic to account for either kind of BPE marker suffix. This diff adds a corresponding test. I also refactored the test setup to reduce the number of boolean args when setting up test data.
Reviewed By: xianxl
Differential Revision: D12919428
fbshipit-source-id: 405e9f346dce6e736c1305288721dfc7b63e872a
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/340
This allows us to do a lot less copy paste when adding new word shuffle function tests
Reviewed By: xianxl
Differential Revision: D12810304
fbshipit-source-id: a56b5df093d17be2b73837897c526978cab92b70
Summary:
We'd like to resue the noising functions and DenoisingDataset in
adversarial training. However, current noising functions assume the input are
subword tokens. The goal of this diff is to extend it so the noising can be
applied to word tokens. Since we're mostly interested in the word shuffle
noising, so I only modified the WordShuffle class.
Reviewed By: liezl200
Differential Revision: D10523177
fbshipit-source-id: 1e5d27362850675010e73cd38850c890d42652ab
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/306
This uses a source dataset to generate a batch of {source: noisy source, target: original clean source} which allows us to train a denoising autoencoding component as part of a seq2seq model.
Reviewed By: xianxl
Differential Revision: D10078981
fbshipit-source-id: 026225984d4a97062ac05dc3a36e79b5c841fe9c
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/305
Previously, noising code assumed that every sentence had an EOS which had to be excluded from noising operations (since we shouldn't drop, blank, or shuffle EOS). This logic allows the noising module to handle sentences with EOS and without EOS
Reviewed By: xianxl
Differential Revision: D10114425
fbshipit-source-id: 04ec8547343eb94266bda1ac7fca3d8a1991c9f4
Summary:
If we want our parallel data to have EOS at the end of source, we keep the EOS at the end of the generated source dialect backtranslation.
If we don't want our parallel data to have EOS at the end of source, we **remove** the EOS at the end of the generated source dialect backtranslation.
Note: we always want EOS at the end of our target / reference in parallel data so our model can learn to generate a sentence at any arbitrary length. So we make sure that the original target has an EOS before returning a batch of {generated src, original target}. If our original targets in tgt dataset doesn't have an EOS, we append EOS to each tgt sample before collating.
We only do this for the purpose of collating a {generated src, original tgt} batch AFTER generating the backtranslations. We don't enforce any EOS before passing tgt to the tgt->src model for generating the backtranslation. The users of this dataset is expected to format tgt dataset examples in the correct format that the tgt->src model expects.
Reviewed By: jmp84
Differential Revision: D10157725
fbshipit-source-id: eb6a15f13c651f7c435b8db28103c9a8189845fb
Summary: This generalizes BacktranslationDataset to allow us to use any SequenceGenerator class. For example, if we want to use this model in PyTorch Translate, we can pass the following to BacktraanslationDataset init: (1) a PyTorch Translate SequenceGenerator class as generator_class and (2) the appropriate args for initializing that class as kwargs.
Reviewed By: xianxl
Differential Revision: D10156552
fbshipit-source-id: 0495d825bf4727da96d0d9a40dc434135ff3486c
Summary:
Using argparse Namespace hides the actual args that are expected and makes code harder to read.
Note the difference in style for the args list
def __init__(
self,
tgt_dataset,
tgt_dict,
backtranslation_model,
unkpen,
sampling,
beam,
max_len_a,
max_len_b,
):
instead of
def __init__(
self, tgt_dataset, tgt_dict, backtranslation_model, unkpen, sampling,
beam, max_len_a, max_len_b,
):
Reviewed By: dpacgopinath
Differential Revision: D10152331
fbshipit-source-id: 6539ccba09d48acf23759996b7e32fb329b3e3f6
A Task defines the data format, stores shared state (e.g., dictionaries) and provides helpers for building the model/criterion and calculating the loss.
Changes:
- Add TranslationTask and LanguageModelingTask. New tasks can be registered with @register_task decorator.
- Add EpochBatchIterator to encapsulate batching and saving/restoring dataloader position
- Remove LEFT_PAD_* constants and make them configurable per task
This implements convolutional language model from https://arxiv.org/pdf/1612.08083.pdf
There are 3 modes for constructing batches:
- token block: fill each sample with a specified number of tokens without regard for sentence delimiters - this is what was used for training in the paper
- complete: fill each sample with a specified number of tokens but make sure it contains only complete sentences (i.e. if next sentence goes over token block limit, move it to the next sample) - this was used for evaluation in the paper
- eos: one sentence per sample (skip blank lines)
some results:
GCNN-13 - GBW - 37.46
GCNN-14B - GBW - 33.88
GCNN-8 - Wiki103 - 43.76
GCNN-14 - Wiki103 - 35.66
train:
python train.py /private/home/abaevski/data/wiki103 --save-dir /tmp --fp16 --max-epoch 35 --save-interval 1 --save-interval-updates 1000 --keep-interval-updates 25 --arch fconv_lm --optimizer nag --lr 1.0 --lr-scheduler reduce_lr_on_plateau --lr-shrink 0.5 --decoder-embed-dim 280 --decoder-layers '[(850, 6)] * 3 + [(850,1)] + [(850,5)] * 4 + [(850,1)] + [(850,4)] * 3 + [(1024,4)] + [(2048, 4)]' --clip-norm 0.1 --dropout 0.2 --weight-decay 5e-06 --criterion cross_entropy --max-tokens 1024 --max-target-positions 1024 --seed 1 --log-format json --log-interval 500
eval:
python eval_lm.py ~abaevski/data/wiki103 --path '/checkpoint02/abaevski/2018-04-27/lm_wiki.fp16.mxup300000.fconv.adam.lrs=reduce_lr_on_plateau.emb280.layers(850,6)*3+(850,1)+(850,5)*4+(850,1)+(850,4)*3+(1024,1)+(2048,4).lr0.0005.clp0.1.drp0.3.wd0.0.crt=cross_entropy.mxtk2048.smptk256.seed1.ngpu8/checkpoint_last.pt'
Changes:
- 7d19e36: Add `--sampling` flag to generate.py to sample instead of doing beam search
- c777340: Add `scripts/average_checkpoints.py` to average multiple checkpoints into a combined model
- 3ea882c: Add `--max-update` option to train.py to stop training after a given number of updates
- small bugfixes for distributed training, LSTM, inverse square root LR scheduler
This PR includes breaking API changes to modularize fairseq-py and adds support for distributed training across multiple nodes.
Changes:
- c7033ef: add support for distributed training! See updated README for usage.
- e016299: modularize fairseq-py, adding support for register_model, register_criterion, register_optimizer, etc.
- 154e440: update LSTM implementation to use PackedSequence objects in the encoder, better following best practices and improving perf
- 90c2973 and 1da6265: improve unit test coverage
* Move some functionality out of FConvModel into FairseqModel base class
* Move incremental decoding functionality into FairseqIncrementalDecoder module
* Refactor positional embeddings to be more specific to FConvModel
