Summary:
Pull Request resolved: https://github.com/facebookresearch/fairseq/pull/4513
With some fixes to torchscript using dual copies.
Reland this diff.
Reviewed By: erichan1
Differential Revision: D37371293
fbshipit-source-id: 4fcfc4083955b6f5fc4ef8600f1b517b6ba69aae
Summary:
Context: https://fburl.com/7vdj7vhl
Backing out due to breaking our TorchScript test:
```
RuntimeError:
method cannot be used as a value:
File "/dev/shm/uid-30041/54641b26-seed-nspid4026533396_cgpid7154327-ns-4026533393/fairseq/modules/transformer_layer.py", line 307
self.in_proj_weight,
self.in_proj_bias,
self.self_attn.out_proj.weight,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
self.self_attn.out_proj.bias,
self.activation_relu_or_gelu == 2,
Stack trace:
Exception type: torch::jit::ErrorReport
```
https://fburl.com/sandcastle/4pzqemf5
Original commit changeset: 984266f850fc
Original Phabricator Diff: D37082681 (3a757d7ab2)
Differential Revision: D37303846
fbshipit-source-id: 1757ea5dae98be5beb4d08f70b0c3001d6ea336f
Summary:
Pull Request resolved: https://github.com/facebookresearch/fairseq/pull/4480
as titled and depends on D36057338
Fork the inference path inside the forward function. If loaded the checkpoint file and perform the inference, we will deploy BT. Otherwise, fairseq take the position.
In summary:
Accuracy: accuracy loss due to the fp16, the maximum diff is around 0.009. If we set it to fp32, there is no accuracy loss
Perf: the current fairseq has similar speed as vanilla version. After the enablement, the speedup is similar to standalone BT test.
With batch size=64
For V100, the speedup reaches to 1.23x
For A100, the speedup reaches to 1.38x
After enable nested tensor,
For V100, the speedup reaches to 2.46x
Reviewed By: mikekgfb
Differential Revision: D37082681
fbshipit-source-id: 984266f850fc30603e48be56e41ac2c67da080f5
Summary:
# Before submitting
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/main/CONTRIBUTING.md)?
- [ ] Did you make sure to update the docs?
- [ ] Did you write any new necessary tests?
## What does this PR do?
Fixes # (issue).
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/2678
Reviewed By: Mortimerp9
Differential Revision: D32653381
Pulled By: dianaml0
fbshipit-source-id: 2810d14867cd7d64f4d340740e2b590b82de47fe
Summary:
# Before submitting
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
- [x] Did you make sure to update the docs?
- [x] Did you write any new necessary tests?
## What does this PR do?
Fixes https://github.com/pytorch/fairseq/issues/3246
Fixes https://github.com/pytorch/fairseq/issues/3248
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/pytorch/fairseq/pull/3247
Reviewed By: myleott
Differential Revision: D26513267
Pulled By: lematt1991
fbshipit-source-id: 958de0b3a58a0dd2a56bd6c6d7fb2644a89f6746
Summary: Moving ``_test_save_and_load()` up top top-level for possible reuse across classes.
Reviewed By: cndn
Differential Revision: D20971566
fbshipit-source-id: b9d9c554d03f26cd43eee9f209e1c1367679af72
Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1653
Earlier we had some issues at pickling. Type information gets lost. Fixed in https://github.com/pytorch/pytorch/pull/32569.
These save_and_load tests are added for protection in the future.
Reviewed By: myleott
Differential Revision: D19435988
fbshipit-source-id: 560ea65ed3493bebcf394327818364b3fcd6fc92
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1011
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1620
Make Fairseq transformer scriptable. Discussion points on possible code refactoring:
(1) Original decoder output is a tuple (x, {"attn": attn, "inner_states": inner_states}). TorchScript does not support dictionary with values of different types (attn: Tensor, inner_states: List[Tensor]). Current workaround is to use [attn] for attention field and access via output["attn"][0] in downstream. This is currently used in fairspeq custom transformer code. Another (maybe) cleaner alternative is to use namedtuple for decoder output but involves tons of downstream changes too.
(2) Currently TorchScript doesn't support **kwargs. Some unused arguments might get passed in due to polymorphism. Now the only workaround I can think of is to add possible unused arguments, (e.g. line 666 in transformer.py)
Reviewed By: myleott
Differential Revision: D19234599
fbshipit-source-id: db3dd364ecf3ae14fb7ac8c0928bd0ebe250f19d
Summary:
* Now that we have `FairseqIncrementalState`, we can move `get_incremental_state` and `set_incremental_state` as methods in that class, instead of having the helper functions in `utils.py`. I think this will eventually help with type checking too.
* The incremental ID logic was overly complicated, we can just use `uuid` to generate a unique ID for every instance.
* Add missing `with_incremental_state` to light/dynamic conv modules.
* Add additional unit test: `test_incremental_state_multihead_attention`
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1005
Test Plan:
* unit tests
Also confirmed this matches master:
```
$ python generate.py ~/data/data-bin/wmt16_en_de_bpe32k --path /checkpoint/myleott/s3/models/wmt16.en-de.joined-dict.transformer/model.pt --beam 4 --lenpen 0.6 --remove-bpe --quiet
(...)
2020-01-22 09:53:38 | INFO | fairseq_cli.generate | Generate test with beam=4: BLEU4 = 29.28, 60.8/35.1/22.8/15.3 (BP=0.997, ratio=0.997, syslen=62859, reflen=63078)
```
Reviewed By: cndn
Differential Revision: D19517908
Pulled By: myleott
fbshipit-source-id: a406490e342d0d30a9231bf823d3350999bda4c0
Summary:
Pull Request resolved: https://github.com/pytorch/translate/pull/683
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1612
Make SinusoidalPositionalEmbedding scriptable. Mostly adding types. The only change that affects lots of downstream code is to have max_positions as member variable instead of method.
Reviewed By: myleott
Differential Revision: D18924939
fbshipit-source-id: 2b6486563e9ec5cc34bcf11acdff9054658f4674
Summary:
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1002
Pull Request resolved: https://github.com/pytorch/translate/pull/681
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1524
Make fairseq MultiheadAttention scriptable. Looking for feedbacks.
1. Add types
2. Move incremental state management logic from util functions to initializers. TorchScript in general doesn't support global dict. As a result modules with multihead attention in it would assign itself fairseq_instance_id in the initializer.
3. There might be opportunities to make assertions and annotations cleaner.
Reviewed By: myleott
Differential Revision: D18772594
fbshipit-source-id: 377aef4bbb7ef51da5b6bac9a87a6f7b03b16fe1
