Summary:
https://github.com/facebookresearch/fairseq/issues/4501
# Before submitting
- [x] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/main/CONTRIBUTING.md)?
- [x] Did you make sure to update the docs?
- [x] Did you write any new necessary tests?
## What does this PR do?
Fixes https://github.com/facebookresearch/fairseq/issues/4501 .
## PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
## Did you have fun?
Make sure you had fun coding �
Pull Request resolved: https://github.com/facebookresearch/fairseq/pull/4511
Reviewed By: cbalioglu
Differential Revision: D37415937
Pulled By: dianaml0
fbshipit-source-id: bed8cd8e6db23f161fc96e32a0b14535b2373a43