mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-09-11 17:25:31 +03:00
Rename references from master -> main in preparation for branch name change (#2297)
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? Fixes # (issue). ## PR review Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged. ## Did you have fun? Make sure you had fun coding � Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/2297 Reviewed By: alexeib Differential Revision: D30906090 Pulled By: dianaml0 fbshipit-source-id: 941d30db7f766c9077a1b5bb2a04680f57e2e070
This commit is contained in:
parent
f6abcc2a67
commit
5adfeaccf9
4
.github/ISSUE_TEMPLATE/bug_report.md
vendored
4
.github/ISSUE_TEMPLATE/bug_report.md
vendored
@ -19,7 +19,7 @@ Steps to reproduce the behavior (**always include the command you ran**):
|
|||||||
|
|
||||||
|
|
||||||
#### Code sample
|
#### Code sample
|
||||||
<!-- Ideally attach a minimal code sample to reproduce the decried issue.
|
<!-- Ideally attach a minimal code sample to reproduce the decried issue.
|
||||||
Minimal means having the shortest code but still preserving the bug. -->
|
Minimal means having the shortest code but still preserving the bug. -->
|
||||||
|
|
||||||
### Expected behavior
|
### Expected behavior
|
||||||
@ -28,7 +28,7 @@ Minimal means having the shortest code but still preserving the bug. -->
|
|||||||
|
|
||||||
### Environment
|
### Environment
|
||||||
|
|
||||||
- fairseq Version (e.g., 1.0 or master):
|
- fairseq Version (e.g., 1.0 or main):
|
||||||
- PyTorch Version (e.g., 1.0)
|
- PyTorch Version (e.g., 1.0)
|
||||||
- OS (e.g., Linux):
|
- OS (e.g., Linux):
|
||||||
- How you installed fairseq (`pip`, source):
|
- How you installed fairseq (`pip`, source):
|
||||||
|
10
.github/ISSUE_TEMPLATE/how-to-question.md
vendored
10
.github/ISSUE_TEMPLATE/how-to-question.md
vendored
@ -6,9 +6,9 @@ labels: 'question, needs triage'
|
|||||||
|
|
||||||
## ❓ Questions and Help
|
## ❓ Questions and Help
|
||||||
|
|
||||||
### Before asking:
|
### Before asking:
|
||||||
1. search the issues.
|
1. search the issues.
|
||||||
2. search the docs.
|
2. search the docs.
|
||||||
|
|
||||||
<!-- If you still can't find what you need: -->
|
<!-- If you still can't find what you need: -->
|
||||||
|
|
||||||
@ -16,13 +16,13 @@ labels: 'question, needs triage'
|
|||||||
|
|
||||||
#### Code
|
#### Code
|
||||||
|
|
||||||
<!-- Please paste a code snippet if your question requires it! -->
|
<!-- Please paste a code snippet if your question requires it! -->
|
||||||
|
|
||||||
#### What have you tried?
|
#### What have you tried?
|
||||||
|
|
||||||
#### What's your environment?
|
#### What's your environment?
|
||||||
|
|
||||||
- fairseq Version (e.g., 1.0 or master):
|
- fairseq Version (e.g., 1.0 or main):
|
||||||
- PyTorch Version (e.g., 1.0)
|
- PyTorch Version (e.g., 1.0)
|
||||||
- OS (e.g., Linux):
|
- OS (e.g., Linux):
|
||||||
- How you installed fairseq (`pip`, source):
|
- How you installed fairseq (`pip`, source):
|
||||||
|
10
.github/PULL_REQUEST_TEMPLATE.md
vendored
10
.github/PULL_REQUEST_TEMPLATE.md
vendored
@ -1,15 +1,15 @@
|
|||||||
# Before submitting
|
# Before submitting
|
||||||
|
|
||||||
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
|
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
|
||||||
- [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)?
|
- [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/main/CONTRIBUTING.md)?
|
||||||
- [ ] Did you make sure to update the docs?
|
- [ ] Did you make sure to update the docs?
|
||||||
- [ ] Did you write any new necessary tests?
|
- [ ] Did you write any new necessary tests?
|
||||||
|
|
||||||
## What does this PR do?
|
## What does this PR do?
|
||||||
Fixes # (issue).
|
Fixes # (issue).
|
||||||
|
|
||||||
## PR review
|
## PR review
|
||||||
Anyone in the community is free to review the PR once the tests have passed.
|
Anyone in the community is free to review the PR once the tests have passed.
|
||||||
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
|
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
|
||||||
|
|
||||||
## Did you have fun?
|
## Did you have fun?
|
||||||
|
4
.github/workflows/build.yml
vendored
4
.github/workflows/build.yml
vendored
@ -1,10 +1,10 @@
|
|||||||
name: build
|
name: build
|
||||||
|
|
||||||
on:
|
on:
|
||||||
# Trigger the workflow on push to master or any pull request
|
# Trigger the workflow on push to main or any pull request
|
||||||
push:
|
push:
|
||||||
branches:
|
branches:
|
||||||
- master
|
- main
|
||||||
pull_request:
|
pull_request:
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
|
@ -5,7 +5,7 @@ possible.
|
|||||||
## Pull Requests
|
## Pull Requests
|
||||||
We actively welcome your pull requests.
|
We actively welcome your pull requests.
|
||||||
|
|
||||||
1. Fork the repo and create your branch from `master`.
|
1. Fork the repo and create your branch from `main`.
|
||||||
2. If you've added code that should be tested, add tests.
|
2. If you've added code that should be tested, add tests.
|
||||||
3. If you've changed APIs, update the documentation.
|
3. If you've changed APIs, update the documentation.
|
||||||
4. Ensure the test suite passes.
|
4. Ensure the test suite passes.
|
||||||
|
@ -2,7 +2,7 @@
|
|||||||
<img src="docs/fairseq_logo.png" width="150">
|
<img src="docs/fairseq_logo.png" width="150">
|
||||||
<br />
|
<br />
|
||||||
<br />
|
<br />
|
||||||
<a href="https://github.com/pytorch/fairseq/blob/master/LICENSE"><img alt="MIT License" src="https://img.shields.io/badge/license-MIT-blue.svg" /></a>
|
<a href="https://github.com/pytorch/fairseq/blob/main/LICENSE"><img alt="MIT License" src="https://img.shields.io/badge/license-MIT-blue.svg" /></a>
|
||||||
<a href="https://github.com/pytorch/fairseq/releases"><img alt="Latest Release" src="https://img.shields.io/github/release/pytorch/fairseq.svg" /></a>
|
<a href="https://github.com/pytorch/fairseq/releases"><img alt="Latest Release" src="https://img.shields.io/github/release/pytorch/fairseq.svg" /></a>
|
||||||
<a href="https://github.com/pytorch/fairseq/actions?query=workflow:build"><img alt="Build Status" src="https://github.com/pytorch/fairseq/workflows/build/badge.svg" /></a>
|
<a href="https://github.com/pytorch/fairseq/actions?query=workflow:build"><img alt="Build Status" src="https://github.com/pytorch/fairseq/workflows/build/badge.svg" /></a>
|
||||||
<a href="https://fairseq.readthedocs.io/en/latest/?badge=latest"><img alt="Documentation Status" src="https://readthedocs.org/projects/fairseq/badge/?version=latest" /></a>
|
<a href="https://fairseq.readthedocs.io/en/latest/?badge=latest"><img alt="Documentation Status" src="https://readthedocs.org/projects/fairseq/badge/?version=latest" /></a>
|
||||||
@ -48,7 +48,7 @@ We provide reference implementations of various sequence modeling papers:
|
|||||||
+ [Linformer: Self-Attention with Linear Complexity (Wang et al., 2020)](examples/linformer/README.md)
|
+ [Linformer: Self-Attention with Linear Complexity (Wang et al., 2020)](examples/linformer/README.md)
|
||||||
+ [Cross-lingual Retrieval for Iterative Self-Supervised Training (Tran et al., 2020)](examples/criss/README.md)
|
+ [Cross-lingual Retrieval for Iterative Self-Supervised Training (Tran et al., 2020)](examples/criss/README.md)
|
||||||
+ [Deep Transformers with Latent Depth (Li et al., 2020)](examples/latent_depth/README.md)
|
+ [Deep Transformers with Latent Depth (Li et al., 2020)](examples/latent_depth/README.md)
|
||||||
+ [Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al., 2020)](https://arxiv.org/abs/2006.13979)
|
+ [Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al., 2020)](https://arxiv.org/abs/2006.13979)
|
||||||
+ [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (Hsu, et al., 2021)](https://arxiv.org/abs/2104.01027)
|
+ [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (Hsu, et al., 2021)](https://arxiv.org/abs/2104.01027)
|
||||||
+ [Unsupervised Speech Recognition (Baevski, et al., 2021)](https://arxiv.org/abs/2105.11084)
|
+ [Unsupervised Speech Recognition (Baevski, et al., 2021)](https://arxiv.org/abs/2105.11084)
|
||||||
* **Non-autoregressive Transformers**
|
* **Non-autoregressive Transformers**
|
||||||
@ -93,7 +93,7 @@ We provide reference implementations of various sequence modeling papers:
|
|||||||
* April 2020: [Initial model parallel support and 11B parameters unidirectional LM released](examples/megatron_11b/README.md)
|
* April 2020: [Initial model parallel support and 11B parameters unidirectional LM released](examples/megatron_11b/README.md)
|
||||||
* March 2020: [Byte-level BPE code released](examples/byte_level_bpe/README.md)
|
* March 2020: [Byte-level BPE code released](examples/byte_level_bpe/README.md)
|
||||||
* February 2020: [mBART model and code released](examples/mbart/README.md)
|
* February 2020: [mBART model and code released](examples/mbart/README.md)
|
||||||
* February 2020: [Added tutorial for back-translation](https://github.com/pytorch/fairseq/tree/master/examples/backtranslation#training-your-own-model-wmt18-english-german)
|
* February 2020: [Added tutorial for back-translation](https://github.com/pytorch/fairseq/tree/main/examples/backtranslation#training-your-own-model-wmt18-english-german)
|
||||||
* December 2019: [fairseq 0.9.0 released](https://github.com/pytorch/fairseq/releases/tag/v0.9.0)
|
* December 2019: [fairseq 0.9.0 released](https://github.com/pytorch/fairseq/releases/tag/v0.9.0)
|
||||||
* November 2019: [VizSeq released (a visual analysis toolkit for evaluating fairseq models)](https://facebookresearch.github.io/vizseq/docs/getting_started/fairseq_example)
|
* November 2019: [VizSeq released (a visual analysis toolkit for evaluating fairseq models)](https://facebookresearch.github.io/vizseq/docs/getting_started/fairseq_example)
|
||||||
* November 2019: [CamemBERT model and code released](examples/camembert/README.md)
|
* November 2019: [CamemBERT model and code released](examples/camembert/README.md)
|
||||||
|
@ -55,7 +55,7 @@ project = "fairseq"
|
|||||||
copyright = "Facebook AI Research (FAIR)"
|
copyright = "Facebook AI Research (FAIR)"
|
||||||
author = "Facebook AI Research (FAIR)"
|
author = "Facebook AI Research (FAIR)"
|
||||||
|
|
||||||
github_doc_root = "https://github.com/pytorch/fairseq/tree/master/docs/"
|
github_doc_root = "https://github.com/pytorch/fairseq/tree/main/docs/"
|
||||||
|
|
||||||
# The version info for the project you're documenting, acts as replacement for
|
# The version info for the project you're documenting, acts as replacement for
|
||||||
# |version| and |release|, also used in various other places throughout the
|
# |version| and |release|, also used in various other places throughout the
|
||||||
|
@ -4,7 +4,7 @@ Adaptive Span is a novel self-attention mechanism that can learn its optimal
|
|||||||
attention span. This allows us to extend significantly the maximum context size
|
attention span. This allows us to extend significantly the maximum context size
|
||||||
used in Transformer, while maintaining control over their memory footprint
|
used in Transformer, while maintaining control over their memory footprint
|
||||||
and computational time. It uses the Truncated BPTT technique for training,
|
and computational time. It uses the Truncated BPTT technique for training,
|
||||||
as in [transformerXL](https://github.com/pytorch/fairseq/blob/master/examples/truncated_bptt/README.md).
|
as in [transformerXL](https://github.com/pytorch/fairseq/blob/main/examples/truncated_bptt/README.md).
|
||||||
|
|
||||||
Adaptive Span was introduced by paper:
|
Adaptive Span was introduced by paper:
|
||||||
[Adaptive Attention Span in Transformers](https://arxiv.org/abs/1905.07799),
|
[Adaptive Attention Span in Transformers](https://arxiv.org/abs/1905.07799),
|
||||||
|
@ -12,7 +12,7 @@ Constrained search is enabled by adding the command-line argument `--constraints
|
|||||||
Constraints are appended to each line of input, separated by tabs. Each constraint (one or more tokens)
|
Constraints are appended to each line of input, separated by tabs. Each constraint (one or more tokens)
|
||||||
is a separate field.
|
is a separate field.
|
||||||
|
|
||||||
The following command, using [Fairseq's WMT19 German--English model](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md),
|
The following command, using [Fairseq's WMT19 German--English model](https://github.com/pytorch/fairseq/blob/main/examples/wmt19/README.md),
|
||||||
translates the sentence *Die maschinelle Übersetzung ist schwer zu kontrollieren.* with the constraints
|
translates the sentence *Die maschinelle Übersetzung ist schwer zu kontrollieren.* with the constraints
|
||||||
"hard" and "to influence".
|
"hard" and "to influence".
|
||||||
|
|
||||||
|
@ -38,7 +38,7 @@ source_sentence_L_hypo_1
|
|||||||
source_sentence_L_hypo_N
|
source_sentence_L_hypo_N
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Download the [XLMR model](https://github.com/fairinternal/fairseq-py/tree/master/examples/xlmr#pre-trained-models).
|
2. Download the [XLMR model](https://github.com/fairinternal/fairseq-py/tree/main/examples/xlmr#pre-trained-models).
|
||||||
```
|
```
|
||||||
wget https://dl.fbaipublicfiles.com/fairseq/models/xlmr.base.tar.gz
|
wget https://dl.fbaipublicfiles.com/fairseq/models/xlmr.base.tar.gz
|
||||||
tar zxvf xlmr.base.tar.gz
|
tar zxvf xlmr.base.tar.gz
|
||||||
|
@ -29,9 +29,9 @@ This framework provides a great way to utlize strong target language models trai
|
|||||||
|
|
||||||
### Training Translation Models and Language Models
|
### Training Translation Models and Language Models
|
||||||
|
|
||||||
For training Transformer models in fairseq for machine translation, refer to instructions [here](https://github.com/pytorch/fairseq/tree/master/examples/translation)
|
For training Transformer models in fairseq for machine translation, refer to instructions [here](https://github.com/pytorch/fairseq/tree/main/examples/translation)
|
||||||
|
|
||||||
For training Transformer models in fairseq for language modeling, refer to instructions [here](https://github.com/pytorch/fairseq/tree/master/examples/language_model)
|
For training Transformer models in fairseq for language modeling, refer to instructions [here](https://github.com/pytorch/fairseq/tree/main/examples/language_model)
|
||||||
|
|
||||||
### Generation with Language Model for German-English translation with fairseq
|
### Generation with Language Model for German-English translation with fairseq
|
||||||
|
|
||||||
|
@ -126,9 +126,9 @@ This model override command overrides the training parameters and updates the mo
|
|||||||
|
|
||||||
Looking to reproduce the results in the paper?
|
Looking to reproduce the results in the paper?
|
||||||
|
|
||||||
1. For Translation on WMT16 en-de, we followed this setting [here](https://github.com/pytorch/fairseq/blob/master/examples/scaling_nmt/README.md)
|
1. For Translation on WMT16 en-de, we followed this setting [here](https://github.com/pytorch/fairseq/blob/main/examples/scaling_nmt/README.md)
|
||||||
2. To train RoBERTa, we followed this setting [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta)
|
2. To train RoBERTa, we followed this setting [here](https://github.com/pytorch/fairseq/tree/main/examples/roberta)
|
||||||
3. To train Language Models on Wikitext-103, we followed this setting [here](https://github.com/pytorch/fairseq/tree/master/examples/language_model)
|
3. To train Language Models on Wikitext-103, we followed this setting [here](https://github.com/pytorch/fairseq/tree/main/examples/language_model)
|
||||||
|
|
||||||
|
|
||||||
## Tips
|
## Tips
|
||||||
|
@ -82,7 +82,7 @@ fairseq-preprocess \
|
|||||||
|
|
||||||
3. **Training Scripts**
|
3. **Training Scripts**
|
||||||
|
|
||||||
To reproduce the training of our models, we train with fairseq-py's multilingual translation [task](https://github.com/pytorch/fairseq/tree/master/examples/multilingual). If you are interested in model parallel training, also check out [fairscale](https://github.com/facebookresearch/fairscale).
|
To reproduce the training of our models, we train with fairseq-py's multilingual translation [task](https://github.com/pytorch/fairseq/tree/main/examples/multilingual). If you are interested in model parallel training, also check out [fairscale](https://github.com/facebookresearch/fairscale).
|
||||||
|
|
||||||
4. **Generation**
|
4. **Generation**
|
||||||
|
|
||||||
|
@ -17,9 +17,9 @@ This work is for training multilingual translation models with multiple bitext d
|
|||||||
- --finetune-from-model to specify the path from which to load the pretrained model
|
- --finetune-from-model to specify the path from which to load the pretrained model
|
||||||
|
|
||||||
## Preprocessing data
|
## Preprocessing data
|
||||||
Multilingual training requires a joint BPE vocab. Please follow [mBART's preprocessing steps](https://github.com/pytorch/fairseq/tree/master/examples/mbart#bpe-data) to reuse our pretrained sentence-piece model.
|
Multilingual training requires a joint BPE vocab. Please follow [mBART's preprocessing steps](https://github.com/pytorch/fairseq/tree/main/examples/mbart#bpe-data) to reuse our pretrained sentence-piece model.
|
||||||
|
|
||||||
You can also train a joint BPE model on your own dataset and then follow the steps in [[link]](https://github.com/pytorch/fairseq/tree/master/examples/translation#multilingual-translation).
|
You can also train a joint BPE model on your own dataset and then follow the steps in [[link]](https://github.com/pytorch/fairseq/tree/main/examples/translation#multilingual-translation).
|
||||||
|
|
||||||
## Training
|
## Training
|
||||||
|
|
||||||
@ -49,7 +49,7 @@ fairseq-train $path_2_data \
|
|||||||
```
|
```
|
||||||
|
|
||||||
## Finetuning
|
## Finetuning
|
||||||
We can also finetune multilingual models from a monolingual pretrained models, e.g. [mMBART](https://github.com/pytorch/fairseq/tree/master/examples/mbart).
|
We can also finetune multilingual models from a monolingual pretrained models, e.g. [mMBART](https://github.com/pytorch/fairseq/tree/main/examples/mbart).
|
||||||
```bash
|
```bash
|
||||||
lang_pairs=<language pairs to be trained, e.g. "en-cs,cs-en">
|
lang_pairs=<language pairs to be trained, e.g. "en-cs,cs-en">
|
||||||
path_2_data=<set to data path>
|
path_2_data=<set to data path>
|
||||||
|
@ -33,7 +33,7 @@ Unlike the section [Iterative Product Quantization](#iterative-product-quantizat
|
|||||||
|
|
||||||
#### Training
|
#### Training
|
||||||
|
|
||||||
Scalar quantization with Quant-Noise consists in randomly quantizing a proportion `p` of the weights during training. Scalar quantization is implemented [here](https://github.com/pytorch/fairseq/tree/master/fairseq/modules/quantization/scalar) under the form of Fake Quantization, meaning that we emulate int8 on GPU by quantizing and de-quantizing both the weights and the activations. We rely on PyTorch's [quantization primitives](https://github.com/pytorch/pytorch/tree/master/torch/quantization).
|
Scalar quantization with Quant-Noise consists in randomly quantizing a proportion `p` of the weights during training. Scalar quantization is implemented [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quantization/scalar) under the form of Fake Quantization, meaning that we emulate int8 on GPU by quantizing and de-quantizing both the weights and the activations. We rely on PyTorch's [quantization primitives](https://github.com/pytorch/pytorch/tree/master/torch/quantization).
|
||||||
|
|
||||||
To train a model with Quant-Noise, add the following flag:
|
To train a model with Quant-Noise, add the following flag:
|
||||||
```
|
```
|
||||||
@ -49,7 +49,7 @@ When evaluating a network, all quantized modules and activation hooks automatica
|
|||||||
#### Integration with your own code
|
#### Integration with your own code
|
||||||
|
|
||||||
Looking to quantize your own models with Quant-Noise + Scalar Quantization?
|
Looking to quantize your own models with Quant-Noise + Scalar Quantization?
|
||||||
- Use the function `quantize_model_` implemented [here](https://github.com/pytorch/fairseq/tree/master/fairseq/modules/quantization/scalar/utils.py) to (1) replace all your modules by their quantized counterparts and (2) add hooks to those modules to quantize the activations.
|
- Use the function `quantize_model_` implemented [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quantization/scalar/utils.py) to (1) replace all your modules by their quantized counterparts and (2) add hooks to those modules to quantize the activations.
|
||||||
- Then, perform your training as usual. Note that in `eval()` mode, the network is always fully quantized (weights and activations) by default (`p=1`).
|
- Then, perform your training as usual. Note that in `eval()` mode, the network is always fully quantized (weights and activations) by default (`p=1`).
|
||||||
|
|
||||||
|
|
||||||
@ -66,12 +66,12 @@ To train a model with Quant-Noise, add the following flags:
|
|||||||
--quant-noise-pq 0.1 --quant-noise-pq-block-size 8
|
--quant-noise-pq 0.1 --quant-noise-pq-block-size 8
|
||||||
```
|
```
|
||||||
`quant-noise-pq` controls how much dropout is applied to the blocks of the weight matrix. `quant-noise-pq-block-size` controls the size of the weight matrix blocks.
|
`quant-noise-pq` controls how much dropout is applied to the blocks of the weight matrix. `quant-noise-pq-block-size` controls the size of the weight matrix blocks.
|
||||||
We recommend training with 0.05 to 0.2 Quant-Noise, a value that worked well in our experiments. For the block-size, we recommend training with block-size of 8. Note that the block size must be a multiple of `input_features`, see the size checks [here](https://github.com/pytorch/fairseq/tree/master/fairseq/modules/quant_noise.py). Large block sizes result in higher compression ratio but may induce a loss in accuracy.
|
We recommend training with 0.05 to 0.2 Quant-Noise, a value that worked well in our experiments. For the block-size, we recommend training with block-size of 8. Note that the block size must be a multiple of `input_features`, see the size checks [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quant_noise.py). Large block sizes result in higher compression ratio but may induce a loss in accuracy.
|
||||||
|
|
||||||
We currently support training Transformer based models, such as sequence-to-sequence, language models, and BERT architectures. The `quant_noise` function [here](https://github.com/pytorch/fairseq/tree/master/fairseq/modules/quant_noise.py) wraps a module. It splits a weight matrix into blocks and applies random dropout to these blocks.
|
We currently support training Transformer based models, such as sequence-to-sequence, language models, and BERT architectures. The `quant_noise` function [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quant_noise.py) wraps a module. It splits a weight matrix into blocks and applies random dropout to these blocks.
|
||||||
In the Transformer architectures, quant-noise is applied to the input and output embeddings, the attention, and the FFN.
|
In the Transformer architectures, quant-noise is applied to the input and output embeddings, the attention, and the FFN.
|
||||||
|
|
||||||
Quant-Noise can also be combined with **LayerDrop** (see [here](https://github.com/pytorch/fairseq/tree/master/examples/layerdrop)) to add its pruning effect to the quantized model and make the model even smaller. We recommend training with LayerDrop 0.1 or 0.2.
|
Quant-Noise can also be combined with **LayerDrop** (see [here](https://github.com/pytorch/fairseq/tree/main/examples/layerdrop)) to add its pruning effect to the quantized model and make the model even smaller. We recommend training with LayerDrop 0.1 or 0.2.
|
||||||
|
|
||||||
#### Quantization
|
#### Quantization
|
||||||
|
|
||||||
@ -84,8 +84,8 @@ For the particular case of PQ, quantization is made sequentially. We recommend f
|
|||||||
#### Integration with your own code
|
#### Integration with your own code
|
||||||
|
|
||||||
Looking to quantize your own models with Quant-Noise + iPQ?
|
Looking to quantize your own models with Quant-Noise + iPQ?
|
||||||
- First wrap your modules with the `quant_noise` function [here](https://github.com/pytorch/fairseq/tree/master/fairseq/modules/quant_noise.py), which is module-agnostic and train your favorite model.
|
- First wrap your modules with the `quant_noise` function [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quant_noise.py), which is module-agnostic and train your favorite model.
|
||||||
- Then, quantize your trained model using the code [here](https://github.com/pytorch/fairseq/tree/master/fairseq/modules/quantization/pq). This can be done *without any changes to your training loop*. Below is an example code for integration.
|
- Then, quantize your trained model using the code [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quantization/pq). This can be done *without any changes to your training loop*. Below is an example code for integration.
|
||||||
Note that we tried our approach only on Transformers and various Convolutional Models such as EfficientNets.
|
Note that we tried our approach only on Transformers and various Convolutional Models such as EfficientNets.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@ -128,7 +128,7 @@ We detail below how to reproduce the state-of-the-art results in reported in the
|
|||||||
|
|
||||||
### Training with Quant-Noise
|
### Training with Quant-Noise
|
||||||
|
|
||||||
To **train** RoBERTa + QuantNoise, we followed this setting [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).
|
To **train** RoBERTa + QuantNoise, we followed this setting [here](https://github.com/pytorch/fairseq/tree/main/examples/roberta).
|
||||||
The following command can be used to train a RoBERTa Base + QuantNoise model:
|
The following command can be used to train a RoBERTa Base + QuantNoise model:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@ -158,7 +158,7 @@ fairseq-train $DATA_DIR \
|
|||||||
--quant-noise-pq 0.2 --quant-noise-pq-block-size 8 --untie-weights-roberta
|
--quant-noise-pq 0.2 --quant-noise-pq-block-size 8 --untie-weights-roberta
|
||||||
```
|
```
|
||||||
|
|
||||||
To **finetune** RoBERTa + QuantNoise, we followed this setting [here](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.glue.md).
|
To **finetune** RoBERTa + QuantNoise, we followed this setting [here](https://github.com/pytorch/fairseq/blob/main/examples/roberta/README.glue.md).
|
||||||
The following command can be used to finetune a RoBERTa Base + QuantNoise model on the RTE dataset:
|
The following command can be used to finetune a RoBERTa Base + QuantNoise model on the RTE dataset:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@ -193,7 +193,7 @@ fairseq-train /path/to/rte/data/ \
|
|||||||
--quant-noise-pq 0.2 --quant-noise-pq-block-size 8
|
--quant-noise-pq 0.2 --quant-noise-pq-block-size 8
|
||||||
```
|
```
|
||||||
|
|
||||||
To **train** Language Models on Wikitext-103, we followed this setting [here](https://github.com/pytorch/fairseq/tree/master/examples/language_model).
|
To **train** Language Models on Wikitext-103, we followed this setting [here](https://github.com/pytorch/fairseq/tree/main/examples/language_model).
|
||||||
The following command can be used to train a Transformer + QuantNoise model on Wikitext-103:
|
The following command can be used to train a Transformer + QuantNoise model on Wikitext-103:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
@ -8,13 +8,13 @@ RoBERTa iterates on BERT's pretraining procedure, including training the model l
|
|||||||
|
|
||||||
### What's New:
|
### What's New:
|
||||||
|
|
||||||
- December 2020: German model (GottBERT) is available: [GottBERT](https://github.com/pytorch/fairseq/tree/master/examples/gottbert).
|
- December 2020: German model (GottBERT) is available: [GottBERT](https://github.com/pytorch/fairseq/tree/main/examples/gottbert).
|
||||||
- January 2020: Italian model (UmBERTo) is available from Musixmatch Research: [UmBERTo](https://github.com/musixmatchresearch/umberto).
|
- January 2020: Italian model (UmBERTo) is available from Musixmatch Research: [UmBERTo](https://github.com/musixmatchresearch/umberto).
|
||||||
- November 2019: French model (CamemBERT) is available: [CamemBERT](https://github.com/pytorch/fairseq/tree/master/examples/camembert).
|
- November 2019: French model (CamemBERT) is available: [CamemBERT](https://github.com/pytorch/fairseq/tree/main/examples/camembert).
|
||||||
- November 2019: Multilingual encoder (XLM-RoBERTa) is available: [XLM-R](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).
|
- November 2019: Multilingual encoder (XLM-RoBERTa) is available: [XLM-R](https://github.com/pytorch/fairseq/tree/main/examples/xlmr).
|
||||||
- September 2019: TensorFlow and TPU support via the [transformers library](https://github.com/huggingface/transformers).
|
- September 2019: TensorFlow and TPU support via the [transformers library](https://github.com/huggingface/transformers).
|
||||||
- August 2019: RoBERTa is now supported in the [pytorch-transformers library](https://github.com/huggingface/pytorch-transformers).
|
- August 2019: RoBERTa is now supported in the [pytorch-transformers library](https://github.com/huggingface/pytorch-transformers).
|
||||||
- August 2019: Added [tutorial for finetuning on WinoGrande](https://github.com/pytorch/fairseq/tree/master/examples/roberta/wsc#roberta-training-on-winogrande-dataset).
|
- August 2019: Added [tutorial for finetuning on WinoGrande](https://github.com/pytorch/fairseq/tree/main/examples/roberta/wsc#roberta-training-on-winogrande-dataset).
|
||||||
- August 2019: Added [tutorial for pretraining RoBERTa using your own data](README.pretraining.md).
|
- August 2019: Added [tutorial for pretraining RoBERTa using your own data](README.pretraining.md).
|
||||||
|
|
||||||
## Pre-trained models
|
## Pre-trained models
|
||||||
|
@ -96,4 +96,4 @@ print('Accuracy: ' + str(ncorrect / float(nsamples)))
|
|||||||
```
|
```
|
||||||
|
|
||||||
The above snippet is not batched, which makes it quite slow. See [instructions
|
The above snippet is not batched, which makes it quite slow. See [instructions
|
||||||
for batched prediction with RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/roberta#batched-prediction).
|
for batched prediction with RoBERTa](https://github.com/pytorch/fairseq/tree/main/examples/roberta#batched-prediction).
|
||||||
|
@ -40,7 +40,7 @@ For more results on probing tasks, please refer to [our paper](https://arxiv.org
|
|||||||
|
|
||||||
## Example Usage
|
## Example Usage
|
||||||
|
|
||||||
Follow the same usage as in [RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/roberta) to load and test your models:
|
Follow the same usage as in [RoBERTa](https://github.com/pytorch/fairseq/tree/main/examples/roberta) to load and test your models:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# Download roberta.base.shuffle.n1 model
|
# Download roberta.base.shuffle.n1 model
|
||||||
@ -53,11 +53,11 @@ roberta = RoBERTaModel.from_pretrained('/path/to/roberta.base.shuffle.n1', check
|
|||||||
roberta.eval() # disable dropout (or leave in train mode to finetune)
|
roberta.eval() # disable dropout (or leave in train mode to finetune)
|
||||||
```
|
```
|
||||||
|
|
||||||
**Note**: The model trained without positional embeddings (`roberta.base.nopos`) is a modified `RoBERTa` model, where the positional embeddings are not used. Thus, the typical `from_pretrained` method on fairseq version of RoBERTa will not be able to load the above model weights. To do so, construct a new `RoBERTaModel` object by setting the flag `use_positional_embeddings` to `False` (or [in the latest code](https://github.com/pytorch/fairseq/blob/master/fairseq/models/roberta/model.py#L543), set `no_token_positional_embeddings` to `True`), and then load the individual weights.
|
**Note**: The model trained without positional embeddings (`roberta.base.nopos`) is a modified `RoBERTa` model, where the positional embeddings are not used. Thus, the typical `from_pretrained` method on fairseq version of RoBERTa will not be able to load the above model weights. To do so, construct a new `RoBERTaModel` object by setting the flag `use_positional_embeddings` to `False` (or [in the latest code](https://github.com/pytorch/fairseq/blob/main/fairseq/models/roberta/model.py#L543), set `no_token_positional_embeddings` to `True`), and then load the individual weights.
|
||||||
|
|
||||||
## Fine-tuning Evaluation
|
## Fine-tuning Evaluation
|
||||||
|
|
||||||
We provide the trained fine-tuned models on MNLI here for each model above for quick evaluation (1 seed for each model). Please refer to [finetuning details](README.finetuning.md) for the parameters of these models. Follow [RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/roberta) instructions to evaluate these models.
|
We provide the trained fine-tuned models on MNLI here for each model above for quick evaluation (1 seed for each model). Please refer to [finetuning details](README.finetuning.md) for the parameters of these models. Follow [RoBERTa](https://github.com/pytorch/fairseq/tree/main/examples/roberta) instructions to evaluate these models.
|
||||||
|
|
||||||
| Model | MNLI M Dev Accuracy | Link |
|
| Model | MNLI M Dev Accuracy | Link |
|
||||||
| :----------------------------------------- | :------------------ | :--------------------------------------------------------------------------------------------------------------- |
|
| :----------------------------------------- | :------------------ | :--------------------------------------------------------------------------------------------------------------- |
|
||||||
|
@ -38,7 +38,7 @@ For your convenience, we provide pre-computed
|
|||||||
[force-alignment](https://dl.fbaipublicfiles.com/fairseq/s2/ljspeech_mfa.zip) from
|
[force-alignment](https://dl.fbaipublicfiles.com/fairseq/s2/ljspeech_mfa.zip) from
|
||||||
[Montreal Forced Aligner](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner) and
|
[Montreal Forced Aligner](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner) and
|
||||||
[pseudo-text units](s3://dl.fbaipublicfiles.com/fairseq/s2/ljspeech_hubert.tsv) from
|
[pseudo-text units](s3://dl.fbaipublicfiles.com/fairseq/s2/ljspeech_hubert.tsv) from
|
||||||
[HuBERT](https://github.com/pytorch/fairseq/tree/master/examples/hubert). You can also generate them by yourself using
|
[HuBERT](https://github.com/pytorch/fairseq/tree/main/examples/hubert). You can also generate them by yourself using
|
||||||
a different software or model.
|
a different software or model.
|
||||||
|
|
||||||
|
|
||||||
@ -106,7 +106,7 @@ use `--sample-rate 16000` for `get_eval_manifest.py`.
|
|||||||
|
|
||||||
|
|
||||||
#### WER/CER metric
|
#### WER/CER metric
|
||||||
We use wav2vec 2.0 ASR model as example. [Download](https://github.com/pytorch/fairseq/tree/master/examples/wav2vec)
|
We use wav2vec 2.0 ASR model as example. [Download](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec)
|
||||||
the model checkpoint and dictionary, then compute WER/CER with
|
the model checkpoint and dictionary, then compute WER/CER with
|
||||||
```bash
|
```bash
|
||||||
python -m examples.speech_synthesis.evaluation.eval_asr \
|
python -m examples.speech_synthesis.evaluation.eval_asr \
|
||||||
|
@ -3,7 +3,7 @@
|
|||||||
* [Paper](https://arxiv.org/abs/2102.01192)
|
* [Paper](https://arxiv.org/abs/2102.01192)
|
||||||
* [Demo](https://speechbot.github.io/gslm/index.html)
|
* [Demo](https://speechbot.github.io/gslm/index.html)
|
||||||
|
|
||||||
We build and evaluate generative speech2speech systems using [Log Mel Filtebank](https://pytorch.org/audio/stable/compliance.kaldi.html#fbank), [Modified CPC](https://github.com/facebookresearch/CPC_audio), [HuBERT Base](https://github.com/pytorch/fairseq/tree/master/examples/hubert) and [Wav2Vec 2.0 Large](https://github.com/pytorch/fairseq/tree/master/examples/wav2vec). Our system is composed of three components, namely, *speech2unit*, *ulm* and *unit2speech*. We explain about models and usage of these components in their respective sub-directories. See the links below.
|
We build and evaluate generative speech2speech systems using [Log Mel Filtebank](https://pytorch.org/audio/stable/compliance.kaldi.html#fbank), [Modified CPC](https://github.com/facebookresearch/CPC_audio), [HuBERT Base](https://github.com/pytorch/fairseq/tree/main/examples/hubert) and [Wav2Vec 2.0 Large](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec). Our system is composed of three components, namely, *speech2unit*, *ulm* and *unit2speech*. We explain about models and usage of these components in their respective sub-directories. See the links below.
|
||||||
|
|
||||||
## Speech to Unit Model (speech2unit)
|
## Speech to Unit Model (speech2unit)
|
||||||
Speech to unit model is used for quantizing raw speech into learned discrete speech units. [More details](speech2unit)
|
Speech to unit model is used for quantizing raw speech into learned discrete speech units. [More details](speech2unit)
|
||||||
@ -18,4 +18,4 @@ Unit to speech model is used for synthesizing speech from discrete speech units.
|
|||||||
We show how to compute ASR based metrics as well as zero-shot metrics proposed in our paper [here](metrics).
|
We show how to compute ASR based metrics as well as zero-shot metrics proposed in our paper [here](metrics).
|
||||||
|
|
||||||
## Tools
|
## Tools
|
||||||
We share two tools to resynthesize a given spoken utterance, and generate novel spoken language given a spoken prompt. [More detail](tools)
|
We share two tools to resynthesize a given spoken utterance, and generate novel spoken language given a spoken prompt. [More detail](tools)
|
||||||
|
@ -1,6 +1,6 @@
|
|||||||
# wav2vec Unsupervised (wav2vec-U)
|
# wav2vec Unsupervised (wav2vec-U)
|
||||||
|
|
||||||
Wav2vec Unsupervised (wav2vec-U) is a framework for building speech recognition systems without any labeled training data as described in [Unsupervised Speech Recognition (Baevski et al., 2021)](https://ai.facebook.com/research/publications/unsupervised-speech-recognition). The model takes as input wav2vec 2.0 or XLSR representations (see [pretrained models](https://github.com/pytorch/fairseq/blob/master/examples/wav2vec)) as well as unlabeled speech and text data.
|
Wav2vec Unsupervised (wav2vec-U) is a framework for building speech recognition systems without any labeled training data as described in [Unsupervised Speech Recognition (Baevski et al., 2021)](https://ai.facebook.com/research/publications/unsupervised-speech-recognition). The model takes as input wav2vec 2.0 or XLSR representations (see [pretrained models](https://github.com/pytorch/fairseq/blob/main/examples/wav2vec)) as well as unlabeled speech and text data.
|
||||||
|
|
||||||
The wav2vec-U training procedure consists of three consecutive main steps:
|
The wav2vec-U training procedure consists of three consecutive main steps:
|
||||||
* Preparation of speech representations and text data
|
* Preparation of speech representations and text data
|
||||||
@ -8,7 +8,7 @@ Wav2vec Unsupervised (wav2vec-U) is a framework for building speech recognition
|
|||||||
* Iterative self-training + Kaldi LM-decoding
|
* Iterative self-training + Kaldi LM-decoding
|
||||||
|
|
||||||
## Preparation of speech and text data
|
## Preparation of speech and text data
|
||||||
Similar to [wav2vec 2.0](https://github.com/pytorch/fairseq/blob/master/examples/wav2vec/README.md), data folders contain {train,valid,test}.{tsv,wrd,phn} files, where audio paths are stored in tsv files, and word, letter or phoneme transcriptions are stored in .{wrd,ltr,phn}.
|
Similar to [wav2vec 2.0](https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/README.md), data folders contain {train,valid,test}.{tsv,wrd,phn} files, where audio paths are stored in tsv files, and word, letter or phoneme transcriptions are stored in .{wrd,ltr,phn}.
|
||||||
|
|
||||||
In **/path/to/data/with_silence** you need a *train.tsv* file as well as (optionally) *{valid,test}.{tsv,wrd,phn}*. It is nice to have *10h.{tsv,phn}* files there too for reproducing the ablation study on layer selection. In **/path/to/data/without_silence** you have the same files, except *.tsv* files contain audios with silences removed using rVAD.
|
In **/path/to/data/with_silence** you need a *train.tsv* file as well as (optionally) *{valid,test}.{tsv,wrd,phn}*. It is nice to have *10h.{tsv,phn}* files there too for reproducing the ablation study on layer selection. In **/path/to/data/without_silence** you have the same files, except *.tsv* files contain audios with silences removed using rVAD.
|
||||||
|
|
||||||
|
@ -23,7 +23,7 @@ logger = logging.getLogger(__name__)
|
|||||||
class BARTHubInterface(GeneratorHubInterface):
|
class BARTHubInterface(GeneratorHubInterface):
|
||||||
"""A simple PyTorch Hub interface to BART.
|
"""A simple PyTorch Hub interface to BART.
|
||||||
|
|
||||||
Usage: https://github.com/pytorch/fairseq/tree/master/examples/bart
|
Usage: https://github.com/pytorch/fairseq/tree/main/examples/bart
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, cfg, task, model):
|
def __init__(self, cfg, task, model):
|
||||||
|
@ -14,7 +14,7 @@ from fairseq.data import encoders
|
|||||||
class RobertaHubInterface(nn.Module):
|
class RobertaHubInterface(nn.Module):
|
||||||
"""A simple PyTorch Hub interface to RoBERTa.
|
"""A simple PyTorch Hub interface to RoBERTa.
|
||||||
|
|
||||||
Usage: https://github.com/pytorch/fairseq/tree/master/examples/roberta
|
Usage: https://github.com/pytorch/fairseq/tree/main/examples/roberta
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, cfg, task, model):
|
def __init__(self, cfg, task, model):
|
||||||
|
Loading…
Reference in New Issue
Block a user