mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-09-11 17:25:31 +03:00
bbb4120b00
Summary: The `preprocess.py` script has been refactored in order to: 1. Use the `options` module for command line arguments parsing. This will give to `preprocess.py` the ability to load custom modules with `--user-dir` flag (already implemented to all other binaries) 2. Dictionary loading and building code has moved to Task implementation. This allows custom Dictionary classes to be used during the data generation step. Pull Request resolved: https://github.com/pytorch/fairseq/pull/448 Differential Revision: D13674819 Pulled By: myleott fbshipit-source-id: b40648a98ed6c08284577e5ec25876e018d8c822 |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_average_checkpoints.py | ||
| test_backtranslation_dataset.py | ||
| test_binaries.py | ||
| test_character_token_embedder.py | ||
| test_convtbc.py | ||
| test_dictionary.py | ||
| test_iterators.py | ||
| test_label_smoothing.py | ||
| test_noising.py | ||
| test_reproducibility.py | ||
| test_sequence_generator.py | ||
| test_sequence_scorer.py | ||
| test_token_block_dataset.py | ||
| test_train.py | ||
| test_utils.py | ||
| utils.py | ||