mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-09-11 17:25:31 +03:00
0add50c2e0
Summary: Co-authored-by: myleott <myleott@fb.com> Changing `data` to be `str` with colon separated list for loading sharded datasets. This change is useful for loading large datasets that cannot fit into, memory. The large dataset can be sharded and then each shard is loaded in one epoch in roudrobin manner. For example, if there are `5` shards of data and `10` epochs then the shards will be iterated upon `[0, 1, 2, 3, 4, 0, 1, 2, 3, 4]`. myleott We need to look into `translation.py` as it currently already expects a list and then concats the datasets. Pull Request resolved: https://github.com/pytorch/fairseq/pull/696 Differential Revision: D15214049 fbshipit-source-id: 03e43a7b69c7aefada2ca668abf1eac1969fe013 |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_average_checkpoints.py | ||
| test_backtranslation_dataset.py | ||
| test_binaries.py | ||
| test_character_token_embedder.py | ||
| test_convtbc.py | ||
| test_dictionary.py | ||
| test_iterators.py | ||
| test_label_smoothing.py | ||
| test_multi_corpus_sampled_dataset.py | ||
| test_noising.py | ||
| test_reproducibility.py | ||
| test_sequence_generator.py | ||
| test_sequence_scorer.py | ||
| test_token_block_dataset.py | ||
| test_train.py | ||
| test_utils.py | ||
| utils.py | ||