mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-08-16 20:10:40 +03:00
1cc8e95cec
Summary: Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1268 We previously had a memory leak when using sharded datasets. In particular, each sharded dataset is a new FairseqDataset instance, and the cache is keyed by the `dataset` instance. Since we never clear the cache, this would eventually cause the system to run out of CPU RAM. This diff disables caching when using sharded datasets. Note that we also change the signature to `get_batch_iterator`, which needs to propagate to many places. We previously avoided this update when adding `data_buffer_size`, so I'm also adding that everywhere. Reviewed By: ngoyal2707 Differential Revision: D23319135 fbshipit-source-id: 6bcd6aee141ad9cc234448c49106a8dbf8ea1800 |
||
|---|---|---|
| .. | ||
| _static | ||
| command_line_tools.rst | ||
| conf.py | ||
| criterions.rst | ||
| data.rst | ||
| docutils.conf | ||
| fairseq_logo.png | ||
| fairseq.gif | ||
| getting_started.rst | ||
| index.rst | ||
| lr_scheduler.rst | ||
| make.bat | ||
| Makefile | ||
| models.rst | ||
| modules.rst | ||
| optim.rst | ||
| overview.rst | ||
| requirements.txt | ||
| tasks.rst | ||
| tutorial_classifying_names.rst | ||
| tutorial_simple_lstm.rst | ||
