2018-09-03 18:57:53 +03:00
|
|
|
.. role:: hidden
|
|
|
|
|
:class: hidden-section
|
|
|
|
|
|
|
|
|
|
.. module:: fairseq.data
|
|
|
|
|
|
|
|
|
|
Data Loading and Utilities
|
|
|
|
|
==========================
|
|
|
|
|
|
|
|
|
|
.. _datasets:
|
|
|
|
|
|
|
|
|
|
Datasets
|
|
|
|
|
--------
|
|
|
|
|
|
|
|
|
|
**Datasets** define the data format and provide helpers for creating
|
|
|
|
|
mini-batches.
|
|
|
|
|
|
|
|
|
|
.. autoclass:: fairseq.data.FairseqDataset
|
|
|
|
|
:members:
|
|
|
|
|
.. autoclass:: fairseq.data.LanguagePairDataset
|
|
|
|
|
:members:
|
|
|
|
|
.. autoclass:: fairseq.data.MonolingualDataset
|
|
|
|
|
:members:
|
|
|
|
|
|
2019-01-05 07:00:49 +03:00
|
|
|
**Helper Datasets**
|
|
|
|
|
|
|
|
|
|
These datasets wrap other :class:`fairseq.data.FairseqDataset` instances and
|
|
|
|
|
provide additional functionality:
|
|
|
|
|
|
|
|
|
|
.. autoclass:: fairseq.data.BacktranslationDataset
|
|
|
|
|
:members:
|
|
|
|
|
.. autoclass:: fairseq.data.ConcatDataset
|
|
|
|
|
:members:
|
2019-09-23 22:25:44 +03:00
|
|
|
.. autoclass:: fairseq.data.ResamplingDataset
|
|
|
|
|
:members:
|
2019-01-05 07:00:49 +03:00
|
|
|
.. autoclass:: fairseq.data.RoundRobinZipDatasets
|
|
|
|
|
:members:
|
|
|
|
|
.. autoclass:: fairseq.data.TransformEosDataset
|
|
|
|
|
:members:
|
|
|
|
|
|
2018-09-03 18:57:53 +03:00
|
|
|
|
|
|
|
|
Dictionary
|
|
|
|
|
----------
|
|
|
|
|
|
|
|
|
|
.. autoclass:: fairseq.data.Dictionary
|
|
|
|
|
:members:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Iterators
|
|
|
|
|
---------
|
|
|
|
|
|
|
|
|
|
.. autoclass:: fairseq.data.CountingIterator
|
|
|
|
|
:members:
|
|
|
|
|
.. autoclass:: fairseq.data.EpochBatchIterator
|
|
|
|
|
:members:
|
2018-09-06 21:46:54 +03:00
|
|
|
.. autoclass:: fairseq.data.GroupedIterator
|
|
|
|
|
:members:
|
2018-09-03 18:57:53 +03:00
|
|
|
.. autoclass:: fairseq.data.ShardedIterator
|
|
|
|
|
:members:
|
