Summary:
Cythonized token block dataset code, it's `> 100x` faster. Token block for entire `bookwiki+CC+stories+openweb` is just ~`39.9` seconds.
TODO:
1) I think, I can make it 2x more faster.
2) cleanup.
EDIT History:
~~First pass at parellelizing `token_block_dataset`. The code feels somewhat complicated and cluttered.
This is 2-3x faster though on my tests on `bookwiki` dataset with both `complete` and `complete_doc` modes.
myleott Can you take a look for correctness as I am still not 100% sure that I am not missing corner cases.~~
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/834
Test Plan:
Imported from GitHub, without a `Test Plan:` line.
Test workflow: f133816198
Reviewed By: myleott
Differential Revision: D16970257
Pulled By: myleott
fbshipit-source-id: ec45a308193c9e9f3e7075336c15df4723228d6f
Summary:
The previous BSD+PATENTS license was controversial. We have been
approved to relicense fairseq under the MIT license.
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/786
Differential Revision: D16560654
Pulled By: myleott
fbshipit-source-id: f78b1beb4f2895dd7b9bfc79f5f952a2bfb94034
Summary:
No major API changes since the last release. Cutting a new release since we'll be merging significant (possibly breaking) changes to logging, data loading and the masked LM implementation soon.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/891
Differential Revision: D16377132
Pulled By: myleott
fbshipit-source-id: f1cb88e671ccd510e53334d0f449fe18585268c7
Summary:
Fairseq wouldn't install on macOS.
A workaround was found here: https://github.com/pytorch/fairseq/issues/289
This is now automatic in setup.py, maybe be there's a cleaner way to do it.
I checked that it compiles fine on Linux and macOS.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/862
Differential Revision: D16142105
Pulled By: myleott
fbshipit-source-id: 998ac7781d7a1ac047f4f9239c1fe16eab4be0dd
Summary:
Notable (possibly breaking) changes:
- d45db80: Remove checkpoint utility functions from utils.py into checkpoint_utils.py
- f2563c2: Move LM definitions into separate files
- dffb167: Updates to model API:
- `FairseqModel` -> `FairseqEncoderDecoderModel`
- add `FairseqDecoder.extract_features` and `FairseqDecoder.output_layer`
- `encoder_out_dict` -> `encoder_out`
- rm unused `remove_head` functions
- 34726d5: Move `distributed_init` into `DistributedFairseqModel`
- cf17068: Simplify distributed launch by automatically launching multiprocessing on each node for all visible GPUs (allows launching just one job per node instead of one per GPU)
- d45db80: Change default LR scheduler from `reduce_lr_on_plateau` to `fixed`
- 96ac28d: Rename `--sampling-temperature` -> `--temperature`
- fc1a19a: Deprecate dummy batches
- a1c997b: Add memory mapped datasets
- 0add50c: Allow cycling over multiple datasets, where each one becomes an "epoch"
Plus many additional features and bugfixes
Pull Request resolved: https://github.com/pytorch/fairseq/pull/817
Differential Revision: D15913844
Pulled By: myleott
fbshipit-source-id: d5b5d678efdd9dd3e4d7ca848ddcf1ec2b21bf6b
Summary:
See #467. Ping myleott to review.
This is a work-related contribution. Ping lark to review.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/794
Differential Revision: D15756816
Pulled By: myleott
fbshipit-source-id: 6dce3ff3a713bf5f60e5782bc260b2ca9d2c0a9b
- no more FP16Trainer, we just have an FP16Optimizer wrapper
- most of the distributed code is moved to a new wrapper class called DistributedFairseqModel, which behaves like DistributedDataParallel and a FairseqModel at the same time
- Trainer now requires an extra dummy_batch argument at initialization, which we do fwd/bwd on when there's an uneven number of batches per worker. We hide the gradients from these dummy batches by multiplying the loss by 0
- Trainer.train_step now takes a list of samples, which will allow cleaner --update-freq
This PR includes breaking API changes to modularize fairseq-py and adds support for distributed training across multiple nodes.
Changes:
- c7033ef: add support for distributed training! See updated README for usage.
- e016299: modularize fairseq-py, adding support for register_model, register_criterion, register_optimizer, etc.
- 154e440: update LSTM implementation to use PackedSequence objects in the encoder, better following best practices and improving perf
- 90c2973 and 1da6265: improve unit test coverage
Release notes:
- 5c7f495: Added simple LSTM model with input feeding and attention
- 6e4b7e2: Refactored model definitions and incremental generation to be cleaner
- 7ae79c1: Split interactive generation out of generate.py and into a new binary: interactive.py
- 19a3865: Subtle correctness fix in beam search decoder. Previously, for a beam size of k, we might emit a hypotheses
if the <eos> was among the top 2*k candidates. Now we only emit hypotheses for which the <eos> is among the
top-k candidates. This may subtly change generation results, and in the case of k=1 we will now produce
strictly greedy outputs.
- 97d7fcb: Fixed bug in padding direction, where previously we right-padded the source and left-padded the target. We
now left-pad the source and right-pad the target. This should not effect existing trained models, but may
change (usually improves) the quality of new models.
- f442f89: Add support for batching based on the number of sentences (`--max-sentences`) in addition to the number of
tokens (`--max-tokens`). When batching by the number of sentences, one can optionally normalize the gradients
by the number of sentences with `--sentence-avg` (the default is to normalize by the number of tokens).
- c6d6256: Add `--log-format` option and JSON logger
