mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-08-16 20:10:40 +03:00
279796224f
Summary: This is the equivalent to PR https://github.com/fairinternal/fairseq-py/issues/2697 but on top of main instead of gshard (cherry-picked and merged the squash): * reorganize preprocess.py code a bit * use Binarizers objects in the multiprocess code * clean up the make_binary * multiprocess logic * learn to count * format and doc string * add basic test for vocab binarizer * generalize to one line * move multiprocess in binarizer Testing: ``` python -m fairseq_cli.preprocess --only-source --trainpref ~/fixathon/small_vocab_test/train.in --destdir ~/fixathon/small_vocab_test/data-bin.cherry --workers 20 python -m fairseq_cli.preprocess --only-source --trainpref ~/fixathon/small_vocab_test/train.in --destdir ~/fixathon/small_vocab_test/data-bin.main --workers 20 ``` ``` md5sum ~/fixathon/small_vocab_test/data-bin.cherry/train.bin == md5sum ~/fixathon/small_vocab_test/data-bin.main/train.bin ``` ``` diff ~/fixathon/small_vocab_test/data-bin.main/dict.txt ~/fixathon/small_vocab_test/data-bin.cherry/dict.tx ``` Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/2738 Reviewed By: sshleifer, dianaml0 Differential Revision: D32830875 Pulled By: Mortimerp9 fbshipit-source-id: e7463d5cdd96a877691bf39666daa319ebb3dcb8 |
||
|---|---|---|
| .. | ||
| distributed | ||
| gpu | ||
| speech | ||
| speech_recognition | ||
| __init__.py | ||
| test_activation_checkpointing.py | ||
| test_amp_optimizer.py | ||
| test_average_checkpoints.py | ||
| test_backtranslation_dataset.py | ||
| test_binaries.py | ||
| test_binarizer.py | ||
| test_character_token_embedder.py | ||
| test_checkpoint_utils.py | ||
| test_concat_dataset.py | ||
| test_constraints.py | ||
| test_convtbc.py | ||
| test_data_utils.py | ||
| test_dataclass_utils.py | ||
| test_dataset.py | ||
| test_dictionary.py | ||
| test_ema.py | ||
| test_espnet_multihead_attention.py | ||
| test_export.py | ||
| test_file_chunker_utils.py | ||
| test_file_io.py | ||
| test_fp16_optimizer.py | ||
| test_hf_hub.py | ||
| test_huffman.py | ||
| test_inference_dropout.py | ||
| test_iopath.py | ||
| test_iterators.py | ||
| test_label_smoothing.py | ||
| test_lm_context_window.py | ||
| test_lstm_jitable.py | ||
| test_memory_efficient_fp16.py | ||
| test_metrics.py | ||
| test_multi_corpus_dataset.py | ||
| test_multi_corpus_sampled_dataset.py | ||
| test_multihead_attention.py | ||
| test_noising.py | ||
| test_online_backtranslation.py | ||
| test_plasma_utils.py | ||
| test_positional_encoding.py | ||
| test_reproducibility.py | ||
| test_resampling_dataset.py | ||
| test_roberta.py | ||
| test_rotary_positional_embedding.py | ||
| test_sequence_generator.py | ||
| test_sequence_scorer.py | ||
| test_sparse_multihead_attention.py | ||
| test_token_block_dataset.py | ||
| test_train.py | ||
| test_transformer.py | ||
| test_utils.py | ||
| test_valid_subset_checks.py | ||
| utils.py | ||