mirror of
https://github.com/facebookresearch/fairseq.git
synced 2024-08-16 20:10:40 +03:00
Summary: This is the equivalent to PR https://github.com/fairinternal/fairseq-py/issues/2697 but on top of main instead of gshard (cherry-picked and merged the squash): * reorganize preprocess.py code a bit * use Binarizers objects in the multiprocess code * clean up the make_binary * multiprocess logic * learn to count * format and doc string * add basic test for vocab binarizer * generalize to one line * move multiprocess in binarizer Testing: ``` python -m fairseq_cli.preprocess --only-source --trainpref ~/fixathon/small_vocab_test/train.in --destdir ~/fixathon/small_vocab_test/data-bin.cherry --workers 20 python -m fairseq_cli.preprocess --only-source --trainpref ~/fixathon/small_vocab_test/train.in --destdir ~/fixathon/small_vocab_test/data-bin.main --workers 20 ``` ``` md5sum ~/fixathon/small_vocab_test/data-bin.cherry/train.bin == md5sum ~/fixathon/small_vocab_test/data-bin.main/train.bin ``` ``` diff ~/fixathon/small_vocab_test/data-bin.main/dict.txt ~/fixathon/small_vocab_test/data-bin.cherry/dict.tx ``` Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/2738 Reviewed By: sshleifer, dianaml0 Differential Revision: D32830875 Pulled By: Mortimerp9 fbshipit-source-id: e7463d5cdd96a877691bf39666daa319ebb3dcb8 |
||
---|---|---|
.. | ||
distributed | ||
gpu | ||
speech | ||
speech_recognition | ||
__init__.py | ||
test_activation_checkpointing.py | ||
test_amp_optimizer.py | ||
test_average_checkpoints.py | ||
test_backtranslation_dataset.py | ||
test_binaries.py | ||
test_binarizer.py | ||
test_character_token_embedder.py | ||
test_checkpoint_utils.py | ||
test_concat_dataset.py | ||
test_constraints.py | ||
test_convtbc.py | ||
test_data_utils.py | ||
test_dataclass_utils.py | ||
test_dataset.py | ||
test_dictionary.py | ||
test_ema.py | ||
test_espnet_multihead_attention.py | ||
test_export.py | ||
test_file_chunker_utils.py | ||
test_file_io.py | ||
test_fp16_optimizer.py | ||
test_hf_hub.py | ||
test_huffman.py | ||
test_inference_dropout.py | ||
test_iopath.py | ||
test_iterators.py | ||
test_label_smoothing.py | ||
test_lm_context_window.py | ||
test_lstm_jitable.py | ||
test_memory_efficient_fp16.py | ||
test_metrics.py | ||
test_multi_corpus_dataset.py | ||
test_multi_corpus_sampled_dataset.py | ||
test_multihead_attention.py | ||
test_noising.py | ||
test_online_backtranslation.py | ||
test_plasma_utils.py | ||
test_positional_encoding.py | ||
test_reproducibility.py | ||
test_resampling_dataset.py | ||
test_roberta.py | ||
test_rotary_positional_embedding.py | ||
test_sequence_generator.py | ||
test_sequence_scorer.py | ||
test_sparse_multihead_attention.py | ||
test_token_block_dataset.py | ||
test_train.py | ||
test_transformer.py | ||
test_utils.py | ||
test_valid_subset_checks.py | ||
utils.py |