tldr/pages/common/parallel.md
Agniva De Sarker 3da76e4150 Applying the snake_case convention throughout the repo (#967)
* Applying the snake_case convention throughout the repo

- Also removing the file extension where not needed
- Adding {{ }} on a few old pages

* Addressing concerns

- Added {{ }} where they were missed out
- Removed spaces inside {{ }}
- Reverting "file" to "filename" to make it clearer

* Fixing the comments on nc page
2016-07-22 22:24:06 +02:00

687 B

GNU Parallel

Run commands on multiple CPU cores.

  • Gzip several files at once, using all cores:

parallel gzip ::: {{file1}} {{file2}} {{file3}}

  • Read arguments from stdin, run 4 jobs at once:

ls *.txt | parallel -j4 gzip

  • Convert JPG images to PNG using replacement strings:

parallel convert {} {.}.png ::: *.jpg

  • Parallel xargs, cram as many args as possible onto one command:

{{args}} | parallel -X {{command}}

  • Break stdin into ~1M blocks, feed each block to stdin of new command:

cat {{big_file.txt}} | parallel --pipe --block 1M {{command}}

  • Run on multiple machines via SSH:

parallel -S {{machine1}},{{machine2}} {{command}} ::: {{arg1}} {{arg2}}