moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2025-01-08 20:46:59 +03:00
Code Issues Packages Projects Releases Wiki Activity

Minimal buffering for multi_moses.py

Browse Source 
Speeds things up when using multi-threaded instances
This commit is contained in:
Michael Denkowski 2015-11-18 13:54:54 -05:00
parent edacfbb9fd
commit b002fade50
1 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
scripts/generic/multi_moses.py
View File

@ -90,9 +90,10 @@ def run_instance(cmd_base, threads, tasks, n_best=False):
cmd.append('--threads')
cmd.append(str(threads))
try:
# Queue of tasks instance is currently working on, limited to the number of
# threads. The queue should be kept full for optimal CPU usage.
work = Queue.Queue(maxsize=threads)
# Queue of tasks instance is currently working on, limited to the number
# of threads * 2 (minimal buffering). The queue should be kept full for
# optimal CPU usage.
work = Queue.Queue(maxsize=(threads * 2))
# Multi-threaded instance
moses = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)