sapling/eden/scm/edenscmnative
Durham Goode 63d19e1eca workers: bulk fetch data in worker thread
Summary:
During an hg update we first prefetch all the data, then write all the
data to disk. There are cases where the prefetched data is not available during
the writing phase, in which case we fall back to fetching the files one-by-one.
This has truly atrocious performance.

Let's allow the worker threads to check for missing data then do bulk fetching
of it. In the case where the cache was completely lost for some reason, this
would reduce the number of serial fetches by 100x.

Note, the background workers already spawn their own ssh connection's, so
they're already getting some level of parallelism even when they're doing 1-by-1
fetching. That's why we aren't seeing a 100x improvement in performance.

Reviewed By: xavierd

Differential Revision: D23766424

fbshipit-source-id: d88a1e55b1c21e9cea7e50fc6dbfd8a27bd97bb0
2020-09-21 11:27:12 -07:00
..
bindings workers: bulk fetch data in worker thread 2020-09-21 11:27:12 -07:00
__init__.py remediation of S205607 2020-07-17 17:16:13 -07:00
clindex.pyi add type annotations for a few more edenscmnative modules 2020-03-20 10:41:32 -07:00
clindex.pyx repoview: further remove repoview references 2020-07-06 14:04:27 -07:00
linelog.pyx py3/cython: set the language_level to 3str 2020-01-30 12:52:06 -08:00
patchrmdir.pyi add type annotations for a few more edenscmnative modules 2020-03-20 10:41:32 -07:00
patchrmdir.pyx py3: fix extdiff 2020-08-27 19:15:22 -07:00
traceprof.pyx py3/cython: set the language_level to 3str 2020-01-30 12:52:06 -08:00