sapling/blobstore_sync_queue
Stanislau Hlebik d3e9dce296 RFC mononoke: do batch writes to blobstore sync queue
Summary:
We've hit an issue of slow pushes to Mononoke when a commit modifies a lot of
files (>500 in our case). turned out that the problem was in the fact that
we have only one master write connection open, and each blobstore write
requires a write to mysql because of multiplexed blobstore. Because we have
only one connection open all our mysql writes are serialized, and the push is
taking too much time. It's especially bad in non-master regions.

To mitigate the issue let's add a batching in the blobstore sync queue. When
clients call `blobstore_sync_queue.add(...)` we'll send this new entry via the
channel to a separate task that would send writes in batches. That allows us to
increase throughput significantly.

Reviewed By: jsgf

Differential Revision: D15248288

fbshipit-source-id: 22bab284b0cbe552b4b51bab4027813b4278fd14
2019-05-21 12:25:45 -07:00
..
schemas mononoke: rename crates to contain underscores instead of dashes 2019-03-06 07:18:28 -08:00
src RFC mononoke: do batch writes to blobstore sync queue 2019-05-21 12:25:45 -07:00
test RFC mononoke: do batch writes to blobstore sync queue 2019-05-21 12:25:45 -07:00