Summary:
Sometimes we can hit an Idle timeout error while talking to MySQL, because we open a connection and go idle for a long time. Then when we finally send a query, server returns an error: the connection is expired. This is the issue we found and fixed in D27503062 (a856799489) that blocked MySQL client release.
## Future starvation
Imagine you have a stream in which you're connecting to a server, fetching and preparing some values:
```
let v = vec![1u32, 2, 3, 4, 5];
let mut s = stream::iter(v)
.map(|key| async move {
let conn = connect(..).await?;
conn.fetch(..).await
})
.buffered(2)
.map(|item| async move { prepare(..) })
.buffered(2);
```
Now you want to asynchronously process those prepared values one by one:
```
while let Some(result) = s.next().await {
let value = result?;
process(value).await?;
}
```
This async `process(..)` call can be talking to some service to take these values or something else that doesn't require much of a CPU time. Although the operation can be long.
**Now what happens when we do s.next().await?**
Because the stream is `buffered(2)` we wait for the first 2 futures. When the first item is ready, it returns the result and polls next stream item with a key - 3. The third future only makes a `connect(..)` call and gets switched.
When we've got a next value from the stream, we're waiting on the `process(value)` call and not polling the underlying stream till the processing is done.
**As I mentioned earlier, it is not expensive...**
But what if it takes > 10s to complete anyway?
The third future from the stream, that was polled earlier, **will wait for all these > 10s till it is polled again**.
More details [in this post](https://fb.workplace.com/groups/learningrust/permalink/2890621307875402/).
## Solution
In this case spawning a future with connection and query steps is a way to fix the issue.
This diff spawns queries in `shed::sql::sql_common::mysql_wrapper` - this covers all the places in Mononoke where we talk to MySQL. Also I removed the spawn from hg sync code, because it is not needed anymore and to illustrate that this approach works.
Reviewed By: StanislavGlebik
Differential Revision: D27639629
fbshipit-source-id: edaa2ce8f5948bf44e1899a19b443935920e33ef
442775f79f