Thomas Orozco ab4f7adaeb mononoke/newfilenodes: introduce a queue-conscious filenodes implementation ... Summary: This introduces a new implementation of filenodes that maintains its own queuing on top of the queuing enforced by the SQL crate. Later in this stack, the goal is for this implementation to avoid dispatching duplicate queries when there is a lot of contention talking to MySQL, which happens when large changes land and suddenly everyone wants the updated code. The underlying goal is to avoid dispatching a lot of duplicate queries when there is contention. Indeed, if there is contention, then the latency between query and response increases. As a result, without visibility in the queue, the following can happen: - Task 1 looks for A in the cache. It misses - Task 1 dispatches a SQL query - Task 2 looks for A in the cache. It misses - Task 2 dispatches a SQL query - Task 3 looks for A in the cache. It misses - Task 3 dispatches a SQL query - ... - Task 1's SQL query finally executes and fills the cache. - All other queries execute anyway. The longer the dispatch queue, the longer it takes to run those queries. Looking at Mononoke's stats in prod, this happens pretty often: https://pxl.cl/10xxmo (the spike at 3pm was a 10K-files change in fbsource, for example). The goal of this stack is to avoid this effect, by checking the cache only once we know we're ready to go to SQL. In this particular diff, what's added is: - The SQL read and write implementation. This is all implemented using new futures, but the logic should be largely unchanged from before (i.e. we store filenodes and their associated copy info in shards by the filenode's path — not the source path if there is copy info —, and paths in their own shard). The queries themselves largely unchanged from the existing filenodes, with only a few tweaks: - Filenodes and copy info are now selected in one go. - There are types to distinguish path hashes and paths. - The structs to support this implementation. Reviewed By: StanislavGlebik Differential Revision: D19905397 fbshipit-source-id: bec981e7bfb396d62eb06e5ce249c21555afc64b		2020-02-27 12:34:19 -08:00
..
sqlite-filenodes.sql	mononoke/newfilenodes: introduce a queue-conscious filenodes implementation	2020-02-27 12:34:19 -08:00