mirror of
https://github.com/facebook/sapling.git
synced 2024-10-10 08:47:12 +03:00
d5097a2d8c
Summary: This modifies our object popularity mechanism to account for the size of the objects being downloaded. Indeed, considering our bottleneck is bandwidth, we forcing similar routing for 10 downloads of a 10MB object and 10 downloads of a 1GB object doesn't make that much sense. This diffs updates our counting so that we now record the object size instead of a count. I'll set up routing so that we disallow consistent routing when a single object exceeds 250MiB/s of throughput ( = 1/4th of a task). It's worth noting that this will be equivalent to what we have right now for our most problematic objects (GraphQL schemas in Fbsource, 35M each), given that we "unroute" at 150 requests / 20 seconds (`150 * 35 / 20 = 262`). The key difference here is that this will work for all objects. This does mean LFS needs to cache and know about content metadata. That's not actually a big deal. Indeed, over a week, we serve 25K distinct objects (https://fburl.com/scuba/mononoke_lfs/a2d26s1a), so considering content metadata is a hundred bytes (and so is the key), we are looking at a few MBs of cache space here. As part of this, I've also refactored our config handling to stop duplicating structures in Configerator and LFS by using the Thrift objects directly (we still have a few dedicated structs when post-processing is necessary, but we no longer have anything that deserializes straight from JSON). Note that one further refinement here would be to consistently route but to more tasks (i.e. return one of 2 routing tokens for an object that is being downloaded at 500MiB/s). We'll see if we need that. Reviewed By: HarveyHunt Differential Revision: D24361314 fbshipit-source-id: 49e1f86cf49357f60f1eac298a753e0c1fcbdbe5 |
||
|---|---|---|
| .. | ||
| blobrepo_hg | ||
| blobsync | ||
| changeset_fetcher | ||
| common | ||
| errors | ||
| factory | ||
| override | ||
| repo_blobstore | ||
| src | ||
| test | ||
| Cargo.toml | ||