sapling/eden/fs/store
Genevieve Helsel 31b5ed02fd deduplicate multiple fetches for the same trees and blobs
Summary:
If multiple requests to fetch a tree come in at the same time fast enough (where the first request hasn't had the chance to retrieve the data from Mercurial and save it in the cache), we will request to download the tree multiple times. If the tree is not in the hgcache, this means we will make an extra round trip to the server. There is no limit to how many concurrent requests can be made, meaning we could make a large amount of round trips to the server for the same tree.

This adds a tracking mechanism in which we track in progress tree fetches, and if we get a request that is already in the queue or being processed, we just return a future that will be fulfilled by the first request, instead of placing this duplicate request in the queue as well.

This also makes sure if we get a duplicate request, but the duplicate request has a higher priority than the request already in the queue, we will update the priority of the request in the queue.

Reviewed By: chadaustin

Differential Revision: D26355499

fbshipit-source-id: 8d3192cf0f5628c650715f4597c92fc8c9238650
2021-02-26 11:49:02 -08:00
..
git model: remove test-only constructor for TreeEntry 2021-01-05 14:08:14 -08:00
hg deduplicate multiple fetches for the same trees and blobs 2021-02-26 11:49:02 -08:00
test include more error message context when a LocalStore lookup misses 2021-02-26 00:28:46 -08:00
BackingStore.h enable skipping Metadata prefetches during eden prefetches 2020-11-11 16:30:02 -08:00
BackingStoreLogger.cpp Apply clang-format update fixes 2021-01-10 10:06:29 -08:00
BackingStoreLogger.h add fetch type to data fetch logging 2020-08-21 17:38:14 -07:00
BlobAccess.cpp use ObjectFetchContext for priority 2020-07-02 12:00:45 -07:00
BlobAccess.h use ObjectFetchContext for priority 2020-07-02 12:00:45 -07:00
BlobCache.cpp Tidy up license headers 2019-10-11 05:28:23 -07:00
BlobCache.h Tidy up license headers 2019-10-11 05:28:23 -07:00
BlobMetadata.h Tidy up license headers 2019-10-11 05:28:23 -07:00
CMakeLists.txt remove Mononoke usage in HgBackingStore 2020-05-05 20:59:02 -07:00
Diff.cpp remove dependency on glog 2020-11-10 16:31:15 -08:00
Diff.h add an ObjectFetchContext interface 2020-02-05 13:15:01 -08:00
DiffCallback.h Tidy up license headers 2019-10-11 05:28:23 -07:00
DiffContext.cpp add an ObjectFetchContext interface 2020-02-05 13:15:01 -08:00
DiffContext.h count fetch statistics during diff and checkout 2020-02-05 16:05:29 -08:00
eden_store_util.cpp eden: use fmt::format instead of folly::to 2020-09-23 21:14:20 -07:00
EmptyBackingStore.cpp enable skipping Metadata prefetches during eden prefetches 2020-11-11 16:30:02 -08:00
EmptyBackingStore.h enable skipping Metadata prefetches during eden prefetches 2020-11-11 16:30:02 -08:00
ImportPriority.h Add noexcept around ImportPriority 2021-01-27 00:33:18 -08:00
IObjectStore.h use ObjectFetchContext for priority 2020-07-02 12:00:45 -07:00
KeySpace.cpp eden: fix compilation on latest vs2017 2020-05-06 15:05:37 -07:00
KeySpace.h Apply clang-format update fixes 2021-01-10 10:06:29 -08:00
LocalStore.cpp remove dependency on glog 2020-11-10 16:31:15 -08:00
LocalStore.h tree metdata storage 2020-07-10 16:03:32 -07:00
MemoryLocalStore.cpp include more error message context when a LocalStore lookup misses 2021-02-26 00:28:46 -08:00
MemoryLocalStore.h restructure KeySpace and move it into KeySpace.h 2020-01-21 10:33:10 -08:00
ObjectFetchContext.cpp add ObjectFetchContext with causeDetail field 2020-09-01 03:39:18 -07:00
ObjectFetchContext.h enable skipping Metadata prefetches during eden prefetches 2020-11-11 16:30:02 -08:00
ObjectStore.cpp deduplicate multiple fetches for the same trees and blobs 2021-02-26 11:49:02 -08:00
ObjectStore.h win: remove ProcessNameCache from Stub.h 2020-09-14 19:25:48 -07:00
PathLoader.cpp add eden glob command 2021-02-23 19:58:03 -08:00
PathLoader.h add eden glob command 2021-02-23 19:58:03 -08:00
RocksDbLocalStore.cpp include more error message context when a LocalStore lookup misses 2021-02-26 00:28:46 -08:00
RocksDbLocalStore.h add independent cache size limits per object type 2020-01-21 19:41:47 -08:00
ScmStatusDiffCallback.cpp Migrate to field_ref Thrift API 2020-07-21 11:23:35 -07:00
ScmStatusDiffCallback.h Tidy up license headers 2019-10-11 05:28:23 -07:00
SerializedBlobMetadata.cpp introduce tree metadata storage in local store 2020-07-10 16:03:32 -07:00
SerializedBlobMetadata.h introduce tree metadata storage in local store 2020-07-10 16:03:32 -07:00
SqliteLocalStore.cpp include more error message context when a LocalStore lookup misses 2021-02-26 00:28:46 -08:00
SqliteLocalStore.h restructure KeySpace and move it into KeySpace.h 2020-01-21 10:33:10 -08:00
StatsFetchContext.cpp remove dependency on glog 2020-11-10 16:31:15 -08:00
StatsFetchContext.h logging for thrift object fetch: checkout 2020-07-26 23:09:40 -07:00
StoreResult.cpp include more error message context when a LocalStore lookup misses 2021-02-26 00:28:46 -08:00
StoreResult.h include more error message context when a LocalStore lookup misses 2021-02-26 00:28:46 -08:00
TreeMetadata.cpp remove dependency on glog 2020-11-10 16:31:15 -08:00
TreeMetadata.h introduce tree metadata storage in local store 2020-07-10 16:03:32 -07:00