sapling/eden/fs/store
Matt Glazar 3b9a0310a1 Fix deadlock when restarting during RocksDbLocalStore::get()
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:

1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.

Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.

Reviewed By: simpkins

Differential Revision: D14337058

fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
2019-03-12 19:29:35 -07:00
..
git [] Future<T>::then Future<T>::then(not-try-task) -> Future<T>::thenValue(task). - 2/11 2018-09-07 11:07:25 -07:00
hg Fix deadlock when restarting during RocksDbLocalStore::get() 2019-03-12 19:29:35 -07:00
mononoke Reuse CURL* handle from curl_easy interface 2019-03-11 14:34:09 -07:00
test Fix deadlock when restarting during RocksDbLocalStore::get() 2019-03-12 19:29:35 -07:00
BackingStore.h remove fallback for correcting empty files 2018-10-22 20:27:26 -07:00
BlobAccess.cpp add a BlobAccess API for stateless blob reading 2018-11-26 10:46:51 -08:00
BlobAccess.h add a BlobAccess API for stateless blob reading 2018-11-26 10:46:51 -08:00
BlobCache.cpp only evict when interest handle is dropped if blob wasn't reloaded 2018-12-10 19:30:28 -08:00
BlobCache.h only evict when interest handle is dropped if blob wasn't reloaded 2018-12-10 19:30:28 -08:00
BlobMetadata.h run clang-format across all C++ files 2017-11-03 16:02:03 -07:00
CMakeLists.txt Use MononokeCurlBackingStore in HgBackingStore 2019-03-11 14:34:09 -07:00
Diff.cpp Future<T>::then Future<T>::then() -> Future<T>::thenValue or thenTry. 2018-09-14 17:10:57 -07:00
Diff.h add a function for diffing source control commits 2018-03-20 16:47:12 -07:00
EmptyBackingStore.cpp run clang-format across all C++ files 2017-11-03 16:02:03 -07:00
EmptyBackingStore.h run clang-format across all C++ files 2017-11-03 16:02:03 -07:00
IObjectStore.h clang-format 2018-10-02 10:07:38 -07:00
LocalStore.cpp add kEmptySha1 hash constant 2018-11-28 16:16:16 -08:00
LocalStore.h split the RocksDB local store into its own Buck target 2019-02-13 15:53:46 -08:00
MemoryLocalStore.cpp pass ReloadableConfig through to LocalStore 2018-11-09 11:22:03 -08:00
MemoryLocalStore.h pass ReloadableConfig through to LocalStore 2018-11-09 11:22:03 -08:00
ObjectStore.cpp folly::Optional -> std::optional 2018-10-23 17:05:11 -07:00
ObjectStore.h add an in-memory LRU for blob metadata 2018-10-22 20:27:27 -07:00
RocksDbLocalStore.cpp Fix deadlock when restarting during RocksDbLocalStore::get() 2019-03-12 19:29:35 -07:00
RocksDbLocalStore.h Fix deadlock when restarting during RocksDbLocalStore::get() 2019-03-12 19:29:35 -07:00
SerializedBlobMetadata.cpp move SerializedBlobMetadata into its own file 2018-10-31 11:50:39 -07:00
SerializedBlobMetadata.h move SerializedBlobMetadata into its own file 2018-10-31 11:50:39 -07:00
SqliteLocalStore.cpp pass ReloadableConfig through to LocalStore 2018-11-09 11:22:03 -08:00
SqliteLocalStore.h pass ReloadableConfig through to LocalStore 2018-11-09 11:22:03 -08:00
StoreResult.cpp Enabled additional compiler warnings in Eden. 2018-06-01 11:39:01 -07:00
StoreResult.h Enabled additional compiler warnings in Eden. 2018-06-01 11:39:01 -07:00