sapling/eden/fs/store/RocksDbLocalStore.h
Matt Glazar 3b9a0310a1 Fix deadlock when restarting during RocksDbLocalStore::get()
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:

1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.

Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.

Reviewed By: simpkins

Differential Revision: D14337058

fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
2019-03-12 19:29:35 -07:00

62 lines
1.9 KiB
C++

/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
*/
#pragma once
#include "eden/fs/rocksdb/RocksHandles.h"
#include "eden/fs/store/LocalStore.h"
#include "eden/fs/utils/UnboundedQueueExecutor.h"
namespace facebook {
namespace eden {
class FaultInjector;
class ReloadableConfig;
/** An implementation of LocalStore that uses RocksDB for the underlying
* storage.
*/
class RocksDbLocalStore : public LocalStore {
public:
/**
* The given FaultInjector must be valid during the lifetime of this
* RocksDbLocalStore object.
*/
explicit RocksDbLocalStore(
AbsolutePathPiece pathToRocksDb,
FaultInjector*,
std::shared_ptr<ReloadableConfig> config = nullptr);
~RocksDbLocalStore();
void close() override;
void clearKeySpace(KeySpace keySpace) override;
void compactKeySpace(KeySpace keySpace) override;
StoreResult get(LocalStore::KeySpace keySpace, folly::ByteRange key)
const override;
FOLLY_NODISCARD folly::Future<StoreResult> getFuture(
KeySpace keySpace,
folly::ByteRange key) const override;
FOLLY_NODISCARD folly::Future<std::vector<StoreResult>> getBatch(
KeySpace keySpace,
const std::vector<folly::ByteRange>& keys) const override;
bool hasKey(LocalStore::KeySpace keySpace, folly::ByteRange key)
const override;
void put(
LocalStore::KeySpace keySpace,
folly::ByteRange key,
folly::ByteRange value) override;
std::unique_ptr<WriteBatch> beginWrite(size_t bufSize = 0) override;
private:
FaultInjector& faultInjector_;
RocksHandles dbHandles_;
mutable UnboundedQueueExecutor ioPool_;
};
} // namespace eden
} // namespace facebook