Summary:
Update the KeySpace enum values to start at 0 instead of 1. This simplifies
the code to avoid having to skip over 0 in a few places.
This also makes the `kKeySpaceRecords` array slightly less confusing. Unlike
the `columns` array used by `RocksDbLocalStore` and the `tableNames` array
used by `SqliteLocalStore`, the `kKeySpaceRecords` array was not previously
indexed by the `KeySpace` enum values.
Reviewed By: wez, strager
Differential Revision: D15307393
fbshipit-source-id: ae8392d02396b4dc3c18e9ee94b198fcbb9b1a34
Summary:
Add a command to force a call to RepairDB() on the local store.
This is similar to using `ldb repair`, but invokes `RepairDB()` with the same
set of column family options as normally used by edenfs.
Reviewed By: chadaustin
Differential Revision: D15043210
fbshipit-source-id: 2c4c0e2d3410a50cb1e523611f569f1701604ae6
Summary:
This moves some logic from the RocksHandles class up to RocksDbLocalStore.
The main thing moved here is the logic to automatically try and repair the DB
if opening fails. This will make it easier in a subsequent diff to make the
repair logic a bit smarter and more aware of our column family semantics.
This keeps RocksHandles a pretty dumb wrapper around the RocksDB object and
column family handles, whose only purpose is to manage destroying these two
things in the correct order.
Reviewed By: chadaustin
Differential Revision: D15043208
fbshipit-source-id: ee2d5619ac7781a892e1ba151712eee9e3ebfb14
Summary:
This effectively reverts D14452214, which caused Eden to write an `id` entry
to each RocksDB column family and then flush the column family each time
edenfs started.
There was relatively little benefit to this in practice. It only matters in
cases where the RocksDB column families never had enough data written to them
to get flushed automatically and then a repair is required.
On the other hand it does have some material downsides: it flushing the column
families can be fairly expensive, and can require a substantial amount of free
disk space. This flush caused some users to not be able to start up edenfs
when they did not have enough free disk space.
Reviewed By: chadaustin
Differential Revision: D14947235
fbshipit-source-id: a29f98163fa87185b028bb47945b6fab75700fd6
Summary:
Change the `eden gc` implementation to use RocksDB's `DeleteFilesInRange()`
function, as well as its new-ish `DeleteRange()` method.
This makes the garbage collection much faster, and also require much less free
disk space than previously.
`DeleteFilesInRange()` asks RocksDB to simply delete all SST files from disk
if they only contain keys in the specified range. Since the range we specify
should include all keys in the DB this should simply drop all SST files for
this column family.
We also call `DeleteRange()` after this, just in case. This API is relatively
recent, and writes a single tombstone saying that the specified range has been
deleted.
Reviewed By: wez
Differential Revision: D14910345
fbshipit-source-id: c76bdc1c8e07cb2def66673ea892e7f455c9dc7a
Summary:
This message was added in D14337058. It is logged at the `INFO` level, which
is enabled by default, but doesn't seem to add much value to normal production
logs.
Reviewed By: chadaustin
Differential Revision: D14712654
fbshipit-source-id: 5a86d883ace30e22d299046e33a6cd6247432857
Summary:
When opening the RocksDB, write one entry to each column family and then
explicitly flush that column family. This ensures that the column family
information has actually been flushed to an SST file. Without this some
column families may only have been written out to the write-ahead log files.
(Even calling `db->Flush()` does not appear to be sufficient; each column
family has to be explicitly flushed.)
The RocksDB' `RepairDB()` function (used by `ldb repair`) currently ends up
deleting column families that do not have any data defined in an SST file.
The repair tool ends up deleting column families that only have data in log
files.
The fact that we haven't been doing these explicit flushes previously probably
isn't too much of a concern in practice: once we write out enough data RocksDB
will automatically trigger a flush. This only matters in cases where we have
not yet written out enough data to trigger an automatic flush.
Note that with this change we re-write these `id` keys each time we open the
RocksDB store, even if they were already present.
Reviewed By: chadaustin, strager
Differential Revision: D14452214
fbshipit-source-id: 3f1b17e240cc89fe00e3d31105d16452795e754d
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:
1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.
Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.
Reviewed By: simpkins
Differential Revision: D14337058
fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
Summary:
This makes it possible to change configuration options
for the LocalStore while the server is running.
As you'll see in the next diff, our current layering makes using
the config a bit more awkward, but at least this diff doesn't
look gross :-p
This diff doesn't introduce any new functionality or configuration.
Reviewed By: strager
Differential Revision: D12949577
fbshipit-source-id: cf897ba676b9359f92865170faa42ff17329b85f
Summary:
Add the beginnings of an eden gc command. Today it's equivalent to
`eden debug clear_local_caches` followed by `eden
debug_compact_local_storage`, except that it compacts each column as
they're cleared to minimize peak disk consumption.
Eventually, it will also unload in-memory inodes, flush data from the
overlay, and clear the kernel's VFS cache too.
Reviewed By: wez
Differential Revision: D9138305
fbshipit-source-id: b303a63f601014cf38ca94c9e6f7c04394159ea8
Summary:
Add a clearCaches function to LocalStore that deletes all data from
LocalStore that could be retrieved from Mercurial.
Reviewed By: wez
Differential Revision: D8101365
fbshipit-source-id: d46d0db94e6f85aaf542d9f6b9b96fbdcc548b57
Summary:
this overrides the LocalStore::getFuture to use its own
thread pool.
Reviewed By: chadaustin
Differential Revision: D7888344
fbshipit-source-id: 76b18d9417b28dc0ab72af8d070bc9e037c73bc3
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
This enables dropping in alternative implementations
of LocalStore and adds a MemoryLocalStore implementation for
use in our tests.
This diff doesn't change the default storage option for the
eden server. I'll look at adding such an option in a follow up diff.
Reviewed By: chadaustin
Differential Revision: D6910413
fbshipit-source-id: 018bf04e0bff101e1f0ab35e8580ca2a2622e5ef