Summary:
Unify how inode metadata is modified across inode types. This allows
changing the permission bits on a directory.
Reviewed By: simpkins
Differential Revision: D7767254
fbshipit-source-id: 35e9cf652c84c7d8680cc22dec7942e94e9f5af1
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This is not at all clear from cppreference.com, but per
https://www.youtube.com/watch?v=dTeKf5Oek2c, it sounds to me like
recommended practice is to either:
`using namespace std::chrono_literals` (or string_literals or
whatever) to pull in a focused set of literals.
or
`using namespace std::literals` to pull in all standard literals.
or
`using namespace std` to pull in everything.
`using namespace std::literals::chrono_literals` is unnecessarily
verbose.
Adopt those standards in Eden.
Reviewed By: simpkins
Differential Revision: D8060944
fbshipit-source-id: 4d9dd4329698b7ff5e5c81b5b28780ca4d81a2a1
Summary:
I'd misunderstood the point of SharedMutex's upgrade locks -
unless they're used in rare paths, they don't allow for increased
concurrency. This diff and D7885245 remove all of Eden's ulocks,
replacing them with a helper which checks once with an rlock held, and
if the check fails, switches to a wlock (and checks again).
Reviewed By: yfeldblum
Differential Revision: D7886046
fbshipit-source-id: 545bb0dbb4898cbb71412efc6222ef12e4ee374e
Summary:
This allows multiple inodes to update their metadata simultaneously
without contending on InodeTable's locks.
Reviewed By: simpkins
Differential Revision: D7885245
fbshipit-source-id: cc8ab6cd90b7424beec314a115852e08b64026ae
Summary:
igorsugak reported an LSAN failure in Eden. Relatively straightforward to
track down - pending requests could sustain an Inode which ultimately
sustains EdenMount which owns the BackingStore, resulting in a
reference cycle for the duration of a fetch request. During EdenMount
shutdown in general we may want to consider aborting or discarding any
pending fetch requests.
Reviewed By: bolinfest, igorsugak
Differential Revision: D8047650
fbshipit-source-id: 32dafaff60570cf54a74ca4f57da61b6657d8ccb
Summary:
Update the folly::Init code to define a `--logging` command line flag, and call
`folly::initLoggingOrDie()` with the value of this command line during
initialization.
This is similar to the existing code that initializes the glog library.
(Programs can use both glog and folly logging together in the same program, and
I expect that many programs will do so as parts get converted to folly::logging
and parts remain using glog.)
Reviewed By: yfeldblum
Differential Revision: D7827344
fbshipit-source-id: 8aa239fbad43bc0b551cbe40cad7b92fa97fcdde
Summary:
While running the secfs filesystem validation tests against Eden, I
discovered a test that caused the eden process to abort. I bisected
and found that D7451330 regressed renaming a directory onto an empty
one. This fixes that case.
Reviewed By: simpkins
Differential Revision: D7945727
fbshipit-source-id: 592ede1b391528c02cd12b2b6ebbf3733fe8f503
Summary:
TestMount used to call through the EdenDispatcher when creating nodes
in the filesystem which had the unfortunate side effect of
incrementing the FUSE refcount. This diff asserts the FUSE refcounts
remain zero after inodes are created.
Reviewed By: simpkins
Differential Revision: D7957177
fbshipit-source-id: 7b0865a37ebbf39fdb34db409edd70d606295a0f
Summary:
I got tired of typing PathComponentPiece{"..."} in tests so here are
some operator literals.
Reviewed By: simpkins
Differential Revision: D7956732
fbshipit-source-id: 85d9f3fd725853a54da9e70fc659bd7eb9e0862c
Summary:
This allows using multiple cores when supported by
the BackingStore, and improves the throughput of prefetches.
Reviewed By: chadaustin
Differential Revision: D7888343
fbshipit-source-id: 1747f4ec4edf9ace02d54a4fb0ea3e8f509f51e5
Summary:
this overrides the LocalStore::getFuture to use its own
thread pool.
Reviewed By: chadaustin
Differential Revision: D7888344
fbshipit-source-id: 76b18d9417b28dc0ab72af8d070bc9e037c73bc3
Summary:
Adds a dumb `getFuture` implementation in the LocalStore
base class that simply calls `get`. Different store implementations
may choose to override this to allow making use of multiple cores
if appropriate.
Reviewed By: chadaustin
Differential Revision: D7888345
fbshipit-source-id: 20ba2db91cd7d62e5594f7d3bc3fca594dd107aa
Summary: This old Overlay code is no longer necessary.
Reviewed By: simpkins
Differential Revision: D7903912
fbshipit-source-id: 4a39d6ce7d1f6f81eb13715f2d5d17b22c10d413
Summary:
A persistent (but notably non-durable) mapping from inode
number to a fixed-size record stored in a memory-mapped file. The two
primary goals here are:
1. efficiently (and lazily) reify timestamps for inodes that aren't in the overlay
2. allow the kernel's page cache to drop pages under memory pressure
Reviewed By: simpkins
Differential Revision: D6877361
fbshipit-source-id: a4366b12e21e2bf483c83069cd93ef150829b2ac
Summary:
Make it clear (especially for the upcoming InodeMetadata struct) which
operations with EdenTimestamp and InodeTimestamps will never throw.
Reviewed By: simpkins
Differential Revision: D7920219
fbshipit-source-id: 5917da51b8128455893a1480def6f2c1c8de13d4
Summary:
simpkins was curious how data format migrations would be handled in
the upcoming InodeTable. This diff implements the bulk of the logic
which is largely at the MappedDiskVector level. The existing file
format supported record version negotiation and this diff hooks it up
with some type-level operations.
Reviewed By: simpkins
Differential Revision: D7836249
fbshipit-source-id: 00e36bc67068c7524956e908b3872c80a79241c0
Summary:
Per yfeldblum's comment in D7886046, we can use folly::unit instead
of folly::Unit{}. We weren't using folly::unit anywhere, so this diff
replaces folly::Unit{} elsewhere in the Eden code.
Reviewed By: yfeldblum
Differential Revision: D7913462
fbshipit-source-id: fa6ab44ceb406d38713e0f4649224a74e6e51abd
Summary:
Historically, we have seen a number of messages like the following in the Eden
logs:
```
Journal for .hg/blackbox.log holds invalid Created, Created sequence
```
Apparently we were getting these invalid sequences because we were not always
recording a "rename" correctly. The "rename" constructor for a `JournalDelta`
assumed that the source path should be included in the list of "removed" files
while the destination path should be included in the list of "created" files.
However, that is not accurate if the destination path already existed before
the user ran `mv`.
Fortunately, we already check whether the destination file exists in
`TreeInode::doRename()`, so it is straightforward to determine whether the
action is a "rename" (destination does not exist) or an "replace" (destination
already exists) and then classify the destination path accordingly.
As demonstrated by the new test introduced in this commit
(`JournalUpdateTest::moveFileReplace`), in the old implementation,
a file that was removed after it was overwritten would not show up as
removed in the merged `JournalDelta`. Because Watchman relies on
`JournalDelta::merge()` via the Thrift method `getFilesChangedSince()`,
this would cause Watchman to report such a file as still existing even
though it was removed.
This definitely caused bugs in Nuclide. It is likely that other tools that rely
on Watchman in Eden (such as Buck) may have also done incorrect things
because of this bug, so this could explain past reported issues.
Reviewed By: simpkins
Differential Revision: D7888249
fbshipit-source-id: 3e57963f27c5421a6175d1a759db8d9597ed76f3
Summary:
We need to introduce a new `includeDotfiles` option to `glob()`. [As we have
done for all of our Thrift API, to date], rather than define `glob()` so that it
takes a single struct, we specified the parameters individually, so we can no
longer add new params to `glob()`.
In particular, we need to support `includeDotfiles` because we often configure
Buck to use Watchman to implement `glob()` in `BUCK` files, and when Watchman is
used in Eden, it leverages Eden's Thrift API to implement `glob()`. Because
Buck's `glob()` has an `include_dotfiles` option, we must be able to honor it
and pass it all the way through to Eden's `glob()` implementation.
Rather than name the new API `glob2()`, I'm electing to go with `globFiles()`.
(Perhaps once we eliminate all known users of `glob()` in the wild, which
requires turning over the current version of Watchman we have deployed, we can
redefine `glob()` in `eden.thrift` to be the same as `globFiles()` and then
update everyone to use `glob()` again so it has the more intuitive name.)
Reviewed By: wez
Differential Revision: D7748870
fbshipit-source-id: 92438f9c41e4fbdbd6cdccca5fce0e41cc3e9b07
Summary:
To provide the ability to ignore dotfiles, we update `GlobNode()` to take a
`bool includeDotfiles` that is used to determine the options used to create
the `GlobMatcher` associated with the node. Whereas the patterns
`**` and `*` were assumed to always match, that is now true only when
`includeDotfiles` is true, as well.
Finally, this adds a special case when `**` is used as the `pattern` for a `GlobNode`
and `includeDotfiles` is `false`. Because `GlobMatcher` does not accept `**` as
input, we specify the pattern as `**/*` to `GlobMatcher`, which it accepts and is
functionally equivalent in our case.
This is safe because we only do this trick when `**` is specified as a leaf `GlobNode`.
Reviewed By: wez
Differential Revision: D7741508
fbshipit-source-id: 9e6a50cb4dab09be2497393c641f176c84316a07
Summary:
Although we have not seen any ASAN failures or anything as a result of the
existing implementation, it certainly seems that it should be possible to
use a `StringPiece pattern` that is passed to `GlobNode.parse()` and not require
the `pattern` to outlive the `GlobNode`.
Reviewed By: wez
Differential Revision: D7795497
fbshipit-source-id: cefdd3fc69f8095c3aef3306b45f60ab4f82c737
Summary:
When `GlobMatcher` is used to implement `glob()` for Eden, `**` should not
include dotfiles by default (at least when it is used to implement `glob()` in Buck),
so we need to make this configurable. To this end, this adds a `GlobOptions`
parameter to `GlobMatcher::create()`. The key option this revision introduces is
`GlobOptions::IGNORE_DOTFILES`.
We implement this new functionality by associating a `matchCanStartWithDot`
boolean with the following opcodes in `GlobMatcher`:
* `GLOB_STAR`
* `GLOB_STAR_STAR_END`
* `GLOB_STAR_STAR_SLASH`
* `GLOB_ENDS_WITH`
The value of `matchCanStartWithDot` is largely determined by
`GlobOptions::IGNORE_DOTFILES`, though some extra checking is done
when assigning this for `GLOB_STAR`.
Originally, `GLOB_ENDS_WITH` required some funny business in how it
manipulated the `result` vector. This revision introduces some new funny
business to preserve the desired optimization.
Most of the work in this revision is new logic to ensure `matchCanStartWithDot`
is honored appropriately for each opcode.
Reviewed By: simpkins
Differential Revision: D7787621
fbshipit-source-id: f2c42e0f0948db74d48dc163d40aa3b13bbb4c3d
Summary:
I frequently find myself forgetting how to make the compiler see that
non-moved-from EDEN_BUG's destructor is noreturn, so add a simple
throwException function to it.
Reviewed By: simpkins
Differential Revision: D7834182
fbshipit-source-id: f279b9ca24f90efb4ad3ac318606dbd2dd002665
Summary:
When comparing two source control blob hashes, identical hashes can be assumed
to mean that the file contents are equal. However, differing hashes does not
necessarily mean that the file contents differ. In particular, mercurial
hashes history metadata in addition to the file contents when computing the
blob hash.
This updates Eden to always compare the file contents when the source control
blob hashes differ, rather than assuming that the file contents are different.
Reviewed By: wez
Differential Revision: D7825900
fbshipit-source-id: e611124a66cdd5c44589f20d1d4665a603286530
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
To avoid bugs similar to the ones fixed in D7781691, mark a bunch of
folly::Future<folly::Unit> functions with FOLLY_NODISCARD.
Reviewed By: simpkins
Differential Revision: D7782224
fbshipit-source-id: 23ba42aa63011cc33e5a6e18d5bc6d00403a78d3
Summary:
Disallow any kind of mutation operation inside of the .eden directory. We had some
code in place to prevent some of this already, but errors (including EPERM) weren't
passed out from unlink and rename out to FUSE.
Reviewed By: simpkins
Differential Revision: D7781691
fbshipit-source-id: aaecf13779eca75d6ee8765fc8bb3727ce9341de
Summary: The InodeTable work will homogenize mode_t storage, access, and modification across file and tree inodes. In preparation, have InodeBase keep track of the initial mode bits instead of a dtype_t.
Reviewed By: simpkins
Differential Revision: D7031924
fbshipit-source-id: f2e6e4467cecfc0ca06ad998cce0af18a99cc251
Summary: Also print the inode path if the assertions in EXPECT_FILE_INODE fail.
Reviewed By: simpkins
Differential Revision: D7035517
fbshipit-source-id: 6c50acb588d1c985c7e7a1586c06f04a657698f9
Summary:
Now that Eden writes tree data into the overlay even when not
materialized, checkouts can be slow as Eden actively tries to forget
allocated inode data for potentially large trees when
kPreciseInodeMemory is false. This pushes the bulk of that garbage
collection work onto a background thread, making checkouts fast again.
Reviewed By: simpkins
Differential Revision: D7683120
fbshipit-source-id: 2a22e4612a4438b3ed2527670343c49dfcd902bc
Summary: This gets rid of those pesky #ifdefs in FileInode.cpp and TreeInode.cpp.
Reviewed By: wez
Differential Revision: D7735914
fbshipit-source-id: d810461984e21f72670f43ca2d1b4f5aacbf376e
Summary:
Update the Eden mercurial extension to read the `.eden/root` symlink to
determine what Eden thinks the mount path is. This might be different from
what directory mercurial thinks it is in if a parent directory of the Eden
mount has been bind-mounted to an alternate location.
Maybe in the future we should update thrift clients to pass in the client ID
(currently readable via `.eden/client`) rather than the mount path. That would
make it less likely for clients to accidentally forget to read `.eden/root` and
pass in the wrong mount path.
Reviewed By: wez
Differential Revision: D7705655
fbshipit-source-id: 7bd1e8013b99a52ff06dd45f63d6669b66bdf577
Summary:
This updates the code to store ServerState using a shared_ptr rather than
having it be an inlined member variable of EdenServer.
Previously EdenMount objects contained a raw pointer to the ServerState, with
the reasoning that the EdenServer object should outlive the EdenMount objects.
It turns out that this is not quite true in practice--EdenMount::destroy() will
normally be called before the EdenServer is destroyed, but this may not
actually destroy the EdenMount object immediately.
This fixes a race condition in the FuseTest.initMount() test that could cause
this test to occasionally fail when run on a heavily loaded system.
Reviewed By: chadaustin
Differential Revision: D7720509
fbshipit-source-id: 056ff5985587c8d8c32c11d17ba637ebd7598677
Summary:
I was worried we might leak data in the overlay in this particular
sequence of operations, so here's a test that shows we don't.
Reviewed By: simpkins
Differential Revision: D7663419
fbshipit-source-id: b4b0d08f863fe7b23501e9e4943d18ba0b746e42
Summary:
When we start storing permission changes and other inode
metadata keyed on inode number, we will need to remember inode numbers
before they're accessed to handle the case where the Eden process dies
before the inode is unloaded. Otherwise, data could be left in an
inconsistent state and some writes could be missed.
This change depends on saving (and forgetting) directories being
cheap.
Reviewed By: simpkins
Differential Revision: D7598094
fbshipit-source-id: 9ab127b30a9c09ab62aa08b50c13b3eaa40be60d
Summary:
D7512338 has been deployed for well over a week, so lets update edenfs to
reject the request if any clients still try to call getScmStatus() without
specifying the hash.
Reviewed By: chadaustin
Differential Revision: D7658022
fbshipit-source-id: ebd49034e076720f156f1ff2c551417ef058167a
Summary:
Split the modifyConflict() test into 3 separate test functions. This test
runs the same checks in 54 separate variations (3 path names, 3 checkout types,
with 6 inode load options).
For each variation we set up a new test mount, creating an overlay and
LocalStore on disk. Simply setting up and tearing down the test mounts can be
expensive if the temporary directory is served from spinning disk. This causes
this test to occasionally time out when run on our continuous build hosts.
Splitting the test up into 3 separate test functions will effectively give it
3x longer to run, since each test function will have a separate timeout. This
will also enable these test functions to run in parallel (although that may not
actually help with performance if they are bottlenecked on disk I/O).
Reviewed By: chadaustin
Differential Revision: D7665409
fbshipit-source-id: 4bddd68e75f38b1b6cc2d57512a5b52855f3bade
Summary:
Update the UnixSocket code to limit the number of iovecs that we send at one
time to folly::kIovMax. (This constant is set from from IOV_MAX on platforms
that provide this setting.)
The thrift serializer code emits data into 16kB chunks, so if we had many
megabytes of data to send we could end up with thousands of iovecs. The kernel
would fail the send with EMSGSIZE in this case.
Reviewed By: chadaustin
Differential Revision: D7665147
fbshipit-source-id: b7a60238d3fca973604b9037f22883cae80891ab
Summary:
The takeover data for a fully-loaded monorepo was 20 MB. Two
monorepos was 40 MB, exceeding the previous 32 MiB limit and causing
takeover to fail. Bump to 512 MiB to minimize the chance of
allocating too much given bogus data while still supporting large
numbers of checkouts.
Reviewed By: simpkins
Differential Revision: D7661637
fbshipit-source-id: 1dc54055e7ac3ce0f6cdbf4a7e5d372c2674dfee
Summary: Temporary fix to bump the takeover timeout to five minutes
Reviewed By: simpkins
Differential Revision: D7660642
fbshipit-source-id: 5b3a0347aa1443e395e9461d33459c62ffce5b2d
Summary:
UnixSocket traverses the IOBuf chain twice. Refactor that
into a common function because the next diff caps the size of
individual iovecs.
Reviewed By: simpkins
Differential Revision: D7659062
fbshipit-source-id: 88b7d63669d8189b96434c38a6e499ed3b5ebbe6
Summary:
fdatasync() has a substantial cost and is unnecessary given
our durability guarantees (and the fact that most filesystems on Linux
try to avoid data loss in this common write + rename situation anyway)
Reviewed By: simpkins
Differential Revision: D7641131
fbshipit-source-id: d041e7090dc05a4d4400f86cad9501aa8a6988a9
Summary:
Drop the log level of the "no dir data for inode" message from DBG2 to DBG3 so
we do not have it enabled by default. (Our default log settings are
eden=DBG2.)
Reviewed By: chadaustin
Differential Revision: D7620918
fbshipit-source-id: 25cf30af08054f3ba879bd57f3841b1035e0c75e
