Summary: Now that permissions on directories work, verify umask works as intended.
Reviewed By: simpkins
Differential Revision: D7783743
fbshipit-source-id: 635221cd3255cc20e9ffa26b6838922c4a4110f3
Summary:
Unify how inode metadata is modified across inode types. This allows
changing the permission bits on a directory.
Reviewed By: simpkins
Differential Revision: D7767254
fbshipit-source-id: 35e9cf652c84c7d8680cc22dec7942e94e9f5af1
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This is not at all clear from cppreference.com, but per
https://www.youtube.com/watch?v=dTeKf5Oek2c, it sounds to me like
recommended practice is to either:
`using namespace std::chrono_literals` (or string_literals or
whatever) to pull in a focused set of literals.
or
`using namespace std::literals` to pull in all standard literals.
or
`using namespace std` to pull in everything.
`using namespace std::literals::chrono_literals` is unnecessarily
verbose.
Adopt those standards in Eden.
Reviewed By: simpkins
Differential Revision: D8060944
fbshipit-source-id: 4d9dd4329698b7ff5e5c81b5b28780ca4d81a2a1
Summary:
I'd misunderstood the point of SharedMutex's upgrade locks -
unless they're used in rare paths, they don't allow for increased
concurrency. This diff and D7885245 remove all of Eden's ulocks,
replacing them with a helper which checks once with an rlock held, and
if the check fails, switches to a wlock (and checks again).
Reviewed By: yfeldblum
Differential Revision: D7886046
fbshipit-source-id: 545bb0dbb4898cbb71412efc6222ef12e4ee374e
Summary:
This allows multiple inodes to update their metadata simultaneously
without contending on InodeTable's locks.
Reviewed By: simpkins
Differential Revision: D7885245
fbshipit-source-id: cc8ab6cd90b7424beec314a115852e08b64026ae
Summary:
igorsugak reported an LSAN failure in Eden. Relatively straightforward to
track down - pending requests could sustain an Inode which ultimately
sustains EdenMount which owns the BackingStore, resulting in a
reference cycle for the duration of a fetch request. During EdenMount
shutdown in general we may want to consider aborting or discarding any
pending fetch requests.
Reviewed By: bolinfest, igorsugak
Differential Revision: D8047650
fbshipit-source-id: 32dafaff60570cf54a74ca4f57da61b6657d8ccb
Summary:
Some tools/script may be blindly assuming the presence of this file,
so rather than fight them, just create it.
Reviewed By: simpkins
Differential Revision: D8020274
fbshipit-source-id: 712e3bf31a0aefe27cc20f5361a0edb59c7deb9f
Summary:
Update the folly::Init code to define a `--logging` command line flag, and call
`folly::initLoggingOrDie()` with the value of this command line during
initialization.
This is similar to the existing code that initializes the glog library.
(Programs can use both glog and folly logging together in the same program, and
I expect that many programs will do so as parts get converted to folly::logging
and parts remain using glog.)
Reviewed By: yfeldblum
Differential Revision: D7827344
fbshipit-source-id: 8aa239fbad43bc0b551cbe40cad7b92fa97fcdde
Summary:
This updates the Eden mercurial extension to no longer invoke the Eden
`resetParentCommits()` thrift call when `setparents()` is called on the
dirstate map. Instead we now defer the call to `resetParentCommits()` until
`write()` is called to write the dirstate data to disk.
Informing edenfs of the parent change as soon as `setparents()` was called was
problematic, as this made edenfs reflect the change before the transaction was
committed. Some mercurial commands, notably `hg status` also call
`setparents()` on the dirstate but never write this back to disk at all. This
is problematic since `hg status` calls `setparents()` without holding any
mercurial locks. As a result it may call `setparents()` with the "wrong"
parent if another mercurial process is running and is in the middle of a
transaction.
Reviewed By: bolinfest, chadaustin
Differential Revision: D7980375
fbshipit-source-id: 4f5e4391fd291d4ea5fc93bb9d49ed0380fc1721
Summary:
While running the secfs filesystem validation tests against Eden, I
discovered a test that caused the eden process to abort. I bisected
and found that D7451330 regressed renaming a directory onto an empty
one. This fixes that case.
Reviewed By: simpkins
Differential Revision: D7945727
fbshipit-source-id: 592ede1b391528c02cd12b2b6ebbf3733fe8f503
Summary:
TestMount used to call through the EdenDispatcher when creating nodes
in the filesystem which had the unfortunate side effect of
incrementing the FUSE refcount. This diff asserts the FUSE refcounts
remain zero after inodes are created.
Reviewed By: simpkins
Differential Revision: D7957177
fbshipit-source-id: 7b0865a37ebbf39fdb34db409edd70d606295a0f
Summary:
I got tired of typing PathComponentPiece{"..."} in tests so here are
some operator literals.
Reviewed By: simpkins
Differential Revision: D7956732
fbshipit-source-id: 85d9f3fd725853a54da9e70fc659bd7eb9e0862c
Summary:
This moves some things around in order to facilitate adding the migration
command in a separate file.
Reviewed By: bolinfest
Differential Revision: D7946842
fbshipit-source-id: 54a554fb02e83a12f1d626b81377bc042fac41aa
Summary:
This allows using multiple cores when supported by
the BackingStore, and improves the throughput of prefetches.
Reviewed By: chadaustin
Differential Revision: D7888343
fbshipit-source-id: 1747f4ec4edf9ace02d54a4fb0ea3e8f509f51e5
Summary:
this overrides the LocalStore::getFuture to use its own
thread pool.
Reviewed By: chadaustin
Differential Revision: D7888344
fbshipit-source-id: 76b18d9417b28dc0ab72af8d070bc9e037c73bc3
Summary:
Adds a dumb `getFuture` implementation in the LocalStore
base class that simply calls `get`. Different store implementations
may choose to override this to allow making use of multiple cores
if appropriate.
Reviewed By: chadaustin
Differential Revision: D7888345
fbshipit-source-id: 20ba2db91cd7d62e5594f7d3bc3fca594dd107aa
Summary: This old Overlay code is no longer necessary.
Reviewed By: simpkins
Differential Revision: D7903912
fbshipit-source-id: 4a39d6ce7d1f6f81eb13715f2d5d17b22c10d413
Summary:
A persistent (but notably non-durable) mapping from inode
number to a fixed-size record stored in a memory-mapped file. The two
primary goals here are:
1. efficiently (and lazily) reify timestamps for inodes that aren't in the overlay
2. allow the kernel's page cache to drop pages under memory pressure
Reviewed By: simpkins
Differential Revision: D6877361
fbshipit-source-id: a4366b12e21e2bf483c83069cd93ef150829b2ac
Summary:
Make it clear (especially for the upcoming InodeMetadata struct) which
operations with EdenTimestamp and InodeTimestamps will never throw.
Reviewed By: simpkins
Differential Revision: D7920219
fbshipit-source-id: 5917da51b8128455893a1480def6f2c1c8de13d4
Summary:
simpkins was curious how data format migrations would be handled in
the upcoming InodeTable. This diff implements the bulk of the logic
which is largely at the MappedDiskVector level. The existing file
format supported record version negotiation and this diff hooks it up
with some type-level operations.
Reviewed By: simpkins
Differential Revision: D7836249
fbshipit-source-id: 00e36bc67068c7524956e908b3872c80a79241c0
Summary:
Per yfeldblum's comment in D7886046, we can use folly::unit instead
of folly::Unit{}. We weren't using folly::unit anywhere, so this diff
replaces folly::Unit{} elsewhere in the Eden code.
Reviewed By: yfeldblum
Differential Revision: D7913462
fbshipit-source-id: fa6ab44ceb406d38713e0f4649224a74e6e51abd
Summary:
Historically, we have seen a number of messages like the following in the Eden
logs:
```
Journal for .hg/blackbox.log holds invalid Created, Created sequence
```
Apparently we were getting these invalid sequences because we were not always
recording a "rename" correctly. The "rename" constructor for a `JournalDelta`
assumed that the source path should be included in the list of "removed" files
while the destination path should be included in the list of "created" files.
However, that is not accurate if the destination path already existed before
the user ran `mv`.
Fortunately, we already check whether the destination file exists in
`TreeInode::doRename()`, so it is straightforward to determine whether the
action is a "rename" (destination does not exist) or an "replace" (destination
already exists) and then classify the destination path accordingly.
As demonstrated by the new test introduced in this commit
(`JournalUpdateTest::moveFileReplace`), in the old implementation,
a file that was removed after it was overwritten would not show up as
removed in the merged `JournalDelta`. Because Watchman relies on
`JournalDelta::merge()` via the Thrift method `getFilesChangedSince()`,
this would cause Watchman to report such a file as still existing even
though it was removed.
This definitely caused bugs in Nuclide. It is likely that other tools that rely
on Watchman in Eden (such as Buck) may have also done incorrect things
because of this bug, so this could explain past reported issues.
Reviewed By: simpkins
Differential Revision: D7888249
fbshipit-source-id: 3e57963f27c5421a6175d1a759db8d9597ed76f3
Summary:
Before I fixed an issue with `nuclide-connections` in D7833410, it could throw
an error. Although there was a try/except around the `check_output` call in
`eden doctor`, it caught `OSError` rather than `CalledProcessError`, which
seems like a mistake (or maybe an inadvertent evolution of the code).
While there, I also extended it to catch `ValueError` in case the stdout of the
subprocess is not valid JSON, in which case `json.loads()` will raise a
`ValueError`.
Reviewed By: chadaustin
Differential Revision: D7890571
fbshipit-source-id: 184f6f669e9d62a5fb04db29bcbab450defc226e
Summary:
For sure, there is still more we can do to improve the output of `eden doctor`
to make it easier to scan, but I thought some color to classify the status
would be helpful. With this change, `eden doctor`:
* Prints green when no issues are encountered.
* Prints red when there are issues that could not be fixed.
* Prints yellow when there were issues that were fixed.
* Prints yellow when there were issues that were not fixed because it was a dry run.
In making this change, I took `StdoutPrinter` from `eden/cli/debug.py` and moved
it into its own file, `eden/cli/stdout_printer.py`. While there, I introduced an
`AnsiEscapeCodes` class that can be passed to the constructor of `StdoutPrinter`
so the client can specify its own ANSI escape codes. The unit test uses these to
ensure the test output will be the same independent of the value of
`sys.stdout.isatty()` when the test is run.
Reviewed By: chadaustin
Differential Revision: D7890525
fbshipit-source-id: a95ff8c1685b48c2d239923cf08456ec6de757fe
Summary:
This diff works a little harder to be able to successfully
stop buck in a repo. It does so by performing a single level glob
to find the main buckconfig files and then invoking buck kill in
each of those locations.
The output from buck is suppressed as we've had reports that it
was confusing.
I've removed the code that shutdown chg; it's been causing us
problems in our integration tests, and the problematic behavior
will soon be addressed in chg itself.
Reviewed By: chadaustin
Differential Revision: D7874975
fbshipit-source-id: e9755099b1d22f2b4e3684280eb95cb9c9d11a41
Summary:
I have an upcoming change that touches this file and I don't want to pull in
unrelated edits. (I thought we had already autoformatted all of this code?)
Reviewed By: chadaustin
Differential Revision: D7855072
fbshipit-source-id: 2d2ab2ce19d438af73c30471199d15db98fa4e3a
Summary:
We need to introduce a new `includeDotfiles` option to `glob()`. [As we have
done for all of our Thrift API, to date], rather than define `glob()` so that it
takes a single struct, we specified the parameters individually, so we can no
longer add new params to `glob()`.
In particular, we need to support `includeDotfiles` because we often configure
Buck to use Watchman to implement `glob()` in `BUCK` files, and when Watchman is
used in Eden, it leverages Eden's Thrift API to implement `glob()`. Because
Buck's `glob()` has an `include_dotfiles` option, we must be able to honor it
and pass it all the way through to Eden's `glob()` implementation.
Rather than name the new API `glob2()`, I'm electing to go with `globFiles()`.
(Perhaps once we eliminate all known users of `glob()` in the wild, which
requires turning over the current version of Watchman we have deployed, we can
redefine `glob()` in `eden.thrift` to be the same as `globFiles()` and then
update everyone to use `glob()` again so it has the more intuitive name.)
Reviewed By: wez
Differential Revision: D7748870
fbshipit-source-id: 92438f9c41e4fbdbd6cdccca5fce0e41cc3e9b07
Summary:
To provide the ability to ignore dotfiles, we update `GlobNode()` to take a
`bool includeDotfiles` that is used to determine the options used to create
the `GlobMatcher` associated with the node. Whereas the patterns
`**` and `*` were assumed to always match, that is now true only when
`includeDotfiles` is true, as well.
Finally, this adds a special case when `**` is used as the `pattern` for a `GlobNode`
and `includeDotfiles` is `false`. Because `GlobMatcher` does not accept `**` as
input, we specify the pattern as `**/*` to `GlobMatcher`, which it accepts and is
functionally equivalent in our case.
This is safe because we only do this trick when `**` is specified as a leaf `GlobNode`.
Reviewed By: wez
Differential Revision: D7741508
fbshipit-source-id: 9e6a50cb4dab09be2497393c641f176c84316a07
Summary:
Although we have not seen any ASAN failures or anything as a result of the
existing implementation, it certainly seems that it should be possible to
use a `StringPiece pattern` that is passed to `GlobNode.parse()` and not require
the `pattern` to outlive the `GlobNode`.
Reviewed By: wez
Differential Revision: D7795497
fbshipit-source-id: cefdd3fc69f8095c3aef3306b45f60ab4f82c737
Summary:
When `GlobMatcher` is used to implement `glob()` for Eden, `**` should not
include dotfiles by default (at least when it is used to implement `glob()` in Buck),
so we need to make this configurable. To this end, this adds a `GlobOptions`
parameter to `GlobMatcher::create()`. The key option this revision introduces is
`GlobOptions::IGNORE_DOTFILES`.
We implement this new functionality by associating a `matchCanStartWithDot`
boolean with the following opcodes in `GlobMatcher`:
* `GLOB_STAR`
* `GLOB_STAR_STAR_END`
* `GLOB_STAR_STAR_SLASH`
* `GLOB_ENDS_WITH`
The value of `matchCanStartWithDot` is largely determined by
`GlobOptions::IGNORE_DOTFILES`, though some extra checking is done
when assigning this for `GLOB_STAR`.
Originally, `GLOB_ENDS_WITH` required some funny business in how it
manipulated the `result` vector. This revision introduces some new funny
business to preserve the desired optimization.
Most of the work in this revision is new logic to ensure `matchCanStartWithDot`
is honored appropriately for each opcode.
Reviewed By: simpkins
Differential Revision: D7787621
fbshipit-source-id: f2c42e0f0948db74d48dc163d40aa3b13bbb4c3d
Summary:
I frequently find myself forgetting how to make the compiler see that
non-moved-from EDEN_BUG's destructor is noreturn, so add a simple
throwException function to it.
Reviewed By: simpkins
Differential Revision: D7834182
fbshipit-source-id: f279b9ca24f90efb4ad3ac318606dbd2dd002665
Summary:
When comparing two source control blob hashes, identical hashes can be assumed
to mean that the file contents are equal. However, differing hashes does not
necessarily mean that the file contents differ. In particular, mercurial
hashes history metadata in addition to the file contents when computing the
blob hash.
This updates Eden to always compare the file contents when the source control
blob hashes differ, rather than assuming that the file contents are different.
Reviewed By: wez
Differential Revision: D7825900
fbshipit-source-id: e611124a66cdd5c44589f20d1d4665a603286530
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
While testing the fbclone --eden changes I found that
we were failing to pick up the bind mounts. The root cause of
this is that the default head rev for hg repos is `.`, so if
the source repo is bare then we won't find an `.arcconfig` file
and thus won't be able to set up the bind mounts.
Reviewed By: simpkins
Differential Revision: D7782489
fbshipit-source-id: f41d3a7daf39ecd0946707cb2c3211d70c36eea2
Summary:
These daemon processes tend to keep file handles open
and block shutdown in various annoying ways, so we terminate
them prior to stopping the eden server.
Note that the chg shutdown is linux specific. We need a supported
command that does the right thing because this is fragile and won't
work on macos. cc: quark-zju.
We'd talked about prompting the user about this; I can add prompting
as a follow on diff.
Reviewed By: chadaustin
Differential Revision: D7784903
fbshipit-source-id: 47a2f6395f99b2bfde3dbd049b6ca4804d011373
Summary:
We've been talking about this for a little while;
this changes the terminology to refer to checkouts rather
than clients because that concept feels more intuitive.
Reviewed By: simpkins
Differential Revision: D7784902
fbshipit-source-id: 5e1e9be9438eed7915a95ea4141c4452e86adbfc
Summary:
To avoid bugs similar to the ones fixed in D7781691, mark a bunch of
folly::Future<folly::Unit> functions with FOLLY_NODISCARD.
Reviewed By: simpkins
Differential Revision: D7782224
fbshipit-source-id: 23ba42aa63011cc33e5a6e18d5bc6d00403a78d3
Summary:
Disallow any kind of mutation operation inside of the .eden directory. We had some
code in place to prevent some of this already, but errors (including EPERM) weren't
passed out from unlink and rename out to FUSE.
Reviewed By: simpkins
Differential Revision: D7781691
fbshipit-source-id: aaecf13779eca75d6ee8765fc8bb3727ce9341de
Summary: We set `metavar=''` to suppress the default and make things look nicer.
Reviewed By: simpkins
Differential Revision: D7783417
fbshipit-source-id: 05f970e8f964cb38e153c509cd5c71dcaa3143d0
Summary: The InodeTable work will homogenize mode_t storage, access, and modification across file and tree inodes. In preparation, have InodeBase keep track of the initial mode bits instead of a dtype_t.
Reviewed By: simpkins
Differential Revision: D7031924
fbshipit-source-id: f2e6e4467cecfc0ca06ad998cce0af18a99cc251
Summary:
By introducing `_assert_glob()`, it eliminates a bit of the boilerplate in the
individual test cases. I think it also makes things easier to read because now
the glob patterns appear before the result of the glob.
Though most importantly, this will help with a subsequent change where I am
going to change the contract of the Thrift API, as now it can be done within
`_assert_glob()`.
Reviewed By: wez
Differential Revision: D7748871
fbshipit-source-id: 9609cde104979e892f0858e7a0c7e53e976ff8e2
Summary:
In the spirit of TDD, I am writing the tests first and annotating them with
`unittest.skip()` so the build still succeeds.
Reviewed By: chadaustin
Differential Revision: D7741507
fbshipit-source-id: 4ede0b933c75f9be4016e399936ff2e47eb4e538
Summary:
I think this makes it easier to reason about the coverage of the space of
inputs to `glob()`.
Reviewed By: chadaustin
Differential Revision: D7741509
fbshipit-source-id: 5882d859df95279189512716004263dd5320ff3f
