Summary:
Fix the FileHandle code that was calling getPathBuggy() when writing journal
entries. This fixes a crash when writing to an unlinked file.
Also remove an unused call to getPathBuggy() in TreeInodeDirHandle.
These were the last places calling getPathBuggy(), so I have now deleted this
method entirely.
Reviewed By: bolinfest
Differential Revision: D4570606
fbshipit-source-id: a1c4aab10df96e2aeee78fca6d5808db28ad0cbf
Summary:
When the privhelper process detects that the connection to the main eden
process has closed, have it forcibly unmount the eden mount points.
(This does the equivalent of "umount -l")
This is needed to properly clean up mount points which contain bind mounts
inside them. If eden has already exited we will not be able to unmount the
bind mounts, and therefore we won't be able to unmount the parent eden mount
directory either without using MNT_DETACH.
In the long run we will probably want to update the eden process to shut down
gracefully on SIGINT or SIGTERM. This would let us unmount everything cleanly
if we did the unmount operations before we stop responding to FUSE requests.
However, the privhelper still needs to be able to clean things up properly if
eden crashes, so this lazy unmount behavior will still be desirable in the long
run.
Reviewed By: wez
Differential Revision: D4548030
fbshipit-source-id: b60189181bccc6f126998c92fd9a2e47593c96b3
Summary:
In the privhelper code we want to wait for the bind unmount to really complete
before proceeding. Previously this was done by repeatedly calling statfs()
until it fails. However, on my system the statfs() call continues to succeed
even after the unmount operation, causing bindUnmount() to never think that the
unmount has finished. (I'm currently running Linux 4.6.7)
This updates the code to check the file system ID before and after the unmount,
and also stop once we see the filesystem ID change. I also replaced the
Linux-specific statfs() call with the POSIX-standard statvfs() call.
Reviewed By: wez
Differential Revision: D4547938
fbshipit-source-id: 51f53c0c901331de80fb908ae30033a0b56091b4
Summary:
This is the initial code for implementing checkout.
This isn't quite 100% implemented yet, but I think it's worth checking in this
code as-is, and getting the remaining functionality done in separate diffs.
In particular, a few operations aren't implemented:
- Removing a directory that was deleted in the new revision
- Replacing a directory that was replaced with a file or symlink in the new
revision
- When doing a forced update, replacing a file or directory that did not exist
in the old revision, but that was created locally in the working directory,
and also exists in the new revision.
Reviewed By: wez
Differential Revision: D4538516
fbshipit-source-id: 5bb4889b02f23ab2048fcae2c8b7614340181aa6
Summary:
If the eden process times out waiting on a response from the privhelper, the
next time it tries to send a request the old response may be available now.
In this case it will read the old response when expecting to read the response
for its new request.
This diff updates the code to discard the old response in this case and retry
reading another request. This allows us to get back in sync with the
privhelper after a timed-out request. Previously the code would simply give up
after finding a mismatching transaction ID in the response, causing it to
always be one response off, and preventing any future requests from succeeding.
This is still kind of hacky, and could definitely be improved in the future,
but for now it is better than always failing all future privhelper requests.
Reviewed By: wez
Differential Revision: D4547769
fbshipit-source-id: 396b57eccb28d938998e29594d336f35ba4fb1fb
Summary:
Update PathFuncs so that only PathComponent defines comparison functions
between PathComponent and PathComponentPiece. Previously PathComponentPiece
also defined methods for performing comparisions between PathComponent and
PathComponentPiece, resulting in ambiguous overload errors when you tried to
use them. (The same also applied to the RelativePath and AbsolutePath types.)
Reviewed By: wez
Differential Revision: D4546326
fbshipit-source-id: 158f72138e6d81d02beaeb925bfcc59db673c51a
Summary:
Update PrivHelperServer::bindUnmount() to give up after 2 seconds instead of
possibly looping forever.
Also fix the unmount code in the eden process to rethrow the original exception
as-is. Previously it was re-throwing it as a std::exception, losing the
exception details.
I think there are still other bugs here in this unmount code--it repeatedly
hangs for me on unmount. (I believe it is trying to unmount things that aren't
actually bind mounts.) However, I'll look into fixing those in a separate
diff.
Reviewed By: wez
Differential Revision: D4547711
fbshipit-source-id: af6bce0262e8314361450874ef998ebf7ad590e3
Summary:
Refactor the Overlay code to store data using inode numbers rather than the
affected file's path in the repository. This simplifies the TreeInode code a
bit, as we no longer have to rename overlay files to stay in sync with the file
paths. This also eliminates some crashes when trying to update overlay files
for inodes that have been unlinked (and hence no longer have a path). This
also includes a few fixes to avoid writing journal entries for unlinked files
too. Additionally this contains a few fixes to how mode bits are stored in the
overlay, and fixes a bug where create() was ignoring the mode argument.
Reviewed By: wez
Differential Revision: D4517578
fbshipit-source-id: c1e31497dcf62c322b0deff72b0a02675b0509ab
Summary:
InodeBase::getNodeId() already returns the inode number, so there is no need
for the TreeInode subclass to provide an alternatively-named method that does
the same thing.
Reviewed By: bolinfest
Differential Revision: D4500215
fbshipit-source-id: fbc595cf9e8f43efe44ed352a97e3a4974b5aed0
Summary:
It turns out that the manifest class provides an iterentries() method to
iterate through the (path, hash, flags) values. Update the code to use this
rather than iterating through the paths and then having to look up the hash and
flags using the path.
This dramatically speeds up importing a commit with 1M files: the import time
drops from around 8 seconds to under 1.5 seconds. (Of the 1.5 seconds, about
half is spent reading the manifest data, and half is spent formatting and
writing the chunks.)
Reviewed By: wez
Differential Revision: D4512538
fbshipit-source-id: ccfff162082d327d525224b2d7dc87f67bda5e22
Summary:
This updates the TreeInode::rename() code to handle concurrency better.
In particular:
- The code now ensures that both the source inode being renamed and destination
inode (if it exists) are loaded. This simplifies issues when an inode is
being loaded at the same time a rename is in progress. This ensures that any
pending load is processed before the rename takes place. (All promises for
the load might not be fulfilled before the rename completes, but the relevant
TreeInode and InodeMap data structures are updated before the rename occurs.)
This does mean that the rename code potentially might have to retry several
times if the inode it began loading is no longer the affected source or
destination or child once the load completes. However, this seems like a
reasonable trade-off, compared to dealing with the complications that would
arise with the load code having to handle renames occuring before load
completion.
- The code now implements proper lock ordering, to avoid acquiring locks in
conflicting orders that might cause deadlock with other threads also trying
to acquire the same locks. The InodeLocks.md document has been updated to
clarify the TreeInode lock ordering requirements.
Reviewed By: wez
Differential Revision: D4493526
fbshipit-source-id: 627393fafad90eb551aea62be7762d59ed043abe
Summary:
Move the integration tests from eden/fs/integration up one directory, to
eden/integration.
The main benefit is that this makes it easy to run just the edenfs unit tests
by running "buck test eden/fs/...". These unit tests complete much more
quickly than the full set of integration tests, providing a faster test suite
to re-run repeatedly during development. The integration tests can be run with
"buck test eden/integration/...", and the full set of tests can still be run
with "buck test eden/..."
Reviewed By: wez
Differential Revision: D4490247
fbshipit-source-id: 5ceb5a19526f56e1cb926f352fa30ad2f1212c05
Summary:
Most of the FileData logic assumes that file_ is open if the file is
materialized. Fix the FileData() constructor to actually open file_ if the
entry is marked materialized, so that this assumption is actually true.
Reviewed By: wez
Differential Revision: D4471365
fbshipit-source-id: b399e6e5e25eaf12f4204f5835db01baa5a6cd78
Summary:
Update the code to release the parent TreeInode's contents lock when processing
its children. This makes sure we do not hold the root lock for the entire
duration of this function.
Reviewed By: wez
Differential Revision: D4471289
fbshipit-source-id: dc4b50bd06fcceb7a7df3d7dd04cfc0d41285556
Summary:
This is to facilitate the watchman integration and draws on the
watchman glob implementation; the approach is to split the glob strings into
path components and evaluate the components step by step as the tree is walked.
Components that do not include any glob special characters can be handled as
a direct lookup from the directory contents (O(1) rather than O(num-entries)).
The glob method returns a set of filenames that match a list of
of glob patterns.
Recursive globs are supported. It is worth noting that a glob like "**/*" will
return a list of every entry in the filesystem. This is potentially expensive
and should be avoided. simpkins is in favor of disallowing this as a forcing
function to encourage tool-makers to adopt patterns that don't rely on a
complete listing of the filesystem.
For now I'd like to get this in without such a restriction; it's also worth
noting that running `find .` in the root of the mount point has a similar
effect and we can't prevent that from happening, so the effect of the overly
broad glob is something that we need to be able to withstand in any case.
Unrestricted recursive globs will make it easier to connect certain watchman
queries in the interim, until we have a more expressive thrift API for walking
and filtering the list of files.
Note: I've removed the wildmatch flags that I'd put in the API when I stubbed
it out originally. Since this is built on top of our GlobMatcher code and that
doesn't have those flags, I thought it would be simplest to just remove them.
If we find that we need them, we can figure out how to add them later.
Also Note: the evaluation of the glob is parallel-ready but currently limited
to 1 at a time by constraining the folly::window call to 1. We could make this
larger but would need a more intelligent constraint. For example, a recursive
glob could initiate N concurrent futures per level where N is the number of
sub-dirs at a given level. Using a custom Executor for these futures may be a
better option to set an upper bound on the number of concurrent jobs allowed
for a given glob call.
Depends on D4361197
Reviewed By: simpkins
Differential Revision: D4371934
fbshipit-source-id: 444735600bc16d2c2185f2277ddc5b51f672600a
Summary:
As part of the mount process, make sure we load InodeBase objects for all files
in the mount that are already materialized.
Other parts of the code assume that InodeBase objects are always loaded for
materialized files. We never unload materialized inodes, but if the mount
point was unmounted then remounted we previously did not ensure to load the
materialized InodeBase objects. This diff makes sure we load all materialized
inodes before starting the mount.
Reviewed By: bolinfest
Differential Revision: D4461193
fbshipit-source-id: 70d06fd01e2df333ce2816d5d7a392b0a5d6e1e6
Summary:
Fix two issues with inode load completion:
1) Make sure we release the parent's TreeInode lock before fulfilling promises
waiting on the inode load to complete. Otherwise this can result in deadlock
if the promises have callbacks that then try to acquire the TreeInode lock.
2) Use unique_ptr<InodeBase> in the load completion code before we have
inserted the Inode into the InodeMap, rather than using InodePtr. The new
InodePtr class relies on the InodeMap to handle ownership, and it cannot
destroy Inode objects unless they are in the InodeMap. Using a unique_ptr
instead ensures that the InodeBase will be destroyed properly if an error
occurs between when we construct the InodeBase and when we insert it into the
map.
Reviewed By: bolinfest
Differential Revision: D4427889
fbshipit-source-id: 2627a4956735f4acebe5c259abd8de0430ff6177
Summary:
Rename the current EdenMount::getInodeBase() function to
EdenMount::getInodeBaseBlocking() and add a new getInodeBase() function that
performs the lookup asynchronously and returns a Future<InodePtr>.
This is implemented with a TreeInode::getChildRecursive() function that allows
asynchronous recursive lookups at any point in the tree.
Reviewed By: bolinfest
Differential Revision: D4427855
fbshipit-source-id: aca55e681a48c848b085b7fc6a13efe6cf0a4e3a
Summary:
Add methods to InodeBase to get the parent of the current Inode, and remove the
existing TreeInode::getParent() function. The old TreeInode parent_ member was
never updated properly after rename or unlink operations, so it could return
incorrect information.
Reviewed By: bolinfest
Differential Revision: D4427103
fbshipit-source-id: cb66a014f745b4af31033e0a3c5405e29791b869
Summary:
Thesis: Since we don't yet have `hg update` implemented, these attributes can be cached forever.
When we implement `hg update` we will explicitly inform the kernel to invalidate anyway.
... and it turns out that the `scmRemove` thrift call is making changes that
aren't visible to the kernel already so we need to hook up some basic
invalidation.
This diff includes the bare minimum code to facilitate this; I've added a
helper method to the fusell::RequestData class to test whether we are in the
context of a FUSE request.
In the EdenDispatcher::unlink method we need to explicitly invalidate if we are
not being called via FUSE.
I have mixed feelings about putting this code in here. Given the initial
mental model I had in the prototype, this is the right place to put it, but it
does mean that all tree mutations must have via the Dispatcher in order for
invalidations to be routed to the correct places. Do we want to move away from
this?
It wouldn't be the end of the world to expand this diff a bit to add
similar/appropriate invalidation calls to all the mutation methods in our
dispatcher implementations while we figure out if we want to keep this.
Reviewed By: simpkins
Differential Revision: D4452961
fbshipit-source-id: 9471f145242fce0620c6872b74b02c56c8d78af1
Summary:
This mirrors the capability added in D4444058.
I don't think the `getenv` calls are awesome, but also don't think that we're
likely to want to override these outside of the environment either, so it
doesn't warrant plumbing these all the way through the eden cli, edenfs command
line flags and the layers to get to instantiate this object.
Reviewed By: simpkins
Differential Revision: D4446320
fbshipit-source-id: d5661e4f3e8dee82617eb6edddbcb9da5f4296d2
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures. These were caused by the integration tests picking up the
default post-clone hook configuration.
This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.
Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.
The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.
Reviewed By: bolinfest
Differential Revision: D4446321
fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
Summary:
It's been bothering me that we claim to run and complete running
the globally installed hooks when they aren't there, so tidy up the logging.
Reviewed By: bolinfest
Differential Revision: D4446318
fbshipit-source-id: acda223e1df2302602ad9967b1aa1b334b8477c7
Summary:
This adopts the new InterpolatedPropertyTree class as
the configuration storage.
It doesn't introduce any actual configuration of the interpolation
replacements; that will be in a follow-on diff.
Reviewed By: bolinfest
Differential Revision: D4444157
fbshipit-source-id: 18ead8e9074d23b1154e81f012f0c90efced1350
Summary:
This makes it possible to deploy via packaging (we'll put the par file
in the same dir as the extension) and eliminates a recursive buck build call
that was slowing down our integration tests; we can now just look up the path
from the environment that we set up in the TARGETS file to locate the par files
in the integration tests.
Reviewed By: bolinfest
Differential Revision: D4444096
fbshipit-source-id: 2ad6b6aa783563dc54a1cfd347a509c9919bb17a
Summary:
I don't know that I'm happy with the name; it started out as
just a helper wrapper to apply interpolation, but grew a bit to better
encapsulate how we access the config from the eden server.
This is a utility class that I'd like to use in place of the raw boost
property tree in our ClientConfig code.
It encapsulates the underlying property tree and provides a couple of simple
methods; one for loading data from a config file and one for looking up
a configuration value.
The configuration value lookup will replace tokens of the form `${KEY}`
with the value of `KEY` from its internal replacements dictionary.
This mirrors the interpolation handling that we do in the equivalent python
code for the eden client.
Reviewed By: bolinfest
Differential Revision: D4444075
fbshipit-source-id: 5d46d63f87caad4f409fbb981aa83165fcd6596d
Summary:
I'm going to follow up with some changes to substitute
some core env vars; this is just some prep.
Reviewed By: bolinfest
Differential Revision: D4444030
fbshipit-source-id: ab5d0e39aba14cdba39bb4867ec24955665723c6
Summary:
`eden config` dumps all parsed config values.
`eden config --get SECTION.OPTION` prints the value of the corresponding key.
Reviewed By: bolinfest
Differential Revision: D4434084
fbshipit-source-id: 63d9f72aa8794371e89a6d9e527bacaab17540de
Summary:
This makes it easier to reason about where these things can be found,
and in particular, allows passing the path through to the post-clone hook
script. Although post-clone accepts some command line configuration options,
our hooks scheme doesn't seem to allow setting that in the eden configuration
file.
Reviewed By: bolinfest
Differential Revision: D4443872
fbshipit-source-id: ed92dbac3b91b7f7dc86a39a85b465907f290b47
Summary:
We implement some sanity checking in wrappers, preventing some unsafe
consequences of using raw Buck UI in fbcode.
Reviewed By: Gownta
Differential Revision: D4453147
fbshipit-source-id: 3069898069c7e89223b133224f7c87d1a6b5886a
Summary:
Fix issues flagged by running "arc lint" on all eden files. There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.
Reviewed By: bolinfest
Differential Revision: D4446615
fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
Summary:
10 seconds isn't long enough for the tests to pass consistently on my devserver (I have a windows VM running on here, so there is more background crap running).
Raising this up to 30 gave me a successful test run.
Reviewed By: bolinfest
Differential Revision: D4442300
fbshipit-source-id: 5c3ae1fc0d0acb21c5bb44e36dae88d5180749d1
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Previously we'd emit an AttributeError. This diff
fixes things up so that we show the help when no arguments are provided.
Reviewed By: simpkins
Differential Revision: D4434060
fbshipit-source-id: 838b1cab47118d8517e4dc0c5fde17fc69752b13
Summary:
This lock protects all operations that change the location of existing inodes
(rename(), unlink(), and rmdir()). This is needed to avoid race conditions
during rename operations.
I also included some FIXME comments about properly updating location
information for unloaded inodes. These should be taken care of in a subsequent
diff.
Reviewed By: wez
Differential Revision: D4361197
fbshipit-source-id: 5aaa1b8250f196b23207be8f3c0fd7f2603e6ae8
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.
The inode lookup itself still happens serially at the moment. We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.
Reviewed By: wez
Differential Revision: D4361091
fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
Summary:
This updates the InodePtr and InodeBase code to actually implement Inode
unloading and destruction.
At the moment we keep Inode objects as long as possible, and only unload them
during shutdown, or when the last reference to an unlinked inode goes away.
However, it should be straightforward to add on-demand unloading in the future
for Inodes that have not been accessed in a while. The
TreeInode::unloadChildrenNow() function provides a template for what this would
look like (it would simply need to be changed to check an access time when
doing on-demand unloading).
Reviewed By: wez
Differential Revision: D4360765
fbshipit-source-id: a46b355f0ac0c25f873a156e62af5184317de735
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function. This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.
This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.
Reviewed By: wez
Differential Revision: D4360558
fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
Summary:
This defines our own custom smart pointer type for Inode objects. This will
provide us with more control over Inode lifetime, allowing us to decide if we
want to unload them immediately when they become unreferenced, or keep them
around for a while.
This will also allow us to fix some memory management issues around EdenMount
destruction. Currently we destroy the EdenMount immediately when it is
unmounted. This can cause issues if other parts of the code are still holding
references to Inode objects from this EdenMount. Our custom InodePtr class
will also allow us to delay destroying the EdenMount until all of its Inodes
have been destroyed.
This diff adds the new pointer types and updates the code to use them, but does
not actually implement destroying unreferenced inodes yet. The logic for that
has proven to be slightly subtle; I will split it out into its own separate
diff.
Reviewed By: wez
Differential Revision: D4351072
fbshipit-source-id: 7a9d81cbd226c9662a79a2f2ceda82fe2651f312
Summary: Folly will soon be dropping support for GCC 4.8, which means that `folly::make_unique` will be going away, so codemod away as much as possible before then.
Reviewed By: yfeldblum
Differential Revision: D4411221
fbshipit-source-id: 31d61425f6595d0750ea2e4c95c7d42bb5a5d955
Summary: I fixed an unrelated bug in HTE that made some more paths lintable - and some more move-abuses visible :-(
Reviewed By: meyering
Differential Revision: D4397732
fbshipit-source-id: 560db05f64ecf160e8198caf80dbd988e0dad75d
Summary:
I ported the C++ test I wrote for `hg add` in D4317871 as an integration test.
Note that this caught a real bug in the Hg extension!
Reviewed By: wez
Differential Revision: D4385970
fbshipit-source-id: 97ee555d463f22af54791f334f9349d88861f45a
Summary:
This constructor is unsafe. The check it uses before doing the comparison isn't a safe enough check to see if the cast is valid. For example, this is broken in the presence of multiple inheritance because it doesn't adjust the pointer offset to the correct vtable.
e.g.
struct A {
virtual ~A() {};
virtual void doSomething() = 0;
};
struct B {
virtual ~B() {}
virtual void doSomethingElse() = 0;
};
struct C : public B, public A {
virtual ~C() {}
void doSomething() override {
std::cout << "Something!" << std::endl;
}
void doSomethingElse() override {
std::cout << "Something Else!" << std::endl;
}
};
int main (int argc, char **argv) {
auto c = folly::makeFuture<std::shared_ptr<C>>(std::make_shared<C>());
folly::Future<std::shared_ptr<A>> a = std::move(c);
a.get()->doSomething();
return 0;
}
This code will print "Something else!" when run.
Reviewed By: siyengar
Differential Revision: D3679673
fbshipit-source-id: dcbf40ca82d458f17ee11191591f8b8daf58c919
Summary:
This describes the behavior around Inode ownership, loading and unloading, as
well as the synchronization around these events.
The InodeMap and InodePtr diffs implementing this behavior are still in flight.
Reviewed By: bolinfest
Differential Revision: D4356591
fbshipit-source-id: 298ba2c6aa98feba8b90f858e5633ca793276a1e
Summary:
If the EDEN_DAEMON_ARGS environment variable is set when running integration
tests, append it to the edenfs daemon command line.
For instance, this makes it easy to pass in a "--vmodule" flag to turn up
verbose logging in particular parts of the code.
Reviewed By: bolinfest
Differential Revision: D4359306
fbshipit-source-id: 6b549ac7220f6c8d7a1dfad3827921d6462d0f77
Summary:
Add a document describing the various inode related locks and the order that
they may be acquired.
Reviewed By: bolinfest
Differential Revision: D4356023
fbshipit-source-id: 44d4ade984f6cb49bb5f09deeeef0fd5439f129c