Commit Graph

277 Commits

Author SHA1 Message Date
Adam Simpkins
4d60f09c1c speed up importing manifest data from mercurial
Summary:
It turns out that the manifest class provides an iterentries() method to
iterate through the (path, hash, flags) values.  Update the code to use this
rather than iterating through the paths and then having to look up the hash and
flags using the path.

This dramatically speeds up importing a commit with 1M files: the import time
drops from around 8 seconds to under 1.5 seconds.  (Of the 1.5 seconds, about
half is spent reading the manifest data, and half is spent formatting and
writing the chunks.)

Reviewed By: wez

Differential Revision: D4512538

fbshipit-source-id: ccfff162082d327d525224b2d7dc87f67bda5e22
2017-02-06 12:28:05 -08:00
Adam Simpkins
8b4c984b28 always load affected inodes in rename(), and update lock ordering
Summary:
This updates the TreeInode::rename() code to handle concurrency better.
In particular:

- The code now ensures that both the source inode being renamed and destination
  inode (if it exists) are loaded.  This simplifies issues when an inode is
  being loaded at the same time a rename is in progress.  This ensures that any
  pending load is processed before the rename takes place.  (All promises for
  the load might not be fulfilled before the rename completes, but the relevant
  TreeInode and InodeMap data structures are updated before the rename occurs.)

  This does mean that the rename code potentially might have to retry several
  times if the inode it began loading is no longer the affected source or
  destination or child once the load completes.  However, this seems like a
  reasonable trade-off, compared to dealing with the complications that would
  arise with the load code having to handle renames occuring before load
  completion.

- The code now implements proper lock ordering, to avoid acquiring locks in
  conflicting orders that might cause deadlock with other threads also trying
  to acquire the same locks.  The InodeLocks.md document has been updated to
  clarify the TreeInode lock ordering requirements.

Reviewed By: wez

Differential Revision: D4493526

fbshipit-source-id: 627393fafad90eb551aea62be7762d59ed043abe
2017-02-03 18:35:03 -08:00
Adam Simpkins
8884b46b3f move integration tests to eden/integration
Summary:
Move the integration tests from eden/fs/integration up one directory, to
eden/integration.

The main benefit is that this makes it easy to run just the edenfs unit tests
by running "buck test eden/fs/...".  These unit tests complete much more
quickly than the full set of integration tests, providing a faster test suite
to re-run repeatedly during development.  The integration tests can be run with
"buck test eden/integration/...", and the full set of tests can still be run
with "buck test eden/..."

Reviewed By: wez

Differential Revision: D4490247

fbshipit-source-id: 5ceb5a19526f56e1cb926f352fa30ad2f1212c05
2017-01-31 14:41:14 -08:00
Adam Simpkins
19a1d3c6b6 make sure FileData always open the overlay file for materialized files
Summary:
Most of the FileData logic assumes that file_ is open if the file is
materialized.  Fix the FileData() constructor to actually open file_ if the
entry is marked materialized, so that this assumption is actually true.

Reviewed By: wez

Differential Revision: D4471365

fbshipit-source-id: b399e6e5e25eaf12f4204f5835db01baa5a6cd78
2017-01-31 13:33:57 -08:00
Adam Simpkins
2c16e33026 do not hold tree locks recursively in getMaterializedEntriesRecursive()
Summary:
Update the code to release the parent TreeInode's contents lock when processing
its children.  This makes sure we do not hold the root lock for the entire
duration of this function.

Reviewed By: wez

Differential Revision: D4471289

fbshipit-source-id: dc4b50bd06fcceb7a7df3d7dd04cfc0d41285556
2017-01-27 16:47:57 -08:00
Wez Furlong
932ef52a55 implement glob thrift method
Summary:
This is to facilitate the watchman integration and draws on the
watchman glob implementation; the approach is to split the glob strings into
path components and evaluate the components step by step as the tree is walked.
Components that do not include any glob special characters can be handled as
a direct lookup from the directory contents (O(1) rather than O(num-entries)).

The glob method returns a set of filenames that match a list of
of glob patterns.

Recursive globs are supported.  It is worth noting that a glob like "**/*" will
return a list of every entry in the filesystem.  This is potentially expensive
and should be avoided.  simpkins is in favor of disallowing this as a forcing
function to encourage tool-makers to adopt patterns that don't rely on a
complete listing of the filesystem.

For now I'd like to get this in without such a restriction; it's also worth
noting that running `find .` in the root of the mount point has a similar
effect and we can't prevent that from happening, so the effect of the overly
broad glob is something that we need to be able to withstand in any case.

Unrestricted recursive globs will make it easier to connect certain watchman
queries in the interim, until we have a more expressive thrift API for walking
and filtering the list of files.

Note: I've removed the wildmatch flags that I'd put in the API when I stubbed
it out originally.  Since this is built on top of our GlobMatcher code and that
doesn't have those flags, I thought it would be simplest to just remove them.
If we find that we need them, we can figure out how to add them later.

Also Note: the evaluation of the glob is parallel-ready but currently limited
to 1 at a time by constraining the folly::window call to 1.  We could make this
larger but would need a more intelligent constraint.  For example, a recursive
glob could initiate N concurrent futures per level where N is the number of
sub-dirs at a given level.  Using a custom Executor for these futures may be a
better option to set an upper bound on the number of concurrent jobs allowed
for a given glob call.

Depends on D4361197

Reviewed By: simpkins

Differential Revision: D4371934

fbshipit-source-id: 444735600bc16d2c2185f2277ddc5b51f672600a
2017-01-26 12:47:05 -08:00
Adam Simpkins
4a6aaa5de4 load all materialized inodes when starting a mount point
Summary:
As part of the mount process, make sure we load InodeBase objects for all files
in the mount that are already materialized.

Other parts of the code assume that InodeBase objects are always loaded for
materialized files.  We never unload materialized inodes, but if the mount
point was unmounted then remounted we previously did not ensure to load the
materialized InodeBase objects.  This diff makes sure we load all materialized
inodes before starting the mount.

Reviewed By: bolinfest

Differential Revision: D4461193

fbshipit-source-id: 70d06fd01e2df333ce2816d5d7a392b0a5d6e1e6
2017-01-25 16:56:12 -08:00
Adam Simpkins
f8042c24d4 clean up some subtle issues with inode loading completion
Summary:
Fix two issues with inode load completion:

1) Make sure we release the parent's TreeInode lock before fulfilling promises
waiting on the inode load to complete.  Otherwise this can result in deadlock
if the promises have callbacks that then try to acquire the TreeInode lock.

2) Use unique_ptr<InodeBase> in the load completion code before we have
inserted the Inode into the InodeMap, rather than using InodePtr.  The new
InodePtr class relies on the InodeMap to handle ownership, and it cannot
destroy Inode objects unless they are in the InodeMap.  Using a unique_ptr
instead ensures that the InodeBase will be destroyed properly if an error
occurs between when we construct the InodeBase and when we insert it into the
map.

Reviewed By: bolinfest

Differential Revision: D4427889

fbshipit-source-id: 2627a4956735f4acebe5c259abd8de0430ff6177
2017-01-25 16:56:12 -08:00
Adam Simpkins
d1556c3601 implement Future-based recursive Inode lookup
Summary:
Rename the current EdenMount::getInodeBase() function to
EdenMount::getInodeBaseBlocking() and add a new getInodeBase() function that
performs the lookup asynchronously and returns a Future<InodePtr>.

This is implemented with a TreeInode::getChildRecursive() function that allows
asynchronous recursive lookups at any point in the tree.

Reviewed By: bolinfest

Differential Revision: D4427855

fbshipit-source-id: aca55e681a48c848b085b7fc6a13efe6cf0a4e3a
2017-01-25 16:56:12 -08:00
Adam Simpkins
20aaa1e2b8 Add InodeBase::getParent()
Summary:
Add methods to InodeBase to get the parent of the current Inode, and remove the
existing TreeInode::getParent() function.  The old TreeInode parent_ member was
never updated properly after rename or unlink operations, so it could return
incorrect information.

Reviewed By: bolinfest

Differential Revision: D4427103

fbshipit-source-id: cb66a014f745b4af31033e0a3c5405e29791b869
2017-01-25 15:06:05 -08:00
Wez Furlong
1e0397db50 raise the attribute TTL to maximum possible, add basic invalidation
Summary:
Thesis: Since we don't yet have `hg update` implemented, these attributes can be cached forever.

When we implement `hg update` we will explicitly inform the kernel to invalidate anyway.

... and it turns out that the `scmRemove` thrift call is making changes that
aren't visible to the kernel already so we need to hook up some basic
invalidation.

This diff includes the bare minimum code to facilitate this; I've added a
helper method to the fusell::RequestData class to test whether we are in the
context of a FUSE request.

In the EdenDispatcher::unlink method we need to explicitly invalidate if we are
not being called via FUSE.

I have mixed feelings about putting this code in here.  Given the initial
mental model I had in the prototype, this is the right place to put it, but it
does mean that all tree mutations must have via the Dispatcher in order for
invalidations to be routed to the correct places.  Do we want to move away from
this?

It wouldn't be the end of the world to expand this diff a bit to add
similar/appropriate invalidation calls to all the mutation methods in our
dispatcher implementations while we figure out if we want to keep this.

Reviewed By: simpkins

Differential Revision: D4452961

fbshipit-source-id: 9471f145242fce0620c6872b74b02c56c8d78af1
2017-01-24 09:39:06 -08:00
Wez Furlong
fb957d58ec allow interpolating USER and HOME in the ClientConfig class
Summary:
This mirrors the capability added in D4444058.

I don't think the `getenv` calls are awesome, but also don't think that we're
likely to want to override these outside of the environment either, so it
doesn't warrant plumbing these all the way through the eden cli, edenfs command
line flags and the layers to get to instantiate this object.

Reviewed By: simpkins

Differential Revision: D4446320

fbshipit-source-id: d5661e4f3e8dee82617eb6edddbcb9da5f4296d2
2017-01-23 23:54:38 -08:00
Wez Furlong
4829dfe319 change systemConfigDir to etcEdenDir
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures.  These were caused by the integration tests picking up the
default post-clone hook configuration.

This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.

Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.

The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.

Reviewed By: bolinfest

Differential Revision: D4446321

fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
2017-01-23 23:54:38 -08:00
Wez Furlong
220a2d4e1e clarify logging around running post-clone when it doesn't exist
Summary:
It's been bothering me that we claim to run and complete running
the globally installed hooks when they aren't there, so tidy up the logging.

Reviewed By: bolinfest

Differential Revision: D4446318

fbshipit-source-id: acda223e1df2302602ad9967b1aa1b334b8477c7
2017-01-23 23:54:38 -08:00
Wez Furlong
2e4610d35d cut ClientConfig over to InterpolatedPropertyTree
Summary:
This adopts the new InterpolatedPropertyTree class as
the configuration storage.

It doesn't introduce any actual configuration of the interpolation
replacements; that will be in a follow-on diff.

Reviewed By: bolinfest

Differential Revision: D4444157

fbshipit-source-id: 18ead8e9074d23b1154e81f012f0c90efced1350
2017-01-23 23:54:38 -08:00
Wez Furlong
774bc5b032 fixup paths used in post-clone and LameThriftClient
Summary:
This makes it possible to deploy via packaging (we'll put the par file
in the same dir as the extension) and eliminates a recursive buck build call
that was slowing down our integration tests; we can now just look up the path
from the environment that we set up in the TARGETS file to locate the par files
in the integration tests.

Reviewed By: bolinfest

Differential Revision: D4444096

fbshipit-source-id: 2ad6b6aa783563dc54a1cfd347a509c9919bb17a
2017-01-23 23:54:38 -08:00
Wez Furlong
7117c836d1 add InterpolatedPropertyTree class
Summary:
I don't know that I'm happy with the name; it started out as
just a helper wrapper to apply interpolation, but grew a bit to better
encapsulate how we access the config from the eden server.

This is a utility class that I'd like to use in place of the raw boost
property tree in our ClientConfig code.

It encapsulates the underlying property tree and provides a couple of simple
methods; one for loading data from a config file and one for looking up
a configuration value.

The configuration value lookup will replace tokens of the form `${KEY}`
with the value of `KEY` from its internal replacements dictionary.

This mirrors the interpolation handling that we do in the equivalent python
code for the eden client.

Reviewed By: bolinfest

Differential Revision: D4444075

fbshipit-source-id: 5d46d63f87caad4f409fbb981aa83165fcd6596d
2017-01-23 23:54:38 -08:00
Wez Furlong
dff225b60e add limited interpolation to eden config parsing
Summary: This allows us to templatize centrally managed config files

Reviewed By: bolinfest

Differential Revision: D4444058

fbshipit-source-id: a44372084d32dcf0b27922490ab40a48478a720d
2017-01-23 23:54:38 -08:00
Wez Furlong
bafc809c36 refactor and centralize ConfigParser calls to _loadConfig
Summary:
I'm going to follow up with some changes to substitute
some core env vars; this is just some prep.

Reviewed By: bolinfest

Differential Revision: D4444030

fbshipit-source-id: ab5d0e39aba14cdba39bb4867ec24955665723c6
2017-01-23 23:54:38 -08:00
Wez Furlong
8b8d6d319f add eden config command
Summary:
`eden config` dumps all parsed config values.

`eden config --get SECTION.OPTION` prints the value of the corresponding key.

Reviewed By: bolinfest

Differential Revision: D4434084

fbshipit-source-id: 63d9f72aa8794371e89a6d9e527bacaab17540de
2017-01-23 23:54:38 -08:00
Wez Furlong
9ff3def69f pass down paths for eden, edenfs, post-clone via environment in tests
Summary:
This makes it easier to reason about where these things can be found,
and in particular, allows passing the path through to the post-clone hook
script.  Although post-clone accepts some command line configuration options,
our hooks scheme doesn't seem to allow setting that in the eden configuration
file.

Reviewed By: bolinfest

Differential Revision: D4443872

fbshipit-source-id: ed92dbac3b91b7f7dc86a39a85b465907f290b47
2017-01-23 23:54:37 -08:00
Andrew Gallagher
14fa8d476e buckification: switch some raw Buck rules to use the buck_ wrappers
Summary:
We implement some sanity checking in wrappers, preventing some unsafe
consequences of using raw Buck UI in fbcode.

Reviewed By: Gownta

Differential Revision: D4453147

fbshipit-source-id: 3069898069c7e89223b133224f7c87d1a6b5886a
2017-01-23 19:30:02 -08:00
Adam Simpkins
e31889d295 fix a variety of lint issues
Summary:
Fix issues flagged by running "arc lint" on all eden files.  There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.

Reviewed By: bolinfest

Differential Revision: D4446615

fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
2017-01-23 18:03:09 -08:00
Wez Furlong
7e29c0eafe raise the default startup timeout in the integration tests
Summary:
10 seconds isn't long enough for the tests to pass consistently on my devserver (I have a windows VM running on here, so there is more background crap running).

Raising this up to 30 gave me a successful test run.

Reviewed By: bolinfest

Differential Revision: D4442300

fbshipit-source-id: 5c3ae1fc0d0acb21c5bb44e36dae88d5180749d1
2017-01-23 09:10:58 -08:00
Adam Simpkins
251da81f36 update all copyright statements to "2016-present"
Summary:
Update copyright statements to "2016-present".  This makes our updated lint
rules happy and complies with the recommended license header statement.

Reviewed By: wez, bolinfest

Differential Revision: D4433594

fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
2017-01-20 22:03:02 -08:00
Wez Furlong
26f5b43a93 fix running "eden" with no arguments
Summary:
Previously we'd emit an AttributeError.  This diff
fixes things up so that we show the help when no arguments are provided.

Reviewed By: simpkins

Differential Revision: D4434060

fbshipit-source-id: 838b1cab47118d8517e4dc0c5fde17fc69752b13
2017-01-18 21:56:57 -08:00
Adam Simpkins
09e9aa0819 add a mountpoint-wide rename lock
Summary:
This lock protects all operations that change the location of existing inodes
(rename(), unlink(), and rmdir()).  This is needed to avoid race conditions
during rename operations.

I also included some FIXME comments about properly updating location
information for unloaded inodes.  These should be taken care of in a subsequent
diff.

Reviewed By: wez

Differential Revision: D4361197

fbshipit-source-id: 5aaa1b8250f196b23207be8f3c0fd7f2603e6ae8
2017-01-17 15:03:20 -08:00
Adam Simpkins
a3af576c5a parallelize getSHA1()
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.

The inode lookup itself still happens serially at the moment.  We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.

Reviewed By: wez

Differential Revision: D4361091

fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
2017-01-17 15:03:20 -08:00
Adam Simpkins
a598e8c334 implement Inode unloading
Summary:
This updates the InodePtr and InodeBase code to actually implement Inode
unloading and destruction.

At the moment we keep Inode objects as long as possible, and only unload them
during shutdown, or when the last reference to an unlinked inode goes away.
However, it should be straightforward to add on-demand unloading in the future
for Inodes that have not been accessed in a while.  The
TreeInode::unloadChildrenNow() function provides a template for what this would
look like (it would simply need to be changed to check an access time when
doing on-demand unloading).

Reviewed By: wez

Differential Revision: D4360765

fbshipit-source-id: a46b355f0ac0c25f873a156e62af5184317de735
2017-01-17 15:03:20 -08:00
Adam Simpkins
0237a0c56d add an EdenMount::destroy() function
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function.  This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.

This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.

Reviewed By: wez

Differential Revision: D4360558

fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
2017-01-17 15:03:20 -08:00
Adam Simpkins
3a73253057 add new InodePtr classes
Summary:
This defines our own custom smart pointer type for Inode objects.  This will
provide us with more control over Inode lifetime, allowing us to decide if we
want to unload them immediately when they become unreferenced, or keep them
around for a while.

This will also allow us to fix some memory management issues around EdenMount
destruction.  Currently we destroy the EdenMount immediately when it is
unmounted.  This can cause issues if other parts of the code are still holding
references to Inode objects from this EdenMount.  Our custom InodePtr class
will also allow us to delay destroying the EdenMount until all of its Inodes
have been destroyed.

This diff adds the new pointer types and updates the code to use them, but does
not actually implement destroying unreferenced inodes yet.  The logic for that
has proven to be slightly subtle; I will split it out into its own separate
diff.

Reviewed By: wez

Differential Revision: D4351072

fbshipit-source-id: 7a9d81cbd226c9662a79a2f2ceda82fe2651f312
2017-01-17 15:03:20 -08:00
Andrew Gallagher
26c2526080 Fixup more rule names in TARGETS
Summary: Manually fixup rule names which the codemod scripting didn't catch.

Reviewed By: igorsugak

Differential Revision: D4420657

fbshipit-source-id: 20794b17732ca5c37686bd22edc1538837098022
2017-01-16 11:40:07 -08:00
Andrew Gallagher
bd862c50e5 codemod: fixup more build targets in TARGETS
Summary:
Ran script from:
differential/diff/21475830/

Reviewed By: Gownta

Differential Revision: D4419263

fbshipit-source-id: 7631ea2c2cb3b6e27756ecc9da9e492b2a3b2dab
2017-01-14 01:44:01 -08:00
Christopher Dykes
138bc12add Codemod folly::make_unique to std::make_unique
Summary: Folly will soon be dropping support for GCC 4.8, which means that `folly::make_unique` will be going away, so codemod away as much as possible before then.

Reviewed By: yfeldblum

Differential Revision: D4411221

fbshipit-source-id: 31d61425f6595d0750ea2e4c95c7d42bb5a5d955
2017-01-13 14:22:04 -08:00
Phil Willoughby
bf25b184ab Yet more std::move shenanigans
Summary: I fixed an unrelated bug in HTE that made some more paths lintable - and some more move-abuses visible :-(

Reviewed By: meyering

Differential Revision: D4397732

fbshipit-source-id: 560db05f64ecf160e8198caf80dbd988e0dad75d
2017-01-12 07:09:00 -08:00
Michael Bolin
1764b09db8 Add an integration test for hg add.
Summary:
I ported the C++ test I wrote for `hg add` in D4317871 as an integration test.
Note that this caught a real bug in the Hg extension!

Reviewed By: wez

Differential Revision: D4385970

fbshipit-source-id: 97ee555d463f22af54791f334f9349d88861f45a
2017-01-09 13:24:09 -08:00
Nick Wolchko
0aa6855557 Remove folly::Future conversion constructor
Summary:
This constructor is unsafe. The check it uses before doing the comparison isn't a safe enough check to see if the cast is valid. For example, this is broken in the presence of multiple inheritance because it doesn't adjust the pointer offset to the correct vtable.

e.g.
  struct A {
    virtual ~A() {};
    virtual void doSomething() = 0;
  };
  struct B {
    virtual ~B() {}
    virtual void doSomethingElse() = 0;
  };
  struct C : public B, public A {
    virtual ~C() {}
    void doSomething() override {
      std::cout << "Something!" << std::endl;
    }
    void doSomethingElse() override {
      std::cout << "Something Else!" << std::endl;
    }
  };
  int main (int argc, char **argv) {
    auto c = folly::makeFuture<std::shared_ptr<C>>(std::make_shared<C>());
    folly::Future<std::shared_ptr<A>> a = std::move(c);
    a.get()->doSomething();
    return 0;
  }
This code will print "Something else!" when run.

Reviewed By: siyengar

Differential Revision: D3679673

fbshipit-source-id: dcbf40ca82d458f17ee11191591f8b8daf58c919
2017-01-09 09:21:57 -08:00
Adam Simpkins
594d5c152f allow passing extra arguments to edenfs in the integration tests
Summary:
If the EDEN_DAEMON_ARGS environment variable is set when running integration
tests, append it to the edenfs daemon command line.

For instance, this makes it easy to pass in a "--vmodule" flag to turn up
verbose logging in particular parts of the code.

Reviewed By: bolinfest

Differential Revision: D4359306

fbshipit-source-id: 6b549ac7220f6c8d7a1dfad3827921d6462d0f77
2016-12-22 18:16:01 -08:00
Adam Simpkins
a5a0563fd3 add log statements to EdenDispatcher
Summary:
Add VLOG() statements for every FUSE entry point.  Turning up the VLOG level in
EdenDispatcher now allows us to have a record of all FUSE calls made using
inode numbers.  (Calls involving file handles or directory handles currently go
directly to the handle in question and do not pass through the EdenDispatcher.
We should probably add log statements for those too, but that can wait for a
later diff.)

Reviewed By: bolinfest

Differential Revision: D4359066

fbshipit-source-id: b378d829c39fdb61faf63d2d400f3ff556c376e8
2016-12-22 18:13:02 -08:00
Adam Simpkins
3e41ecaa26 Remove FileInode::parentInode_
Summary:
This member variable was not updated properly when files were renamed.
InodeBase now tracks our parent properly, so we don't need our own copy.

This does still call getParentBuggy() (which does not perform proper locking)
in a couple places for performing overlay operations.  We'll need to fix this
later when addressing other overlay concurrency handling issues.

Reviewed By: bolinfest

Differential Revision: D4348481

fbshipit-source-id: 19c1ffced6f63e1ff041d0bab2363fecdb93d5a3
2016-12-22 15:36:29 -08:00
Adam Simpkins
9ba08b9d1e Update FileData to store a pointer to its FileInode
Summary:
Have FileData objects store a pointer to the FileInode that owns them, rather
than just to the EdenMount.

FileData objects still store a direct pointer to their FileInode's mutex_ and
entry_.  It's potentially worth just accessing these through inode_ in the
future.

Reviewed By: bolinfest

Differential Revision: D4348103

fbshipit-source-id: 1f8497979bfc89c6a192ca0195209335db0d911c
2016-12-22 15:36:29 -08:00
Adam Simpkins
0156f01d51 Update InodeBase to contain a pointer to its EdenMount
Summary:
Add an EdenMount* member variable to InodeBase.  Previously each TreeInode kept
a pointer to its EdenMount, and this moves that into the InodeBase class.

This is needed to support upcoming diffs that will change InodeBase memory
management.  InodeBase objects will be responsible for notifying the InodeMap
when they become unreferenced.

Reviewed By: bolinfest

Differential Revision: D4348079

fbshipit-source-id: 6bf7ea908c6096aa2bca5b21290c09cbd58d5af7
2016-12-22 15:36:29 -08:00
Adam Simpkins
5b346bbff2 update code to use InodeMap
Summary:
This updates all of the eden code to use the new InodeMap class.  This replaces
the InodeNameManager class and the unordered_map previously stored in the
EdenDispatcher.

Reviewed By: bolinfest

Differential Revision: D4325750

fbshipit-source-id: d80ae7581ba79ca2b63155e184995a3e83e85dc1
2016-12-22 15:36:29 -08:00
Adam Simpkins
435a0ca2a1 introduce a new InodeMap class
Summary:
This diff starts adding a new InodeMap class.  This class will eventually
consolidate the functionality of InodeNameMap plus the inode map stored in
EdenDispatcher.

This new class will bring several new benefits:

- All inode mapping logic consolidated into a single class, with a single lock
  protecting the maps.
- A well-defined model for loaded vs unloaded inodes.  InodeMap explicitly
  tracks inodes that have InodeBase objects created for them vs inodes that
  have an inode number allocated (for FUSE) but do not have an InodeBase object
  in memory.  This will make it possible to unload Inode objects on demand to
  reduce memory usage.
- Tracking of pending loads, and de-duplicating load requests.  This ensures
  that only one Inode object ever exists for a given inode number / path.  If a
  second request to load an Inode arrives when a previous load request is still
  in progress, InodeMap deals with this situation properly.
- Better support for using Inode objects without FUSE.  With the old code,
  attempts to interact with Inode objects without going through the FUSE
  dispatch (say, when processing a thrift call) could result in inconsistent
  state.  New inodes created would not be put into the EdenDispatcher map,
  which could result in problems.
- More convenient child inode access from TreeInode.  With this change, the
  TreeInode class can easily tell which of its children are loaded.  This makes
  it easier to do tasks which only need to operate on existing loaded inode
  state (for instance, dirstate computation).
- Support for saving and loading state, to implement graceful restart..
  InodeMap provides a central place to write out inode state on shutdown and
  restoring it on startup.  Saved inodes can easily be restored to an
  "unloaded" state on startup.  This code is not implemented yet as part of
  this diff, but it should be straightforward to add in future diffs.

Reviewed By: bolinfest

Differential Revision: D4318060

fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-22 15:36:29 -08:00
Michael Bolin
a3003bf596 Introduce integration tests for the Mercurial extension.
Summary:
This introduces an `HgExtensionTestBase` class that can be subclassed
to create tests for Mercurial integration with Eden. It also introduces
`status_test.py`, which includes some basic integration tests for `hg status`
in Eden. Although our existing `.cpp` tests give us a lot of coverage, they
do not verify the logic in the Python code for our Mercurial extension,
so this will help me sleep better at night.

Note that `HgExtensionTestBase` is a subclass of `testcase.EdenHgTest`
rather than using the `testcase.eden_repo_test` decorator because these
tests will only be used in the Mercurial case for now, so there is no point in
trying to run them when creating the mount from a Git repo, as well.

Further note that this also required a small change to `eden/fs/integration/lib/testcase.py`
because the Eden directory was not being created at `$HOME/local/.eden`, which
is something that the Mercurial extension relies on.

Reviewed By: simpkins

Differential Revision: D4337660

fbshipit-source-id: ea825134420ae6eb794d37b6c75da19a6aadaae5
2016-12-21 21:58:16 -08:00
Michael Bolin
c05b6153fb Refactor find_executables logic for integration tests.
Summary:
This preserves the existing behavior while making `REPO_ROOT` and `BUCK_OUT`
available to other test code.

Reviewed By: simpkins

Differential Revision: D4358875

fbshipit-source-id: a7b2cafe7e56673831beac59a85fcab436e637a5
2016-12-21 17:51:10 -08:00
Michael Bolin
3f2f22d8fa Normalize build targets to always start with //.
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.

This revision was initially created by running:

```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```

And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.

But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.

Reviewed By: simpkins

Differential Revision: D4356724

fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
2016-12-21 16:28:02 -08:00
Adam Simpkins
1a6ba19f67 implement a FakeBackingStore class
Summary:
Rename the existing TestBackingStore class to FakeBackingStore, and fill it out
with an implementation that allows test code to control the store.

The test code can populate the store with Trees and Blobs to return, and can
control when the Futures returned by the store are fulfilled.

Reviewed By: bolinfest

Differential Revision: D4338577

fbshipit-source-id: 79221b04d844bd6011078b799e55182de4ccdfdc
2016-12-20 16:24:17 -08:00
Adam Simpkins
7de3f50a2e update hg_import_helper.py to work with the latest mercurial code
Summary:
Upstream mercurial commit d83ca854fa21bef17bf3f097087c691c7d07bb5b changed the
mercurial.ui API, breaking existing scripts using it.

ui objects now have to be created using the ui.load() classmethod, rather than
by constructing them directly.  (Creating a ui object directly no longer loads
the global or user configs.)

Reviewed By: bolinfest

Differential Revision: D4354629

fbshipit-source-id: 650db3abab9f2eebaefa2b4d8029c169ccc30217
2016-12-20 15:11:00 -08:00
Mathieu Baudet
89ad59d673 fbcode: remove unused includes from .cpp files with no #if and #define
Summary:
This is a first diff to remove the "easiest" unused includes in fbcode.

* For safety, we only touch .cpp files without #if and #define,
* We do not try to remove redundant systems headers (aka. "packing").

The diff was generated as follows:
```
foundation/scripts/ls-cpp-dirs | grep -v '^\(\.\.\|external/\|.*/external\)' | xargs ffmr -o /tmp/ffmr-diff-1 codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh

cat /tmp/ffmr-diff-1/*.diff | patch -p2
hg commit -m something
arc diff --prepare --nolint --nounit --less-context --excuse refactoring
```

Note: `grep -v` is just an optimization. The actual configuration is in these two files:
diffusion/FBS/browse/master/fbcode/codegraph/analysis/config.py
diffusion/FBS/browse/master/fbcode/codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh

See the task for more context, and the recent "safety" improvements on the tool.

depends on D4317825 for very few cases where `nolint` had to be manually added.

Reviewed By: igorsugak

Differential Revision: D4312617

fbshipit-source-id: ecc1f0addfd0651fa4770fcc43cd1314661a311a
2016-12-17 18:24:09 -08:00