Commit Graph

14419 Commits

Author SHA1 Message Date
Adam Simpkins
582e9ea49d add an EdenClient.run_cmd() helper function for the tests
Summary:
Add a run_cmd() function that's slightly simpler to use compared to
_get_eden_args().

Reviewed By: bolinfest

Differential Revision: D3448844

fbshipit-source-id: 06b654b2f4bc773d67a4f9d1f6effdad2ce9aa2a
2016-06-17 15:16:02 -07:00
Adam Simpkins
94a26b5503 improvements to the daemon and shutdown commands
Summary:
- Update "eden daemon" to wait until edenfs is healthy before returning.
- Accept additional command line arguments and pass them through to edenfs.
- When starting edenfs as a background daemon, direct stdout and stderr to a
  log file in the eden config directory.
- Update "eden shutdown" command to wait for edenfs to exit.
- Update "eden shutdown" command to print an error message if edenfs was not
  running.

Reviewed By: bolinfest

Differential Revision: D3446403

fbshipit-source-id: 2ca1874256b7d124a85092b886548bea8c198327
2016-06-17 15:12:48 -07:00
Adam Simpkins
382ad643c1 improve start-up handling of hg_import_helper.py
Summary:
Look for hg_import_helper.py relative to the edenfs binary, rather than
relative to the current working directory.  Also check in a couple places, so
we can work in both normal deployment locations as well as inside a source
repository.

Additionally, update hg_import_helper.py to emit a response chunk as soon as it
starts.  This contains either message indicating that it has started
successfully, or an error message.  This allows us to propagate error
information back to the thrift caller if something goes wrong starting
hg_import_helper.py for a given repository.

Reviewed By: bolinfest

Differential Revision: D3447027

fbshipit-source-id: 4bffd1d03ab6475c09cb3c8385145555c7fc7361
2016-06-17 15:12:48 -07:00
Yedidya Feldblum
d950fdeaed Wrappers for some of OpenSSL's crypto hash functions
Summary:
[Folly] Wrappers for some of OpenSSL's crypto hash functions.

Wraps some of the OpenSSL crypto hash functions with variants that take `ByteRange` for input and `MutableByteRange` for output, and also variants that take `const IOBuf&` for input as well.

These are a bit nicer to use than passing pointers and lengths separately.

Reviewed By: ivmaykov

Differential Revision: D3434562

fbshipit-source-id: 3688ef11680a029b7664ac417a7781e70f9c6926
2016-06-16 18:30:50 -07:00
Adam Simpkins
cbb493f716 support running in the foreground, and improve test cleanup
Summary:
This adds a --foreground flag to the "eden daemon" command, which causes the
daemon to run in the foreground, rather than daemonizing as a background
process.  Additionally, this drops the --preserve-environment flag and instead
updates the CLI to always build a sane environment to run with.

This also updates the integration test code to use this flag, and wait for eden
to shut down during test clean up.  Without this, the test case code could try
to clean up temporary directories before they were unmounted, which would fail.
This would leave many temporary directories behind after test runs.

This also re-organizes the temporary directory set up a bit, so that each test
uses a single top-level temporary directory.  All other directories it creates
are put inside this one directory.

Reviewed By: bolinfest

Differential Revision: D3439232

fbshipit-source-id: 85305f7db60da6bae589d28c802ee260e42e4dea
2016-06-16 12:44:06 -07:00
Michael Bolin
26403df852 Exclude the cpp_binary for the daemon from the python_binary for the CLI.
Summary:
This changes the way that Eden is built and deployed.

* To build the binary that must be run as `root` (but quickly drops privileges), run `buck build eden-daemon`.
* To build the CLI that communicates with the daemon (and does not require privileges), run `buck build eden-cli`.
* To build both, run `buck build eden`.

There is an example of how to build the various parts of Eden using
Buck and how to package them up in the `install` script introduced by this revision.

While here, I also cleaned up some of our build files and changed them to be
parameterized between internal and external use. In both cases, the user gets the
"unadorned" version of their primary build targets. This ensures that shortcuts such as:

```
buck test eden/fs/integration
```

do the right thing by default.

Finally, I also made `find_default_config_dir()` and `find_default_daemon_binary()`
lazy whereas `find_default_config_dir()` was previously eager.

Reviewed By: simpkins

Differential Revision: D3436245

fbshipit-source-id: 4dfbd59ed0d198620324f0705c462334bb5a7daf
2016-06-15 17:07:58 -07:00
Adam Simpkins
1eed0364e3 always show tree contents for (non-opaque) directories in the overlay
Summary:
If a directory is present in the overlay, we still need to check if a TreeEntry
exists from the source control data structures.  Previously this was causing us
to incorrectly report directories as empty if they exist in the local overlay,
even when they had contents from the main Tree.

Reviewed By: wez

Differential Revision: D3434219

fbshipit-source-id: f872f90075602dfdc7b217f50eefcd7c248512e7
2016-06-15 14:24:12 -07:00
Adam Simpkins
183b6f208e add some debug logging in ObjectStore.cpp
Summary:
Add some verbose logging about when trees and blobs are loaded in the object
store.

Reviewed By: bolinfest

Differential Revision: D3434182

fbshipit-source-id: 3e8d2617290604f119e6164d15d63324a4c9a2aa
2016-06-15 14:24:12 -07:00
Adam Simpkins
5f639c037b support retrieving file data from mercurial
Summary:
Update the HgImporter class to support retrieving file contents from mercurial.

This also includes simple code for storing the data in the LocalStore using
git's blob serialization format.  In the future I think it would perhaps be
better to drop the "blob<length>" prefix, and instead just use a RocksDB column
family to separate blob data from other types of data.  However, for now using
the git format is simplest for keeping compatibility with the getBlob() code.

Reviewed By: bolinfest

Differential Revision: D3416691

fbshipit-source-id: 268787533be2172b2dbedc3bf06464eabf3d2c5e
2016-06-15 14:24:11 -07:00
Adam Simpkins
169f050b7f update HgBackingStore to track already imported commits
Summary:
Record mercurial commits that have been previously imported, and avoid
re-importing them when checking out a commit that was previously processed and
already has a tree present in the LocalStore.

Reviewed By: bolinfest

Differential Revision: D3416654

fbshipit-source-id: 80f70fa2204d3a2961991b634ad934a623b42b82
2016-06-13 15:16:30 -07:00
Adam Simpkins
6a9f974f31 add a generic LocalStore get() and put() methods
Summary:
Add APIs for storing arbitrary (key, value) data.

This will allow BackingStore implementations to store additional metadata, such
as mapping mercurial commit IDs to the eden root tree ID.

Eventually we may want to use RocksDB column families to partition the
different types of data being put into the LocalStore.  However, for now this
just uses a single key space.  We can add column family support in a separate
diff, if desired.

Reviewed By: bolinfest

Differential Revision: D3409866

fbshipit-source-id: 19a1d340b65bff2081981bf5daf32d5ad15b60c4
2016-06-13 15:16:30 -07:00
Adam Simpkins
eae8ee41e9 start adding an HgBackingStore implementation
Summary:
This adds an HgBackingStore implementation which can load tree data from a
mercurial repository.  Blob loading is not implemented yet, but will come in a
separate diff.

This also adds a minimal GitBackingStore class.  The GitBackingStore has nearly
no functionality, but is needed to keep the existing git functionality working.

Reviewed By: bolinfest

Differential Revision: D3409743

fbshipit-source-id: dbebf53e9de08bd1469e489baa48b84cbf889511
2016-06-13 15:16:30 -07:00
Adam Simpkins
d9be0757b8 add a BackingStore API
Summary:
Add the basic BackingStore interface, plus a NullBackingStore implementation
that always returns null.  This updates the ObjectStore to query the
BackingStore if data is not found in the LocalStore.

Additionally, this updates EdenServer to manage the BackingStore objects.  It
maintains a map of the BackingStore objects created for each known repository.

Reviewed By: bolinfest

Differential Revision: D3409602

fbshipit-source-id: 2920dc4c24ee1ec37efb542f058d0d121ceb5532
2016-06-13 15:16:29 -07:00
Adam Simpkins
346a4cdc4e fix CLI issues initializing mercurial mount points
Summary:
The _get_hg_dir() function had an incorrect return statement caused "eden init"
to fail with mercurial repositories, unless the path pointed to a shared
working directory.

Also update the code to store the path to the mercurial repository, rather than
the path to the .hg directory.  Unlike git, mercurial requires the repository
path itself, and refuses to work if given a path to the .hg directory.

Reviewed By: bolinfest

Differential Revision: D3409808

fbshipit-source-id: 22ee7748422b614f5af57aff07fcfa517af846d9
2016-06-13 15:16:29 -07:00
Michael Bolin
11169bcf95 Add an eden shutdown command.
Summary:
This revision introduces two complementary changes:
* `eden daemon` no longer runs in the foreground.
* There is now an `eden shutdown` command to kill the daemon.

When `shutdown` is called, it tells the Thrift server to shutdown.
In turn, this causes `EdenServer::runThriftServer()` to exit,
which causes `EdenServer::run()` to exit.

Reviewed By: simpkins

Differential Revision: D3402347

fbshipit-source-id: 80032ba53eb69b3f69bef9d7cd169f93500c833c
2016-06-10 14:16:09 -07:00
Caren Thomas
27fe1e2306 track latency for eden fuse operations
Summary: Add stats tracking for eden. Thought it would be best to collect some feedback before I go ahead and add tracking for all of the operations in the "dispatcher_ops" structure.

Reviewed By: simpkins

Differential Revision: D3387902

fbshipit-source-id: 2fdb22d3b7d0248f67255614f82442379e53e212
2016-06-10 11:27:09 -07:00
Adam Simpkins
fab40060f1 unbreak gcc-4.9 builds
Summary:
D3406773 included a change which compiles on clang and gcc-5.x, but fails to
build with gcc-4.9.

This looks like a bug in gcc-4.9's handling of list initialization.  Overload
resolution for non-initializer-list constructors should be attempted if
no suitable initializer-list constructors are found, but gcc-4.9 does not
appear to do this.

Reviewed By: bolinfest

Differential Revision: D3410142

fbshipit-source-id: f34125000eb3fa949c2427aa4ffbd4ef92942cd7
2016-06-09 22:15:05 -07:00
Adam Simpkins
e7a8605e0d update deserializeGitBlob() to accept an IOBuf
Summary:
Update deserializeGitBlob() to work on an IOBuf, rather than an rvalue
reference to a string.

The ugliness about having to wrap a std::string in a managed IOBuf is now
hidden inside the StoreResult class, rather than being something that the
GitBlob code has to know about.

Reviewed By: bolinfest

Differential Revision: D3403977

fbshipit-source-id: 0c58c019557050d6e201c1a462fa051c2526674a
2016-06-08 19:01:14 -07:00
Adam Simpkins
1b36d4bf83 add a StoreResult class
Summary:
Add a new StoreResult which wraps the std::string returned by RocksDB.

This replaces the std::unique<string> that LocalStore::get() used to return.
This lets us avoid a memory allocation.  StoreResult can also represent a "not
found" result, so that this case can be processed efficiently without having to
throw an exception.

Additionally, StoreResult is move-only so we can't ever unintentionally copy
the string data, which is potentially expensive.  It also provides APIs for
creating IOBuf wrappers, or moving the string to the heap so we can create an
managed IOBuf around it.

Reviewed By: bolinfest

Differential Revision: D3403958

fbshipit-source-id: ab0c304988a53eda50341ecc2f96ae5235e5260c
2016-06-08 19:01:13 -07:00
Adam Simpkins
32f4c458fe begin adding a new ObjectStore class
Summary:
Add a new ObjectStore class, which will eventually contain both a LocalStore
and a BackingStore.  The LocalStore will be a cache of data loaded from the
authoritative BackingStore.  The ObjectStore API will hide the work of querying
the BackingStore and updating the LocalStore when data is not already available
in the LocalStore.

For now ObjectStore only contains the LocalStore, but I will add BackingStore
functionality in subsequent diffs.  This diff simply updates all call sites to
use the ObjectStore instead of directly accessing the LocalStore.

Reviewed By: bolinfest

Differential Revision: D3403898

fbshipit-source-id: 47b8c51a7717a4c7c29911a7085b382521a8c0db
2016-06-08 19:01:13 -07:00
Adam Simpkins
4147c7b937 make Hash objects assignable, and add a default constructor
Summary:
Previously Hash objects could not be assigned to after they were created, since
they contained a const member.  This makes the data non-const, so a Hash
variable can be replaced to contain new contents after it is created.

This also adds a default constructor, which zero-initializes the hash.  The
default constructor makes it possible to declare a Hash with a 0-value at one
location, and then set it to the desired value at some later point.

Reviewed By: bolinfest

Differential Revision: D3406773

fbshipit-source-id: 41e2c7e3ad5bc4d14813be4adaa03866701380f6
2016-06-08 16:16:59 -07:00
Adam Simpkins
96cea91e54 various minor efficiency improvements in LocalStore
Summary:
- Add a Sha1Key class that can more efficiently compute the key for
  file content SHA-1 values, without having to copy it into a new std::string
  object.  (In practice fbstring would have avoided having to actually allocate
  memory, but it was still an extra data copy.)

- The code was always converting the hash keys to hex on get and put
  operations, just in case it needed it if an error occurred.  This diff
  changes the code to only compute the hex value if an error actually occurred.

Reviewed By: bolinfest

Differential Revision: D3403889

fbshipit-source-id: 5abd8ef202cb00677a84a03a82e2a3d21f16cd2f
2016-06-08 14:54:01 -07:00
Adam Simpkins
947dc27e3e use std::array when possible
Summary:
Update several places to use std::array rather than plain C arrays, using
folly::make_array() to automatically deduce the correct type when necessary.

Reviewed By: wez

Differential Revision: D3370445

fbshipit-source-id: b7642cf3a9b08eac817988bf95679bf5e584ef72
2016-06-08 00:15:22 -07:00
Adam Simpkins
5b3af5db6d add initial mercurial tree import code
Summary:
Add an HgImporter class for importing mercurial data into the eden local store.
At the moment this only includes code for importing revision manifest data as
tree objects, and does not yet include code for importing file blob data.

Reviewed By: bolinfest

Differential Revision: D3367958

fbshipit-source-id: 58049bf1594b3c27d676c5ebe778917b4043fccf
2016-06-08 00:15:22 -07:00
Adam Simpkins
5b65743a38 update deserializeGitTree() to work with IOBuf
Summary:
Update deserializeGitTree() to accept an IOBuf object.  IOBuf objects can
easily wrap other buffers, so this can still easily support ByteRange objects
as well.

Being able to use IOBuf's Cursor class ended up simplifying the logic a bit as
well.

Note that using IOBuf does require copying the name and mode data out of the
buffer when we read it (using the readTerminatedString() API).  This is
necessary since the data may not be stored contiguously in the IOBuf.  However,
this shouldn't impact performance much: we already need to copy the name data
into a std::string anyway.  For the mode, most modern platforms can avoid doing
a heap allocation for this small string.

Reviewed By: bolinfest

Differential Revision: D3357255

fbshipit-source-id: 5b6e1bc93199849327409a8039266d7dc4f3afdf
2016-06-08 00:15:22 -07:00
Adam Simpkins
d414ee1021 add logic for serializing git trees
Summary: Add a GitTreeSerializer class for serializing git tree data.

Reviewed By: bolinfest

Differential Revision: D3356770

fbshipit-source-id: d04bc9788117272504c2faa335b3648e4ac93e81
2016-06-08 00:15:21 -07:00
Caren Thomas
d2ef81d6cf clean up serialize/parse functions
Summary: create generic serialize/parse functions to be called by mount, unmount, bindmount serialize/parse methods

Reviewed By: wez

Differential Revision: D3374062

fbshipit-source-id: 6d1f2b505fee86a0dd0b53d862f4060142a54bf5
2016-06-07 13:32:03 -07:00
Michael Bolin
7a48628bb8 Create dot eden folder if it does not already exist when running eden daemon.
Summary:
Previous to this change, if `~/local/.eden` did not exist when `eden daemon` was
run, then it would fail.

Now the logic to create `~/local/.eden` is encapsulated in `_ensure_dot_eden_folder_exists()`.
We do not call this for all subcommands because we want to make sure that running
`eden --help` does not have the side-effect of writing `~/local/.eden` if it does not exist.

Reviewed By: wez

Differential Revision: D3397057

fbshipit-source-id: a3f974f367058d9e4ebd515c78423e54edc179cc
2016-06-07 13:03:13 -07:00
Michael Bolin
5e69f112eb Add an eden health subcommand.
Summary:
`eden health` will return with exit code 0 if both of the following are true:

* The Thrift client is up and running.
* The status of the client is either ALIVE or STARTING.

Reviewed By: wez

Differential Revision: D3395582

fbshipit-source-id: ba668d26acae73a51fbae8aca2b4979156c0c50f
2016-06-07 13:03:13 -07:00
Michael Bolin
1391f7725c Fix some Python lint warnings.
Reviewed By: simpkins

Differential Revision: D3394466

fbshipit-source-id: 3dd670785e0eb2586ed38be87e74caeb3262966b
2016-06-06 21:01:03 -07:00
Adam Simpkins
bff4754bfe ignore EPERM errors when trying to kill eden
Summary:
When running the CLI "daemon" command, we try killing the underlying eden
process group when we receive SIGTERM or SIGINT.  (We really only want to kill
the main eden process, but we currently have to kill the entire process group
due to how sudo works.)

Since the privhelper process runs as root and is part of this process group, we
can get an EPERM error back.  This was causing the CLI to fail with an
unhandled exception backtrace.  This diff updates the code to ignore EPERM.

Reviewed By: bolinfest, wez

Differential Revision: D3384121

fbshipit-source-id: 39b2364d8c921b1d84a8902566fe9af2a370e4e5
2016-06-06 13:01:58 -07:00
Adam Simpkins
e66d3f81dc add a kDirSeparator constant
Summary:
Define a constant for the path directory separator ('/').  We currently plan to
normalize all pathnames to use forward slash as a directory separator, even on
Windows.  This will simplify the bulk of the eden code.

Even though we don't plan to make this separator character configurable, it
still seems useful to define a symbolic constant for it.

Reviewed By: bolinfest

Differential Revision: D3377110

fbshipit-source-id: cf6b158824e0cb318cc94838618506e78d868c9f
2016-06-02 22:08:15 -07:00
Adam Simpkins
aa3b5aa8ed Add Path::findParent(), isSubdirOf() and isParentDirOf()
Summary: Add functions for checking if one path is in a subdirectory of another path.

Reviewed By: bolinfest

Differential Revision: D3367035

fbshipit-source-id: 3bd10d88bded2ccf0066d273bc5fe5c447287cb6
2016-06-02 22:08:15 -07:00
Adam Simpkins
45ff446f41 fix issues with reverse path iterators
Summary:
The reverse path iterators derived from the forward iterators, and overrode the
pre-increment operator.  Unfortunately they didn't override post-increment,
which meant that post increment actually advanced them forwards instead of
backwards.  (The base class's post-increment calls the base class's
pre-increment, since this method is not virtual.)  The reverse iterators also
weren't overriding the decrement operators added in D3366877, so decremening a
reverse iterator would move it in the wrong direction.

This fixes all of those issues by making the ComposedPathIterator template
smarter so it can correctly handle both forward and reverse iteration.
It also now contains the logic that the "begin" iteration for absolute paths
needs to start one character in.  ComposedPathIterator needs to be aware of
this in order to stop at the correct character when iterating backwards.

Reviewed By: bolinfest

Differential Revision: D3376727

fbshipit-source-id: c61c86c1a9233507f8297015439e416680aaa39b
2016-06-02 22:08:15 -07:00
Adam Simpkins
892e078f8c changes to path iterator behavior
Summary:
This modifies the iterator behavior to so the behavior is a bit cleaner
with respect to empty paths.  It is valid to have an empty relative path,
and there are legitimate use cases where this is useful.  For instance,
calling dirname() on a RelativePath with a single component will result in
an empty path.  It is useful to use this empty path to refer to the parent
directory, to which the path is relative.  Therefore it is also useful to
be able to include the empty path when iterating through the parent
directories of a path.

This removes RelativePath::begin() and RelativePath::end(), and replaces
them with a RelativePath::paths() function.  paths() returns a struct with
a begin() and end() function, so it can be used in range-based for loops,
and has the same behavior that begin()/end() did.  This also adds a
RelativePath::allPaths() function, which also includes the empty relative
path in the results.

Reviewed By: bolinfest

Differential Revision: D3366877

fbshipit-source-id: 3d92b600f07b993925f88d4f1e619b6c1705fb82
2016-06-02 22:08:15 -07:00
Adam Simpkins
a34502cfd8 delete ComposedPathIterator::operator->()
Summary:
Unfortunately operator->() can't really be implemented properly, since
ComposedPathIterator doesn't actually point to an already-existing path piece.

This removes the broken implementation, and leaves a comment in its place.

Reviewed By: wez

Differential Revision: D3366836

fbshipit-source-id: acbd0332a0a7a1fc9eac114ef0c06ed605adbbf3
2016-06-02 22:08:14 -07:00
Caren Thomas
1117f21e19 handle unmount through privhelper process
Summary:
PrivHelper serializes messages and sends it over to PrivHelperServer who verifies that mount point exists, cleans up bind mounts for the FUSE mount, and undoes FUSE mount.

Some repeated code in this diff since I was unsure on the protocol for that - let me know if/where I should generalize functions to avoid this.

Reviewed By: simpkins

Differential Revision: D3361955

fbshipit-source-id: a7324fb9660912d6c2b753e15b1fa6061c0d5261
2016-05-31 13:17:03 -07:00
Michael Bolin
499f72a9f8 Introduce TreeEntryFileInode::getSHA1()
Summary:
This avoids translation from string->Hash in the common case
where the file is unmodified and its hash is read directly from
the store rather than computed from the overlay.

I'm guessing I should use `unique_ptr` as the return value throughout?

Reviewed By: simpkins

Differential Revision: D3355773

fbshipit-source-id: 50dff879a78b3d6ff49f86b856866ca28808c4f7
2016-05-27 18:17:07 -07:00
Michael Bolin
5f7a0c287f Add a Thrift API to get the SHA-1 of a file.
Summary:
Other tools, such as Buck, will benefit from being able to get
the SHA-1 of a file without having to read the entire contents
of the file (or do the associated computation that is proportional
to the size of the contents of the file).

Reviewed By: simpkins

Differential Revision: D3345828

fbshipit-source-id: 360bb268793369af75f408208e8211d8b9db146d
2016-05-27 18:17:07 -07:00
Caren Thomas
203be051a1 add unmount parser and wrapper functions
Summary: Updated python CLI to include subparser for unmount command and added wrapper functions that hand over execution to privhelper process. Unmount currently requires client_name at the command line.

Reviewed By: simpkins

Differential Revision: D3359517

fbshipit-source-id: ff05e90bcdb96ecad63f37634c69dbeef429c90f
2016-05-27 17:41:07 -07:00
Adam Simpkins
c769088f16 add Hash::sha1() factory functions
Summary:
Add some static helper functions to create Hash objects by running a SHA1 hash
on input data.

Reviewed By: wez, bolinfest

Differential Revision: D3354594

fbshipit-source-id: 6d6bfb835175e7a25c1e6e2539438bee5887a863
2016-05-27 16:36:14 -07:00
Adam Simpkins
106717e4e7 update Hash::getBytes() to return a folly::ByteRange
Summary:
Change Hash::getBytes() to return a folly::ByteRange rather than a
std::array<uint8_t, 20>.  This makes Hash more convienent to use with existing
APIs that accept a ByteRange.  (For instance, IOBuf.)

There were only 2 call sites using the existing getBytes() functionality,
and they only used the data() method on the returned std::array, so they don't
have to be updated at all to use a ByteRange.

Reviewed By: bolinfest

Differential Revision: D3354581

fbshipit-source-id: 8f2a3c196e59620fb5b0fb2caf4d1d7f26e1d2c4
2016-05-27 16:36:14 -07:00
Michael Bolin
4897086780 Introduce InodeDispatcher::lookupInodeBase().
Summary:
This is an alternative to `InodeDispatcher::lookup()` that does not result
in `RequestData::get()` being called, which is important for requests that
do not originate from FUSE.

The `::lookup()` call takes the `InodeBase` and invokes its `getattr()` method,
which in the case of `TreeEntryFileInode::getattr()`, creates a `fusell::Dispatcher::Attr`
whose constructor calls the following:

```
Dispatcher::Attr::Attr() : timeout(1.0) {
  memset(&st, 0, sizeof(st));
  auto& req = RequestData::get();
  auto mount = req.getChannel().getMountPoint();
  st.st_uid = mount->getUid();
  st.st_gid = mount->getGid();
}
```

So the only reason this is done at all is to set `st_uid` and `st_gid`.

And then the only reason `Attr` is needed is to set the following fields on a `fuse_entry_param`:

```
attr
attr_timeout
entry_timeout
```

So it is possible that we can find a better way to streamline all of this, but this was
the easiest shortcut I could implement myself right now.

Reviewed By: wez

Differential Revision: D3351819

fbshipit-source-id: b095c085ee4a9b9a81438db093869fd0acf1f8ad
2016-05-26 09:50:11 -07:00
Michael Bolin
4db509b508 Reformat InodeDispatcher.cpp
Summary:
I am going to submit some changes to `InodeDispatcher.cpp` in a
follow-up diff where I definitely want to benefit from auto-format,
so to make that diff less noisy, I'm doing the auto-format in this revision.

Reviewed By: wez

Differential Revision: D3351888

fbshipit-source-id: f8fdebe43f247e2522f1f14b3ee4bac37f564c70
2016-05-26 09:50:11 -07:00
Wez Furlong
dde572cf5e eden: sha1 attributes on overlay [2/2]
Summary:
Add a function to compute the sha1 content hash for an overlay file.

We persist the computed hash in an extended attribute in the underlying overlay
file so that a subsequent read of the attribute doesn't require opening the
file to recompute it.

Each time the file is mutated, we blow the cached status of the file.
Each time the sha1 attribute is read, if the cache is blown, the content
hash will be recomputed and set in the overlay file.

Each time the file is flushed or sync'd, if the cache is blown, the content
hash will be computed and set in the overlay file.

Reviewed By: bolinfest

Differential Revision: D3302412

fbshipit-source-id: bd45c7a24b732bd0b7474b7f96e82936870b2117
2016-05-26 08:23:11 -07:00
Wez Furlong
056d08bbbe eden: sha1 attributes on overlay [1/2]
Summary:
This is part 1 of 2 diffs.  This one adds some plumbing to make it possible
to read the xattr attribute from an overlay.

It doesn't do anything to ensure that it is set; the next diff in this series will take care of that.

Reviewed By: bolinfest

Differential Revision: D3302410

fbshipit-source-id: 47406a9c75f29743691d396676c691bcb99c4760
2016-05-26 08:23:11 -07:00
Michael Bolin
6e4cb1ebf1 Refactor logic to create a EdenService.Client in Python.
Summary: This logic should be shared by the Eden CLI as well as unit tests.

Reviewed By: simpkins

Differential Revision: D3348300

fbshipit-source-id: c87b1f03f16560323f3d7685063bb6466c39efe2
2016-05-25 21:44:07 -07:00
Wez Furlong
ff4bc88da6 eden: remove InodeNamgeManager singleton
Summary:
We look this up via the mount point or eden mount object instead.

I've also removed the mercurial library stuff that was added to support the now defunct lamehg fuse we had in the earlier days.
simpkins' new importer doesn't use these and it resolves our CI mismatch issue.

Reviewed By: bolinfest

Differential Revision: D3349698

fbshipit-source-id: 5f4ec16b76042959cd1e3184f46bb3526fbaf74c
2016-05-25 19:34:16 -07:00
Adam Simpkins
98dcc73503 initial CLI support for creating hg mounts
Summary:
Update the CLI to support running "init" with a mercurial repository.

This is just some bare bones framework code at the moment.  It doesn't actually
import any data from mercurial at the moment, and mounting doesn't work.

Reviewed By: bolinfest, wez

Differential Revision: D3345426

fbshipit-source-id: 72c31ac8d2aac2a16e0a7d6f0425eb4ca218d487
2016-05-25 13:15:01 -07:00
Adam Simpkins
c8ee028d3a display rocksdb key in hex if LocalStore::get() fails
Summary:
Use the hex-encoded version of the key in the RocksException if _get() fails,
rather than the raw binary data.

Reviewed By: bolinfest

Differential Revision: D3345355

fbshipit-source-id: cd8dc644a56ca3d5f3b9a9a0f5cc789b142f0bda
2016-05-24 22:14:14 -07:00
Michael Bolin
a65e7fb98a Update directions for building Java thrift bindings with Java 7.
Summary:
Buck is [currently] built with Java 7, so it can only use third-party dependencies
that are also Java 7.

Reviewed By: simpkins

Differential Revision: D3342367

fbshipit-source-id: 4370fd152e7d2055495e783de68a6bb59867bee5
2016-05-24 21:53:00 -07:00
Michael Bolin
d6d5d6c695 Set up bind mounts for a client when mounting it.
Summary:
This adds a new API to `PrivHelper`: `privilegedBindMount()`.
Similar to `privilegedFuseMount()`, this sends a message to the privileged helper,
which is running as `root`, so it can set up the specified bind mount.
The changes in the `privhelper` directory parrot what was done to support `privilegedFuseMount()`.

Now, once the primary mount for a client is created, any bind mounts listed in the
config for the client are set up. This logic is introduced in `EdenServer.cpp`.

Reviewed By: simpkins

Differential Revision: D3296660

fbshipit-source-id: 61296f35e5c3a6f232a1c17e0f296dd5d3b5ec06
2016-05-23 21:33:20 -07:00
Michael Bolin
20ce44db52 Fix some errors I was seeing about raw pointer/unique_ptr consistency.
Reviewed By: wez

Differential Revision: D3333566

fbshipit-source-id: aae3bdbace416ab90ca32607951be3117dd4dbbb
2016-05-23 14:13:04 -07:00
Michael Bolin
cec2037036 Use glob() and local include.
Summary:
`glob()` prevents us from adding `.cpp` files to this directory that do not get compiled.

The header without the long path makes it clear that it is in the same directory as `PrivHelperTest.cpp`.

Reviewed By: simpkins

Differential Revision: D3301264

fbshipit-source-id: f58351a4f2857c8911f0bf82ae67136920fcb998
2016-05-23 12:24:58 -07:00
Adam Simpkins
53e821eb23 add an EdenMount class
Summary:
Add a new class to serve as a single location where we can store all
information about a single eden mount point.  Currently this contains the
MountPoint, LocalStore, and Overlay objects.  This allows the TreeInode class
to just store a single pointer to the EdenMount, rather than having to track
these three objects separately.

In the future we could consider also keeping a copy of the ClientConfig in the
EdenMount object, but I haven't done that for now.

Reviewed By: bolinfest

Differential Revision: D3321355

fbshipit-source-id: 8a39bb49822ca8e90c88b2a834b59230d2f91435
2016-05-20 10:34:07 -07:00
Wez Furlong
498a3b8aba eden: add mkdir support
Summary:
Enables mkdir in the overlay area.

I had to add some `lstat` calls in to the overlay dir reader because we depend
on knowing at least whether a node is a dir or not at the next level up.

When I run the test suite, the mounts are on my `/tmp` filesystem.  When I run
eden manually, they are on my `/data` filesystem.  The latter (xfs) does not
populate the type bits.  This meant that the test suite passed but manual
testing did not.

Adding the `lstat` calls is a little unfortunate.  On OS X there is a bulk
operation that combines `readdir` and `lstat` so that there are fewer syscalls.
We don't have an equivalent for Linux.

Reviewed By: bolinfest

Differential Revision: D3301532

fbshipit-source-id: e228f4a392f90aa491fec62e8b98471a8acecff2
2016-05-18 12:24:00 -07:00
Wez Furlong
103224b5df eden: remove naked pointers from FileInode::open and DirInode::opendir
Summary:
We still have naked pointers at the handoff to the kernel, but now
have a cleaner implementation at the level that we're going to be working at
day to day.

I also renamed `FileHandle::release` to `FileHandle::releasefile` so that it
isn't visually ambiguous with `std::unique_ptr::release` in the
`Dispatcher.cpp` code: `fh.release()` vs `fh->release()` look similar but are
dramatically different in behavior.

Reviewed By: bolinfest

Differential Revision: D3309455

fbshipit-source-id: f8cf055bcd51121048a20f0202988cf0aef1f085
2016-05-17 18:17:11 -07:00
Wez Furlong
f06b404b25 eden: remove PassThru inode classes
Summary: Now that we have integrated this functionality, we don't need these

Reviewed By: simpkins

Differential Revision: D3307034

fbshipit-source-id: 590469e9e7dec9c6b7d4f7b4e9a8d8c40d6ff552
2016-05-17 18:17:11 -07:00
Wez Furlong
deaf75a6b6 eden: init: require --mount and --repo options
Summary: This avoids an ugly error when --mount isn't passed.

Reviewed By: bolinfest

Differential Revision: D3302970

fbshipit-source-id: e60475e6b94fb4228e75fb392fb472886274aa0a
2016-05-17 18:17:11 -07:00
Wez Furlong
88c1b44aab eden: remove PassThru file handle usage
Summary:
`creat(2)` and `open(2)` could decide to create a PassThru file handle.

This diff removes that usage.  There is a TODO here around handling `O_EXCL` properly.
I'm punting this to a follow-up diff.

Reviewed By: bolinfest

Differential Revision: D3301387

fbshipit-source-id: d35104c536396e7fd064d786f3d5592ecfcbfecf
2016-05-17 17:23:02 -07:00
Wez Furlong
25dd9997d9 eden: add FileData::write, enable writes
Summary:
Centralize and delegate most (all?) of the content sensitive portions of file accesses into the FileData class.

Add tests to show that we can write to the overlay file and that the stat data is consistent with the result.

Reviewed By: bolinfest

Differential Revision: D3301251

fbshipit-source-id: a09316ad61c6ef4c656bc5d6dbd43f906abb7932
2016-05-16 14:59:49 -07:00
Wez Furlong
e9157c8b8b eden: add FileData::materialize
Summary:
This is the workhorse for adjusting the state that we track for the file data.

It handles both overlay and Tree backed data cases and moving from the latter
to the former depending on the open flags provided when a file handle is
opened.

This diff handles more cases than we have tests for.  Those will be covered later in this stack of diffs.

Reviewed By: bolinfest

Differential Revision: D3301213

fbshipit-source-id: c1dab40c0ad205ce6cee820043b70dd886e78431
2016-05-16 14:59:49 -07:00
Wez Furlong
1e3314f2dd eden: remove OverlayFileInode, fold into TreeEntryFileInode
Summary:
Another step towards making TreeEntryFileInode overlay aware, this diff implements:

- stat
- readlink (although it is not possible to create symlinks yet)
- open

Reviewed By: bolinfest

Differential Revision: D3255158

fbshipit-source-id: 3f90b624e629ef279d6cc32e1d82787ee24796eb
2016-05-16 14:59:49 -07:00
Wez Furlong
a34c8f1eac eden: factor out the filedata from the inode instance
Summary:
This allows us to share the same state between multiple open files and
also helps to reduce the size of inode instances that are otherwise inactive;
when there are no outstanding references to the data, we can drop it and forget
it.

Reviewed By: bolinfest

Differential Revision: D3301198

fbshipit-source-id: f0b3fc73a666ec2033c7a22e9eb587d3212cf966
2016-05-16 14:59:49 -07:00
Adam Simpkins
509467399a fix gcc opt build
Summary:
Update the PrivHelperConn code to copy file descriptors in and out of the cmsg
data using memcpy(), rather than dereferencing reinterpret_cast'ed pointers.
These cast's break C/C++'s strict aliasing rules, and gcc complains about this.

Reviewed By: bolinfest, wez

Differential Revision: D3306762

fbshipit-source-id: d36fb5ee02a52da2b60e86fb4a0fdb4910fc72d7
2016-05-16 13:18:05 -07:00
Michael Bolin
c2a0aa3af5 Modify EdenServiceHandler to get client info through ClientConfig.
Summary: This should set us up to have `eden mount` perform the bind mounts.

Reviewed By: simpkins

Differential Revision: D3296370

fbshipit-source-id: 5d8c21308074b357bad3ace72cec157adb5f8b56
2016-05-13 15:10:22 -07:00
Michael Bolin
3feb4c5c9a Change ClientConfig.getSnapshotID() to return a Hash instead of a string.
Reviewed By: wez

Differential Revision: D3296485

fbshipit-source-id: aef7abea6e4931c2129239dbfd44443b1141d549
2016-05-13 15:10:22 -07:00
Facebook Github Bot 8
83f42a9fa6 Include build files that were inadvertently excluded from the initial export.
fbshipit-source-id: 2c76f0d5e55d84859ad9f4841cbe6994a62446f8
2016-05-12 16:08:34 -07:00
Facebook Github Bot 5
2eeea32117 Initial commit
fbshipit-source-id: 2bcefbd0cd127cc5ea982e074ea6819d7aac3d7a
2016-05-12 14:09:13 -07:00