Summary:
- Add a Sha1Key class that can more efficiently compute the key for
file content SHA-1 values, without having to copy it into a new std::string
object. (In practice fbstring would have avoided having to actually allocate
memory, but it was still an extra data copy.)
- The code was always converting the hash keys to hex on get and put
operations, just in case it needed it if an error occurred. This diff
changes the code to only compute the hex value if an error actually occurred.
Reviewed By: bolinfest
Differential Revision: D3403889
fbshipit-source-id: 5abd8ef202cb00677a84a03a82e2a3d21f16cd2f
Summary:
Update several places to use std::array rather than plain C arrays, using
folly::make_array() to automatically deduce the correct type when necessary.
Reviewed By: wez
Differential Revision: D3370445
fbshipit-source-id: b7642cf3a9b08eac817988bf95679bf5e584ef72
Summary:
Add an HgImporter class for importing mercurial data into the eden local store.
At the moment this only includes code for importing revision manifest data as
tree objects, and does not yet include code for importing file blob data.
Reviewed By: bolinfest
Differential Revision: D3367958
fbshipit-source-id: 58049bf1594b3c27d676c5ebe778917b4043fccf
Summary:
Update deserializeGitTree() to accept an IOBuf object. IOBuf objects can
easily wrap other buffers, so this can still easily support ByteRange objects
as well.
Being able to use IOBuf's Cursor class ended up simplifying the logic a bit as
well.
Note that using IOBuf does require copying the name and mode data out of the
buffer when we read it (using the readTerminatedString() API). This is
necessary since the data may not be stored contiguously in the IOBuf. However,
this shouldn't impact performance much: we already need to copy the name data
into a std::string anyway. For the mode, most modern platforms can avoid doing
a heap allocation for this small string.
Reviewed By: bolinfest
Differential Revision: D3357255
fbshipit-source-id: 5b6e1bc93199849327409a8039266d7dc4f3afdf
Summary: Add a GitTreeSerializer class for serializing git tree data.
Reviewed By: bolinfest
Differential Revision: D3356770
fbshipit-source-id: d04bc9788117272504c2faa335b3648e4ac93e81
Summary:
Previous to this change, if `~/local/.eden` did not exist when `eden daemon` was
run, then it would fail.
Now the logic to create `~/local/.eden` is encapsulated in `_ensure_dot_eden_folder_exists()`.
We do not call this for all subcommands because we want to make sure that running
`eden --help` does not have the side-effect of writing `~/local/.eden` if it does not exist.
Reviewed By: wez
Differential Revision: D3397057
fbshipit-source-id: a3f974f367058d9e4ebd515c78423e54edc179cc
Summary:
`eden health` will return with exit code 0 if both of the following are true:
* The Thrift client is up and running.
* The status of the client is either ALIVE or STARTING.
Reviewed By: wez
Differential Revision: D3395582
fbshipit-source-id: ba668d26acae73a51fbae8aca2b4979156c0c50f
Summary:
When running the CLI "daemon" command, we try killing the underlying eden
process group when we receive SIGTERM or SIGINT. (We really only want to kill
the main eden process, but we currently have to kill the entire process group
due to how sudo works.)
Since the privhelper process runs as root and is part of this process group, we
can get an EPERM error back. This was causing the CLI to fail with an
unhandled exception backtrace. This diff updates the code to ignore EPERM.
Reviewed By: bolinfest, wez
Differential Revision: D3384121
fbshipit-source-id: 39b2364d8c921b1d84a8902566fe9af2a370e4e5
Summary:
Define a constant for the path directory separator ('/'). We currently plan to
normalize all pathnames to use forward slash as a directory separator, even on
Windows. This will simplify the bulk of the eden code.
Even though we don't plan to make this separator character configurable, it
still seems useful to define a symbolic constant for it.
Reviewed By: bolinfest
Differential Revision: D3377110
fbshipit-source-id: cf6b158824e0cb318cc94838618506e78d868c9f
Summary: Add functions for checking if one path is in a subdirectory of another path.
Reviewed By: bolinfest
Differential Revision: D3367035
fbshipit-source-id: 3bd10d88bded2ccf0066d273bc5fe5c447287cb6
Summary:
The reverse path iterators derived from the forward iterators, and overrode the
pre-increment operator. Unfortunately they didn't override post-increment,
which meant that post increment actually advanced them forwards instead of
backwards. (The base class's post-increment calls the base class's
pre-increment, since this method is not virtual.) The reverse iterators also
weren't overriding the decrement operators added in D3366877, so decremening a
reverse iterator would move it in the wrong direction.
This fixes all of those issues by making the ComposedPathIterator template
smarter so it can correctly handle both forward and reverse iteration.
It also now contains the logic that the "begin" iteration for absolute paths
needs to start one character in. ComposedPathIterator needs to be aware of
this in order to stop at the correct character when iterating backwards.
Reviewed By: bolinfest
Differential Revision: D3376727
fbshipit-source-id: c61c86c1a9233507f8297015439e416680aaa39b
Summary:
This modifies the iterator behavior to so the behavior is a bit cleaner
with respect to empty paths. It is valid to have an empty relative path,
and there are legitimate use cases where this is useful. For instance,
calling dirname() on a RelativePath with a single component will result in
an empty path. It is useful to use this empty path to refer to the parent
directory, to which the path is relative. Therefore it is also useful to
be able to include the empty path when iterating through the parent
directories of a path.
This removes RelativePath::begin() and RelativePath::end(), and replaces
them with a RelativePath::paths() function. paths() returns a struct with
a begin() and end() function, so it can be used in range-based for loops,
and has the same behavior that begin()/end() did. This also adds a
RelativePath::allPaths() function, which also includes the empty relative
path in the results.
Reviewed By: bolinfest
Differential Revision: D3366877
fbshipit-source-id: 3d92b600f07b993925f88d4f1e619b6c1705fb82
Summary:
Unfortunately operator->() can't really be implemented properly, since
ComposedPathIterator doesn't actually point to an already-existing path piece.
This removes the broken implementation, and leaves a comment in its place.
Reviewed By: wez
Differential Revision: D3366836
fbshipit-source-id: acbd0332a0a7a1fc9eac114ef0c06ed605adbbf3
Summary:
PrivHelper serializes messages and sends it over to PrivHelperServer who verifies that mount point exists, cleans up bind mounts for the FUSE mount, and undoes FUSE mount.
Some repeated code in this diff since I was unsure on the protocol for that - let me know if/where I should generalize functions to avoid this.
Reviewed By: simpkins
Differential Revision: D3361955
fbshipit-source-id: a7324fb9660912d6c2b753e15b1fa6061c0d5261
Summary:
This avoids translation from string->Hash in the common case
where the file is unmodified and its hash is read directly from
the store rather than computed from the overlay.
I'm guessing I should use `unique_ptr` as the return value throughout?
Reviewed By: simpkins
Differential Revision: D3355773
fbshipit-source-id: 50dff879a78b3d6ff49f86b856866ca28808c4f7
Summary:
Other tools, such as Buck, will benefit from being able to get
the SHA-1 of a file without having to read the entire contents
of the file (or do the associated computation that is proportional
to the size of the contents of the file).
Reviewed By: simpkins
Differential Revision: D3345828
fbshipit-source-id: 360bb268793369af75f408208e8211d8b9db146d
Summary: Updated python CLI to include subparser for unmount command and added wrapper functions that hand over execution to privhelper process. Unmount currently requires client_name at the command line.
Reviewed By: simpkins
Differential Revision: D3359517
fbshipit-source-id: ff05e90bcdb96ecad63f37634c69dbeef429c90f
Summary:
Change Hash::getBytes() to return a folly::ByteRange rather than a
std::array<uint8_t, 20>. This makes Hash more convienent to use with existing
APIs that accept a ByteRange. (For instance, IOBuf.)
There were only 2 call sites using the existing getBytes() functionality,
and they only used the data() method on the returned std::array, so they don't
have to be updated at all to use a ByteRange.
Reviewed By: bolinfest
Differential Revision: D3354581
fbshipit-source-id: 8f2a3c196e59620fb5b0fb2caf4d1d7f26e1d2c4
Summary:
This is an alternative to `InodeDispatcher::lookup()` that does not result
in `RequestData::get()` being called, which is important for requests that
do not originate from FUSE.
The `::lookup()` call takes the `InodeBase` and invokes its `getattr()` method,
which in the case of `TreeEntryFileInode::getattr()`, creates a `fusell::Dispatcher::Attr`
whose constructor calls the following:
```
Dispatcher::Attr::Attr() : timeout(1.0) {
memset(&st, 0, sizeof(st));
auto& req = RequestData::get();
auto mount = req.getChannel().getMountPoint();
st.st_uid = mount->getUid();
st.st_gid = mount->getGid();
}
```
So the only reason this is done at all is to set `st_uid` and `st_gid`.
And then the only reason `Attr` is needed is to set the following fields on a `fuse_entry_param`:
```
attr
attr_timeout
entry_timeout
```
So it is possible that we can find a better way to streamline all of this, but this was
the easiest shortcut I could implement myself right now.
Reviewed By: wez
Differential Revision: D3351819
fbshipit-source-id: b095c085ee4a9b9a81438db093869fd0acf1f8ad
Summary:
I am going to submit some changes to `InodeDispatcher.cpp` in a
follow-up diff where I definitely want to benefit from auto-format,
so to make that diff less noisy, I'm doing the auto-format in this revision.
Reviewed By: wez
Differential Revision: D3351888
fbshipit-source-id: f8fdebe43f247e2522f1f14b3ee4bac37f564c70
Summary:
Add a function to compute the sha1 content hash for an overlay file.
We persist the computed hash in an extended attribute in the underlying overlay
file so that a subsequent read of the attribute doesn't require opening the
file to recompute it.
Each time the file is mutated, we blow the cached status of the file.
Each time the sha1 attribute is read, if the cache is blown, the content
hash will be recomputed and set in the overlay file.
Each time the file is flushed or sync'd, if the cache is blown, the content
hash will be computed and set in the overlay file.
Reviewed By: bolinfest
Differential Revision: D3302412
fbshipit-source-id: bd45c7a24b732bd0b7474b7f96e82936870b2117
Summary:
This is part 1 of 2 diffs. This one adds some plumbing to make it possible
to read the xattr attribute from an overlay.
It doesn't do anything to ensure that it is set; the next diff in this series will take care of that.
Reviewed By: bolinfest
Differential Revision: D3302410
fbshipit-source-id: 47406a9c75f29743691d396676c691bcb99c4760
Summary: This logic should be shared by the Eden CLI as well as unit tests.
Reviewed By: simpkins
Differential Revision: D3348300
fbshipit-source-id: c87b1f03f16560323f3d7685063bb6466c39efe2
Summary:
We look this up via the mount point or eden mount object instead.
I've also removed the mercurial library stuff that was added to support the now defunct lamehg fuse we had in the earlier days.
simpkins' new importer doesn't use these and it resolves our CI mismatch issue.
Reviewed By: bolinfest
Differential Revision: D3349698
fbshipit-source-id: 5f4ec16b76042959cd1e3184f46bb3526fbaf74c
Summary:
Update the CLI to support running "init" with a mercurial repository.
This is just some bare bones framework code at the moment. It doesn't actually
import any data from mercurial at the moment, and mounting doesn't work.
Reviewed By: bolinfest, wez
Differential Revision: D3345426
fbshipit-source-id: 72c31ac8d2aac2a16e0a7d6f0425eb4ca218d487
Summary:
Use the hex-encoded version of the key in the RocksException if _get() fails,
rather than the raw binary data.
Reviewed By: bolinfest
Differential Revision: D3345355
fbshipit-source-id: cd8dc644a56ca3d5f3b9a9a0f5cc789b142f0bda
Summary:
Buck is [currently] built with Java 7, so it can only use third-party dependencies
that are also Java 7.
Reviewed By: simpkins
Differential Revision: D3342367
fbshipit-source-id: 4370fd152e7d2055495e783de68a6bb59867bee5
Summary:
This adds a new API to `PrivHelper`: `privilegedBindMount()`.
Similar to `privilegedFuseMount()`, this sends a message to the privileged helper,
which is running as `root`, so it can set up the specified bind mount.
The changes in the `privhelper` directory parrot what was done to support `privilegedFuseMount()`.
Now, once the primary mount for a client is created, any bind mounts listed in the
config for the client are set up. This logic is introduced in `EdenServer.cpp`.
Reviewed By: simpkins
Differential Revision: D3296660
fbshipit-source-id: 61296f35e5c3a6f232a1c17e0f296dd5d3b5ec06
Summary:
`glob()` prevents us from adding `.cpp` files to this directory that do not get compiled.
The header without the long path makes it clear that it is in the same directory as `PrivHelperTest.cpp`.
Reviewed By: simpkins
Differential Revision: D3301264
fbshipit-source-id: f58351a4f2857c8911f0bf82ae67136920fcb998
Summary:
Add a new class to serve as a single location where we can store all
information about a single eden mount point. Currently this contains the
MountPoint, LocalStore, and Overlay objects. This allows the TreeInode class
to just store a single pointer to the EdenMount, rather than having to track
these three objects separately.
In the future we could consider also keeping a copy of the ClientConfig in the
EdenMount object, but I haven't done that for now.
Reviewed By: bolinfest
Differential Revision: D3321355
fbshipit-source-id: 8a39bb49822ca8e90c88b2a834b59230d2f91435
Summary:
Enables mkdir in the overlay area.
I had to add some `lstat` calls in to the overlay dir reader because we depend
on knowing at least whether a node is a dir or not at the next level up.
When I run the test suite, the mounts are on my `/tmp` filesystem. When I run
eden manually, they are on my `/data` filesystem. The latter (xfs) does not
populate the type bits. This meant that the test suite passed but manual
testing did not.
Adding the `lstat` calls is a little unfortunate. On OS X there is a bulk
operation that combines `readdir` and `lstat` so that there are fewer syscalls.
We don't have an equivalent for Linux.
Reviewed By: bolinfest
Differential Revision: D3301532
fbshipit-source-id: e228f4a392f90aa491fec62e8b98471a8acecff2
Summary:
We still have naked pointers at the handoff to the kernel, but now
have a cleaner implementation at the level that we're going to be working at
day to day.
I also renamed `FileHandle::release` to `FileHandle::releasefile` so that it
isn't visually ambiguous with `std::unique_ptr::release` in the
`Dispatcher.cpp` code: `fh.release()` vs `fh->release()` look similar but are
dramatically different in behavior.
Reviewed By: bolinfest
Differential Revision: D3309455
fbshipit-source-id: f8cf055bcd51121048a20f0202988cf0aef1f085
Summary: Now that we have integrated this functionality, we don't need these
Reviewed By: simpkins
Differential Revision: D3307034
fbshipit-source-id: 590469e9e7dec9c6b7d4f7b4e9a8d8c40d6ff552
Summary:
`creat(2)` and `open(2)` could decide to create a PassThru file handle.
This diff removes that usage. There is a TODO here around handling `O_EXCL` properly.
I'm punting this to a follow-up diff.
Reviewed By: bolinfest
Differential Revision: D3301387
fbshipit-source-id: d35104c536396e7fd064d786f3d5592ecfcbfecf
Summary:
Centralize and delegate most (all?) of the content sensitive portions of file accesses into the FileData class.
Add tests to show that we can write to the overlay file and that the stat data is consistent with the result.
Reviewed By: bolinfest
Differential Revision: D3301251
fbshipit-source-id: a09316ad61c6ef4c656bc5d6dbd43f906abb7932
Summary:
This is the workhorse for adjusting the state that we track for the file data.
It handles both overlay and Tree backed data cases and moving from the latter
to the former depending on the open flags provided when a file handle is
opened.
This diff handles more cases than we have tests for. Those will be covered later in this stack of diffs.
Reviewed By: bolinfest
Differential Revision: D3301213
fbshipit-source-id: c1dab40c0ad205ce6cee820043b70dd886e78431
Summary:
Another step towards making TreeEntryFileInode overlay aware, this diff implements:
- stat
- readlink (although it is not possible to create symlinks yet)
- open
Reviewed By: bolinfest
Differential Revision: D3255158
fbshipit-source-id: 3f90b624e629ef279d6cc32e1d82787ee24796eb
Summary:
This allows us to share the same state between multiple open files and
also helps to reduce the size of inode instances that are otherwise inactive;
when there are no outstanding references to the data, we can drop it and forget
it.
Reviewed By: bolinfest
Differential Revision: D3301198
fbshipit-source-id: f0b3fc73a666ec2033c7a22e9eb587d3212cf966
Summary:
Update the PrivHelperConn code to copy file descriptors in and out of the cmsg
data using memcpy(), rather than dereferencing reinterpret_cast'ed pointers.
These cast's break C/C++'s strict aliasing rules, and gcc complains about this.
Reviewed By: bolinfest, wez
Differential Revision: D3306762
fbshipit-source-id: d36fb5ee02a52da2b60e86fb4a0fdb4910fc72d7
Summary: This should set us up to have `eden mount` perform the bind mounts.
Reviewed By: simpkins
Differential Revision: D3296370
fbshipit-source-id: 5d8c21308074b357bad3ace72cec157adb5f8b56
