Summary:
Start adding utility code to make it easier for the integration tests to
manipulate git and mercurial repositories.
This adds an HgRepository class, but does not update the tests to use it yet.
I will do that in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3449328
fbshipit-source-id: caea5befb683a388cbad30a29017c566e22702af
Summary:
Move the eden/fs/importer/hg code to eden/fs/store/hg. This import code is
only used by the HgBackingStore, so logically it belongs together.
The eden/fs/importer/git directory was already moved into eden/fs/store/git by
D3448752, so this makes the mercurial code similar.
Reviewed By: bolinfest
Differential Revision: D3458128
fbshipit-source-id: 54b1f707197e41ff5f5f4a163a34d3d1ab432e21
Summary:
This moves all of the test library code into a lib/ subdirectory, just to help
distinguish tests from utility code.
This also changes the test so that we no longer pack the eden CLI and daemon
binaries into the python archives. This results in very large archives when
building in dbg and opt modes, and isn't really necessary. Instead
edenclient.py simply finds the CLI and daemon binaries relative to the test
binary. We pass in an EDENFS_SUFFIX variable to tell it which flavor of the
daemon to use.
Additionally, this changes the tests to run with python 3.
Reviewed By: bolinfest
Differential Revision: D3449013
fbshipit-source-id: 82533137090325766a52cd067aa97dd8391ae088
Summary:
This moves git import logic from the GitImporter class to GitBackingStore.
The logic is simpler now, since GitBackingStore only needs to import a single
Tree or Blob at a time.
Reviewed By: bolinfest
Differential Revision: D3448752
fbshipit-source-id: da2d59f953ada714d8512545ae83dd48e5d3e410
Summary:
Add a run_cmd() function that's slightly simpler to use compared to
_get_eden_args().
Reviewed By: bolinfest
Differential Revision: D3448844
fbshipit-source-id: 06b654b2f4bc773d67a4f9d1f6effdad2ce9aa2a
Summary:
- Update "eden daemon" to wait until edenfs is healthy before returning.
- Accept additional command line arguments and pass them through to edenfs.
- When starting edenfs as a background daemon, direct stdout and stderr to a
log file in the eden config directory.
- Update "eden shutdown" command to wait for edenfs to exit.
- Update "eden shutdown" command to print an error message if edenfs was not
running.
Reviewed By: bolinfest
Differential Revision: D3446403
fbshipit-source-id: 2ca1874256b7d124a85092b886548bea8c198327
Summary:
Look for hg_import_helper.py relative to the edenfs binary, rather than
relative to the current working directory. Also check in a couple places, so
we can work in both normal deployment locations as well as inside a source
repository.
Additionally, update hg_import_helper.py to emit a response chunk as soon as it
starts. This contains either message indicating that it has started
successfully, or an error message. This allows us to propagate error
information back to the thrift caller if something goes wrong starting
hg_import_helper.py for a given repository.
Reviewed By: bolinfest
Differential Revision: D3447027
fbshipit-source-id: 4bffd1d03ab6475c09cb3c8385145555c7fc7361
Summary:
[Folly] Wrappers for some of OpenSSL's crypto hash functions.
Wraps some of the OpenSSL crypto hash functions with variants that take `ByteRange` for input and `MutableByteRange` for output, and also variants that take `const IOBuf&` for input as well.
These are a bit nicer to use than passing pointers and lengths separately.
Reviewed By: ivmaykov
Differential Revision: D3434562
fbshipit-source-id: 3688ef11680a029b7664ac417a7781e70f9c6926
Summary:
This adds a --foreground flag to the "eden daemon" command, which causes the
daemon to run in the foreground, rather than daemonizing as a background
process. Additionally, this drops the --preserve-environment flag and instead
updates the CLI to always build a sane environment to run with.
This also updates the integration test code to use this flag, and wait for eden
to shut down during test clean up. Without this, the test case code could try
to clean up temporary directories before they were unmounted, which would fail.
This would leave many temporary directories behind after test runs.
This also re-organizes the temporary directory set up a bit, so that each test
uses a single top-level temporary directory. All other directories it creates
are put inside this one directory.
Reviewed By: bolinfest
Differential Revision: D3439232
fbshipit-source-id: 85305f7db60da6bae589d28c802ee260e42e4dea
Summary:
This changes the way that Eden is built and deployed.
* To build the binary that must be run as `root` (but quickly drops privileges), run `buck build eden-daemon`.
* To build the CLI that communicates with the daemon (and does not require privileges), run `buck build eden-cli`.
* To build both, run `buck build eden`.
There is an example of how to build the various parts of Eden using
Buck and how to package them up in the `install` script introduced by this revision.
While here, I also cleaned up some of our build files and changed them to be
parameterized between internal and external use. In both cases, the user gets the
"unadorned" version of their primary build targets. This ensures that shortcuts such as:
```
buck test eden/fs/integration
```
do the right thing by default.
Finally, I also made `find_default_config_dir()` and `find_default_daemon_binary()`
lazy whereas `find_default_config_dir()` was previously eager.
Reviewed By: simpkins
Differential Revision: D3436245
fbshipit-source-id: 4dfbd59ed0d198620324f0705c462334bb5a7daf
Summary:
If a directory is present in the overlay, we still need to check if a TreeEntry
exists from the source control data structures. Previously this was causing us
to incorrectly report directories as empty if they exist in the local overlay,
even when they had contents from the main Tree.
Reviewed By: wez
Differential Revision: D3434219
fbshipit-source-id: f872f90075602dfdc7b217f50eefcd7c248512e7
Summary:
Add some verbose logging about when trees and blobs are loaded in the object
store.
Reviewed By: bolinfest
Differential Revision: D3434182
fbshipit-source-id: 3e8d2617290604f119e6164d15d63324a4c9a2aa
Summary:
Update the HgImporter class to support retrieving file contents from mercurial.
This also includes simple code for storing the data in the LocalStore using
git's blob serialization format. In the future I think it would perhaps be
better to drop the "blob<length>" prefix, and instead just use a RocksDB column
family to separate blob data from other types of data. However, for now using
the git format is simplest for keeping compatibility with the getBlob() code.
Reviewed By: bolinfest
Differential Revision: D3416691
fbshipit-source-id: 268787533be2172b2dbedc3bf06464eabf3d2c5e
Summary:
Record mercurial commits that have been previously imported, and avoid
re-importing them when checking out a commit that was previously processed and
already has a tree present in the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3416654
fbshipit-source-id: 80f70fa2204d3a2961991b634ad934a623b42b82
Summary:
Add APIs for storing arbitrary (key, value) data.
This will allow BackingStore implementations to store additional metadata, such
as mapping mercurial commit IDs to the eden root tree ID.
Eventually we may want to use RocksDB column families to partition the
different types of data being put into the LocalStore. However, for now this
just uses a single key space. We can add column family support in a separate
diff, if desired.
Reviewed By: bolinfest
Differential Revision: D3409866
fbshipit-source-id: 19a1d340b65bff2081981bf5daf32d5ad15b60c4
Summary:
This adds an HgBackingStore implementation which can load tree data from a
mercurial repository. Blob loading is not implemented yet, but will come in a
separate diff.
This also adds a minimal GitBackingStore class. The GitBackingStore has nearly
no functionality, but is needed to keep the existing git functionality working.
Reviewed By: bolinfest
Differential Revision: D3409743
fbshipit-source-id: dbebf53e9de08bd1469e489baa48b84cbf889511
Summary:
Add the basic BackingStore interface, plus a NullBackingStore implementation
that always returns null. This updates the ObjectStore to query the
BackingStore if data is not found in the LocalStore.
Additionally, this updates EdenServer to manage the BackingStore objects. It
maintains a map of the BackingStore objects created for each known repository.
Reviewed By: bolinfest
Differential Revision: D3409602
fbshipit-source-id: 2920dc4c24ee1ec37efb542f058d0d121ceb5532
Summary:
The _get_hg_dir() function had an incorrect return statement caused "eden init"
to fail with mercurial repositories, unless the path pointed to a shared
working directory.
Also update the code to store the path to the mercurial repository, rather than
the path to the .hg directory. Unlike git, mercurial requires the repository
path itself, and refuses to work if given a path to the .hg directory.
Reviewed By: bolinfest
Differential Revision: D3409808
fbshipit-source-id: 22ee7748422b614f5af57aff07fcfa517af846d9
Summary:
This revision introduces two complementary changes:
* `eden daemon` no longer runs in the foreground.
* There is now an `eden shutdown` command to kill the daemon.
When `shutdown` is called, it tells the Thrift server to shutdown.
In turn, this causes `EdenServer::runThriftServer()` to exit,
which causes `EdenServer::run()` to exit.
Reviewed By: simpkins
Differential Revision: D3402347
fbshipit-source-id: 80032ba53eb69b3f69bef9d7cd169f93500c833c
Summary:
D3406773 included a change which compiles on clang and gcc-5.x, but fails to
build with gcc-4.9.
This looks like a bug in gcc-4.9's handling of list initialization. Overload
resolution for non-initializer-list constructors should be attempted if
no suitable initializer-list constructors are found, but gcc-4.9 does not
appear to do this.
Reviewed By: bolinfest
Differential Revision: D3410142
fbshipit-source-id: f34125000eb3fa949c2427aa4ffbd4ef92942cd7
Summary:
Update deserializeGitBlob() to work on an IOBuf, rather than an rvalue
reference to a string.
The ugliness about having to wrap a std::string in a managed IOBuf is now
hidden inside the StoreResult class, rather than being something that the
GitBlob code has to know about.
Reviewed By: bolinfest
Differential Revision: D3403977
fbshipit-source-id: 0c58c019557050d6e201c1a462fa051c2526674a
Summary:
Add a new StoreResult which wraps the std::string returned by RocksDB.
This replaces the std::unique<string> that LocalStore::get() used to return.
This lets us avoid a memory allocation. StoreResult can also represent a "not
found" result, so that this case can be processed efficiently without having to
throw an exception.
Additionally, StoreResult is move-only so we can't ever unintentionally copy
the string data, which is potentially expensive. It also provides APIs for
creating IOBuf wrappers, or moving the string to the heap so we can create an
managed IOBuf around it.
Reviewed By: bolinfest
Differential Revision: D3403958
fbshipit-source-id: ab0c304988a53eda50341ecc2f96ae5235e5260c
Summary:
Add a new ObjectStore class, which will eventually contain both a LocalStore
and a BackingStore. The LocalStore will be a cache of data loaded from the
authoritative BackingStore. The ObjectStore API will hide the work of querying
the BackingStore and updating the LocalStore when data is not already available
in the LocalStore.
For now ObjectStore only contains the LocalStore, but I will add BackingStore
functionality in subsequent diffs. This diff simply updates all call sites to
use the ObjectStore instead of directly accessing the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3403898
fbshipit-source-id: 47b8c51a7717a4c7c29911a7085b382521a8c0db
Summary:
Previously Hash objects could not be assigned to after they were created, since
they contained a const member. This makes the data non-const, so a Hash
variable can be replaced to contain new contents after it is created.
This also adds a default constructor, which zero-initializes the hash. The
default constructor makes it possible to declare a Hash with a 0-value at one
location, and then set it to the desired value at some later point.
Reviewed By: bolinfest
Differential Revision: D3406773
fbshipit-source-id: 41e2c7e3ad5bc4d14813be4adaa03866701380f6
Summary:
- Add a Sha1Key class that can more efficiently compute the key for
file content SHA-1 values, without having to copy it into a new std::string
object. (In practice fbstring would have avoided having to actually allocate
memory, but it was still an extra data copy.)
- The code was always converting the hash keys to hex on get and put
operations, just in case it needed it if an error occurred. This diff
changes the code to only compute the hex value if an error actually occurred.
Reviewed By: bolinfest
Differential Revision: D3403889
fbshipit-source-id: 5abd8ef202cb00677a84a03a82e2a3d21f16cd2f
Summary:
Update several places to use std::array rather than plain C arrays, using
folly::make_array() to automatically deduce the correct type when necessary.
Reviewed By: wez
Differential Revision: D3370445
fbshipit-source-id: b7642cf3a9b08eac817988bf95679bf5e584ef72
Summary:
Add an HgImporter class for importing mercurial data into the eden local store.
At the moment this only includes code for importing revision manifest data as
tree objects, and does not yet include code for importing file blob data.
Reviewed By: bolinfest
Differential Revision: D3367958
fbshipit-source-id: 58049bf1594b3c27d676c5ebe778917b4043fccf
Summary:
Update deserializeGitTree() to accept an IOBuf object. IOBuf objects can
easily wrap other buffers, so this can still easily support ByteRange objects
as well.
Being able to use IOBuf's Cursor class ended up simplifying the logic a bit as
well.
Note that using IOBuf does require copying the name and mode data out of the
buffer when we read it (using the readTerminatedString() API). This is
necessary since the data may not be stored contiguously in the IOBuf. However,
this shouldn't impact performance much: we already need to copy the name data
into a std::string anyway. For the mode, most modern platforms can avoid doing
a heap allocation for this small string.
Reviewed By: bolinfest
Differential Revision: D3357255
fbshipit-source-id: 5b6e1bc93199849327409a8039266d7dc4f3afdf
Summary: Add a GitTreeSerializer class for serializing git tree data.
Reviewed By: bolinfest
Differential Revision: D3356770
fbshipit-source-id: d04bc9788117272504c2faa335b3648e4ac93e81
Summary:
Previous to this change, if `~/local/.eden` did not exist when `eden daemon` was
run, then it would fail.
Now the logic to create `~/local/.eden` is encapsulated in `_ensure_dot_eden_folder_exists()`.
We do not call this for all subcommands because we want to make sure that running
`eden --help` does not have the side-effect of writing `~/local/.eden` if it does not exist.
Reviewed By: wez
Differential Revision: D3397057
fbshipit-source-id: a3f974f367058d9e4ebd515c78423e54edc179cc
Summary:
`eden health` will return with exit code 0 if both of the following are true:
* The Thrift client is up and running.
* The status of the client is either ALIVE or STARTING.
Reviewed By: wez
Differential Revision: D3395582
fbshipit-source-id: ba668d26acae73a51fbae8aca2b4979156c0c50f
Summary:
When running the CLI "daemon" command, we try killing the underlying eden
process group when we receive SIGTERM or SIGINT. (We really only want to kill
the main eden process, but we currently have to kill the entire process group
due to how sudo works.)
Since the privhelper process runs as root and is part of this process group, we
can get an EPERM error back. This was causing the CLI to fail with an
unhandled exception backtrace. This diff updates the code to ignore EPERM.
Reviewed By: bolinfest, wez
Differential Revision: D3384121
fbshipit-source-id: 39b2364d8c921b1d84a8902566fe9af2a370e4e5
Summary:
This modifies the iterator behavior to so the behavior is a bit cleaner
with respect to empty paths. It is valid to have an empty relative path,
and there are legitimate use cases where this is useful. For instance,
calling dirname() on a RelativePath with a single component will result in
an empty path. It is useful to use this empty path to refer to the parent
directory, to which the path is relative. Therefore it is also useful to
be able to include the empty path when iterating through the parent
directories of a path.
This removes RelativePath::begin() and RelativePath::end(), and replaces
them with a RelativePath::paths() function. paths() returns a struct with
a begin() and end() function, so it can be used in range-based for loops,
and has the same behavior that begin()/end() did. This also adds a
RelativePath::allPaths() function, which also includes the empty relative
path in the results.
Reviewed By: bolinfest
Differential Revision: D3366877
fbshipit-source-id: 3d92b600f07b993925f88d4f1e619b6c1705fb82
Summary:
PrivHelper serializes messages and sends it over to PrivHelperServer who verifies that mount point exists, cleans up bind mounts for the FUSE mount, and undoes FUSE mount.
Some repeated code in this diff since I was unsure on the protocol for that - let me know if/where I should generalize functions to avoid this.
Reviewed By: simpkins
Differential Revision: D3361955
fbshipit-source-id: a7324fb9660912d6c2b753e15b1fa6061c0d5261
Summary:
This avoids translation from string->Hash in the common case
where the file is unmodified and its hash is read directly from
the store rather than computed from the overlay.
I'm guessing I should use `unique_ptr` as the return value throughout?
Reviewed By: simpkins
Differential Revision: D3355773
fbshipit-source-id: 50dff879a78b3d6ff49f86b856866ca28808c4f7
Summary:
Other tools, such as Buck, will benefit from being able to get
the SHA-1 of a file without having to read the entire contents
of the file (or do the associated computation that is proportional
to the size of the contents of the file).
Reviewed By: simpkins
Differential Revision: D3345828
fbshipit-source-id: 360bb268793369af75f408208e8211d8b9db146d
Summary: Updated python CLI to include subparser for unmount command and added wrapper functions that hand over execution to privhelper process. Unmount currently requires client_name at the command line.
Reviewed By: simpkins
Differential Revision: D3359517
fbshipit-source-id: ff05e90bcdb96ecad63f37634c69dbeef429c90f
Summary:
Change Hash::getBytes() to return a folly::ByteRange rather than a
std::array<uint8_t, 20>. This makes Hash more convienent to use with existing
APIs that accept a ByteRange. (For instance, IOBuf.)
There were only 2 call sites using the existing getBytes() functionality,
and they only used the data() method on the returned std::array, so they don't
have to be updated at all to use a ByteRange.
Reviewed By: bolinfest
Differential Revision: D3354581
fbshipit-source-id: 8f2a3c196e59620fb5b0fb2caf4d1d7f26e1d2c4
Summary:
Add a function to compute the sha1 content hash for an overlay file.
We persist the computed hash in an extended attribute in the underlying overlay
file so that a subsequent read of the attribute doesn't require opening the
file to recompute it.
Each time the file is mutated, we blow the cached status of the file.
Each time the sha1 attribute is read, if the cache is blown, the content
hash will be recomputed and set in the overlay file.
Each time the file is flushed or sync'd, if the cache is blown, the content
hash will be computed and set in the overlay file.
Reviewed By: bolinfest
Differential Revision: D3302412
fbshipit-source-id: bd45c7a24b732bd0b7474b7f96e82936870b2117
Summary:
This is part 1 of 2 diffs. This one adds some plumbing to make it possible
to read the xattr attribute from an overlay.
It doesn't do anything to ensure that it is set; the next diff in this series will take care of that.
Reviewed By: bolinfest
Differential Revision: D3302410
fbshipit-source-id: 47406a9c75f29743691d396676c691bcb99c4760
Summary: This logic should be shared by the Eden CLI as well as unit tests.
Reviewed By: simpkins
Differential Revision: D3348300
fbshipit-source-id: c87b1f03f16560323f3d7685063bb6466c39efe2
Summary:
We look this up via the mount point or eden mount object instead.
I've also removed the mercurial library stuff that was added to support the now defunct lamehg fuse we had in the earlier days.
simpkins' new importer doesn't use these and it resolves our CI mismatch issue.
Reviewed By: bolinfest
Differential Revision: D3349698
fbshipit-source-id: 5f4ec16b76042959cd1e3184f46bb3526fbaf74c
Summary:
Update the CLI to support running "init" with a mercurial repository.
This is just some bare bones framework code at the moment. It doesn't actually
import any data from mercurial at the moment, and mounting doesn't work.
Reviewed By: bolinfest, wez
Differential Revision: D3345426
fbshipit-source-id: 72c31ac8d2aac2a16e0a7d6f0425eb4ca218d487
Summary:
Use the hex-encoded version of the key in the RocksException if _get() fails,
rather than the raw binary data.
Reviewed By: bolinfest
Differential Revision: D3345355
fbshipit-source-id: cd8dc644a56ca3d5f3b9a9a0f5cc789b142f0bda
Summary:
Buck is [currently] built with Java 7, so it can only use third-party dependencies
that are also Java 7.
Reviewed By: simpkins
Differential Revision: D3342367
fbshipit-source-id: 4370fd152e7d2055495e783de68a6bb59867bee5
Summary:
This adds a new API to `PrivHelper`: `privilegedBindMount()`.
Similar to `privilegedFuseMount()`, this sends a message to the privileged helper,
which is running as `root`, so it can set up the specified bind mount.
The changes in the `privhelper` directory parrot what was done to support `privilegedFuseMount()`.
Now, once the primary mount for a client is created, any bind mounts listed in the
config for the client are set up. This logic is introduced in `EdenServer.cpp`.
Reviewed By: simpkins
Differential Revision: D3296660
fbshipit-source-id: 61296f35e5c3a6f232a1c17e0f296dd5d3b5ec06
Summary:
Add a new class to serve as a single location where we can store all
information about a single eden mount point. Currently this contains the
MountPoint, LocalStore, and Overlay objects. This allows the TreeInode class
to just store a single pointer to the EdenMount, rather than having to track
these three objects separately.
In the future we could consider also keeping a copy of the ClientConfig in the
EdenMount object, but I haven't done that for now.
Reviewed By: bolinfest
Differential Revision: D3321355
fbshipit-source-id: 8a39bb49822ca8e90c88b2a834b59230d2f91435
