Summary:
These changes restructure the eden directory so that 'client' directories are created during the `eden clone` command and are associated with a single mount path.
The new eden directory looks as follows:
~/.eden
config.json
clients/
abcd08d/
edenrc
SNAPSHOT
overlay/
efgh19i/
edenrc
SNAPSHOT
overlay/
...
Where the config.json file holds the mapping of mount paths to their respective client directory which is a hash, and the edenrc files in each client directory is an INI file which holds the name of repository associated with the mount path. This INI file follows the current format:
[repository]
name = fbsource
This restructuring required a couple other changes:
- unmount command now cleans up the client directory and removes the mapping of its mount path from config.json
- eden list command now lists all of the mount paths rather than the client names
Reviewed By: bolinfest
Differential Revision: D3506119
fbshipit-source-id: dc07a8baf1052be731ff335d9cf74a07ab8e661a
Summary: Change the ClientConfig class to parse client data via INI config file rather than json file. This class uses boost::property_tree::ini_parser and the ptree data structure to hold the parsed INI file contents. This change makes it possible for eden to no longer rely on json files for getting client data, and the json files will be completely taken out in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3498567
fbshipit-source-id: 3298047a014beda0c250475c0809a7a1ebd95b2b
Summary: Update the unmount command to use the mount point rather than the client name. This diff also removes the mount command completely since tests are no longer dependent on it.
Reviewed By: bolinfest
Differential Revision: D3489290
fbshipit-source-id: 9bfbcb78a78d1593f6d0950502bdd4a55e3e6ca4
Summary: Introduce new 'clone' command that takes in repo name and mount path. This command replaces the current mount command which uses the client name as an argument
Reviewed By: bolinfest
Differential Revision: D3489098
fbshipit-source-id: 719fb94387da1d4ccab770a0e659f717888a7dc6
Summary: Add a simple eden repository command that takes no arguments. The behavior mimics that of git remote and lists all repositories by parsing both the global and local INI config files. The INI file currently needs to be manually added and updated to maintain information about existing repositories.
Reviewed By: bolinfest
Differential Revision: D3479543
fbshipit-source-id: b0d36f33c9b8e0c7b9fe20ec2f362ca15c0adeb8
Summary:
We're seeing failures like this in our CI system:
```
stderr:
test_create (eden.fs.integration.basic_test.BasicTest) ...
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: no tty present and no askpass program specified
error: edenfs is not running
ERROR
ERROR
======================================================================
ERROR: test_create (eden.fs.integration.basic_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "eden/fs/integration/basic_test.py", line 68, in test_create
eden = self.init_git_eden()
File "eden/fs/integration/lib/testcase.py", line 104, in init_git_eden
return self.init_eden(repo_path)
File "eden/fs/integration/lib/testcase.py", line 54, in init_eden
inst.init(repo_path, *kwargs)
File "eden/fs/integration/lib/edenclient.py", line 160, in init
self.daemon_cmd(timeout)
File "eden/fs/integration/lib/edenclient.py", line 174, in daemon_cmd
self._wait_for_thrift(timeout)
File "eden/fs/integration/lib/edenclient.py", line 99, in _wait_for_thrift
raise Exception("edenfs didn't start within timeout of %s" % timeout)
Exception: edenfs didn't start within timeout of 10
```
This diff improves things by failing faster and printing a little more appropriate context.
Reviewed By: bolinfest
Differential Revision: D3523257
fbshipit-source-id: 14508a1b820d9d6c773713e2bb7c22955c5b2b66
Summary:
Implements basic rename(2) support. We only do this for files at this
time as directory renames require that we recursively materialize a tree and
are a bit more complex. I'll look at that in a follow-on diff; there's
potential for optimizing it to avoid eager materialization, but that depends on
the trie work that I'm tackling concurrently with this.
rename for files is the last piece needed to allow `sed -i` to operate correctly.
Reviewed By: bolinfest
Differential Revision: D3515361
fbshipit-source-id: 9c8cc5f9b8db6b5a9372ca9286336647f50490f8
Summary:
This enables O_EXCL to function by allowing the create routine to
move its folly::File instance down in to the underlying FileData instance.
Previously we would close and then re-open the file; this would discard
any of the natural gating for the open call that is performed by the kernel
for the underlying filesystem in the overlay.
Reviewed By: bolinfest
Differential Revision: D3513758
fbshipit-source-id: 85967a3b7affa1b1df46842be8ba21c8fbb843a6
Summary:
Despite being handy things for filesystems in general, these are
needed to support the `sed -i` integration test.
Reviewed By: bolinfest
Differential Revision: D3513754
fbshipit-source-id: 505b4bd58b254141e2ef996f01e3347fc1a77584
Summary:
setattr is a bit of a multi-purpose interface; depending on the flag
values, this is responsible for:
* ftruncate(2)
* fchmod(2)
* fchown(2)
* futimens(2)
In order to apply any of these things, we have to materialize the file. In
the future we may want to allow setting the utimes without materializing the file.
We don't allow chown to actually chown anything. We may want to relax that in
the future, but at the moment we will return an error if an attempt is made to
change the ownership of a file.
Reviewed By: bolinfest
Differential Revision: D3511011
fbshipit-source-id: 858d2c07686fcbe2dcdb60a07527f739a9726be3
Summary:
Start adding utility code to make it easier for the integration tests to
manipulate git and mercurial repositories.
This adds an HgRepository class, but does not update the tests to use it yet.
I will do that in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3449328
fbshipit-source-id: caea5befb683a388cbad30a29017c566e22702af
Summary:
This moves all of the test library code into a lib/ subdirectory, just to help
distinguish tests from utility code.
This also changes the test so that we no longer pack the eden CLI and daemon
binaries into the python archives. This results in very large archives when
building in dbg and opt modes, and isn't really necessary. Instead
edenclient.py simply finds the CLI and daemon binaries relative to the test
binary. We pass in an EDENFS_SUFFIX variable to tell it which flavor of the
daemon to use.
Additionally, this changes the tests to run with python 3.
Reviewed By: bolinfest
Differential Revision: D3449013
fbshipit-source-id: 82533137090325766a52cd067aa97dd8391ae088
Summary:
Add a run_cmd() function that's slightly simpler to use compared to
_get_eden_args().
Reviewed By: bolinfest
Differential Revision: D3448844
fbshipit-source-id: 06b654b2f4bc773d67a4f9d1f6effdad2ce9aa2a
Summary:
This adds a --foreground flag to the "eden daemon" command, which causes the
daemon to run in the foreground, rather than daemonizing as a background
process. Additionally, this drops the --preserve-environment flag and instead
updates the CLI to always build a sane environment to run with.
This also updates the integration test code to use this flag, and wait for eden
to shut down during test clean up. Without this, the test case code could try
to clean up temporary directories before they were unmounted, which would fail.
This would leave many temporary directories behind after test runs.
This also re-organizes the temporary directory set up a bit, so that each test
uses a single top-level temporary directory. All other directories it creates
are put inside this one directory.
Reviewed By: bolinfest
Differential Revision: D3439232
fbshipit-source-id: 85305f7db60da6bae589d28c802ee260e42e4dea
Summary:
This changes the way that Eden is built and deployed.
* To build the binary that must be run as `root` (but quickly drops privileges), run `buck build eden-daemon`.
* To build the CLI that communicates with the daemon (and does not require privileges), run `buck build eden-cli`.
* To build both, run `buck build eden`.
There is an example of how to build the various parts of Eden using
Buck and how to package them up in the `install` script introduced by this revision.
While here, I also cleaned up some of our build files and changed them to be
parameterized between internal and external use. In both cases, the user gets the
"unadorned" version of their primary build targets. This ensures that shortcuts such as:
```
buck test eden/fs/integration
```
do the right thing by default.
Finally, I also made `find_default_config_dir()` and `find_default_daemon_binary()`
lazy whereas `find_default_config_dir()` was previously eager.
Reviewed By: simpkins
Differential Revision: D3436245
fbshipit-source-id: 4dfbd59ed0d198620324f0705c462334bb5a7daf
Summary:
This revision introduces two complementary changes:
* `eden daemon` no longer runs in the foreground.
* There is now an `eden shutdown` command to kill the daemon.
When `shutdown` is called, it tells the Thrift server to shutdown.
In turn, this causes `EdenServer::runThriftServer()` to exit,
which causes `EdenServer::run()` to exit.
Reviewed By: simpkins
Differential Revision: D3402347
fbshipit-source-id: 80032ba53eb69b3f69bef9d7cd169f93500c833c
Summary:
Previous to this change, if `~/local/.eden` did not exist when `eden daemon` was
run, then it would fail.
Now the logic to create `~/local/.eden` is encapsulated in `_ensure_dot_eden_folder_exists()`.
We do not call this for all subcommands because we want to make sure that running
`eden --help` does not have the side-effect of writing `~/local/.eden` if it does not exist.
Reviewed By: wez
Differential Revision: D3397057
fbshipit-source-id: a3f974f367058d9e4ebd515c78423e54edc179cc
Summary:
`eden health` will return with exit code 0 if both of the following are true:
* The Thrift client is up and running.
* The status of the client is either ALIVE or STARTING.
Reviewed By: wez
Differential Revision: D3395582
fbshipit-source-id: ba668d26acae73a51fbae8aca2b4979156c0c50f
Summary:
PrivHelper serializes messages and sends it over to PrivHelperServer who verifies that mount point exists, cleans up bind mounts for the FUSE mount, and undoes FUSE mount.
Some repeated code in this diff since I was unsure on the protocol for that - let me know if/where I should generalize functions to avoid this.
Reviewed By: simpkins
Differential Revision: D3361955
fbshipit-source-id: a7324fb9660912d6c2b753e15b1fa6061c0d5261
Summary:
Other tools, such as Buck, will benefit from being able to get
the SHA-1 of a file without having to read the entire contents
of the file (or do the associated computation that is proportional
to the size of the contents of the file).
Reviewed By: simpkins
Differential Revision: D3345828
fbshipit-source-id: 360bb268793369af75f408208e8211d8b9db146d
Summary:
Add a function to compute the sha1 content hash for an overlay file.
We persist the computed hash in an extended attribute in the underlying overlay
file so that a subsequent read of the attribute doesn't require opening the
file to recompute it.
Each time the file is mutated, we blow the cached status of the file.
Each time the sha1 attribute is read, if the cache is blown, the content
hash will be recomputed and set in the overlay file.
Each time the file is flushed or sync'd, if the cache is blown, the content
hash will be computed and set in the overlay file.
Reviewed By: bolinfest
Differential Revision: D3302412
fbshipit-source-id: bd45c7a24b732bd0b7474b7f96e82936870b2117
Summary: This logic should be shared by the Eden CLI as well as unit tests.
Reviewed By: simpkins
Differential Revision: D3348300
fbshipit-source-id: c87b1f03f16560323f3d7685063bb6466c39efe2
Summary:
Enables mkdir in the overlay area.
I had to add some `lstat` calls in to the overlay dir reader because we depend
on knowing at least whether a node is a dir or not at the next level up.
When I run the test suite, the mounts are on my `/tmp` filesystem. When I run
eden manually, they are on my `/data` filesystem. The latter (xfs) does not
populate the type bits. This meant that the test suite passed but manual
testing did not.
Adding the `lstat` calls is a little unfortunate. On OS X there is a bulk
operation that combines `readdir` and `lstat` so that there are fewer syscalls.
We don't have an equivalent for Linux.
Reviewed By: bolinfest
Differential Revision: D3301532
fbshipit-source-id: e228f4a392f90aa491fec62e8b98471a8acecff2
Summary:
Centralize and delegate most (all?) of the content sensitive portions of file accesses into the FileData class.
Add tests to show that we can write to the overlay file and that the stat data is consistent with the result.
Reviewed By: bolinfest
Differential Revision: D3301251
fbshipit-source-id: a09316ad61c6ef4c656bc5d6dbd43f906abb7932
