Summary: Restructure the current logic used for loading the config data into a ClientConfig object. Rather than having loadFromClientDirectory iterate through all the config files and parse them to find the necessary information, abstract that logic out into a new method that compiles all of the relevant data so that all loadFromClientDirectory has to do is pull out the needed information. Since this change separates the two steps, this will make it easier to move the first step of compiling config information outside of ClientConfig - the goal here is to have the eden server load all of the config data at start up and cache it in memory so that it doesn't need to be done every time a ClientConfig object is created, and this change is an intermediate step.
Reviewed By: simpkins
Differential Revision: D3580757
fbshipit-source-id: c340a0fe715856066a554238249574f8177bc4d7
Summary: This should facilitate things by making the way to reference the path unambiguous.
Reviewed By: simpkins
Differential Revision: D3672311
fbshipit-source-id: b2a21e4dba60ea5ded28643f76aa402c4250d7ac
Summary:
If remotefilelog is unable to download a file from the server, it can end up
logging data through the UI which gets printed to stdout. Since we use stdout
to communicate with the main edenfs daemon this interferes with that
communication channel: edenfs ends up trying to parse the ASCII data as a
binary chunk header, sees a very large data length, and then ends up waiting
forever trying to read data that isn't coming.
This provides our own custom mercurial.ui.ui subclass to try and make sure that
we print all messages to stderr rather than stdout.
I will also send a separate diff so that we use a different pipe to communicate
with edenfs, rather than just using stdout/stdin. That should be a more
guaranteed way of making sure that nothing inadvertantly gets written to the
communication channel with edenfs.
Reviewed By: DurhamG
Differential Revision: D3673830
fbshipit-source-id: d64d492b10ee09ba63318b86d77f9181c4a0ce29
Summary: The repo_name could be cloned multiple times, which would cause a collision.
Reviewed By: simpkins
Differential Revision: D3672298
fbshipit-source-id: f27e1ec00ab773ed50f4fcf6f485461d01f8ba11
Summary:
Update the privhelper code to properly handle the error if it is asked to
unmount a path that isn't currently mounted.
This fixes the code to send an error response back to the main edenfs process.
Previously the exception propagated all the way up to the privhelper main loop,
causing the privhelper process to exit.
It would be nice to refactor the privhelper code a bit more to create a more
standardized API for implementing privhelper commands. This would provide a
more common place to catch exceptions, and make it harder for the command
implementation code to leak exceptions. However, I'll wait to do that in a
subsequent diff at some point.
Reviewed By: wez
Differential Revision: D3653727
fbshipit-source-id: e14c6ae974c8f99da3b426be250510c1f9b8017d
Summary:
Add back the "mount" CLI command.
While end-users may not need this command under normal circumstances, during
development it is sometimes necessary to re-mount unmounted clients. This adds
back the "mount" command which simply re-mounts an already configured client.
This also adds a "--no-forget" option to "unmount" which causes it to unmount
the client without forgetting its configuration. This allows it to be
remounted with "mount", rather than having to re-run the "clone" command and
specify the repository name again (and potentially restore the desired snapshot
hash).
Reviewed By: wez
Differential Revision: D3653706
fbshipit-source-id: 0e3f3fe1efdf72e9e54bdd1e0fb308ae67b7c058
Summary:
Use the default_permissions option when mounting. This tells the kernel to
enforce the normal file permissions semantics based on the file mode bits.
With this option enabled, the kernel will also implement the access() call for
us automatically.
Reviewed By: wez
Differential Revision: D3653198
fbshipit-source-id: e7227f455077a9ef012a5c102e4e22a976595681
Summary:
I have a follow on diff that needs this to work; this just injects the
existing hash functions that we're using for the boost hash stuff into
`std::hash` and checks that `set` and `unordered_set` compile.
Reviewed By: simpkins
Differential Revision: D3659318
fbshipit-source-id: 57920dbcd5f6e5449fd67769e379f8b3846a47f3
Summary:
It's useful in some debugging situations to understand the context for something like
`folly::throwSystemErrorExplicit(ENOENT, "useful context")`.
This is conceptually as simple as adding some `VLOG` statements in the error
handler for a fuse request, but I thought that it would be worth while to clean
up the macro that `clang-format` was doing a poor job of formatting. In turn,
I noticed that we can simplify the cancellation handling a bit while we're in
here, since we're only ever dealing with `Future<Unit>` in here now.
This reduces some boilerplate in Dispatcher.cpp.
Reviewed By: simpkins
Differential Revision: D3630958
fbshipit-source-id: 8d9dd50797f421ab9794b87f10d4c02c96e6fea8
Summary:
We weren't using these and they made some of our handling more complicated.
We now leave the release behavior to the destructor for the instance.
Reviewed By: simpkins
Differential Revision: D3615326
fbshipit-source-id: de39fb4315dd7b08da7c22cbcbd40e21be487102
Summary:
Include a configPath_ field for EdenServer that holds the path of the user ~/.edenrc config file. The server needs the data from this user config file in order to perform mounts and currently, the path to the home directory is passed via the CLI to the mount command as a field inside the MountInfo struct in order to get the file. As per discussion in D3498567, including the home directory inside the MountInfo struct is logically a bit disjointed, and this change would no longer require the home directory to be passed to the server via MountInfo.
This restructuring also sets up eden for a future change - having the server remount existing mount points on start-up is now possible from the inside. Before this change, mounting anything had to be done via the CLI since the home directory had to be passed in from the outside. This meant that remounting the existing mount points on start up could only be done if Eden was run in the background - running in the foreground would require manual remounting of all existing mount points. Now that the server has access to the config file's path, remounting can be done without any prompting from the CLI in both cases.
Reviewed By: simpkins
Differential Revision: D3580737
fbshipit-source-id: 46667ccd130b470a3a8a9e9aa08e5ec8e8b90336
Summary:
Previously we would simply report the raw pointer address to the kernel and
rely on it to return that same number to us as the file handle, and make sure
that it told us to shut down the handle when it was closed.
This meant that we had no real idea about which files were still open.
For our future live upgrade plans we need to be able to know this so that we
can transfer the appropriate information to our replacement process.
To facilitate this this diff implements a FileHandleMap class that will assign
file handle numbers and keep track of the instances. The number assignment
strategy is the same as it used to be in the common case: we take the address
of the newly created instance and use that 64-bit number as the file handle
number. However, in the future when we transfer the mapping to a new process,
we may experience a collision when subsequently opening a file handle. To deal
with that, we have a bounded number of attempts to assign a random file handle
number.
We don't yet offer a means to iterate the map, but it would be trivial to
expose such an accessor when we're ready to use it.
Since we now keep track of these things via shared_ptr this changes the
appropriate portions of the fuse interface from unique_ptr to shared_ptr.
Reviewed By: simpkins
Differential Revision: D3602364
fbshipit-source-id: dd996339c2838225a2caeee9da16ef99a06c1e2b
Summary:
When the EDEN_GDB environment variable is set, start eden under gdb when
running the integration tests. Have gdb automatically start the daemon,
and exit automatically if everything is successful. If the daemon crashes, gdb
will break and the user can interact with gdb normally.
This is generally only useful when manually running a single test. Trying to
use this in combination with "buck test" will probably cause problems if
multiple gdb instances all try to interact with the terminal together.
Reviewed By: wez
Differential Revision: D3459575
fbshipit-source-id: 1359735a391e93a332a4c673c6525ad44034cc73
Summary: This change updates LocalStore to perform serialization of trees and blobs internally so that its users don't need to be aware of the internal serialization format. Previously, the get and put APIs were asymmetric such that the get APIs returned deserialized Tree and Blob objects, while put required raw serialized bytes. After this change, put will also use deserialized Tree and Blob objects.
Reviewed By: simpkins
Differential Revision: D3589899
fbshipit-source-id: 2e572e6ec5af44d66206b178a03f7a9d619b2290
Summary:
Update the CLI to always close the thrift client socket, to avoid resource leak
warnings on exit.
I also updated the code to just monkey-patch a nicer EdenError.__str__()
method, rather than having to explicitly catch and modify this exception in
multiple different places.
Reviewed By: bolinfest
Differential Revision: D3560662
fbshipit-source-id: 900fe74c793ffd99f4a2c1f1ddd94b96e48f5eb7
Summary:
The "eden repository <name> <path>" command had a bug that it would duplicate
the existing edenrc contents when writing out a new config: it opened the file
in append mode rather than truncate mode when performing the write.
This addresses that issue, but also does a bigger overhaul of the config update
code. We now acquire a lock for the duration of the modification, so that
another eden CLI process cannot modify the file between when we read it and
when we write out our modifications. We also perform the write using an atomic
rename so the file contents are always valid at all points in time.
I also updated the CLI command to print the repositories in sorted order, and
to only catch expected exceptions, and show backtraces for unexpected errors.
Reviewed By: bolinfest
Differential Revision: D3554550
fbshipit-source-id: 5920ccb2447330673eac3f9956a8ae5d8a66a67e
Summary:
This feels awful, but eliminates an abort that shows up when running under ASAN.
Details in the comment.
Reviewed By: bolinfest
Differential Revision: D3533529
fbshipit-source-id: 1717384c716f565b74b1678c66883a35ee60c522
Summary:
For consistency with `git clone`, the destination path must be either:
* non-existent (but `mkdir -p`-able)
* an empty directory
Or else `clone` should fail.
In the process of writing the integration test, I cleaned up some of our test infra
and took advantage of the fact that we require Python 3 (I believe 3.5) now:
* We now use `subprocess.run()` instead of `subprocess.check_output()`.
* Removed the `client_name` argument to `EdenClient.init()` because that is no longer the right terminology and no one appeared to be overriding the default, anyway.
* `EdenClient.clone_cmd()` no longer calls `os.mkdir(self._mount_path)` because that was papering over the issue this revision is trying to solve.
Reviewed By: simpkins
Differential Revision: D3526909
fbshipit-source-id: 66b5799a1b47a51468faf5dc17d87a385926dc89
Summary:
The intention is to use this for the PathTrie that is in a following diff.
The motivation is to avoid making a N copies of a string when traversing a path
of depth N in a PathTrie. The PathMap is backed by a vector and knows how to
compare the Piece variant of our path types.
I have not performed any rigorous comparisons, but based on reading around
other flat_map vs map comparisons, the expectation is that this will use less
memory (and avoid extraneous allocations during lookups) than a regular map,
with most random access being fairly similar. Inserts need to move the memory
around to make room and are thus slower than a regular map. Sorted inserts
perform the best for PathMap as that translates into repeated back-insertions.
My intuition is that this is the right trade off to make for managing the
overlay data structures; the size of these materialized dirs is likely going to
be small compared to the full fbsource tree.
Reviewed By: simpkins
Differential Revision: D3470377
fbshipit-source-id: 6f4f3abe33818da6981d8ab821ebe004a78d382c
Summary: Move adding the mount path to the directory map as the final step of the clone command so that the path is only documented if the mount is actually successful. Previously, if 'eden clone' failed during the mounting, another call to 'eden clone' to try again would fail with 'Error: mount path NAME already exists.' This change ensures that a second call to 'eden clone' will actually try again if the first failed.
Reviewed By: simpkins
Differential Revision: D3543388
fbshipit-source-id: be6412ad6d5d009bd87ad6e0b4110b35e6300fcb
Summary: This change fixes exception messages so that they all match the same format and don't include 'error: ' twice
Reviewed By: simpkins
Differential Revision: D3544034
fbshipit-source-id: 32d600c712d1782d2a1a35331b66e06a842ed8e0
Summary:
[Folly] Move `IPAddress` definitions to source files.
And to internal header files. Keeping headers lightweight can help with build times.
Reviewed By: simpkins
Differential Revision: D3514455
fbshipit-source-id: de78f4ef9e70e7ddd7fb666348ed705c5228531c
Summary:
Add a decorator to more easily declare tests that should be run for both git
and mercurial repository types.
Reviewed By: wez
Differential Revision: D3458883
fbshipit-source-id: 8058873923f9519a6550a8dcb930a5edd02aec10
Summary:
- Refactor the EdenClient code to better reflect how the eden currently works.
This code was originally written when the edenfs daemon only supported
running a single mount point. This updates it to reflect the fact that it
manages a single edenfs daemon, but multiple repositories can be mounted.
- Refactor the EdenTestCase now that tests generally only need a single eden
daemon. EdenTestCase starts the eden daemon by default.
- Add EdenHgTest and EdenGitTest classes. These are subclasses of
EdenTestCase, and they create and mount an hg/git repository before starting
the test function.
- Update the tests to derive from EdenHgTest and EdenGitTest where appropriate.
Reviewed By: wez
Differential Revision: D3458842
fbshipit-source-id: 77349a60ff72a700a2c2526a27e7621b76f9eec2
Summary: Fix a bug with the eden repository command that let it add multiple repositories with the same name. If an entry for repository 'name' already exists in the home ~/.edenrc config file, the eden repository command will now raise an error message saying that the repository already exists. Note that if an entry for repository 'name' exists in any of the global /etc/eden/config.d/* files, the eden repository command will still allow the user to create the new repository and will add an entry for 'name' to the home ~/.edenrc config file (since entries in home config file overwrite those in global)
Reviewed By: wez
Differential Revision: D3529743
fbshipit-source-id: b7c7ceef6ef38fcd3b4ff9a92fbd23a4a0bd6951
Summary:
Mercurial maintains its manifest in sorted order, but since the manifest only tracks file names we can end up with the following sequence:
```
some/path-foo/bar
some/path/bar
```
This is because the `-` sorts ahead of the `/`.
This diff defers passing the entries to the tree serializer, buffering them up
into a temporary vector and using `std::lower_bound` to find the appropriate
insertion point.
Reviewed By: bolinfest
Differential Revision: D3529329
fbshipit-source-id: 395ed16a20c14d17717ec69192a38f0407b51e1d
Summary:
This check was already being done appropriately in our Python code, but we also need
to do it in our C++ code.
Reviewed By: wez
Differential Revision: D3526705
fbshipit-source-id: 3b28b88f63ae768113f363ace58d40a89a8f4b61
Summary: Move the _get_or_create_dir() method from config.py to util.py and rename as mkdir_p(). This change replaces all of the os.makedirs call sites with the new mkdir_p() method.
Reviewed By: bolinfest
Differential Revision: D3512570
fbshipit-source-id: a867049f9af22076934390061f09070bf9ee6397
Summary:
These changes restructure the eden directory so that 'client' directories are created during the `eden clone` command and are associated with a single mount path.
The new eden directory looks as follows:
~/.eden
config.json
clients/
abcd08d/
edenrc
SNAPSHOT
overlay/
efgh19i/
edenrc
SNAPSHOT
overlay/
...
Where the config.json file holds the mapping of mount paths to their respective client directory which is a hash, and the edenrc files in each client directory is an INI file which holds the name of repository associated with the mount path. This INI file follows the current format:
[repository]
name = fbsource
This restructuring required a couple other changes:
- unmount command now cleans up the client directory and removes the mapping of its mount path from config.json
- eden list command now lists all of the mount paths rather than the client names
Reviewed By: bolinfest
Differential Revision: D3506119
fbshipit-source-id: dc07a8baf1052be731ff335d9cf74a07ab8e661a
Summary: Change the ClientConfig class to parse client data via INI config file rather than json file. This class uses boost::property_tree::ini_parser and the ptree data structure to hold the parsed INI file contents. This change makes it possible for eden to no longer rely on json files for getting client data, and the json files will be completely taken out in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3498567
fbshipit-source-id: 3298047a014beda0c250475c0809a7a1ebd95b2b
Summary: Previously the repo snapshot id was collected and stored when the repository was initialized, but this diff moves the update to the clone command so now the snapshot is only stored right before a repository is mounted.
Reviewed By: bolinfest
Differential Revision: D3489550
fbshipit-source-id: 1271c1d7c7709ed332307a636c23e26e913483c9
Summary: Update the unmount command to use the mount point rather than the client name. This diff also removes the mount command completely since tests are no longer dependent on it.
Reviewed By: bolinfest
Differential Revision: D3489290
fbshipit-source-id: 9bfbcb78a78d1593f6d0950502bdd4a55e3e6ca4
Summary: Introduce new 'clone' command that takes in repo name and mount path. This command replaces the current mount command which uses the client name as an argument
Reviewed By: bolinfest
Differential Revision: D3489098
fbshipit-source-id: 719fb94387da1d4ccab770a0e659f717888a7dc6
Summary: Expand functionality of repository command to also create repositories. The current command line format is 'eden repository <name> <path>' and will soon replace the existing init command.
Reviewed By: bolinfest
Differential Revision: D3489064
fbshipit-source-id: 3e6946cbd80a64ad23b52383f3f14a92d4492a21
Summary: Add a simple eden repository command that takes no arguments. The behavior mimics that of git remote and lists all repositories by parsing both the global and local INI config files. The INI file currently needs to be manually added and updated to maintain information about existing repositories.
Reviewed By: bolinfest
Differential Revision: D3479543
fbshipit-source-id: b0d36f33c9b8e0c7b9fe20ec2f362ca15c0adeb8
Summary:
This is a bit ugly and longer term should probably be an
eden configuration that we deploy to our CI tier.
In our CI environment we don't allow unfettered password-less sudo
access and instead must launch privileged processes via a helper
process that is whitelisted. D3524299 has the definition of that
helper as it applies to our internal infra.
Reviewed By: bolinfest
Differential Revision: D3524373
fbshipit-source-id: f36bf29e6b455220c434b681a6d0d551bc21ecab
Summary:
We're seeing failures like this in our CI system:
```
stderr:
test_create (eden.fs.integration.basic_test.BasicTest) ...
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: no tty present and no askpass program specified
error: edenfs is not running
ERROR
ERROR
======================================================================
ERROR: test_create (eden.fs.integration.basic_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "eden/fs/integration/basic_test.py", line 68, in test_create
eden = self.init_git_eden()
File "eden/fs/integration/lib/testcase.py", line 104, in init_git_eden
return self.init_eden(repo_path)
File "eden/fs/integration/lib/testcase.py", line 54, in init_eden
inst.init(repo_path, *kwargs)
File "eden/fs/integration/lib/edenclient.py", line 160, in init
self.daemon_cmd(timeout)
File "eden/fs/integration/lib/edenclient.py", line 174, in daemon_cmd
self._wait_for_thrift(timeout)
File "eden/fs/integration/lib/edenclient.py", line 99, in _wait_for_thrift
raise Exception("edenfs didn't start within timeout of %s" % timeout)
Exception: edenfs didn't start within timeout of 10
```
This diff improves things by failing faster and printing a little more appropriate context.
Reviewed By: bolinfest
Differential Revision: D3523257
fbshipit-source-id: 14508a1b820d9d6c773713e2bb7c22955c5b2b66
Summary:
Implements basic rename(2) support. We only do this for files at this
time as directory renames require that we recursively materialize a tree and
are a bit more complex. I'll look at that in a follow-on diff; there's
potential for optimizing it to avoid eager materialization, but that depends on
the trie work that I'm tackling concurrently with this.
rename for files is the last piece needed to allow `sed -i` to operate correctly.
Reviewed By: bolinfest
Differential Revision: D3515361
fbshipit-source-id: 9c8cc5f9b8db6b5a9372ca9286336647f50490f8
Summary:
This enables O_EXCL to function by allowing the create routine to
move its folly::File instance down in to the underlying FileData instance.
Previously we would close and then re-open the file; this would discard
any of the natural gating for the open call that is performed by the kernel
for the underlying filesystem in the overlay.
Reviewed By: bolinfest
Differential Revision: D3513758
fbshipit-source-id: 85967a3b7affa1b1df46842be8ba21c8fbb843a6
Summary:
Despite being handy things for filesystems in general, these are
needed to support the `sed -i` integration test.
Reviewed By: bolinfest
Differential Revision: D3513754
fbshipit-source-id: 505b4bd58b254141e2ef996f01e3347fc1a77584
Summary:
This eliminates a linear scan from TreeInode and replaces it with a
binary search, exploiting the sorted order of the entries vector.
Two new methods are introduced: getEntryPtr which returns a pointer to the
entry with the matching name, and getEntryAt() which returns a reference
(throwing a range error if there is no such entry).
I wanted to use the PathMap class here, but that would cause us to duplicate
the name string as both the key and value in the map.
Reviewed By: bolinfest
Differential Revision: D3515723
fbshipit-source-id: 4ee0371f3ec08cbcf110cf28f5c1e1529b120fb6
Summary:
I found this by running the tests with ASAN enabled; it was reporting leaks.
The issue is that fuse decides to append some args to the argument list in some
cases. When it does this it tries to realloc the storage in the argv array.
This was not safe to do because that data was owned by the vector and may not
have been allocated directly by malloc.
This diff switches us to use the underlying functions in libfuse that append
arguments and then know how to safely free the result.
SCOPE_EXIT is used to ensure that this gets cleaned up when we return.
Reviewed By: bolinfest
Differential Revision: D3519999
fbshipit-source-id: ca12739f11be1fd9662063ec7515cec90ccf0d57
Summary:
This removes use of the builtin Buck `thrift_library` support from
the macro library. It turns out that a lot of rules incorrectly
added deps onto the raw builtin `thrift_library`, rather than one
of the per-language rules. This is a noop, and removing the builtin
rule exposes this as missing target errors, so this diff removes them.
Reviewed By: Coneko
Differential Revision: D3512451
fbshipit-source-id: dd8beb148ed47a3ad7d3963fae600abd73d030d5
Summary:
setattr is a bit of a multi-purpose interface; depending on the flag
values, this is responsible for:
* ftruncate(2)
* fchmod(2)
* fchown(2)
* futimens(2)
In order to apply any of these things, we have to materialize the file. In
the future we may want to allow setting the utimes without materializing the file.
We don't allow chown to actually chown anything. We may want to relax that in
the future, but at the moment we will return an error if an attempt is made to
change the ownership of a file.
Reviewed By: bolinfest
Differential Revision: D3511011
fbshipit-source-id: 858d2c07686fcbe2dcdb60a07527f739a9726be3
Summary:
Make sure mount points are completely stopped before destroying the EdenServer
object. Previously the EdenServer was destroyed with the MountPoints still
running the fuse channels in background threads. When the privileged helper
process unmounted them, fuse requests from the kernel could arrive and access
memory that had already been destroyed.
Reviewed By: wez
Differential Revision: D3458898
fbshipit-source-id: 365bca716ff0f8315b66af92effeb8c6dc574ce1
Summary:
When running the integration tests when built with gcc, the tests would crash
in TreeInode::create(). It appears that the unique_ptr<FileHandle> object was
getting passed to the lambda before dereferencing it to call getattr().
Reviewed By: wez
Differential Revision: D3459605
fbshipit-source-id: 5e2ce98d268a85731acaf7d7f37f22c77fb571cf
Summary:
This fixes inode number handling problems in TreeInode and TreeEntryFileInode.
Previosly these classes each had an ino_ member variable, despite deriving from
InodeBase which has its own (private) ino_ member.
TreeEntryFileInode never actually initialized its own local ino_ variable.
This was causing problems for many applications which use inode numbers to
cache file data. TreeEntryFileInode returned garbage data in the inode field,
resulting in incorrect cache collisions.
This fixes the inode handling, and also fixes the stat data returned by
FileData. It now sets the uid, gid, and inode fields correctly. It 0s out
several other fields rather than leaving them uninitialized.
Reviewed By: bolinfest
Differential Revision: D3455126
fbshipit-source-id: 631276b01676733f96035bc153219ef84406dcc9
Summary:
Start adding utility code to make it easier for the integration tests to
manipulate git and mercurial repositories.
This adds an HgRepository class, but does not update the tests to use it yet.
I will do that in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3449328
fbshipit-source-id: caea5befb683a388cbad30a29017c566e22702af
