Summary:
It turns out that the manifest class provides an iterentries() method to
iterate through the (path, hash, flags) values. Update the code to use this
rather than iterating through the paths and then having to look up the hash and
flags using the path.
This dramatically speeds up importing a commit with 1M files: the import time
drops from around 8 seconds to under 1.5 seconds. (Of the 1.5 seconds, about
half is spent reading the manifest data, and half is spent formatting and
writing the chunks.)
Reviewed By: wez
Differential Revision: D4512538
fbshipit-source-id: ccfff162082d327d525224b2d7dc87f67bda5e22
Summary:
This updates the TreeInode::rename() code to handle concurrency better.
In particular:
- The code now ensures that both the source inode being renamed and destination
inode (if it exists) are loaded. This simplifies issues when an inode is
being loaded at the same time a rename is in progress. This ensures that any
pending load is processed before the rename takes place. (All promises for
the load might not be fulfilled before the rename completes, but the relevant
TreeInode and InodeMap data structures are updated before the rename occurs.)
This does mean that the rename code potentially might have to retry several
times if the inode it began loading is no longer the affected source or
destination or child once the load completes. However, this seems like a
reasonable trade-off, compared to dealing with the complications that would
arise with the load code having to handle renames occuring before load
completion.
- The code now implements proper lock ordering, to avoid acquiring locks in
conflicting orders that might cause deadlock with other threads also trying
to acquire the same locks. The InodeLocks.md document has been updated to
clarify the TreeInode lock ordering requirements.
Reviewed By: wez
Differential Revision: D4493526
fbshipit-source-id: 627393fafad90eb551aea62be7762d59ed043abe
Summary:
Move the integration tests from eden/fs/integration up one directory, to
eden/integration.
The main benefit is that this makes it easy to run just the edenfs unit tests
by running "buck test eden/fs/...". These unit tests complete much more
quickly than the full set of integration tests, providing a faster test suite
to re-run repeatedly during development. The integration tests can be run with
"buck test eden/integration/...", and the full set of tests can still be run
with "buck test eden/..."
Reviewed By: wez
Differential Revision: D4490247
fbshipit-source-id: 5ceb5a19526f56e1cb926f352fa30ad2f1212c05
Summary:
Most of the FileData logic assumes that file_ is open if the file is
materialized. Fix the FileData() constructor to actually open file_ if the
entry is marked materialized, so that this assumption is actually true.
Reviewed By: wez
Differential Revision: D4471365
fbshipit-source-id: b399e6e5e25eaf12f4204f5835db01baa5a6cd78
Summary:
Update the code to release the parent TreeInode's contents lock when processing
its children. This makes sure we do not hold the root lock for the entire
duration of this function.
Reviewed By: wez
Differential Revision: D4471289
fbshipit-source-id: dc4b50bd06fcceb7a7df3d7dd04cfc0d41285556
Summary:
This is to facilitate the watchman integration and draws on the
watchman glob implementation; the approach is to split the glob strings into
path components and evaluate the components step by step as the tree is walked.
Components that do not include any glob special characters can be handled as
a direct lookup from the directory contents (O(1) rather than O(num-entries)).
The glob method returns a set of filenames that match a list of
of glob patterns.
Recursive globs are supported. It is worth noting that a glob like "**/*" will
return a list of every entry in the filesystem. This is potentially expensive
and should be avoided. simpkins is in favor of disallowing this as a forcing
function to encourage tool-makers to adopt patterns that don't rely on a
complete listing of the filesystem.
For now I'd like to get this in without such a restriction; it's also worth
noting that running `find .` in the root of the mount point has a similar
effect and we can't prevent that from happening, so the effect of the overly
broad glob is something that we need to be able to withstand in any case.
Unrestricted recursive globs will make it easier to connect certain watchman
queries in the interim, until we have a more expressive thrift API for walking
and filtering the list of files.
Note: I've removed the wildmatch flags that I'd put in the API when I stubbed
it out originally. Since this is built on top of our GlobMatcher code and that
doesn't have those flags, I thought it would be simplest to just remove them.
If we find that we need them, we can figure out how to add them later.
Also Note: the evaluation of the glob is parallel-ready but currently limited
to 1 at a time by constraining the folly::window call to 1. We could make this
larger but would need a more intelligent constraint. For example, a recursive
glob could initiate N concurrent futures per level where N is the number of
sub-dirs at a given level. Using a custom Executor for these futures may be a
better option to set an upper bound on the number of concurrent jobs allowed
for a given glob call.
Depends on D4361197
Reviewed By: simpkins
Differential Revision: D4371934
fbshipit-source-id: 444735600bc16d2c2185f2277ddc5b51f672600a
Summary:
As part of the mount process, make sure we load InodeBase objects for all files
in the mount that are already materialized.
Other parts of the code assume that InodeBase objects are always loaded for
materialized files. We never unload materialized inodes, but if the mount
point was unmounted then remounted we previously did not ensure to load the
materialized InodeBase objects. This diff makes sure we load all materialized
inodes before starting the mount.
Reviewed By: bolinfest
Differential Revision: D4461193
fbshipit-source-id: 70d06fd01e2df333ce2816d5d7a392b0a5d6e1e6
Summary:
Fix two issues with inode load completion:
1) Make sure we release the parent's TreeInode lock before fulfilling promises
waiting on the inode load to complete. Otherwise this can result in deadlock
if the promises have callbacks that then try to acquire the TreeInode lock.
2) Use unique_ptr<InodeBase> in the load completion code before we have
inserted the Inode into the InodeMap, rather than using InodePtr. The new
InodePtr class relies on the InodeMap to handle ownership, and it cannot
destroy Inode objects unless they are in the InodeMap. Using a unique_ptr
instead ensures that the InodeBase will be destroyed properly if an error
occurs between when we construct the InodeBase and when we insert it into the
map.
Reviewed By: bolinfest
Differential Revision: D4427889
fbshipit-source-id: 2627a4956735f4acebe5c259abd8de0430ff6177
Summary:
Rename the current EdenMount::getInodeBase() function to
EdenMount::getInodeBaseBlocking() and add a new getInodeBase() function that
performs the lookup asynchronously and returns a Future<InodePtr>.
This is implemented with a TreeInode::getChildRecursive() function that allows
asynchronous recursive lookups at any point in the tree.
Reviewed By: bolinfest
Differential Revision: D4427855
fbshipit-source-id: aca55e681a48c848b085b7fc6a13efe6cf0a4e3a
Summary:
Add methods to InodeBase to get the parent of the current Inode, and remove the
existing TreeInode::getParent() function. The old TreeInode parent_ member was
never updated properly after rename or unlink operations, so it could return
incorrect information.
Reviewed By: bolinfest
Differential Revision: D4427103
fbshipit-source-id: cb66a014f745b4af31033e0a3c5405e29791b869
Summary:
Thesis: Since we don't yet have `hg update` implemented, these attributes can be cached forever.
When we implement `hg update` we will explicitly inform the kernel to invalidate anyway.
... and it turns out that the `scmRemove` thrift call is making changes that
aren't visible to the kernel already so we need to hook up some basic
invalidation.
This diff includes the bare minimum code to facilitate this; I've added a
helper method to the fusell::RequestData class to test whether we are in the
context of a FUSE request.
In the EdenDispatcher::unlink method we need to explicitly invalidate if we are
not being called via FUSE.
I have mixed feelings about putting this code in here. Given the initial
mental model I had in the prototype, this is the right place to put it, but it
does mean that all tree mutations must have via the Dispatcher in order for
invalidations to be routed to the correct places. Do we want to move away from
this?
It wouldn't be the end of the world to expand this diff a bit to add
similar/appropriate invalidation calls to all the mutation methods in our
dispatcher implementations while we figure out if we want to keep this.
Reviewed By: simpkins
Differential Revision: D4452961
fbshipit-source-id: 9471f145242fce0620c6872b74b02c56c8d78af1
Summary:
This mirrors the capability added in D4444058.
I don't think the `getenv` calls are awesome, but also don't think that we're
likely to want to override these outside of the environment either, so it
doesn't warrant plumbing these all the way through the eden cli, edenfs command
line flags and the layers to get to instantiate this object.
Reviewed By: simpkins
Differential Revision: D4446320
fbshipit-source-id: d5661e4f3e8dee82617eb6edddbcb9da5f4296d2
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures. These were caused by the integration tests picking up the
default post-clone hook configuration.
This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.
Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.
The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.
Reviewed By: bolinfest
Differential Revision: D4446321
fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
Summary:
It's been bothering me that we claim to run and complete running
the globally installed hooks when they aren't there, so tidy up the logging.
Reviewed By: bolinfest
Differential Revision: D4446318
fbshipit-source-id: acda223e1df2302602ad9967b1aa1b334b8477c7
Summary:
This adopts the new InterpolatedPropertyTree class as
the configuration storage.
It doesn't introduce any actual configuration of the interpolation
replacements; that will be in a follow-on diff.
Reviewed By: bolinfest
Differential Revision: D4444157
fbshipit-source-id: 18ead8e9074d23b1154e81f012f0c90efced1350
Summary:
This makes it possible to deploy via packaging (we'll put the par file
in the same dir as the extension) and eliminates a recursive buck build call
that was slowing down our integration tests; we can now just look up the path
from the environment that we set up in the TARGETS file to locate the par files
in the integration tests.
Reviewed By: bolinfest
Differential Revision: D4444096
fbshipit-source-id: 2ad6b6aa783563dc54a1cfd347a509c9919bb17a
Summary:
I don't know that I'm happy with the name; it started out as
just a helper wrapper to apply interpolation, but grew a bit to better
encapsulate how we access the config from the eden server.
This is a utility class that I'd like to use in place of the raw boost
property tree in our ClientConfig code.
It encapsulates the underlying property tree and provides a couple of simple
methods; one for loading data from a config file and one for looking up
a configuration value.
The configuration value lookup will replace tokens of the form `${KEY}`
with the value of `KEY` from its internal replacements dictionary.
This mirrors the interpolation handling that we do in the equivalent python
code for the eden client.
Reviewed By: bolinfest
Differential Revision: D4444075
fbshipit-source-id: 5d46d63f87caad4f409fbb981aa83165fcd6596d
Summary:
I'm going to follow up with some changes to substitute
some core env vars; this is just some prep.
Reviewed By: bolinfest
Differential Revision: D4444030
fbshipit-source-id: ab5d0e39aba14cdba39bb4867ec24955665723c6
Summary:
`eden config` dumps all parsed config values.
`eden config --get SECTION.OPTION` prints the value of the corresponding key.
Reviewed By: bolinfest
Differential Revision: D4434084
fbshipit-source-id: 63d9f72aa8794371e89a6d9e527bacaab17540de
Summary:
This makes it easier to reason about where these things can be found,
and in particular, allows passing the path through to the post-clone hook
script. Although post-clone accepts some command line configuration options,
our hooks scheme doesn't seem to allow setting that in the eden configuration
file.
Reviewed By: bolinfest
Differential Revision: D4443872
fbshipit-source-id: ed92dbac3b91b7f7dc86a39a85b465907f290b47
Summary:
We implement some sanity checking in wrappers, preventing some unsafe
consequences of using raw Buck UI in fbcode.
Reviewed By: Gownta
Differential Revision: D4453147
fbshipit-source-id: 3069898069c7e89223b133224f7c87d1a6b5886a
Summary:
Fix issues flagged by running "arc lint" on all eden files. There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.
Reviewed By: bolinfest
Differential Revision: D4446615
fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
Summary:
10 seconds isn't long enough for the tests to pass consistently on my devserver (I have a windows VM running on here, so there is more background crap running).
Raising this up to 30 gave me a successful test run.
Reviewed By: bolinfest
Differential Revision: D4442300
fbshipit-source-id: 5c3ae1fc0d0acb21c5bb44e36dae88d5180749d1
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Previously we'd emit an AttributeError. This diff
fixes things up so that we show the help when no arguments are provided.
Reviewed By: simpkins
Differential Revision: D4434060
fbshipit-source-id: 838b1cab47118d8517e4dc0c5fde17fc69752b13
Summary:
This lock protects all operations that change the location of existing inodes
(rename(), unlink(), and rmdir()). This is needed to avoid race conditions
during rename operations.
I also included some FIXME comments about properly updating location
information for unloaded inodes. These should be taken care of in a subsequent
diff.
Reviewed By: wez
Differential Revision: D4361197
fbshipit-source-id: 5aaa1b8250f196b23207be8f3c0fd7f2603e6ae8
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.
The inode lookup itself still happens serially at the moment. We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.
Reviewed By: wez
Differential Revision: D4361091
fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
Summary:
This updates the InodePtr and InodeBase code to actually implement Inode
unloading and destruction.
At the moment we keep Inode objects as long as possible, and only unload them
during shutdown, or when the last reference to an unlinked inode goes away.
However, it should be straightforward to add on-demand unloading in the future
for Inodes that have not been accessed in a while. The
TreeInode::unloadChildrenNow() function provides a template for what this would
look like (it would simply need to be changed to check an access time when
doing on-demand unloading).
Reviewed By: wez
Differential Revision: D4360765
fbshipit-source-id: a46b355f0ac0c25f873a156e62af5184317de735
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function. This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.
This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.
Reviewed By: wez
Differential Revision: D4360558
fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
Summary:
This defines our own custom smart pointer type for Inode objects. This will
provide us with more control over Inode lifetime, allowing us to decide if we
want to unload them immediately when they become unreferenced, or keep them
around for a while.
This will also allow us to fix some memory management issues around EdenMount
destruction. Currently we destroy the EdenMount immediately when it is
unmounted. This can cause issues if other parts of the code are still holding
references to Inode objects from this EdenMount. Our custom InodePtr class
will also allow us to delay destroying the EdenMount until all of its Inodes
have been destroyed.
This diff adds the new pointer types and updates the code to use them, but does
not actually implement destroying unreferenced inodes yet. The logic for that
has proven to be slightly subtle; I will split it out into its own separate
diff.
Reviewed By: wez
Differential Revision: D4351072
fbshipit-source-id: 7a9d81cbd226c9662a79a2f2ceda82fe2651f312
Summary: Folly will soon be dropping support for GCC 4.8, which means that `folly::make_unique` will be going away, so codemod away as much as possible before then.
Reviewed By: yfeldblum
Differential Revision: D4411221
fbshipit-source-id: 31d61425f6595d0750ea2e4c95c7d42bb5a5d955
Summary: I fixed an unrelated bug in HTE that made some more paths lintable - and some more move-abuses visible :-(
Reviewed By: meyering
Differential Revision: D4397732
fbshipit-source-id: 560db05f64ecf160e8198caf80dbd988e0dad75d
Summary:
I ported the C++ test I wrote for `hg add` in D4317871 as an integration test.
Note that this caught a real bug in the Hg extension!
Reviewed By: wez
Differential Revision: D4385970
fbshipit-source-id: 97ee555d463f22af54791f334f9349d88861f45a
Summary:
This constructor is unsafe. The check it uses before doing the comparison isn't a safe enough check to see if the cast is valid. For example, this is broken in the presence of multiple inheritance because it doesn't adjust the pointer offset to the correct vtable.
e.g.
struct A {
virtual ~A() {};
virtual void doSomething() = 0;
};
struct B {
virtual ~B() {}
virtual void doSomethingElse() = 0;
};
struct C : public B, public A {
virtual ~C() {}
void doSomething() override {
std::cout << "Something!" << std::endl;
}
void doSomethingElse() override {
std::cout << "Something Else!" << std::endl;
}
};
int main (int argc, char **argv) {
auto c = folly::makeFuture<std::shared_ptr<C>>(std::make_shared<C>());
folly::Future<std::shared_ptr<A>> a = std::move(c);
a.get()->doSomething();
return 0;
}
This code will print "Something else!" when run.
Reviewed By: siyengar
Differential Revision: D3679673
fbshipit-source-id: dcbf40ca82d458f17ee11191591f8b8daf58c919
Summary:
This describes the behavior around Inode ownership, loading and unloading, as
well as the synchronization around these events.
The InodeMap and InodePtr diffs implementing this behavior are still in flight.
Reviewed By: bolinfest
Differential Revision: D4356591
fbshipit-source-id: 298ba2c6aa98feba8b90f858e5633ca793276a1e
Summary:
If the EDEN_DAEMON_ARGS environment variable is set when running integration
tests, append it to the edenfs daemon command line.
For instance, this makes it easy to pass in a "--vmodule" flag to turn up
verbose logging in particular parts of the code.
Reviewed By: bolinfest
Differential Revision: D4359306
fbshipit-source-id: 6b549ac7220f6c8d7a1dfad3827921d6462d0f77
Summary:
Add a document describing the various inode related locks and the order that
they may be acquired.
Reviewed By: bolinfest
Differential Revision: D4356023
fbshipit-source-id: 44d4ade984f6cb49bb5f09deeeef0fd5439f129c
Summary:
Add VLOG() statements for every FUSE entry point. Turning up the VLOG level in
EdenDispatcher now allows us to have a record of all FUSE calls made using
inode numbers. (Calls involving file handles or directory handles currently go
directly to the handle in question and do not pass through the EdenDispatcher.
We should probably add log statements for those too, but that can wait for a
later diff.)
Reviewed By: bolinfest
Differential Revision: D4359066
fbshipit-source-id: b378d829c39fdb61faf63d2d400f3ff556c376e8
Summary:
This member variable was not updated properly when files were renamed.
InodeBase now tracks our parent properly, so we don't need our own copy.
This does still call getParentBuggy() (which does not perform proper locking)
in a couple places for performing overlay operations. We'll need to fix this
later when addressing other overlay concurrency handling issues.
Reviewed By: bolinfest
Differential Revision: D4348481
fbshipit-source-id: 19c1ffced6f63e1ff041d0bab2363fecdb93d5a3
Summary:
Have FileData objects store a pointer to the FileInode that owns them, rather
than just to the EdenMount.
FileData objects still store a direct pointer to their FileInode's mutex_ and
entry_. It's potentially worth just accessing these through inode_ in the
future.
Reviewed By: bolinfest
Differential Revision: D4348103
fbshipit-source-id: 1f8497979bfc89c6a192ca0195209335db0d911c
Summary:
Add an EdenMount* member variable to InodeBase. Previously each TreeInode kept
a pointer to its EdenMount, and this moves that into the InodeBase class.
This is needed to support upcoming diffs that will change InodeBase memory
management. InodeBase objects will be responsible for notifying the InodeMap
when they become unreferenced.
Reviewed By: bolinfest
Differential Revision: D4348079
fbshipit-source-id: 6bf7ea908c6096aa2bca5b21290c09cbd58d5af7
Summary:
This updates all of the eden code to use the new InodeMap class. This replaces
the InodeNameManager class and the unordered_map previously stored in the
EdenDispatcher.
Reviewed By: bolinfest
Differential Revision: D4325750
fbshipit-source-id: d80ae7581ba79ca2b63155e184995a3e83e85dc1
Summary:
This diff starts adding a new InodeMap class. This class will eventually
consolidate the functionality of InodeNameMap plus the inode map stored in
EdenDispatcher.
This new class will bring several new benefits:
- All inode mapping logic consolidated into a single class, with a single lock
protecting the maps.
- A well-defined model for loaded vs unloaded inodes. InodeMap explicitly
tracks inodes that have InodeBase objects created for them vs inodes that
have an inode number allocated (for FUSE) but do not have an InodeBase object
in memory. This will make it possible to unload Inode objects on demand to
reduce memory usage.
- Tracking of pending loads, and de-duplicating load requests. This ensures
that only one Inode object ever exists for a given inode number / path. If a
second request to load an Inode arrives when a previous load request is still
in progress, InodeMap deals with this situation properly.
- Better support for using Inode objects without FUSE. With the old code,
attempts to interact with Inode objects without going through the FUSE
dispatch (say, when processing a thrift call) could result in inconsistent
state. New inodes created would not be put into the EdenDispatcher map,
which could result in problems.
- More convenient child inode access from TreeInode. With this change, the
TreeInode class can easily tell which of its children are loaded. This makes
it easier to do tasks which only need to operate on existing loaded inode
state (for instance, dirstate computation).
- Support for saving and loading state, to implement graceful restart..
InodeMap provides a central place to write out inode state on shutdown and
restoring it on startup. Saved inodes can easily be restored to an
"unloaded" state on startup. This code is not implemented yet as part of
this diff, but it should be straightforward to add in future diffs.
Reviewed By: bolinfest
Differential Revision: D4318060
fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
Summary:
This introduces an `HgExtensionTestBase` class that can be subclassed
to create tests for Mercurial integration with Eden. It also introduces
`status_test.py`, which includes some basic integration tests for `hg status`
in Eden. Although our existing `.cpp` tests give us a lot of coverage, they
do not verify the logic in the Python code for our Mercurial extension,
so this will help me sleep better at night.
Note that `HgExtensionTestBase` is a subclass of `testcase.EdenHgTest`
rather than using the `testcase.eden_repo_test` decorator because these
tests will only be used in the Mercurial case for now, so there is no point in
trying to run them when creating the mount from a Git repo, as well.
Further note that this also required a small change to `eden/fs/integration/lib/testcase.py`
because the Eden directory was not being created at `$HOME/local/.eden`, which
is something that the Mercurial extension relies on.
Reviewed By: simpkins
Differential Revision: D4337660
fbshipit-source-id: ea825134420ae6eb794d37b6c75da19a6aadaae5
Summary:
This preserves the existing behavior while making `REPO_ROOT` and `BUCK_OUT`
available to other test code.
Reviewed By: simpkins
Differential Revision: D4358875
fbshipit-source-id: a7b2cafe7e56673831beac59a85fcab436e637a5
Summary: This is much less hacky than what we had before.
Reviewed By: simpkins
Differential Revision: D4359650
fbshipit-source-id: 52fac899a98828c0a3cc731f276327d3ae171301
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.
This revision was initially created by running:
```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```
And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.
But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.
Reviewed By: simpkins
Differential Revision: D4356724
fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
Summary:
Specialize folly::FormatValue for the various path types, so that they can be
used with folly::format().
Reviewed By: bolinfest
Differential Revision: D4343924
fbshipit-source-id: 21165dfbad73612d1b786591c78e47dc26e6cc66
Summary:
Add an EDEN_BUG() macro to use in situations that can only occur if there is a
logic bug in the code.
In some ways this is similar to LOG(DFATAL) except that:
- It throws an exception in non-debug builds, instead of just logging. It
provides APIs to return a folly::exception_wrapper rather than throwing.
- We can extend it in the future to log bug info to a scuba dataset.
Reviewed By: bolinfest
Differential Revision: D4343911
fbshipit-source-id: 11f030a4f594f9997455cec9235a0151362a472f
Summary:
Rename the existing TestBackingStore class to FakeBackingStore, and fill it out
with an implementation that allows test code to control the store.
The test code can populate the store with Trees and Blobs to return, and can
control when the Futures returned by the store are fulfilled.
Reviewed By: bolinfest
Differential Revision: D4338577
fbshipit-source-id: 79221b04d844bd6011078b799e55182de4ccdfdc
Summary:
Upstream mercurial commit d83ca854fa21bef17bf3f097087c691c7d07bb5b changed the
mercurial.ui API, breaking existing scripts using it.
ui objects now have to be created using the ui.load() classmethod, rather than
by constructing them directly. (Creating a ui object directly no longer loads
the global or user configs.)
Reviewed By: bolinfest
Differential Revision: D4354629
fbshipit-source-id: 650db3abab9f2eebaefa2b4d8029c169ccc30217
Summary:
This is a first diff to remove the "easiest" unused includes in fbcode.
* For safety, we only touch .cpp files without #if and #define,
* We do not try to remove redundant systems headers (aka. "packing").
The diff was generated as follows:
```
foundation/scripts/ls-cpp-dirs | grep -v '^\(\.\.\|external/\|.*/external\)' | xargs ffmr -o /tmp/ffmr-diff-1 codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
cat /tmp/ffmr-diff-1/*.diff | patch -p2
hg commit -m something
arc diff --prepare --nolint --nounit --less-context --excuse refactoring
```
Note: `grep -v` is just an optimization. The actual configuration is in these two files:
diffusion/FBS/browse/master/fbcode/codegraph/analysis/config.py
diffusion/FBS/browse/master/fbcode/codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
See the task for more context, and the recent "safety" improvements on the tool.
depends on D4317825 for very few cases where `nolint` had to be manually added.
Reviewed By: igorsugak
Differential Revision: D4312617
fbshipit-source-id: ecc1f0addfd0651fa4770fcc43cd1314661a311a
Summary:
Most of the work for this diff was achieved via:
```
find eden -type f | xargs sed -i -e 's#ThriftHgStatusCode#StatusCode#g'
find eden -type f | xargs sed -i -e 's#HgStatusCode#StatusCode#g'
```
Reviewed By: simpkins
Differential Revision: D4237975
fbshipit-source-id: 2ee04a89101291c8972ac7bd3ff6cca92cbb0799
Summary:
Add a helper function to create human-readable hashes in test code without
having to always specify a full 40-byte long string.
Reviewed By: wez
Differential Revision: D4336161
fbshipit-source-id: cf6af58dd788b5553a2a6daef56db43cddbbc04a
Summary:
Some of the eden/utils test files were already in eden/utils/test, but some
were just in eden/utils. This moves all test code into the test/ subdirectory.
Reviewed By: bolinfest
Differential Revision: D4335334
fbshipit-source-id: 50ef610f5557ed3ec81e5569e5a96c96e6cf15a5
Summary:
EXPECT_THROW_ERRNO() makes sure a std::system_exception is thrown with the
expected errno value.
EXPECT_THROW_RE() is similar to EXPECT_THROW() but also checks that the
exception message matches the specified regular expression pattern.
Reviewed By: wez
Differential Revision: D4335178
fbshipit-source-id: bd4a894325f7e644b14d5e834cbb88e4fe000d34
Summary:
This should make the common case of `hg add` or `hg add .` much more efficient
because it no longer performs a walk of the entire repository from the client
side.
Reviewed By: simpkins
Differential Revision: D4317871
fbshipit-source-id: 7061553fe0de0c4afa84b4f835316965088675e8
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.
This updates the main program to process path arguments using canonicalPath().
Reviewed By: bolinfest
Differential Revision: D4332273
fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
Summary:
Add a canonicalPath() function to accept a non-normalized input string, and
return a normalized AbsolutePath object.
Also added facebook::eden::getcwd() and facebook::eden::realpath() functions,
which are simply wrappers around the standard versions of these functions, but
return AbsolutePath objects. (Let me know if you prefer renaming these to
something else rather than shadowing the names of the same functions in the
global namespace.)
Reviewed By: bolinfest
Differential Revision: D4332266
fbshipit-source-id: cf89eca16f1f0be2acfce3cb8b3c763312f5c4bc
Summary:
This code compiled fine with clang, but gcc apparently requires explicitly
specifying "this->" when referring to members inside a lambda body.
Reviewed By: bolinfest
Differential Revision: D4332357
fbshipit-source-id: 3919dad4fca0b6676731f21781331be73fde8e55
Summary:
The default mode of detection assumes that the test runner par
will be the one from the eden dir.
In the watchman<->eden integration tests this is not the case, so we need
to provide a way to pass down the correct location.
The watchman tests (see D4323716) are using a TARGETS fragment like this:
```
eden_env = env.copy()
eden_env.update({
'EDENFS_CLI_PATH': '$(location @/eden/fs/cli:cli)',
'EDENFS_SERVER_PATH': '$(location @/eden/fs/service:edenfs)',
})
python_unittest(...
env=eden_env,
...)
```
We may want to cut eden over to a similar construction, but I've left
that alone for the moment.
Reviewed By: simpkins
Differential Revision: D4323695
fbshipit-source-id: f48afa6eef82a806c7b10e187d41e6253039d612
Summary:
Fix Dispatcher::symlink() to accept the symlink contents as a StringPiece
rather than a PathComponentPiece. symlink contents can be any arbitrary
string, and are not required to be a valid, normalized path name.
Reviewed By: wez
Differential Revision: D4325380
fbshipit-source-id: 88448bee50ea192c06442dc70042c7d17d49a12f
Summary:
Update EdenServiceHandler::getSHA1ForPath() to replace its own custom path
lookup code with EdenMount::getFileInode().
This also ended up fixing the error message to correctly include the path name
on EISDIR errors.
Reviewed By: wez
Differential Revision: D4325066
fbshipit-source-id: 9aa3932c71c33e6bc11d2c71cc8f1badb4c0dcb7
Summary:
InodeError is a subclass of std::system_error that accepts an InodePtr to the
inode that it refers to. This makes it easier to construct error objects that
retain information about the inode they refer to.
InodeError also avoids computing the inode path until the error message is
actually needed. This should make it less expensive in cases where errors are
thrown and handled internally without ever using the human-readable error
message. It is possible that the file may have been renamed or unlinked by the
time the error message is computed. However, this race condition might still
exist even if we computed the path at the time when the error is constructed.
getLogPath() will construct a usable human-readable string even if the file has
been unlinked.
Reviewed By: wez
Differential Revision: D4325043
fbshipit-source-id: c9683a80b022f281ca4583a9b7f73b15277335bb
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.
For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready. I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.
Reviewed By: bolinfest
Differential Revision: D4318055
fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
Summary:
This will make it easier to implement `hg add <directory>` such that
`<directory>` is expanded on the server rather than on the client.
Reviewed By: simpkins
Differential Revision: D4318735
fbshipit-source-id: cf0e89bd95eb58304cd23e70beb77bc7151f2c5c
Summary:
Previously, `.hg` entries were filtered when calling `hg status` on the client,
but this should be happening on the server. This updates the logic in
`Dirstate.cpp` to properly exclude `.hg` when traversing the overlay for
modified directories, so this will eliminate a bunch of unnecessary computation
and simplify the client.
I'm unsure of how best to implement the ownership relation for the set that
contains `.hg`. Please advise! I know that I could categorically exclude
`".hg"` in `getModifiedDirectoriesRecursive()`, but I haven't used it enough
scenarios yet to be sure that's the right thing to do. For example, if it were
a Git repo, arguably we should consider `.hg` and not `.git`, so I could also
require the set to be a parameter of `Dirstate`, but I want to make sure I get
the ownership stuff right.
Reviewed By: simpkins
Differential Revision: D4316531
fbshipit-source-id: a0f13ca1c3c620b686435c8aa6485ba4e850f043
Summary: These should have been added as part of D4270526.
Reviewed By: simpkins
Differential Revision: D4315382
fbshipit-source-id: 5920ff38f9cc63540e4813e8ab40f79ad46f9ec1
Summary: Follow-up on a comment that came out of the review for D4249214.
Reviewed By: simpkins
Differential Revision: D4314979
fbshipit-source-id: 76384474092e6fd48394f6faf8b84ba6220c556a
Summary:
Move the FakeObjectStoreTest class into fs/testharness, along with the
TestMount and TestBackingStore classes. This simply consolidates the test
utility code into a single location.
Reviewed By: bolinfest
Differential Revision: D4317517
fbshipit-source-id: 4e19590c5ffde88b66f2c8d4a964352ec349031c
Summary:
Hash objects are small enough (20 bytes) that it isn't worth allocating them on
the heap. This updates LocalStore::getSha1ForBlob() to return a
folly::Optional<Hash>, and ObjectStore::getSha1ForBlob() to return a plain
Hash.
Reviewed By: bolinfest
Differential Revision: D4298162
fbshipit-source-id: 9cf54f2997ba8c3b2346db315a2aca41e580b078
Summary:
Add comments in ObjectStore.h documenting the fact that the get* APIs all throw
std::domain_error when the specified ID does not exist, and never return
nullptr.
Also update the FakeObjectStore class used for testing to follow this behavior.
Reviewed By: bolinfest
Differential Revision: D4298160
fbshipit-source-id: c5509bb3aa2ed76619b06b733ad240aaa5f00862
Summary:
In addition to storing the SHA-1 of each file's contents, also store the size.
This will allow us to more quickly look up the file size, without having to
retreive the file size.
I haven't yet added an API to ObjectStore to retreive the full BlobMetadata
object; I will do that in a subsequent diff. One benefit for now is that this
does avoid double-computing the SHA-1 in ObjectStore::getSha1ForBlob() if we
had to load the blob.
Reviewed By: bolinfest
Differential Revision: D4298157
fbshipit-source-id: 4d83ebfa631c93fcef06ca1cd0ba0e1a70a2476d
Summary:
Define InodePtr, TreeInodePtr, and FileInodePtr as aliases for std::shared_ptr
of the underlying inode type. This also updates all of the code to use these
new type names.
This will make it easier swap out std::shared_ptr with a custom pointer type in
the future. (I believe we will need a custom type in the future so that we
can have more precise control of the reference counting so we can load and
unload Inode objects on demand. std::shared_ptr::unique() doesn't quite
provide the flexibility we need, and is also being deprecated in C++17.)
Reviewed By: bolinfest
Differential Revision: D4297791
fbshipit-source-id: 1080945649290e676f62689592159f1166159b20
Summary:
This updates the InodeBase code to track its location in the filesystem.
Since we do not support hard links, each inode has a single path where it
exists.
Tracking this data allows us to implement getPath() as a method of InodeBase.
This code is not really complete yet, but it seems worth getting the current
code in as-is. The location data is not updated properly on unlinks or
renames, but it looks like the existing InodeNameManager code does not get
updated either. I am working on some additional refactoring of inode object
management, and it will be easier to come back and fix the unlink and rename
handling after this refactoring is further along.
Reviewed By: bolinfest
Differential Revision: D4297591
fbshipit-source-id: 82ceb326e4f9c376f627b1d8f49bb7db3cfc2b0b
Summary:
In this revision, we override `committablectx.markcommitted()` to make a Thrift
call to Eden to record the new commit. For now, this defers the need for us to
implement `edendirstate.normal()`, though we expect we will have to provide a
proper implementation at some point.
Because `hg update` is not implemented yet, this puts us in a funny state where
we have to restart eden after `hg commit` to ensure all of our `TreeEntry` and
other in-memory data structures are in the correct state.
Reviewed By: simpkins
Differential Revision: D4249214
fbshipit-source-id: 8ec06dfee67070f008dd93a0ee6c810ce75d2faa
Summary:
This refines the initial support for `hg remove` by adding support for
directories.
Reviewed By: simpkins
Differential Revision: D4270546
fbshipit-source-id: c97dfea555ad489ddda01ad2587f1856b1953e02
Summary:
`getEntryForPath()` is helpful when we don't know (or care) whether the path
corresponds to a file or a directory.
Reviewed By: simpkins
Differential Revision: D4270526
fbshipit-source-id: 6dd2f76df1749e040dda788a74055d9da2156a4d
Summary:
This adds support for `hg remove <file>` in Eden and sets up some of the scaffolding
to support `hg remove <directory>`.
Note that the Thrift API for `scmRemove()` is slightly different than that of `scmAdd()`
in that it returns a list of error messages to display to the user rather than throwing
an exception. In practice, for batch operations, Mercurial will allow some operations
to succeed while others may fail, so it is possible to have multiple error messages to
return.
Unlike the current implementation of `hg add`, this does the directory traversal
on the server rather than on the client. Once we work out how to do this for
`hg remove`, we should figure out how to reuse the logic for `hg add`.
Reviewed By: simpkins
Differential Revision: D4263068
fbshipit-source-id: d084774d562c48c59664f313eba229d4197929fe
Summary:
The check on the type of an exception was inverted, which meant
`hg remove <path>` would throw an exception if the parent directory of `path`
did not exist. This is not correct because the user should be able to expect
to do:
```
mkdir -p /tmp/example
cd /tmp/example
hg init
mkdir mydir
touch mydir/a
hg add mydir/a
rm -rf mydir/a
hg rm mydir/a
```
In this scenario, `mydir` does not exist when `hg rm` is called, but the command
should succeed, making `mydir/a` no longer tracked.
Reviewed By: simpkins
Differential Revision: D4268451
fbshipit-source-id: 517d81252aa8e4b6bd1a32dece14776a9f7dd6f7
Summary:
A `TreeEntry` reports a `mode_t` whose "group" and "other" bits are set in a way
that reflects the "owner" bits (so they are non-zero). By comparison, the `mode`
of a `TreeInode::Entry` will reflect the permissions on disk in the overlay (if
the file is materialized). In general, the overlay bits will probably be the
same as those in the `TreeEntry` since we expect the user will rarely mess
with the "group" and "other" bits, but we're seeing a difference more often
right now because of t14953681.
Reviewed By: simpkins
Differential Revision: D4298214
fbshipit-source-id: 2919be94c6bba61135838ee86bbc68aa4031af7c
Summary:
Move the InodeBase class from the lower-level fusell code up to the
eden/fs/inodes layer, now that everything else that uses it is in
eden/fs/inodes.
I plan to start changing the ownership model of inode objects a bit, and this
will allow the InodeBase class to interact with EdenDispatcher and other
classes in eden/fs/inodes.
Reviewed By: bolinfest
Differential Revision: D4283392
fbshipit-source-id: 9e1d6fb81dc223f905847cbe8d165a40ad0aca4d
Summary:
`stdin`, `stdout` and `stderr` are macros that expand to function calls with the MSVC CRT implementation. This is also the case for musl-libc. This means that Subprocess simply cannot be compiled on those platforms without changing the API.
To solve that, we change the API and deprecate the old API.
For more fun, `stdin`, `stdout` and `stderr` are also macros in glibc, they just expand to other identifiers rather than a function call.
Reviewed By: yfeldblum
Differential Revision: D4229544
fbshipit-source-id: 97f1a3b228b83cfdcaffee56d729063ea235e608
Summary:
Update the TestMount APIs to use the term "TreeInode" instead of "DirInode" now
that the DirInode class is no more.
Reviewed By: bolinfest
Differential Revision: D4257171
fbshipit-source-id: 2407dfc25405f7161987260c0299f1723831e264
Summary:
Now that the EdenDispatcher class has been moved into eden/fs, we no longer
need the distinction between TreeInode and fusell::DirInode, and FileInode and
fusell::FileInode.
This diff deletes the fusell versions of these classes, and updates all of the
code to always directly use TreeInode and FileInode. This allows us to get rid
of the remaining dynamic_casts between these pairs of classes.
Reviewed By: bolinfest
Differential Revision: D4257165
fbshipit-source-id: e2b6f328b9605ca0e2882f5cf7a3983fb4470cdf
Summary:
Move the InodeDispatcher class out of the lower-level fusell namespace in
eden/fuse and into the higher-level eden code in eden/fs/inodes. I also
renamed it from InodeDispatcher to EdenDispatcher, in anticipation of it
getting more eden-specific functionality in the future.
The fusell::MountPoint class is now independent of the Dispatcher type, and can
work with any Dispatcher subclass. Previously the MountPoint class was
responsible for owning the InodeDispatcher object. Now its caller (EdenMount
in our case) is responsible for supplying a Dispatcher object that is owned
externally.
Several parts of EdenDispatcher had to be updated as a result of the namespace
move, but I tried to keep this change somewhat minimal. I did update it from
using fusell::DirInode and fusell::FileInode to eden's TreeInode and FileInode
classes directly. However, there still remains more clean-up work to do. I
will split remaining changes out into upcoming diffs.
Reviewed By: bolinfest
Differential Revision: D4257163
fbshipit-source-id: dc9c2526640798f9f924ae2531218ba2c45d1d0a
Summary:
Drop the MountPoint::setRootInode() method, and have EdenMount perform the
operation directly on the InodeDispatcher.
Reviewed By: bolinfest
Differential Revision: D4257158
fbshipit-source-id: af4a696de2d36979c658972104361f225f482338
Summary:
Update call sites in eden/fs to access the InodeNameManager through the
EdenMount object rather than the MountPoint.
It turns out that there was only one call site in TreeInode, and all other
callers in eden/fs get it indirectly via TreeInode::getNameMgr().
Reviewed By: bolinfest
Differential Revision: D4257156
fbshipit-source-id: 9f0212134b20c8dd8943827c17aa16ee7274bc36
Summary:
Move getInodeBaseForPath(), getDirInodeForPath(), and getFileInodeForPath()
entirely into the EdenMount class, and make sure all call sites are using the
EdenMount methods rather than the MountPoint methods.
Reviewed By: bolinfest
Differential Revision: D4257153
fbshipit-source-id: d528cfad174757c3c9f23e62a0616f8bf1976da7
Summary:
Update all code in eden/fs to call EdenMount::getDispatcher() instead of
getting the underlying MountPoint from the EdenMount and then calling
getDispatcher() on it. This will allow me to move the InodeDispatcher from
MountPoint to EdenMount in a subsequent diff. This also simplifies many of the
callers of this method.
Additionally, add an EdenMount::getRootInode() method, and update call sites to
use this rather than having to look up the InodeDispatcher and call
getRootInode() or getDirInode(FUSE_ROOT_ID) on it.
Reviewed By: bolinfest
Differential Revision: D4257152
fbshipit-source-id: 33e6f6b8853db2a88f4f2c221122eea50e796390
Summary:
This updates the Dirstate to also check if untracked files are ignored or not.
This is somewhat inefficient, as we have perform a separate check for each
untracked file we find. Ideally we should perform all of the dirstate
computation in a single tree walk, and check ignore status as we go. This
would allow us to skip ignored directories entirely, rather than potentially
having to check each file inside them. I intend to work on cleaning this up in
the future, but it will require refactoring some of the inode code first.
Reviewed By: bolinfest
Differential Revision: D4225308
fbshipit-source-id: 49e444c85cbb6ede11cc6e19052fdd16cf8aab9f
Summary:
Update the Dirstate to store a pointer to the EdenMount object that owns it,
rather than storing pointers to the lower-level MountPoint and ObjectStore
objects.
This change is necessary in order for me to move more functionality from
MountPoint to EdenMount. (In particular, I plan to move the InodeDispatcher to
the EdenMount.)
As part of this change I also started moving some APIs from MountPoint to
EdenMount. For now the EdenMount versions are just thin wrappers on top of the
MountPoint APIs. I will move the functionality directly into EdenMount in a
future diff.
Reviewed By: bolinfest
Differential Revision: D4255675
fbshipit-source-id: 93749c6516c3cea4b4ae93de4ca49ddf05f4d260
Summary:
Write the dirstate data using the new folly::writeFileAtomic() function.
This ensures that the dirstate file will always contain full, valid contents,
even if we crash or run out of disk space partway through writing out the data.
This diff also includes a couple other minor tweaks:
- Update Dirstate to store the DirstatePersistence object directly inline,
rather than allocating it separately on the heap.
- Update the DirstatePersistenceTest code to prefix temporary directories with
"eden_test". This just makes it easier to tell if the tests fail to clean up
after themselves for any reason.
Reviewed By: bolinfest
Differential Revision: D4254027
fbshipit-source-id: 6b6601b65aeacdee998a6c4260e972d5fb2426ac
Summary:
This cleans up construction of the EdenMount and Dirstate objects:
- The EdenMount constructor is now responsible for creating the Overlay and
Dirstate objects.
- The Dirstate constructor is now responsible for loading the
DirstatePersistence file.
- The EdenMount now takes ownership of the ClientConfig object, and stores it
for later use.
- The ClientConfig object now has a method to get the path to the
DirstatePersistence file.
- I added a ClientConfig::createTestConfig() method, so that the TestMount code
can now use the same EdenMount constructor as the normal code.
This simplifies the logic in EdenServiceHandler and TestMount, and makes some
of the initialization dependencies a little bit simpler.
This change is necessary in order for me to move some logic from
fusell::MountPoint into EdenMount. The Dirstate object will need a pointer
back to its EdenMount object, and this diff enables that.
Reviewed By: bolinfest
Differential Revision: D4249393
fbshipit-source-id: 439786accbf48c8696dbc6ca4fe77a4c6bdeab65
Summary:
Rename TreeEntryFileInode to FileInode, and TreeEntryFileHandle to FileHandle.
These class names were long and awkward.
It's slightly unfortunate that we now have classes named both
eden::fuse::FileInode and eden::fuse::fusell::FileInode, but I don't believe
this should cause any major problems. If we want to eliminate these name
collisions in the future I would advocate for renaming the fusell versions to
something like "FileInodeIface".
Reviewed By: bolinfest
Differential Revision: D4217909
fbshipit-source-id: 899672a318d7ae39595f2c18e171f8fd6cebedc6
Summary:
It is important to make the local of the Hg Eden extension configurable so we
can develop it.
Reviewed By: simpkins
Differential Revision: D4223733
fbshipit-source-id: 1746d0cda590f24fc48e0d781b557824ff5f60d3
Summary:
Previous to this commit, the `Dirstate` logic only worked correctly when the
changes occurred in the root directory. Obviously that is very limiting, so this
adds support for changes in arbitrary directories at arbitrary depths.
This also introduces support for things like a file being replaced by a
directory of same name or vice versa. The tests have been updated to verify
these cases.
One interesting design change that fell out of this was the addition of the
`removedDirectories` field to the `DirectoryDelta` struct. As you can see,
all entries in a removed directory need to be processed by the new
`addDeletedEntries()` function. These require special handling because deleted
directories do not show up in the traversal of modified directories.
In contrast, new directories do show up in the traversal, so they require a
different type of special handling. Specifically, this call will return `NULL`:
```
auto tree = getTreeForDirectory(directory, rootTree.get(), objectStore);
```
When this happens, we must pass an empty vector of tree entries to
`computeDelta()` rather than `&tree->getTreeEntries()`. Admittedly, the special
case of new directories is much simpler than the special case of deleted ones.
Reviewed By: simpkins
Differential Revision: D4219478
fbshipit-source-id: 4c805ba3d7688c4d12ab2ced003a7f5c19ca07eb
Summary:
This is a better fix for the quick fix introduced by D4198939.
It turns out that the `EdenMount` does not need to take ownership
of the `ClientConfig`, so removing the `std::move()` makes this code
much simpler because instead of declaring a bunch of variables
early in `mountImpl()` so that we can "hold on" to them before `EdenMount`
takes ownership of the `ClientConfig`, we can declare them closer to where they
are actually used.
Note that we may want `EdenMount` to actually take ownership of the
`ClientConfig` in the future, but we'll cross that bridge when we come to it.
Reviewed By: simpkins
Differential Revision: D4199000
fbshipit-source-id: 67411a9a5ef630a9d481aebc94631c79da4ab2c4
Summary:
This also introduces the change where the `EdenMount` creates
and takes ownership of the `Dirstate`.
To clean some of this up, I had to expose a `getEdenDir()` method on `EdenServer`
that returns an `AbsolutePathPiece`. This was previously stored internally as a
`std::string`, so I had to clean up a bunch of path construction that was using `edenDir_`.
Reviewed By: simpkins
Differential Revision: D4123763
fbshipit-source-id: 270b182521c1a84bb054832f4b5f92af849d67e4
Summary:
Previously, `Dirstate` took a `std::shared_ptr<EdenMount>`, but now it takes
pointers to a `MountPoint` and an `ObjectStore` because it does not need the
entire `EdenMount`. Ultimately, this will enable us to have `EdenMount` create
the `Dirstate` itself, but that will be done in a follow-up commit.
Fortunately, it was pretty easy to remove the references to `edenMount_` in
`Dirstate.cpp` and rewrite them in terms of `mountPoint_` or `objectStore_`.
The one thing that I also decided to move was `getModifiedDirectoriesForMount()`
because I already needed to create an `EdenMounts` file (admittedly not a
great name) to collect some utility functions that use members of an `EdenMount`
while not having access to the `EdenMount` itself.
As part of this change, all of the code in `eden/fs/model/hg` has been moved to
`eden/fs/inodes` so that it is alongside `EdenMount`. We are going to change
the `Dirstate` from an Hg-specific concept to a more general concept.
`LocalDirstatePersistence` is no longer one of two implementations of
`DirstatePersistence`. (The other was `FakeDirstatePersistence`.) Now there is
just one concrete implementation called `DirstatePersistence` that takes its
implementation from `LocalDirstatePersistence`. Because there is no longer a
`FakeDirstatePersistence`, `TestMount` must create a `DirstatePersistence` that
uses a `TemporaryFile`.
Because `TestMount` now takes responsibility for creating the `Dirstate`, it
must also give callers the ability to specify the user directives. To that end,
`TestMountBuilder` got an `addUserDirectives()` method while `TestMount` got a
`getDirstate()` method. Surprisingly, `TestMountTest` did not need to be updated
as part of this revision, but `DirstateTest` needed quite a few updates
(which were generally mechanical).
Reviewed By: simpkins
Differential Revision: D4230154
fbshipit-source-id: 9b8cb52b45ef5d75bc8f5e62a58fcd1cddc32bfa
Summary: This is an implementation of DirstatePersistence that persists data to a local file.
Reviewed By: simpkins
Differential Revision: D4181868
fbshipit-source-id: 7177b2ef67cd3aec56e5ad10f41169cc5ec69d81
Summary:
Update the gitignore handling code to perform pattern matching the same way git
does. Previously the code just called the standard fnmatch() function, which
does not handle "**" in patterns the same way git does.
This includes our own new implementation of glob pattern matching. I did
evaluate several other options before writing our own implementation here:
- The wildmatch() code used by git (and watchman, and rsync) has a few
downsides: it is not distributed by itself as a library anywhere else.
Therefore we would probably have to include a copy of this code in our
repository. Making another copy is unfortunate, and somewhat undesirable
from a legal and licensing perspective. This code also only works with
nul-terminated strings, and our code deals primarily with non-terminated
StringPiece objects.
- I did look at translating glob patterns in to regular expressions and using
re2 to perform matching. Unfortunately re2 turns out to be substantially
slower than wildmatch() for typical gitignore patterns.
This new implementation performs some preprocessing on the glob pattern, and
generates a pattern opcode buffer. Eden can perform this glob preprocessing
when it first loads a .gitignore file, and can then save and re-use this result
each time it needs to match a filename. Doing this preprocessing allows
matching to be done 50% to 100% faster than wildmatch() for typical glob
patterns.
Reviewed By: bolinfest
Differential Revision: D4194573
fbshipit-source-id: 46bc6a61b6d8066f4bbdb5d3e74265a3e72e42cc
Summary:
This adds some initial code for handling gitignore files.
I did check to see if there were APIs from libgit2 that we could leverage for
this, but it does not look like we can easily use their functionality. The
libgit2 ignore code seems to tightly coupled with their repository data
structures, and it requires that you actually have a git repository.
This code isn't quite 100% compatible with git's semantics yet. In particular:
- For now we are just using fnmatch() to do the matching. This is currently
inefficient as we have to do string allocations on each match attempt. This
also doesn't quite match git's behavior, particularly with regard to "**"
inside patterns.
- The code currently does not have a mechanism for indicating if a path refers
to a directory or not, so trailing slashes in the pattern are not honored
correctly.
We will probably need to implement our own fnmatch-like function in the future
to solve these issues.
Reviewed By: bolinfest
Differential Revision: D4156480
fbshipit-source-id: 8ceaefd3805358ae2edc29bfc316e5c8f2fb7d31
Summary:
For now, initial state is represented by a
`std::unordered_map<RelativePath, HgUserStatusDirective>`.
Reviewed By: simpkins
Differential Revision: D4123461
fbshipit-source-id: 83a99e1f504dd1efca1bc1ed33cbc3f116787a80
Summary:
This adds the logic to power `hg rm`. There are comprehensive tests that attempt to cover
all of the edge cases.
This evolved to become a complex change because I realized that I needed to change
my internal representation of the dirstate to implement it properly. Specifically, we now maintain
a map (`userDirectives`) of files that have been explicitly scheduled for change via `hg add` or `hg rm`.
To compute the result of `hg status`, we find the changes between the manifest/root tree
and the overlay and also consult `userDirectives`. `Dirstate::getStatus()` was updated
considerably as part of this commit due to the introduction of `userDirectives`.
As such, `Dirstate::remove()` must do several things:
* Defend the integrity of the dirstate by throwing appropriate exceptions for invalid inputs.
* Delete the specified file, if appropriate.
* Update `userDirectives`, if appropriate.
Although `Dirstate::add()` was not the focus of this commit, it also had to be updated to
match the pattern introduced by `Dirstate::remove()`.
Some important features that are still not supported are:
* Handling ignored files correctly.
* Storing copy/move information.
Reviewed By: simpkins
Differential Revision: D4104503
fbshipit-source-id: d5d45a279e16ded584c6cd4d528ba92d2c8e2993
Summary:
D4014598 changed this line but didn't change the test expectations.
Since it seems desirable for the mount point name to be used, I've reverted
back to the prior state for this line.
Reviewed By: bolinfest
Differential Revision: D4202265
fbshipit-source-id: bddc01436e0a5921a3b0b2c01c0fd2c32f5f1960
Summary:
Originally, D3858635 was going to introduce a scheme for hooks where the
repo type was included in the path:
/etc/eden/hooks/hg/post-clone <args...>
But over the course of the review, we decided to make the repo type a
parameter:
/etc/eden/hooks/post-clone hg <other-args...>
Unfortunately, `generate-hooks-dir` was not updated as part of that
change and it is not covered by unit tests. This error was particularly hard
to discover because of how `ENOENT` is handled, so I added a log statement for
that.
Reviewed By: simpkins
Differential Revision: D4200277
fbshipit-source-id: ffffd871cd78dcaeb717be8f1e01893ce9643a47
Summary:
This is a quick and dirty fix for this issue that was causing
and confusing bug where the memory for the `AbsolutePathPiece`
was getting reclaimed, so when it was later read as the value
for a path, it failed because it was binary garbage.
This is mainly caused by the `std::move(config)` that passes
the `ClientConfig` to the `EdenMount` constructor. I will do
some more general cleanup for that in a follow-up revision,
but I wanted to have this change in its own commit that makes
it clear where the failure/fix were coming from.
Reviewed By: simpkins
Differential Revision: D4198939
fbshipit-source-id: 19e0423a1bee924fa6cc2edc8bae534ef472c988
Summary:
Use new, less confusing names for mentioned thrift methods.
Codemod with 'Yes to All'. Reverted changes in thrift/
Reviewed By: yfeldblum
Differential Revision: D4076812
fbshipit-source-id: 4962ec0aead1f6a45efc1ac7fc2778f39c72e1d0
Summary:
This is the start of the C++ dirstate implementation. It's possible that this
commit does too many things at once:
* Introduces `Dirstate` type.
* Includes logic for serializing/deserializing the dirstate's data so that it persists across Eden restarts.
* Includes logic for basic `hg add` calls.
* Includes unit tests where we model Eden usage via the TestMount utility.
I'm backing this up in Phabricator with `--plan-changes` to start until I get
some basic `hg add` functionality working end-to-end. When that looks good, I'll
determine if/how this should be split into smaller commits.
Reviewed By: wez
Differential Revision: D4023232
fbshipit-source-id: 7fc931d547ccadb34f7caae93bc4eb8f91f6ceb8
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
Reviewed By: simpkins
Differential Revision: D4073653
fbshipit-source-id: dda1d8ea8d29aa071a31f8e2afab324f9109e9b2
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
As you can see, this helped uncover a bug in the way we were
inserting blobs into `LocalStore`.
Reviewed By: simpkins
Differential Revision: D4073039
fbshipit-source-id: 42683fd0bfdb0a1e77df9324fcaa79091f45e83d
Summary: This is a follow-up revision from a comment on D4013464.
Reviewed By: wez
Differential Revision: D4050278
fbshipit-source-id: 1e46526f58a07e1eedd8ace1a6d84a919240d899
Summary:
This is analogous to the existing `getEntryForFile()` helper function that we
have, and I was able to rewrite `getEntryForFile()` in terms of
`getTreeForDirectory()`, which simplifies the code considerably.
Also moved things from `eden/fs/model/hg/misc.h` to
`eden/fs/store/ObjectStores.h`, which is much more appropriate.
Reviewed By: wez
Differential Revision: D4032817
fbshipit-source-id: ff4d32120fb050f8b5c5c53b7f2e94b524781648
Summary:
This is not a one-liner and this is needed for the upcoming `Dirstate` class,
so moving this code to a place where it is more easily reusable.
Reviewed By: simpkins
Differential Revision: D4032001
fbshipit-source-id: 7d8d87802665ac2993ec0a3ac73c5f645fe4a1aa
Summary:
Performs a depth-first traversal of the overlay to find modified
directories and returns them in that order.
Reviewed By: simpkins
Differential Revision: D4025309
fbshipit-source-id: 09d8ed41b250dddbfb3fe545643ec3fd755a430e
Summary:
Now that I've done all this work, I'm not sure whether it is a good idea or even
necessary. I'll keep it in my back pocket.
Reviewed By: simpkins
Differential Revision: D4014598
fbshipit-source-id: 6ded3cc29838e964b56833ac24dff19e9de040f5
Summary:
These are new helper methods we need to create test scenarios.
They will be used in upcoming revisions.
Reviewed By: wez
Differential Revision: D4046981
fbshipit-source-id: 9c66c456be57006173e4a65eed603de4a426a438
Summary: This should be useful for my upcoming unit tests for the Hg dirstate.
Reviewed By: simpkins
Differential Revision: D4013464
fbshipit-source-id: 46460186abfa104aa026894068cd160e52c94729
Summary: This will make it easier to compare a `TreeEntry` with a `TreeInode::Entry`.
Reviewed By: simpkins
Differential Revision: D4034298
fbshipit-source-id: 29674e2902661bf46394ea71b81537b35bd4b107
Summary: This should make some of the upcoming test harness work a little easier.
Reviewed By: simpkins
Differential Revision: D4011747
fbshipit-source-id: 87ee80a6d641a29be9027b163b1adee496f4452f
Summary:
I need this for the upcoming test harness so I can avoid creating a
`ClientConfig`, which is currently a huge pain to do from a unit test.
Reviewed By: simpkins
Differential Revision: D4010842
fbshipit-source-id: 03d1e1de9c3047340a6f26202d4b432f4a8620b4
Summary:
This was reported by ASAN.
The major issue was that `FakeObjectStore` was returning a copy of a `Tree`,
so it was not the case that the `TreeEntry*` returned by `getEntryForFile()`
was guaranteed to be "owned by" the `Tree* root` that was passed in. To address
this, we change `getEntryForFile()` to now return a copy of the `TreeEntry*`
that it gets back from `getEntryPtr()`. It really comes down to this line:
```
auto entry = currentDirectory->getEntryPtr(piece.basename());
```
because we cannot guarantee that `currentDirectory` will live past the end of
`getEntryForFile()`, so we cannot guarantee that return return value of
`currentDirectory->getEntryPtr()` will, either.
Special thanks to meyering and yfeldblum for helping me debug this.
Reviewed By: simpkins
Differential Revision: D4024627
fbshipit-source-id: 6295e6f2b1d2f544271b2aebad27a4ad3ae04563
Summary:
Utility function that given a `Tree` and a `RelativePathPiece`, returns the
corresponding `TreeEntry` in the `ObjectStore`, if it exists.
Reviewed By: wez
Differential Revision: D3980261
fbshipit-source-id: 2808a4ca45be84e3a6bb91b0cf2db19a3bf88798
Summary:
In an upcoming revision, I am going to introduce a utility function that takes
an `ObjectStore` (well, now an `IObjectStore`) as a parameter and I want to be
able to test it. Having a `FakeObjectStore` should make this considerably easier
without having to resort to mocks.
Reviewed By: simpkins
Differential Revision: D3980580
fbshipit-source-id: 5886e2055c893e749cc898226e1baade776c3ea7
Summary:
Adds a very basic example of testing eden functionality with hypothesis.
We'll be building on this with stateful testing in a follow on diff tomorrow.
There's some prep/setup work in the base test class that can be removed when an updated version of hypothesis ships and is updated in our third-party repo.
Reviewed By: simpkins
Differential Revision: D3968250
fbshipit-source-id: 46382c3bf2d6a0edbd60ac2b048b1bae26ca2572
Summary: This is necessary so that it can be used as the key in an `unordered_map`.
Reviewed By: simpkins
Differential Revision: D3980575
fbshipit-source-id: d225a98f957f9aae2f2f50a6cc365011d953c92e
Summary:
Apparently we did not have an existing unit test for `Tree`, so this adds one.
The other methods should be tested, as well, but I'm about to use `getEntryPtr()`
elsewhere, which is why I just focused on this one for the moment.
Reviewed By: simpkins
Differential Revision: D3980150
fbshipit-source-id: 33456fd621a1894606605af4fee06ba42d124752
Summary:
We want to use these with Eden
Depends on D3961190
Depends on D3961193
Depends on D3961196
Depends on D3961208
Reviewed By: rhysparry
Differential Revision: D3961232
fbshipit-source-id: 56f5a1811625303514e4398a6d47ea90ba348724
Summary:
The default `casecollisionauditor` reads all of `dirstate._map`. This is
(1) expensive for a large repo and (2) accesses a private property that we would
prefer not to expose via `edendirstate`. Setting `portablefilenames = ignore` by
default avoids this check.
This means that Eden users are currently responsible for not creating directory
entries that would cause a case-insensitive collision. Ideally, we would just do
this check on the server.
Reviewed By: wez
Differential Revision: D3964461
fbshipit-source-id: f351bdeaad0fc06cd70cc637ca1b6fde249dde9c
Summary:
The getMaterializedEntries() would previously try to dereference a null pointer
if the input mount path did not refer to a valid mount piont.
Reviewed By: bolinfest, wez
Differential Revision: D3942600
fbshipit-source-id: 2a8c9aa87d2bd8175f7bc77f3d6293ad25e9c198
Summary:
Add some helper functions for constructing EdenError objects from a few
different types of arguments. Also update eden.thrift to indicate that most
functions can throw EdenErrors on failure.
Reviewed By: bolinfest, wez
Differential Revision: D3942588
fbshipit-source-id: 1b561c5310a8a218f88c38c70499e087fe47bbe0
Summary:
Python 2.x requires the current class name be passed into super().
Add arguments to super so that we can use this inside a mercurial extension.
(Mercurial only supports python 2.x.)
Reviewed By: bolinfest
Differential Revision: D3942573
fbshipit-source-id: 06df55f217631a398004c0d25448d3a612f772e9
Summary:
The keys in the config directory map are normalized, absolute paths to the
mount point. When trying to look up a mount point make sure we also always use
a normalized absolute path.
Reviewed By: bolinfest
Differential Revision: D3942565
fbshipit-source-id: 63db838ffc7139d779925adf07c50f849d73bcc5
Summary:
We were hitting an assertion in the case where we did a `mkdir`
followed by a `rename` followed by `getMaterializedEntries`.
The issue is that our in-memory representation has a boolean to indicate
whether a dir inode is materialized, but our serialization format does
not have this bit. When we loaded the data we were not setting the
field to true and this was caught by the DCHECK.
If we have serialized data for a dir then it is, by definition, materialized
and we should just set that field to true.
Reviewed By: bolinfest
Differential Revision: D3900795
fbshipit-source-id: 62d8281e7a1009056d274888c9aff87664d2e09f
Summary:
This design is inspired by that of Git hooks:
https://git-scm.com/docs/githooks
By default, `/etc/eden/hooks` should be the place where Eden looks for
hooks; however, this can be overridden in `~/.edenrc` on a per-`repository` basis.
This directory should be installed as part of installing Eden.
There is information in `eden/hooks/README.md` about this.
The first hook that is supported is for post-clone logic for a repository.
This change demonstrates the need for an `eden config --get <value>`
analogous to what Git has, as hooks should be able to leverage this in their
own scripts. There introduces a `TODO` in `post-clone.py` where such a
feature would be useful, so that I could add the following to my `~/.edenrc`
to develop the Eden extension for Hg:
```
[hooks]
hg.edenextension = /data/users/mbolin/fbsource/fbcode/eden/hg/eden
[repository fbsource]
path = /data/users/mbolin/fbsource
type = hg
hooks = /data/users/mbolin/eden-hooks
```
Note that this revision also introduces a `generate-hooks-dir` script that can be
used to generate the standard `/etc/eden/hooks` directory that we intend to
distribute with Eden. This is also useful in creating the basis for a custom `hooks`
directory that can be specified as shown above in an `~/.edenrc` file.
Reviewed By: simpkins
Differential Revision: D3858635
fbshipit-source-id: 215ca26379a4b3b0a07d50845fd645b4d9ccf0f2
Summary:
simpkins spotted this; we were passing the wrong path down to the overlay saving dir.
This adds a test to prove that the source and destination directory contents
are correct both immediately after performing the rename and after remounting,
where we just read the serialized data.
Reviewed By: simpkins
Differential Revision: D3888694
fbshipit-source-id: 7f5fb5be417db5c693ac8a07b85abbffdbfe0fff
Summary:
This is pretty straightforward; we just walk back until we hit the
boundary with the requested JournalPosition.sequenceNumber
Reviewed By: simpkins
Differential Revision: D3872970
fbshipit-source-id: 1405f05957346d7ac513070f0407a477548aff1d
Summary:
populate the position from the latest journal delta.
To facilitate this, we also define the mountGeneration value to be a
combination of the pid and the time at which we created the EdenMount object,
as well as a global counter that we bump for each mount.
The precise value and meaning of this bits really doesn't matter, just that we
are unlikely to pick the same value for this same mountPoint path again if we
were to remount in the future.
Since we are now in a position to report JournalPosition values to clients, now
is also a good time to fill out the `currentPosition` field for the
`getMaterializedEntries` thrift call, and to check that this value is
consistent with the value we return via `getCurrentJournalPosition`.
Reviewed By: simpkins
Differential Revision: D3872952
fbshipit-source-id: 2fbc25d2e9711035b66ab1bf5d746507b72de265
Summary:
This just populates the initial snapshot hash in the journal.
The `addDelta` method will propagate this into subsequent deltas if the delta
to be added has hash values that have not been set from the default 0-filled
hash values.
Reviewed By: simpkins
Differential Revision: D3872936
fbshipit-source-id: d0014ded40488a2be04d5a381e1d9815c7f0a638
Summary:
This diff adds a couple more things to our thrift interface:
1. Introduces JournalPosition
2. Adds methods to query the current JournalPosition and obtain a
delta since a given JournalPosition
3. Augments getMaterializedFiles to also return the current JournalPosition
4. Adds a method to evaluate a `glob` against Eden
5. Adds a method using thrift streaming to subscribe to realtime changes
Could probably finesse the naming a little bit.
The JournalPosition allows reasoning about changes to files that are not part
of an Eden snapshot. Internally the journal position is just the
SequenceNumber from the journal datastructures, but when we expose it to
clients we need to be able to distinguish between a sequence number from the
current instance of the eden service and a prior incarnation (eg: if the
process has been restarted, and we have no way to recreate the journal we need
to be able to indicate this to the client if they ask about changes in that
range). For the convenience of the client we also include the `toHash` (the
most recent hash from the journal entry) which is likely useful for the `hg`
dirstate operations; it is useful to know that the snapshot may have changed
since the last query about the dirstate.
The `getFileInformation` method returns the instantaneously available `stat()`
like information about the requested list of files. Since we simply don't
have historical data on how files in the overlay looked (only how they look
now), this method does not allow passing in a JournalPosition. When it comes
to comparing historical data, we will need to add an API that accepts two
snapshot hashes and generates the results from there. This particular method
is geared up to understanding the current state of the world; the obvious use
case is plugging in the file list from `getFilesChangedSince` into this
function to figure out what's what.
* Do we want a function that combines `getFilesChangedSince` + `getFileInformation` into a single RPC?
Why is there a glob method? It's to support a use-case in the watchman/buck
integration. I'm just sketching it out in the thrift interface at this stage.
In the future we also need to be able to express how to carry out a tree walk,
but that will require some query predicates that I don't want to get hung up on
specifying immediately.
Why is the streaming stuff in its own thrift file? We can't generate code for
it in java or perhaps also python. It's only needed to plumb data into
watchman so it's broken out into its own definition. Nothing depends on that
file yet, so it's probably not specified quite right. The important thing is
how the subscribe method looks: it's essentially the same as the method to
query a delta, but it keeps emitting deltas as they are produced. This is
another API that will benefit from query predicates when we get around to
specifying them.
I've added `JournalDelta::fromHash` and `JournalDelta::toHash` to hold the
appropriate snapshot ids in the journal entry; this will allow us to indicate
when we've checked out a new snapshot, or created a new snapshot. We have
no way to populate these yet; I commented on D3762646 about storing the
`snapshotID` that we have during `EdenServiceHandler::mountImpl` into either
the `EdenMount` or the proposed `RootInode` class. Once we have that we
can simply sample it and store it as we generate `JournalDelta`s.
Reviewed By: simpkins
Differential Revision: D3860804
fbshipit-source-id: 896c24c354e6f58328fb45c24b16915d9e937108
Summary:
This is pretty simplistic: we just wlock and add a delta for the set
of file(s) that were changed in a given fuse operation (this is typically 1
file, but rename affects 2).
To reduce boilerplate very slightly, I've added an initializer_list constructor
for JournalDelta that makes it less cumbersome to create a JournalDelta for a
list of files.
Reviewed By: simpkins
Differential Revision: D3866053
fbshipit-source-id: cd918e2c98c022d5ef79430cd8ab4aef88875239
Summary:
This implements a pretty simple change Journal and associated
JournalDelta.
The Journal is intended to be held in memory and not persisted to disk.
The idea is that we'll hold a `Synchronized<Journal>` along with the
other mount data and grab a `wlock` on it each time we want to add
a change record.
This diff doesn't change any other existing functionality.
Reviewed By: simpkins
Differential Revision: D3660162
fbshipit-source-id: a6b6fa28dd12e4d34718956167ee87f8cb2d89ca
Summary:
Adds a thrift call that returns the list of materialized entries from the whole tree.
This is intended to be plugged into the mercurial dirstate extension.
Reviewed By: simpkins
Differential Revision: D3851805
fbshipit-source-id: 8429fdb4eeccc32928e8abc154d4e6fd49343556
Summary:
Was chatting with simpkins the other day and he mentioned that our
instrumented hg wrappers are quite CPU intensive. This diff switches us to
running the underlying `hg.real` or `git.real` in our integration tests when we
find them in the path.
Reviewed By: bolinfest
Differential Revision: D3865996
fbshipit-source-id: d047749356f0c1c0662774e25801f3578f9f9243
Summary:
hg_import_helper.py imports mercurial python modules, which are GPLv2+, so this
code also needs to be licensed under GPLv2+, rather than the BSD-style license
used for the bulk of the eden code base.
Reviewed By: wez
Differential Revision: D3833508
fbshipit-source-id: eb2a8969a5a88c12444a3778875609f24e145e6b
Summary:
Annoying that gcc and clang behave differently here. The compilation
error is due to gcc not seeing the implicit this pointer for some of these
method calls, so we need to explicitly use it.
Reviewed By: simpkins
Differential Revision: D3846973
fbshipit-source-id: 3d5b8b8b8c9bbab1e7935cff0e65677f76d116fb
Summary:
Buck needs this API so that it knows which paths under a project
root it should exclude when deciding whether it can ask Eden for its
SHA-1 or if it must compute it on its own.
Reviewed By: simpkins
Differential Revision: D3840658
fbshipit-source-id: 5eddc0bef423d3b3ee165d2a4b0bbf193f94f61a
Summary:
we now serialize the overlay data for each directory independently.
When we mount, we try to load the root overlay data. The children are lazy
loaded as the inodes are instantiated.
Structural changes cause the overlay data for the impacted dirs to get saved out.
I need to make a pass over this to fixup comments and so on, I just wanted to get this diff out first.
I moved the overlay stuff from `eden/fs/overlay` -> `eden/fs/inodes` since most
of the overlay-ness is handled in `TreeInode` now; the `Overlay` class is
really just for carrying around the paths and providing the serialization
helpers.
Reviewed By: simpkins
Differential Revision: D3787108
fbshipit-source-id: f0e089a829defd953535b9d0a96b102ac729261b
Summary:
It was starting to get pretty complex to manage locking across the
inodes, filedata, overlay and soon the journal, so as a simplifying step, this
folds data that was tracked by the overlay into the TreeInode itself.
This is the first diff in a short series for this. This one:
1. Breaks the persistent overlay information, so shutting down eden and
bringing it back up will lose your changes (to be restored in the
following diff)
2. Allows deferring materialization of file data in more cases
3. Allows renaming dirs.
The approach here is now to keep just one source of information about the
directory contents; when we construct a TreeInode we import this data from the
Tree and then apply mutations to it locally.
Each inode can be mutated indepdently from others; we only need to lock the 1,
2 or 3 participating inodes in the various mutation operations.
I'll tackle persistence of the mutations in the following diff, but the high
level plan for that (to help understand this diff) is to always keep the
directory inodes for mutations alive as inode objects. We make use of the
canForget functionality introduced by D3774269 to ensure that these don't
get evicted early. On startup we'll load this information from the overlay
area.
This model simplifies some of the processing around reading dirs and looking up
children.
Since the overlay data now tracks the appropriate tree or content hash
we can be much more lazy at materializing data, especially in the rename
case. For example, renaming "fbcode" to "fbcod" doesn't require us to
recursively materialize the "fbcode" tree.
Depends on D3653706
Reviewed By: simpkins
Differential Revision: D3657894
fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
Summary:
We can't allow ~EdenServer to delete the memory until we're sure that
the other threads are done. To ensure that, we need to notify the condition
variable while the aux thread still holds the lock. This makes sure that the
thread destroying the EdenServer waits for the aux thread to release the lock
before we check the predicate and proceed to deleting the memory.
```
SUMMARY ThreadSanitizer: data race /
/common/concurrency/Event.cpp:107 in facebook::common::concurrency::Event::set() const
==================
I0909 14:51:18.543072 4147554 main.cpp:173] edenfs performing orderly shutdown
I0909 14:51:18.555794 4148654 Channel.cpp:177] session completed
I0909 14:51:18.556011 4148654 EdenServer.cpp:192] mount point "/tmp/eden_test.0ostuc90/mounts/main" stopped
==================
WARNING: ThreadSanitizer: data race (pid=4147554)
Write of size 8 at 0x7fff9e182d90 by main thread:
#0 pthread_cond_destroy <null> (edenfs+0x00000007671a)
#1 facebook::eden::EdenServer::~EdenServer() /
/eden/fs/service/EdenServer.cpp:93 (libeden_fs_service_server.so+0x0000000b96cd)
#2 main /
/eden/fs/service/main.cpp:176 (edenfs+0x000000018515)
Previous read of size 8 at 0x7fff9e182d90 by thread T73:
#0 pthread_cond_broadcast <null> (edenfs+0x0000000765b7)
#1 __gthread_cond_broadcast /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/include/x86_64-facebook-linux/bits/gthr-default.h:852 (libstdc++.so.6+0x0000000e14f8)
#2 std::condition_variable::notify_all() /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/condition_variable.cc:72 (libstdc
++.so.6+0x0000000e14f8)
#3 facebook::eden::EdenServer::mount(std::shared_ptr<facebook::eden::EdenMount>, std::unique_ptr<facebook::eden::ClientConfig, std::default_delete<facebook::eden::ClientConfig> >)::$_0::operator()() const /
/
/eden/fs/service/EdenServer.cpp:145 (libeden_fs_service_server.so+0x0000000bcdb5)
#4 std::_Function_handler<void (), facebook::eden::EdenServer::mount(std::shared_ptr<facebook::eden::EdenMount>, std::unique_ptr<facebook::eden::ClientConfig, std::default_delete<facebook::eden::ClientConfig>
>)::$_0>::_M_invoke(std::_Any_data const&) /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:2039 (libeden_fs_service_server.so+0x0000000bcab0)
#5 std::function<void ()>::operator()() const /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:2439 (libeden_fuse_fusell.so+0x00000020fbb9)
#6 facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0::operator()() const /
/eden/fuse/MountPoint.cpp:69 (libeden_fuse_fusell.so+0x000000237447
)
#7 void std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()>::_M_invoke<>(std::_Index_tuple<>) /
/third-party-buck/gcc-4.9-
glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:1699 (libeden_fuse_fusell.so+0x000000237048)
#8 std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()>::operator()() /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgc
c/include/c++/trunk/functional:1688 (libeden_fuse_fusell.so+0x000000236ff8)
#9 std:🧵:_Impl<std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()> >::_M_run() /
/third-party-buck/gcc-4.9-glibc-2.
20-fb/build/libgcc/include/c++/trunk/thread:115 (libeden_fuse_fusell.so+0x000000236d8c)
#10 execute_native_thread_routine /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:84 (libstdc++.so.6+0x0000000e6
ec0)
```
Reviewed By: simpkins
Differential Revision: D3844846
fbshipit-source-id: 545474bc1aff8621dbeb487dcd6b54c82828ff3b
Summary:
This did not work because the dict-like object read from
a `ConfigParser` was not JSON-serializable by Python.
I had to add some methods to the `Repository` that we use in our
integration test harness in order to verify everything I wanted to
in my new integration test. I implemented these methods in both
`HgRepository` and `GitRepository`.
Reviewed By: simpkins
Differential Revision: D3837879
fbshipit-source-id: e0bfb5f1bd3add192ef9bdf561591ac8e52bc002
Summary: I did what the linter told me to do.
Reviewed By: wez
Differential Revision: D3836659
fbshipit-source-id: a5d3fc8974cf6cb7c7e2d88a6215ac5c54479780
Summary:
This is useful for inodes that maintain the source of truth about
something and that must not be deleted from our tracking datastructures.
Reviewed By: simpkins
Differential Revision: D3774269
fbshipit-source-id: 623c77769cbb30c35b413a443d0413e57d6c17b3
Summary: Open a file in a subdirectory in append mode and update it.
Reviewed By: wez
Differential Revision: D3802815
fbshipit-source-id: fe1726529d017345dfee530ae5ec84cfb7531602
Summary:
test_utime() seems to fail a reasonable amount of the time in my experience:
the atime returned by lstat() occasionally ends up being 1 or 2 milliseconds
earlier than the saved timestamp. With the existing behavior python and eden
are independently computing the current time, although I haven't investigated
why eden sometimes gets a timestamp slightly earlier than python did.
This fixes the test by asking eden to set an explicit timestamp value, rather
than letting eden independently get the current timestamp.
Reviewed By: wez
Differential Revision: D3802100
fbshipit-source-id: fecfd1d68b8c95d119099ef39f17004be13d3dff
Summary:
`buck-out` now has an extra level of depth due to its new CPU-specific directories,
so we have to change the place we look for `hg_import_helper.py` in development.
Reviewed By: simpkins
Differential Revision: D3807012
fbshipit-source-id: 24d1fc1fa22f3003580f59cfdd46b9b3917f6eeb
Summary: As noted in the comments, these dependencies were making this harder to use in Buck.
Reviewed By: simpkins
Differential Revision: D3805861
fbshipit-source-id: e04898d0e1a3ccc5e38a9629b1d30791853224a5
Summary:
This codemods `TARGETS` under `[a-d]*` directories in fbcode to make
the `headers` parameter explicitly refer to `AutoHeaders.RECURSIVE_GLOB`.
Reviewed By: yfeldblum
Differential Revision: D3801845
fbshipit-source-id: 715c753b6d4ca3a9779db1ff0a0e6632c56c0655
Summary:
Update DirList::add() to accept entry names as a folly::StringPiece instead of
just a null-terminated string. This makes it possible to use PathComponent
objects without having to make a copy of the name just to ensure that it is
null terminated.
The libfuse APIs unfortunately only accept null terminated strings, so we now
simply manually populate the fuse_dirent structs in our buffer rather than
using the libfuse helper methods.
Reviewed By: wez
Differential Revision: D3762623
fbshipit-source-id: d4132b354912e0e003090bddcad0ce912f4ed401
Summary:
Recent changes to our sudo policy seems to be dropping the SSH_AUTH_SOCK
environment variable for me, even when using "sudo -E". (We do explicitly
configure this to be dropped with sudo's env_delete setting.)
This updates the eden CLI to explicitly ask sudo to set environment variables
that we care about.
Reviewed By: bolinfest
Differential Revision: D3674124
fbshipit-source-id: ab14453056961f7b6a7f5d4122bc0c6f5caa6588
Summary:
Use fd 5 rather than stdout (fd 1) for communicating from the hg import helper
process. This way we won't have communication problems even if some of the
mercurial code does somehow ever end up printing to stdout. File descriptor 5
was picked rather arbitrarily. Anything greater than or equal to 3 should work
in practice.
Reviewed By: bolinfest
Differential Revision: D3673941
fbshipit-source-id: ed668542a17c585a5c54bcd3499174a6ed6fa138
Summary:
This is working around a noisy reporting issue in our CI system. Rather than
run and skip the integration tests in that environment, we swap out the base
class with one that won't report any tests to the python test case discovery
mechanism.
Reviewed By: simpkins
Differential Revision: D3634277
fbshipit-source-id: d118dca78a967bf068242790f7b874f865dddecc
Summary: Currently, all existing mount path are unmounted on 'eden shutdown' but are not remounted again after a subsequent 'eden daemon' call, though they appear as mounted when 'eden list' is called. These changes fix this behavior and have the daemon remount the paths that had been mounted before shutdown was called.
Reviewed By: simpkins
Differential Revision: D3580793
fbshipit-source-id: d03beafc20db4bd01662dd7f198a5ab8859b8e3d
Summary:
Load the config data when eden server is started so that it doesn't need to be re-loaded every time a mount is done. The normal use case for eden will not see that many changes to the config data (users adding repositories themselves is expected to be minimal) so this new logic will be more efficient overall.
Currently, the config data IS reloaded before use every time but this is because there is currently no way to reload the config data if any files are modified on disk. I am looking into how to do this now, and this feature will soon be updated to this diff so configData_ does not need to be constantly reloaded.
Reviewed By: simpkins
Differential Revision: D3580777
fbshipit-source-id: 5e23f51e4aab815e9812750617446dcb7e5483cb
Summary: Restructure the current logic used for loading the config data into a ClientConfig object. Rather than having loadFromClientDirectory iterate through all the config files and parse them to find the necessary information, abstract that logic out into a new method that compiles all of the relevant data so that all loadFromClientDirectory has to do is pull out the needed information. Since this change separates the two steps, this will make it easier to move the first step of compiling config information outside of ClientConfig - the goal here is to have the eden server load all of the config data at start up and cache it in memory so that it doesn't need to be done every time a ClientConfig object is created, and this change is an intermediate step.
Reviewed By: simpkins
Differential Revision: D3580757
fbshipit-source-id: c340a0fe715856066a554238249574f8177bc4d7
Summary: This should facilitate things by making the way to reference the path unambiguous.
Reviewed By: simpkins
Differential Revision: D3672311
fbshipit-source-id: b2a21e4dba60ea5ded28643f76aa402c4250d7ac
Summary:
If remotefilelog is unable to download a file from the server, it can end up
logging data through the UI which gets printed to stdout. Since we use stdout
to communicate with the main edenfs daemon this interferes with that
communication channel: edenfs ends up trying to parse the ASCII data as a
binary chunk header, sees a very large data length, and then ends up waiting
forever trying to read data that isn't coming.
This provides our own custom mercurial.ui.ui subclass to try and make sure that
we print all messages to stderr rather than stdout.
I will also send a separate diff so that we use a different pipe to communicate
with edenfs, rather than just using stdout/stdin. That should be a more
guaranteed way of making sure that nothing inadvertantly gets written to the
communication channel with edenfs.
Reviewed By: DurhamG
Differential Revision: D3673830
fbshipit-source-id: d64d492b10ee09ba63318b86d77f9181c4a0ce29
Summary: The repo_name could be cloned multiple times, which would cause a collision.
Reviewed By: simpkins
Differential Revision: D3672298
fbshipit-source-id: f27e1ec00ab773ed50f4fcf6f485461d01f8ba11
Summary:
Update the privhelper code to properly handle the error if it is asked to
unmount a path that isn't currently mounted.
This fixes the code to send an error response back to the main edenfs process.
Previously the exception propagated all the way up to the privhelper main loop,
causing the privhelper process to exit.
It would be nice to refactor the privhelper code a bit more to create a more
standardized API for implementing privhelper commands. This would provide a
more common place to catch exceptions, and make it harder for the command
implementation code to leak exceptions. However, I'll wait to do that in a
subsequent diff at some point.
Reviewed By: wez
Differential Revision: D3653727
fbshipit-source-id: e14c6ae974c8f99da3b426be250510c1f9b8017d
Summary:
Add back the "mount" CLI command.
While end-users may not need this command under normal circumstances, during
development it is sometimes necessary to re-mount unmounted clients. This adds
back the "mount" command which simply re-mounts an already configured client.
This also adds a "--no-forget" option to "unmount" which causes it to unmount
the client without forgetting its configuration. This allows it to be
remounted with "mount", rather than having to re-run the "clone" command and
specify the repository name again (and potentially restore the desired snapshot
hash).
Reviewed By: wez
Differential Revision: D3653706
fbshipit-source-id: 0e3f3fe1efdf72e9e54bdd1e0fb308ae67b7c058
Summary:
Use the default_permissions option when mounting. This tells the kernel to
enforce the normal file permissions semantics based on the file mode bits.
With this option enabled, the kernel will also implement the access() call for
us automatically.
Reviewed By: wez
Differential Revision: D3653198
fbshipit-source-id: e7227f455077a9ef012a5c102e4e22a976595681
Summary:
I have a follow on diff that needs this to work; this just injects the
existing hash functions that we're using for the boost hash stuff into
`std::hash` and checks that `set` and `unordered_set` compile.
Reviewed By: simpkins
Differential Revision: D3659318
fbshipit-source-id: 57920dbcd5f6e5449fd67769e379f8b3846a47f3
Summary:
It's useful in some debugging situations to understand the context for something like
`folly::throwSystemErrorExplicit(ENOENT, "useful context")`.
This is conceptually as simple as adding some `VLOG` statements in the error
handler for a fuse request, but I thought that it would be worth while to clean
up the macro that `clang-format` was doing a poor job of formatting. In turn,
I noticed that we can simplify the cancellation handling a bit while we're in
here, since we're only ever dealing with `Future<Unit>` in here now.
This reduces some boilerplate in Dispatcher.cpp.
Reviewed By: simpkins
Differential Revision: D3630958
fbshipit-source-id: 8d9dd50797f421ab9794b87f10d4c02c96e6fea8
Summary:
We weren't using these and they made some of our handling more complicated.
We now leave the release behavior to the destructor for the instance.
Reviewed By: simpkins
Differential Revision: D3615326
fbshipit-source-id: de39fb4315dd7b08da7c22cbcbd40e21be487102
Summary:
Include a configPath_ field for EdenServer that holds the path of the user ~/.edenrc config file. The server needs the data from this user config file in order to perform mounts and currently, the path to the home directory is passed via the CLI to the mount command as a field inside the MountInfo struct in order to get the file. As per discussion in D3498567, including the home directory inside the MountInfo struct is logically a bit disjointed, and this change would no longer require the home directory to be passed to the server via MountInfo.
This restructuring also sets up eden for a future change - having the server remount existing mount points on start-up is now possible from the inside. Before this change, mounting anything had to be done via the CLI since the home directory had to be passed in from the outside. This meant that remounting the existing mount points on start up could only be done if Eden was run in the background - running in the foreground would require manual remounting of all existing mount points. Now that the server has access to the config file's path, remounting can be done without any prompting from the CLI in both cases.
Reviewed By: simpkins
Differential Revision: D3580737
fbshipit-source-id: 46667ccd130b470a3a8a9e9aa08e5ec8e8b90336
Summary:
Previously we would simply report the raw pointer address to the kernel and
rely on it to return that same number to us as the file handle, and make sure
that it told us to shut down the handle when it was closed.
This meant that we had no real idea about which files were still open.
For our future live upgrade plans we need to be able to know this so that we
can transfer the appropriate information to our replacement process.
To facilitate this this diff implements a FileHandleMap class that will assign
file handle numbers and keep track of the instances. The number assignment
strategy is the same as it used to be in the common case: we take the address
of the newly created instance and use that 64-bit number as the file handle
number. However, in the future when we transfer the mapping to a new process,
we may experience a collision when subsequently opening a file handle. To deal
with that, we have a bounded number of attempts to assign a random file handle
number.
We don't yet offer a means to iterate the map, but it would be trivial to
expose such an accessor when we're ready to use it.
Since we now keep track of these things via shared_ptr this changes the
appropriate portions of the fuse interface from unique_ptr to shared_ptr.
Reviewed By: simpkins
Differential Revision: D3602364
fbshipit-source-id: dd996339c2838225a2caeee9da16ef99a06c1e2b
Summary:
When the EDEN_GDB environment variable is set, start eden under gdb when
running the integration tests. Have gdb automatically start the daemon,
and exit automatically if everything is successful. If the daemon crashes, gdb
will break and the user can interact with gdb normally.
This is generally only useful when manually running a single test. Trying to
use this in combination with "buck test" will probably cause problems if
multiple gdb instances all try to interact with the terminal together.
Reviewed By: wez
Differential Revision: D3459575
fbshipit-source-id: 1359735a391e93a332a4c673c6525ad44034cc73
Summary: This change updates LocalStore to perform serialization of trees and blobs internally so that its users don't need to be aware of the internal serialization format. Previously, the get and put APIs were asymmetric such that the get APIs returned deserialized Tree and Blob objects, while put required raw serialized bytes. After this change, put will also use deserialized Tree and Blob objects.
Reviewed By: simpkins
Differential Revision: D3589899
fbshipit-source-id: 2e572e6ec5af44d66206b178a03f7a9d619b2290
Summary:
Update the CLI to always close the thrift client socket, to avoid resource leak
warnings on exit.
I also updated the code to just monkey-patch a nicer EdenError.__str__()
method, rather than having to explicitly catch and modify this exception in
multiple different places.
Reviewed By: bolinfest
Differential Revision: D3560662
fbshipit-source-id: 900fe74c793ffd99f4a2c1f1ddd94b96e48f5eb7
Summary:
The "eden repository <name> <path>" command had a bug that it would duplicate
the existing edenrc contents when writing out a new config: it opened the file
in append mode rather than truncate mode when performing the write.
This addresses that issue, but also does a bigger overhaul of the config update
code. We now acquire a lock for the duration of the modification, so that
another eden CLI process cannot modify the file between when we read it and
when we write out our modifications. We also perform the write using an atomic
rename so the file contents are always valid at all points in time.
I also updated the CLI command to print the repositories in sorted order, and
to only catch expected exceptions, and show backtraces for unexpected errors.
Reviewed By: bolinfest
Differential Revision: D3554550
fbshipit-source-id: 5920ccb2447330673eac3f9956a8ae5d8a66a67e
Summary:
This feels awful, but eliminates an abort that shows up when running under ASAN.
Details in the comment.
Reviewed By: bolinfest
Differential Revision: D3533529
fbshipit-source-id: 1717384c716f565b74b1678c66883a35ee60c522
Summary:
For consistency with `git clone`, the destination path must be either:
* non-existent (but `mkdir -p`-able)
* an empty directory
Or else `clone` should fail.
In the process of writing the integration test, I cleaned up some of our test infra
and took advantage of the fact that we require Python 3 (I believe 3.5) now:
* We now use `subprocess.run()` instead of `subprocess.check_output()`.
* Removed the `client_name` argument to `EdenClient.init()` because that is no longer the right terminology and no one appeared to be overriding the default, anyway.
* `EdenClient.clone_cmd()` no longer calls `os.mkdir(self._mount_path)` because that was papering over the issue this revision is trying to solve.
Reviewed By: simpkins
Differential Revision: D3526909
fbshipit-source-id: 66b5799a1b47a51468faf5dc17d87a385926dc89
Summary:
The intention is to use this for the PathTrie that is in a following diff.
The motivation is to avoid making a N copies of a string when traversing a path
of depth N in a PathTrie. The PathMap is backed by a vector and knows how to
compare the Piece variant of our path types.
I have not performed any rigorous comparisons, but based on reading around
other flat_map vs map comparisons, the expectation is that this will use less
memory (and avoid extraneous allocations during lookups) than a regular map,
with most random access being fairly similar. Inserts need to move the memory
around to make room and are thus slower than a regular map. Sorted inserts
perform the best for PathMap as that translates into repeated back-insertions.
My intuition is that this is the right trade off to make for managing the
overlay data structures; the size of these materialized dirs is likely going to
be small compared to the full fbsource tree.
Reviewed By: simpkins
Differential Revision: D3470377
fbshipit-source-id: 6f4f3abe33818da6981d8ab821ebe004a78d382c
Summary: Move adding the mount path to the directory map as the final step of the clone command so that the path is only documented if the mount is actually successful. Previously, if 'eden clone' failed during the mounting, another call to 'eden clone' to try again would fail with 'Error: mount path NAME already exists.' This change ensures that a second call to 'eden clone' will actually try again if the first failed.
Reviewed By: simpkins
Differential Revision: D3543388
fbshipit-source-id: be6412ad6d5d009bd87ad6e0b4110b35e6300fcb
Summary: This change fixes exception messages so that they all match the same format and don't include 'error: ' twice
Reviewed By: simpkins
Differential Revision: D3544034
fbshipit-source-id: 32d600c712d1782d2a1a35331b66e06a842ed8e0
Summary:
[Folly] Move `IPAddress` definitions to source files.
And to internal header files. Keeping headers lightweight can help with build times.
Reviewed By: simpkins
Differential Revision: D3514455
fbshipit-source-id: de78f4ef9e70e7ddd7fb666348ed705c5228531c
Summary:
Add a decorator to more easily declare tests that should be run for both git
and mercurial repository types.
Reviewed By: wez
Differential Revision: D3458883
fbshipit-source-id: 8058873923f9519a6550a8dcb930a5edd02aec10
Summary:
- Refactor the EdenClient code to better reflect how the eden currently works.
This code was originally written when the edenfs daemon only supported
running a single mount point. This updates it to reflect the fact that it
manages a single edenfs daemon, but multiple repositories can be mounted.
- Refactor the EdenTestCase now that tests generally only need a single eden
daemon. EdenTestCase starts the eden daemon by default.
- Add EdenHgTest and EdenGitTest classes. These are subclasses of
EdenTestCase, and they create and mount an hg/git repository before starting
the test function.
- Update the tests to derive from EdenHgTest and EdenGitTest where appropriate.
Reviewed By: wez
Differential Revision: D3458842
fbshipit-source-id: 77349a60ff72a700a2c2526a27e7621b76f9eec2
Summary: Fix a bug with the eden repository command that let it add multiple repositories with the same name. If an entry for repository 'name' already exists in the home ~/.edenrc config file, the eden repository command will now raise an error message saying that the repository already exists. Note that if an entry for repository 'name' exists in any of the global /etc/eden/config.d/* files, the eden repository command will still allow the user to create the new repository and will add an entry for 'name' to the home ~/.edenrc config file (since entries in home config file overwrite those in global)
Reviewed By: wez
Differential Revision: D3529743
fbshipit-source-id: b7c7ceef6ef38fcd3b4ff9a92fbd23a4a0bd6951
Summary:
Mercurial maintains its manifest in sorted order, but since the manifest only tracks file names we can end up with the following sequence:
```
some/path-foo/bar
some/path/bar
```
This is because the `-` sorts ahead of the `/`.
This diff defers passing the entries to the tree serializer, buffering them up
into a temporary vector and using `std::lower_bound` to find the appropriate
insertion point.
Reviewed By: bolinfest
Differential Revision: D3529329
fbshipit-source-id: 395ed16a20c14d17717ec69192a38f0407b51e1d
Summary:
This check was already being done appropriately in our Python code, but we also need
to do it in our C++ code.
Reviewed By: wez
Differential Revision: D3526705
fbshipit-source-id: 3b28b88f63ae768113f363ace58d40a89a8f4b61
Summary: Move the _get_or_create_dir() method from config.py to util.py and rename as mkdir_p(). This change replaces all of the os.makedirs call sites with the new mkdir_p() method.
Reviewed By: bolinfest
Differential Revision: D3512570
fbshipit-source-id: a867049f9af22076934390061f09070bf9ee6397
Summary:
These changes restructure the eden directory so that 'client' directories are created during the `eden clone` command and are associated with a single mount path.
The new eden directory looks as follows:
~/.eden
config.json
clients/
abcd08d/
edenrc
SNAPSHOT
overlay/
efgh19i/
edenrc
SNAPSHOT
overlay/
...
Where the config.json file holds the mapping of mount paths to their respective client directory which is a hash, and the edenrc files in each client directory is an INI file which holds the name of repository associated with the mount path. This INI file follows the current format:
[repository]
name = fbsource
This restructuring required a couple other changes:
- unmount command now cleans up the client directory and removes the mapping of its mount path from config.json
- eden list command now lists all of the mount paths rather than the client names
Reviewed By: bolinfest
Differential Revision: D3506119
fbshipit-source-id: dc07a8baf1052be731ff335d9cf74a07ab8e661a
Summary: Change the ClientConfig class to parse client data via INI config file rather than json file. This class uses boost::property_tree::ini_parser and the ptree data structure to hold the parsed INI file contents. This change makes it possible for eden to no longer rely on json files for getting client data, and the json files will be completely taken out in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3498567
fbshipit-source-id: 3298047a014beda0c250475c0809a7a1ebd95b2b
Summary: Previously the repo snapshot id was collected and stored when the repository was initialized, but this diff moves the update to the clone command so now the snapshot is only stored right before a repository is mounted.
Reviewed By: bolinfest
Differential Revision: D3489550
fbshipit-source-id: 1271c1d7c7709ed332307a636c23e26e913483c9
Summary: Update the unmount command to use the mount point rather than the client name. This diff also removes the mount command completely since tests are no longer dependent on it.
Reviewed By: bolinfest
Differential Revision: D3489290
fbshipit-source-id: 9bfbcb78a78d1593f6d0950502bdd4a55e3e6ca4
Summary: Introduce new 'clone' command that takes in repo name and mount path. This command replaces the current mount command which uses the client name as an argument
Reviewed By: bolinfest
Differential Revision: D3489098
fbshipit-source-id: 719fb94387da1d4ccab770a0e659f717888a7dc6
Summary: Expand functionality of repository command to also create repositories. The current command line format is 'eden repository <name> <path>' and will soon replace the existing init command.
Reviewed By: bolinfest
Differential Revision: D3489064
fbshipit-source-id: 3e6946cbd80a64ad23b52383f3f14a92d4492a21
Summary: Add a simple eden repository command that takes no arguments. The behavior mimics that of git remote and lists all repositories by parsing both the global and local INI config files. The INI file currently needs to be manually added and updated to maintain information about existing repositories.
Reviewed By: bolinfest
Differential Revision: D3479543
fbshipit-source-id: b0d36f33c9b8e0c7b9fe20ec2f362ca15c0adeb8
Summary:
This is a bit ugly and longer term should probably be an
eden configuration that we deploy to our CI tier.
In our CI environment we don't allow unfettered password-less sudo
access and instead must launch privileged processes via a helper
process that is whitelisted. D3524299 has the definition of that
helper as it applies to our internal infra.
Reviewed By: bolinfest
Differential Revision: D3524373
fbshipit-source-id: f36bf29e6b455220c434b681a6d0d551bc21ecab
Summary:
We're seeing failures like this in our CI system:
```
stderr:
test_create (eden.fs.integration.basic_test.BasicTest) ...
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: no tty present and no askpass program specified
error: edenfs is not running
ERROR
ERROR
======================================================================
ERROR: test_create (eden.fs.integration.basic_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "eden/fs/integration/basic_test.py", line 68, in test_create
eden = self.init_git_eden()
File "eden/fs/integration/lib/testcase.py", line 104, in init_git_eden
return self.init_eden(repo_path)
File "eden/fs/integration/lib/testcase.py", line 54, in init_eden
inst.init(repo_path, *kwargs)
File "eden/fs/integration/lib/edenclient.py", line 160, in init
self.daemon_cmd(timeout)
File "eden/fs/integration/lib/edenclient.py", line 174, in daemon_cmd
self._wait_for_thrift(timeout)
File "eden/fs/integration/lib/edenclient.py", line 99, in _wait_for_thrift
raise Exception("edenfs didn't start within timeout of %s" % timeout)
Exception: edenfs didn't start within timeout of 10
```
This diff improves things by failing faster and printing a little more appropriate context.
Reviewed By: bolinfest
Differential Revision: D3523257
fbshipit-source-id: 14508a1b820d9d6c773713e2bb7c22955c5b2b66
Summary:
Implements basic rename(2) support. We only do this for files at this
time as directory renames require that we recursively materialize a tree and
are a bit more complex. I'll look at that in a follow-on diff; there's
potential for optimizing it to avoid eager materialization, but that depends on
the trie work that I'm tackling concurrently with this.
rename for files is the last piece needed to allow `sed -i` to operate correctly.
Reviewed By: bolinfest
Differential Revision: D3515361
fbshipit-source-id: 9c8cc5f9b8db6b5a9372ca9286336647f50490f8
Summary:
This enables O_EXCL to function by allowing the create routine to
move its folly::File instance down in to the underlying FileData instance.
Previously we would close and then re-open the file; this would discard
any of the natural gating for the open call that is performed by the kernel
for the underlying filesystem in the overlay.
Reviewed By: bolinfest
Differential Revision: D3513758
fbshipit-source-id: 85967a3b7affa1b1df46842be8ba21c8fbb843a6
Summary:
Despite being handy things for filesystems in general, these are
needed to support the `sed -i` integration test.
Reviewed By: bolinfest
Differential Revision: D3513754
fbshipit-source-id: 505b4bd58b254141e2ef996f01e3347fc1a77584
Summary:
This eliminates a linear scan from TreeInode and replaces it with a
binary search, exploiting the sorted order of the entries vector.
Two new methods are introduced: getEntryPtr which returns a pointer to the
entry with the matching name, and getEntryAt() which returns a reference
(throwing a range error if there is no such entry).
I wanted to use the PathMap class here, but that would cause us to duplicate
the name string as both the key and value in the map.
Reviewed By: bolinfest
Differential Revision: D3515723
fbshipit-source-id: 4ee0371f3ec08cbcf110cf28f5c1e1529b120fb6
Summary:
I found this by running the tests with ASAN enabled; it was reporting leaks.
The issue is that fuse decides to append some args to the argument list in some
cases. When it does this it tries to realloc the storage in the argv array.
This was not safe to do because that data was owned by the vector and may not
have been allocated directly by malloc.
This diff switches us to use the underlying functions in libfuse that append
arguments and then know how to safely free the result.
SCOPE_EXIT is used to ensure that this gets cleaned up when we return.
Reviewed By: bolinfest
Differential Revision: D3519999
fbshipit-source-id: ca12739f11be1fd9662063ec7515cec90ccf0d57
Summary:
This removes use of the builtin Buck `thrift_library` support from
the macro library. It turns out that a lot of rules incorrectly
added deps onto the raw builtin `thrift_library`, rather than one
of the per-language rules. This is a noop, and removing the builtin
rule exposes this as missing target errors, so this diff removes them.
Reviewed By: Coneko
Differential Revision: D3512451
fbshipit-source-id: dd8beb148ed47a3ad7d3963fae600abd73d030d5
Summary:
setattr is a bit of a multi-purpose interface; depending on the flag
values, this is responsible for:
* ftruncate(2)
* fchmod(2)
* fchown(2)
* futimens(2)
In order to apply any of these things, we have to materialize the file. In
the future we may want to allow setting the utimes without materializing the file.
We don't allow chown to actually chown anything. We may want to relax that in
the future, but at the moment we will return an error if an attempt is made to
change the ownership of a file.
Reviewed By: bolinfest
Differential Revision: D3511011
fbshipit-source-id: 858d2c07686fcbe2dcdb60a07527f739a9726be3
Summary:
Make sure mount points are completely stopped before destroying the EdenServer
object. Previously the EdenServer was destroyed with the MountPoints still
running the fuse channels in background threads. When the privileged helper
process unmounted them, fuse requests from the kernel could arrive and access
memory that had already been destroyed.
Reviewed By: wez
Differential Revision: D3458898
fbshipit-source-id: 365bca716ff0f8315b66af92effeb8c6dc574ce1
Summary:
When running the integration tests when built with gcc, the tests would crash
in TreeInode::create(). It appears that the unique_ptr<FileHandle> object was
getting passed to the lambda before dereferencing it to call getattr().
Reviewed By: wez
Differential Revision: D3459605
fbshipit-source-id: 5e2ce98d268a85731acaf7d7f37f22c77fb571cf
Summary:
This fixes inode number handling problems in TreeInode and TreeEntryFileInode.
Previosly these classes each had an ino_ member variable, despite deriving from
InodeBase which has its own (private) ino_ member.
TreeEntryFileInode never actually initialized its own local ino_ variable.
This was causing problems for many applications which use inode numbers to
cache file data. TreeEntryFileInode returned garbage data in the inode field,
resulting in incorrect cache collisions.
This fixes the inode handling, and also fixes the stat data returned by
FileData. It now sets the uid, gid, and inode fields correctly. It 0s out
several other fields rather than leaving them uninitialized.
Reviewed By: bolinfest
Differential Revision: D3455126
fbshipit-source-id: 631276b01676733f96035bc153219ef84406dcc9
Summary:
Start adding utility code to make it easier for the integration tests to
manipulate git and mercurial repositories.
This adds an HgRepository class, but does not update the tests to use it yet.
I will do that in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3449328
fbshipit-source-id: caea5befb683a388cbad30a29017c566e22702af
Summary:
Move the eden/fs/importer/hg code to eden/fs/store/hg. This import code is
only used by the HgBackingStore, so logically it belongs together.
The eden/fs/importer/git directory was already moved into eden/fs/store/git by
D3448752, so this makes the mercurial code similar.
Reviewed By: bolinfest
Differential Revision: D3458128
fbshipit-source-id: 54b1f707197e41ff5f5f4a163a34d3d1ab432e21
Summary:
This moves all of the test library code into a lib/ subdirectory, just to help
distinguish tests from utility code.
This also changes the test so that we no longer pack the eden CLI and daemon
binaries into the python archives. This results in very large archives when
building in dbg and opt modes, and isn't really necessary. Instead
edenclient.py simply finds the CLI and daemon binaries relative to the test
binary. We pass in an EDENFS_SUFFIX variable to tell it which flavor of the
daemon to use.
Additionally, this changes the tests to run with python 3.
Reviewed By: bolinfest
Differential Revision: D3449013
fbshipit-source-id: 82533137090325766a52cd067aa97dd8391ae088
Summary:
This moves git import logic from the GitImporter class to GitBackingStore.
The logic is simpler now, since GitBackingStore only needs to import a single
Tree or Blob at a time.
Reviewed By: bolinfest
Differential Revision: D3448752
fbshipit-source-id: da2d59f953ada714d8512545ae83dd48e5d3e410
Summary:
Add a run_cmd() function that's slightly simpler to use compared to
_get_eden_args().
Reviewed By: bolinfest
Differential Revision: D3448844
fbshipit-source-id: 06b654b2f4bc773d67a4f9d1f6effdad2ce9aa2a
Summary:
- Update "eden daemon" to wait until edenfs is healthy before returning.
- Accept additional command line arguments and pass them through to edenfs.
- When starting edenfs as a background daemon, direct stdout and stderr to a
log file in the eden config directory.
- Update "eden shutdown" command to wait for edenfs to exit.
- Update "eden shutdown" command to print an error message if edenfs was not
running.
Reviewed By: bolinfest
Differential Revision: D3446403
fbshipit-source-id: 2ca1874256b7d124a85092b886548bea8c198327
Summary:
Look for hg_import_helper.py relative to the edenfs binary, rather than
relative to the current working directory. Also check in a couple places, so
we can work in both normal deployment locations as well as inside a source
repository.
Additionally, update hg_import_helper.py to emit a response chunk as soon as it
starts. This contains either message indicating that it has started
successfully, or an error message. This allows us to propagate error
information back to the thrift caller if something goes wrong starting
hg_import_helper.py for a given repository.
Reviewed By: bolinfest
Differential Revision: D3447027
fbshipit-source-id: 4bffd1d03ab6475c09cb3c8385145555c7fc7361
Summary:
[Folly] Wrappers for some of OpenSSL's crypto hash functions.
Wraps some of the OpenSSL crypto hash functions with variants that take `ByteRange` for input and `MutableByteRange` for output, and also variants that take `const IOBuf&` for input as well.
These are a bit nicer to use than passing pointers and lengths separately.
Reviewed By: ivmaykov
Differential Revision: D3434562
fbshipit-source-id: 3688ef11680a029b7664ac417a7781e70f9c6926
Summary:
This adds a --foreground flag to the "eden daemon" command, which causes the
daemon to run in the foreground, rather than daemonizing as a background
process. Additionally, this drops the --preserve-environment flag and instead
updates the CLI to always build a sane environment to run with.
This also updates the integration test code to use this flag, and wait for eden
to shut down during test clean up. Without this, the test case code could try
to clean up temporary directories before they were unmounted, which would fail.
This would leave many temporary directories behind after test runs.
This also re-organizes the temporary directory set up a bit, so that each test
uses a single top-level temporary directory. All other directories it creates
are put inside this one directory.
Reviewed By: bolinfest
Differential Revision: D3439232
fbshipit-source-id: 85305f7db60da6bae589d28c802ee260e42e4dea
Summary:
This changes the way that Eden is built and deployed.
* To build the binary that must be run as `root` (but quickly drops privileges), run `buck build eden-daemon`.
* To build the CLI that communicates with the daemon (and does not require privileges), run `buck build eden-cli`.
* To build both, run `buck build eden`.
There is an example of how to build the various parts of Eden using
Buck and how to package them up in the `install` script introduced by this revision.
While here, I also cleaned up some of our build files and changed them to be
parameterized between internal and external use. In both cases, the user gets the
"unadorned" version of their primary build targets. This ensures that shortcuts such as:
```
buck test eden/fs/integration
```
do the right thing by default.
Finally, I also made `find_default_config_dir()` and `find_default_daemon_binary()`
lazy whereas `find_default_config_dir()` was previously eager.
Reviewed By: simpkins
Differential Revision: D3436245
fbshipit-source-id: 4dfbd59ed0d198620324f0705c462334bb5a7daf
Summary:
If a directory is present in the overlay, we still need to check if a TreeEntry
exists from the source control data structures. Previously this was causing us
to incorrectly report directories as empty if they exist in the local overlay,
even when they had contents from the main Tree.
Reviewed By: wez
Differential Revision: D3434219
fbshipit-source-id: f872f90075602dfdc7b217f50eefcd7c248512e7
Summary:
Add some verbose logging about when trees and blobs are loaded in the object
store.
Reviewed By: bolinfest
Differential Revision: D3434182
fbshipit-source-id: 3e8d2617290604f119e6164d15d63324a4c9a2aa
Summary:
Update the HgImporter class to support retrieving file contents from mercurial.
This also includes simple code for storing the data in the LocalStore using
git's blob serialization format. In the future I think it would perhaps be
better to drop the "blob<length>" prefix, and instead just use a RocksDB column
family to separate blob data from other types of data. However, for now using
the git format is simplest for keeping compatibility with the getBlob() code.
Reviewed By: bolinfest
Differential Revision: D3416691
fbshipit-source-id: 268787533be2172b2dbedc3bf06464eabf3d2c5e
Summary:
Record mercurial commits that have been previously imported, and avoid
re-importing them when checking out a commit that was previously processed and
already has a tree present in the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3416654
fbshipit-source-id: 80f70fa2204d3a2961991b634ad934a623b42b82
Summary:
Add APIs for storing arbitrary (key, value) data.
This will allow BackingStore implementations to store additional metadata, such
as mapping mercurial commit IDs to the eden root tree ID.
Eventually we may want to use RocksDB column families to partition the
different types of data being put into the LocalStore. However, for now this
just uses a single key space. We can add column family support in a separate
diff, if desired.
Reviewed By: bolinfest
Differential Revision: D3409866
fbshipit-source-id: 19a1d340b65bff2081981bf5daf32d5ad15b60c4
Summary:
This adds an HgBackingStore implementation which can load tree data from a
mercurial repository. Blob loading is not implemented yet, but will come in a
separate diff.
This also adds a minimal GitBackingStore class. The GitBackingStore has nearly
no functionality, but is needed to keep the existing git functionality working.
Reviewed By: bolinfest
Differential Revision: D3409743
fbshipit-source-id: dbebf53e9de08bd1469e489baa48b84cbf889511
