Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Previously we'd emit an AttributeError. This diff
fixes things up so that we show the help when no arguments are provided.
Reviewed By: simpkins
Differential Revision: D4434060
fbshipit-source-id: 838b1cab47118d8517e4dc0c5fde17fc69752b13
Summary:
This lock protects all operations that change the location of existing inodes
(rename(), unlink(), and rmdir()). This is needed to avoid race conditions
during rename operations.
I also included some FIXME comments about properly updating location
information for unloaded inodes. These should be taken care of in a subsequent
diff.
Reviewed By: wez
Differential Revision: D4361197
fbshipit-source-id: 5aaa1b8250f196b23207be8f3c0fd7f2603e6ae8
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.
The inode lookup itself still happens serially at the moment. We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.
Reviewed By: wez
Differential Revision: D4361091
fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
Summary:
This updates the InodePtr and InodeBase code to actually implement Inode
unloading and destruction.
At the moment we keep Inode objects as long as possible, and only unload them
during shutdown, or when the last reference to an unlinked inode goes away.
However, it should be straightforward to add on-demand unloading in the future
for Inodes that have not been accessed in a while. The
TreeInode::unloadChildrenNow() function provides a template for what this would
look like (it would simply need to be changed to check an access time when
doing on-demand unloading).
Reviewed By: wez
Differential Revision: D4360765
fbshipit-source-id: a46b355f0ac0c25f873a156e62af5184317de735
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function. This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.
This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.
Reviewed By: wez
Differential Revision: D4360558
fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
Summary:
This defines our own custom smart pointer type for Inode objects. This will
provide us with more control over Inode lifetime, allowing us to decide if we
want to unload them immediately when they become unreferenced, or keep them
around for a while.
This will also allow us to fix some memory management issues around EdenMount
destruction. Currently we destroy the EdenMount immediately when it is
unmounted. This can cause issues if other parts of the code are still holding
references to Inode objects from this EdenMount. Our custom InodePtr class
will also allow us to delay destroying the EdenMount until all of its Inodes
have been destroyed.
This diff adds the new pointer types and updates the code to use them, but does
not actually implement destroying unreferenced inodes yet. The logic for that
has proven to be slightly subtle; I will split it out into its own separate
diff.
Reviewed By: wez
Differential Revision: D4351072
fbshipit-source-id: 7a9d81cbd226c9662a79a2f2ceda82fe2651f312
Summary: Folly will soon be dropping support for GCC 4.8, which means that `folly::make_unique` will be going away, so codemod away as much as possible before then.
Reviewed By: yfeldblum
Differential Revision: D4411221
fbshipit-source-id: 31d61425f6595d0750ea2e4c95c7d42bb5a5d955
Summary: I fixed an unrelated bug in HTE that made some more paths lintable - and some more move-abuses visible :-(
Reviewed By: meyering
Differential Revision: D4397732
fbshipit-source-id: 560db05f64ecf160e8198caf80dbd988e0dad75d
Summary:
I ported the C++ test I wrote for `hg add` in D4317871 as an integration test.
Note that this caught a real bug in the Hg extension!
Reviewed By: wez
Differential Revision: D4385970
fbshipit-source-id: 97ee555d463f22af54791f334f9349d88861f45a
Summary:
This constructor is unsafe. The check it uses before doing the comparison isn't a safe enough check to see if the cast is valid. For example, this is broken in the presence of multiple inheritance because it doesn't adjust the pointer offset to the correct vtable.
e.g.
struct A {
virtual ~A() {};
virtual void doSomething() = 0;
};
struct B {
virtual ~B() {}
virtual void doSomethingElse() = 0;
};
struct C : public B, public A {
virtual ~C() {}
void doSomething() override {
std::cout << "Something!" << std::endl;
}
void doSomethingElse() override {
std::cout << "Something Else!" << std::endl;
}
};
int main (int argc, char **argv) {
auto c = folly::makeFuture<std::shared_ptr<C>>(std::make_shared<C>());
folly::Future<std::shared_ptr<A>> a = std::move(c);
a.get()->doSomething();
return 0;
}
This code will print "Something else!" when run.
Reviewed By: siyengar
Differential Revision: D3679673
fbshipit-source-id: dcbf40ca82d458f17ee11191591f8b8daf58c919
Summary:
This describes the behavior around Inode ownership, loading and unloading, as
well as the synchronization around these events.
The InodeMap and InodePtr diffs implementing this behavior are still in flight.
Reviewed By: bolinfest
Differential Revision: D4356591
fbshipit-source-id: 298ba2c6aa98feba8b90f858e5633ca793276a1e
Summary:
If the EDEN_DAEMON_ARGS environment variable is set when running integration
tests, append it to the edenfs daemon command line.
For instance, this makes it easy to pass in a "--vmodule" flag to turn up
verbose logging in particular parts of the code.
Reviewed By: bolinfest
Differential Revision: D4359306
fbshipit-source-id: 6b549ac7220f6c8d7a1dfad3827921d6462d0f77
Summary:
Add a document describing the various inode related locks and the order that
they may be acquired.
Reviewed By: bolinfest
Differential Revision: D4356023
fbshipit-source-id: 44d4ade984f6cb49bb5f09deeeef0fd5439f129c
Summary:
Add VLOG() statements for every FUSE entry point. Turning up the VLOG level in
EdenDispatcher now allows us to have a record of all FUSE calls made using
inode numbers. (Calls involving file handles or directory handles currently go
directly to the handle in question and do not pass through the EdenDispatcher.
We should probably add log statements for those too, but that can wait for a
later diff.)
Reviewed By: bolinfest
Differential Revision: D4359066
fbshipit-source-id: b378d829c39fdb61faf63d2d400f3ff556c376e8
Summary:
This member variable was not updated properly when files were renamed.
InodeBase now tracks our parent properly, so we don't need our own copy.
This does still call getParentBuggy() (which does not perform proper locking)
in a couple places for performing overlay operations. We'll need to fix this
later when addressing other overlay concurrency handling issues.
Reviewed By: bolinfest
Differential Revision: D4348481
fbshipit-source-id: 19c1ffced6f63e1ff041d0bab2363fecdb93d5a3
Summary:
Have FileData objects store a pointer to the FileInode that owns them, rather
than just to the EdenMount.
FileData objects still store a direct pointer to their FileInode's mutex_ and
entry_. It's potentially worth just accessing these through inode_ in the
future.
Reviewed By: bolinfest
Differential Revision: D4348103
fbshipit-source-id: 1f8497979bfc89c6a192ca0195209335db0d911c
Summary:
Add an EdenMount* member variable to InodeBase. Previously each TreeInode kept
a pointer to its EdenMount, and this moves that into the InodeBase class.
This is needed to support upcoming diffs that will change InodeBase memory
management. InodeBase objects will be responsible for notifying the InodeMap
when they become unreferenced.
Reviewed By: bolinfest
Differential Revision: D4348079
fbshipit-source-id: 6bf7ea908c6096aa2bca5b21290c09cbd58d5af7
Summary:
This updates all of the eden code to use the new InodeMap class. This replaces
the InodeNameManager class and the unordered_map previously stored in the
EdenDispatcher.
Reviewed By: bolinfest
Differential Revision: D4325750
fbshipit-source-id: d80ae7581ba79ca2b63155e184995a3e83e85dc1
Summary:
This diff starts adding a new InodeMap class. This class will eventually
consolidate the functionality of InodeNameMap plus the inode map stored in
EdenDispatcher.
This new class will bring several new benefits:
- All inode mapping logic consolidated into a single class, with a single lock
protecting the maps.
- A well-defined model for loaded vs unloaded inodes. InodeMap explicitly
tracks inodes that have InodeBase objects created for them vs inodes that
have an inode number allocated (for FUSE) but do not have an InodeBase object
in memory. This will make it possible to unload Inode objects on demand to
reduce memory usage.
- Tracking of pending loads, and de-duplicating load requests. This ensures
that only one Inode object ever exists for a given inode number / path. If a
second request to load an Inode arrives when a previous load request is still
in progress, InodeMap deals with this situation properly.
- Better support for using Inode objects without FUSE. With the old code,
attempts to interact with Inode objects without going through the FUSE
dispatch (say, when processing a thrift call) could result in inconsistent
state. New inodes created would not be put into the EdenDispatcher map,
which could result in problems.
- More convenient child inode access from TreeInode. With this change, the
TreeInode class can easily tell which of its children are loaded. This makes
it easier to do tasks which only need to operate on existing loaded inode
state (for instance, dirstate computation).
- Support for saving and loading state, to implement graceful restart..
InodeMap provides a central place to write out inode state on shutdown and
restoring it on startup. Saved inodes can easily be restored to an
"unloaded" state on startup. This code is not implemented yet as part of
this diff, but it should be straightforward to add in future diffs.
Reviewed By: bolinfest
Differential Revision: D4318060
fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
Summary:
This introduces an `HgExtensionTestBase` class that can be subclassed
to create tests for Mercurial integration with Eden. It also introduces
`status_test.py`, which includes some basic integration tests for `hg status`
in Eden. Although our existing `.cpp` tests give us a lot of coverage, they
do not verify the logic in the Python code for our Mercurial extension,
so this will help me sleep better at night.
Note that `HgExtensionTestBase` is a subclass of `testcase.EdenHgTest`
rather than using the `testcase.eden_repo_test` decorator because these
tests will only be used in the Mercurial case for now, so there is no point in
trying to run them when creating the mount from a Git repo, as well.
Further note that this also required a small change to `eden/fs/integration/lib/testcase.py`
because the Eden directory was not being created at `$HOME/local/.eden`, which
is something that the Mercurial extension relies on.
Reviewed By: simpkins
Differential Revision: D4337660
fbshipit-source-id: ea825134420ae6eb794d37b6c75da19a6aadaae5
Summary:
This preserves the existing behavior while making `REPO_ROOT` and `BUCK_OUT`
available to other test code.
Reviewed By: simpkins
Differential Revision: D4358875
fbshipit-source-id: a7b2cafe7e56673831beac59a85fcab436e637a5
Summary: This is much less hacky than what we had before.
Reviewed By: simpkins
Differential Revision: D4359650
fbshipit-source-id: 52fac899a98828c0a3cc731f276327d3ae171301
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.
This revision was initially created by running:
```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```
And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.
But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.
Reviewed By: simpkins
Differential Revision: D4356724
fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
Summary:
Specialize folly::FormatValue for the various path types, so that they can be
used with folly::format().
Reviewed By: bolinfest
Differential Revision: D4343924
fbshipit-source-id: 21165dfbad73612d1b786591c78e47dc26e6cc66
Summary:
Add an EDEN_BUG() macro to use in situations that can only occur if there is a
logic bug in the code.
In some ways this is similar to LOG(DFATAL) except that:
- It throws an exception in non-debug builds, instead of just logging. It
provides APIs to return a folly::exception_wrapper rather than throwing.
- We can extend it in the future to log bug info to a scuba dataset.
Reviewed By: bolinfest
Differential Revision: D4343911
fbshipit-source-id: 11f030a4f594f9997455cec9235a0151362a472f
Summary:
Rename the existing TestBackingStore class to FakeBackingStore, and fill it out
with an implementation that allows test code to control the store.
The test code can populate the store with Trees and Blobs to return, and can
control when the Futures returned by the store are fulfilled.
Reviewed By: bolinfest
Differential Revision: D4338577
fbshipit-source-id: 79221b04d844bd6011078b799e55182de4ccdfdc
Summary:
Upstream mercurial commit d83ca854fa21bef17bf3f097087c691c7d07bb5b changed the
mercurial.ui API, breaking existing scripts using it.
ui objects now have to be created using the ui.load() classmethod, rather than
by constructing them directly. (Creating a ui object directly no longer loads
the global or user configs.)
Reviewed By: bolinfest
Differential Revision: D4354629
fbshipit-source-id: 650db3abab9f2eebaefa2b4d8029c169ccc30217
Summary:
This is a first diff to remove the "easiest" unused includes in fbcode.
* For safety, we only touch .cpp files without #if and #define,
* We do not try to remove redundant systems headers (aka. "packing").
The diff was generated as follows:
```
foundation/scripts/ls-cpp-dirs | grep -v '^\(\.\.\|external/\|.*/external\)' | xargs ffmr -o /tmp/ffmr-diff-1 codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
cat /tmp/ffmr-diff-1/*.diff | patch -p2
hg commit -m something
arc diff --prepare --nolint --nounit --less-context --excuse refactoring
```
Note: `grep -v` is just an optimization. The actual configuration is in these two files:
diffusion/FBS/browse/master/fbcode/codegraph/analysis/config.py
diffusion/FBS/browse/master/fbcode/codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
See the task for more context, and the recent "safety" improvements on the tool.
depends on D4317825 for very few cases where `nolint` had to be manually added.
Reviewed By: igorsugak
Differential Revision: D4312617
fbshipit-source-id: ecc1f0addfd0651fa4770fcc43cd1314661a311a
Summary:
Most of the work for this diff was achieved via:
```
find eden -type f | xargs sed -i -e 's#ThriftHgStatusCode#StatusCode#g'
find eden -type f | xargs sed -i -e 's#HgStatusCode#StatusCode#g'
```
Reviewed By: simpkins
Differential Revision: D4237975
fbshipit-source-id: 2ee04a89101291c8972ac7bd3ff6cca92cbb0799
Summary:
Add a helper function to create human-readable hashes in test code without
having to always specify a full 40-byte long string.
Reviewed By: wez
Differential Revision: D4336161
fbshipit-source-id: cf6af58dd788b5553a2a6daef56db43cddbbc04a
Summary:
Some of the eden/utils test files were already in eden/utils/test, but some
were just in eden/utils. This moves all test code into the test/ subdirectory.
Reviewed By: bolinfest
Differential Revision: D4335334
fbshipit-source-id: 50ef610f5557ed3ec81e5569e5a96c96e6cf15a5
Summary:
EXPECT_THROW_ERRNO() makes sure a std::system_exception is thrown with the
expected errno value.
EXPECT_THROW_RE() is similar to EXPECT_THROW() but also checks that the
exception message matches the specified regular expression pattern.
Reviewed By: wez
Differential Revision: D4335178
fbshipit-source-id: bd4a894325f7e644b14d5e834cbb88e4fe000d34
Summary:
This should make the common case of `hg add` or `hg add .` much more efficient
because it no longer performs a walk of the entire repository from the client
side.
Reviewed By: simpkins
Differential Revision: D4317871
fbshipit-source-id: 7061553fe0de0c4afa84b4f835316965088675e8
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.
This updates the main program to process path arguments using canonicalPath().
Reviewed By: bolinfest
Differential Revision: D4332273
fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
Summary:
Add a canonicalPath() function to accept a non-normalized input string, and
return a normalized AbsolutePath object.
Also added facebook::eden::getcwd() and facebook::eden::realpath() functions,
which are simply wrappers around the standard versions of these functions, but
return AbsolutePath objects. (Let me know if you prefer renaming these to
something else rather than shadowing the names of the same functions in the
global namespace.)
Reviewed By: bolinfest
Differential Revision: D4332266
fbshipit-source-id: cf89eca16f1f0be2acfce3cb8b3c763312f5c4bc
Summary:
This code compiled fine with clang, but gcc apparently requires explicitly
specifying "this->" when referring to members inside a lambda body.
Reviewed By: bolinfest
Differential Revision: D4332357
fbshipit-source-id: 3919dad4fca0b6676731f21781331be73fde8e55
Summary:
The default mode of detection assumes that the test runner par
will be the one from the eden dir.
In the watchman<->eden integration tests this is not the case, so we need
to provide a way to pass down the correct location.
The watchman tests (see D4323716) are using a TARGETS fragment like this:
```
eden_env = env.copy()
eden_env.update({
'EDENFS_CLI_PATH': '$(location @/eden/fs/cli:cli)',
'EDENFS_SERVER_PATH': '$(location @/eden/fs/service:edenfs)',
})
python_unittest(...
env=eden_env,
...)
```
We may want to cut eden over to a similar construction, but I've left
that alone for the moment.
Reviewed By: simpkins
Differential Revision: D4323695
fbshipit-source-id: f48afa6eef82a806c7b10e187d41e6253039d612
Summary:
Fix Dispatcher::symlink() to accept the symlink contents as a StringPiece
rather than a PathComponentPiece. symlink contents can be any arbitrary
string, and are not required to be a valid, normalized path name.
Reviewed By: wez
Differential Revision: D4325380
fbshipit-source-id: 88448bee50ea192c06442dc70042c7d17d49a12f
Summary:
Update EdenServiceHandler::getSHA1ForPath() to replace its own custom path
lookup code with EdenMount::getFileInode().
This also ended up fixing the error message to correctly include the path name
on EISDIR errors.
Reviewed By: wez
Differential Revision: D4325066
fbshipit-source-id: 9aa3932c71c33e6bc11d2c71cc8f1badb4c0dcb7
Summary:
InodeError is a subclass of std::system_error that accepts an InodePtr to the
inode that it refers to. This makes it easier to construct error objects that
retain information about the inode they refer to.
InodeError also avoids computing the inode path until the error message is
actually needed. This should make it less expensive in cases where errors are
thrown and handled internally without ever using the human-readable error
message. It is possible that the file may have been renamed or unlinked by the
time the error message is computed. However, this race condition might still
exist even if we computed the path at the time when the error is constructed.
getLogPath() will construct a usable human-readable string even if the file has
been unlinked.
Reviewed By: wez
Differential Revision: D4325043
fbshipit-source-id: c9683a80b022f281ca4583a9b7f73b15277335bb
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.
For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready. I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.
Reviewed By: bolinfest
Differential Revision: D4318055
fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
Summary:
This will make it easier to implement `hg add <directory>` such that
`<directory>` is expanded on the server rather than on the client.
Reviewed By: simpkins
Differential Revision: D4318735
fbshipit-source-id: cf0e89bd95eb58304cd23e70beb77bc7151f2c5c
Summary:
Previously, `.hg` entries were filtered when calling `hg status` on the client,
but this should be happening on the server. This updates the logic in
`Dirstate.cpp` to properly exclude `.hg` when traversing the overlay for
modified directories, so this will eliminate a bunch of unnecessary computation
and simplify the client.
I'm unsure of how best to implement the ownership relation for the set that
contains `.hg`. Please advise! I know that I could categorically exclude
`".hg"` in `getModifiedDirectoriesRecursive()`, but I haven't used it enough
scenarios yet to be sure that's the right thing to do. For example, if it were
a Git repo, arguably we should consider `.hg` and not `.git`, so I could also
require the set to be a parameter of `Dirstate`, but I want to make sure I get
the ownership stuff right.
Reviewed By: simpkins
Differential Revision: D4316531
fbshipit-source-id: a0f13ca1c3c620b686435c8aa6485ba4e850f043
Summary: These should have been added as part of D4270526.
Reviewed By: simpkins
Differential Revision: D4315382
fbshipit-source-id: 5920ff38f9cc63540e4813e8ab40f79ad46f9ec1
Summary: Follow-up on a comment that came out of the review for D4249214.
Reviewed By: simpkins
Differential Revision: D4314979
fbshipit-source-id: 76384474092e6fd48394f6faf8b84ba6220c556a
Summary:
Move the FakeObjectStoreTest class into fs/testharness, along with the
TestMount and TestBackingStore classes. This simply consolidates the test
utility code into a single location.
Reviewed By: bolinfest
Differential Revision: D4317517
fbshipit-source-id: 4e19590c5ffde88b66f2c8d4a964352ec349031c
