Summary:
Add VLOG() statements for every FUSE entry point. Turning up the VLOG level in
EdenDispatcher now allows us to have a record of all FUSE calls made using
inode numbers. (Calls involving file handles or directory handles currently go
directly to the handle in question and do not pass through the EdenDispatcher.
We should probably add log statements for those too, but that can wait for a
later diff.)
Reviewed By: bolinfest
Differential Revision: D4359066
fbshipit-source-id: b378d829c39fdb61faf63d2d400f3ff556c376e8
Summary:
This member variable was not updated properly when files were renamed.
InodeBase now tracks our parent properly, so we don't need our own copy.
This does still call getParentBuggy() (which does not perform proper locking)
in a couple places for performing overlay operations. We'll need to fix this
later when addressing other overlay concurrency handling issues.
Reviewed By: bolinfest
Differential Revision: D4348481
fbshipit-source-id: 19c1ffced6f63e1ff041d0bab2363fecdb93d5a3
Summary:
Have FileData objects store a pointer to the FileInode that owns them, rather
than just to the EdenMount.
FileData objects still store a direct pointer to their FileInode's mutex_ and
entry_. It's potentially worth just accessing these through inode_ in the
future.
Reviewed By: bolinfest
Differential Revision: D4348103
fbshipit-source-id: 1f8497979bfc89c6a192ca0195209335db0d911c
Summary:
Add an EdenMount* member variable to InodeBase. Previously each TreeInode kept
a pointer to its EdenMount, and this moves that into the InodeBase class.
This is needed to support upcoming diffs that will change InodeBase memory
management. InodeBase objects will be responsible for notifying the InodeMap
when they become unreferenced.
Reviewed By: bolinfest
Differential Revision: D4348079
fbshipit-source-id: 6bf7ea908c6096aa2bca5b21290c09cbd58d5af7
Summary:
This updates all of the eden code to use the new InodeMap class. This replaces
the InodeNameManager class and the unordered_map previously stored in the
EdenDispatcher.
Reviewed By: bolinfest
Differential Revision: D4325750
fbshipit-source-id: d80ae7581ba79ca2b63155e184995a3e83e85dc1
Summary:
This diff starts adding a new InodeMap class. This class will eventually
consolidate the functionality of InodeNameMap plus the inode map stored in
EdenDispatcher.
This new class will bring several new benefits:
- All inode mapping logic consolidated into a single class, with a single lock
protecting the maps.
- A well-defined model for loaded vs unloaded inodes. InodeMap explicitly
tracks inodes that have InodeBase objects created for them vs inodes that
have an inode number allocated (for FUSE) but do not have an InodeBase object
in memory. This will make it possible to unload Inode objects on demand to
reduce memory usage.
- Tracking of pending loads, and de-duplicating load requests. This ensures
that only one Inode object ever exists for a given inode number / path. If a
second request to load an Inode arrives when a previous load request is still
in progress, InodeMap deals with this situation properly.
- Better support for using Inode objects without FUSE. With the old code,
attempts to interact with Inode objects without going through the FUSE
dispatch (say, when processing a thrift call) could result in inconsistent
state. New inodes created would not be put into the EdenDispatcher map,
which could result in problems.
- More convenient child inode access from TreeInode. With this change, the
TreeInode class can easily tell which of its children are loaded. This makes
it easier to do tasks which only need to operate on existing loaded inode
state (for instance, dirstate computation).
- Support for saving and loading state, to implement graceful restart..
InodeMap provides a central place to write out inode state on shutdown and
restoring it on startup. Saved inodes can easily be restored to an
"unloaded" state on startup. This code is not implemented yet as part of
this diff, but it should be straightforward to add in future diffs.
Reviewed By: bolinfest
Differential Revision: D4318060
fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
Summary:
This introduces an `HgExtensionTestBase` class that can be subclassed
to create tests for Mercurial integration with Eden. It also introduces
`status_test.py`, which includes some basic integration tests for `hg status`
in Eden. Although our existing `.cpp` tests give us a lot of coverage, they
do not verify the logic in the Python code for our Mercurial extension,
so this will help me sleep better at night.
Note that `HgExtensionTestBase` is a subclass of `testcase.EdenHgTest`
rather than using the `testcase.eden_repo_test` decorator because these
tests will only be used in the Mercurial case for now, so there is no point in
trying to run them when creating the mount from a Git repo, as well.
Further note that this also required a small change to `eden/fs/integration/lib/testcase.py`
because the Eden directory was not being created at `$HOME/local/.eden`, which
is something that the Mercurial extension relies on.
Reviewed By: simpkins
Differential Revision: D4337660
fbshipit-source-id: ea825134420ae6eb794d37b6c75da19a6aadaae5
Summary:
This preserves the existing behavior while making `REPO_ROOT` and `BUCK_OUT`
available to other test code.
Reviewed By: simpkins
Differential Revision: D4358875
fbshipit-source-id: a7b2cafe7e56673831beac59a85fcab436e637a5
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.
This revision was initially created by running:
```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```
And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.
But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.
Reviewed By: simpkins
Differential Revision: D4356724
fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
Summary:
Rename the existing TestBackingStore class to FakeBackingStore, and fill it out
with an implementation that allows test code to control the store.
The test code can populate the store with Trees and Blobs to return, and can
control when the Futures returned by the store are fulfilled.
Reviewed By: bolinfest
Differential Revision: D4338577
fbshipit-source-id: 79221b04d844bd6011078b799e55182de4ccdfdc
Summary:
Upstream mercurial commit d83ca854fa21bef17bf3f097087c691c7d07bb5b changed the
mercurial.ui API, breaking existing scripts using it.
ui objects now have to be created using the ui.load() classmethod, rather than
by constructing them directly. (Creating a ui object directly no longer loads
the global or user configs.)
Reviewed By: bolinfest
Differential Revision: D4354629
fbshipit-source-id: 650db3abab9f2eebaefa2b4d8029c169ccc30217
Summary:
This is a first diff to remove the "easiest" unused includes in fbcode.
* For safety, we only touch .cpp files without #if and #define,
* We do not try to remove redundant systems headers (aka. "packing").
The diff was generated as follows:
```
foundation/scripts/ls-cpp-dirs | grep -v '^\(\.\.\|external/\|.*/external\)' | xargs ffmr -o /tmp/ffmr-diff-1 codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
cat /tmp/ffmr-diff-1/*.diff | patch -p2
hg commit -m something
arc diff --prepare --nolint --nounit --less-context --excuse refactoring
```
Note: `grep -v` is just an optimization. The actual configuration is in these two files:
diffusion/FBS/browse/master/fbcode/codegraph/analysis/config.py
diffusion/FBS/browse/master/fbcode/codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
See the task for more context, and the recent "safety" improvements on the tool.
depends on D4317825 for very few cases where `nolint` had to be manually added.
Reviewed By: igorsugak
Differential Revision: D4312617
fbshipit-source-id: ecc1f0addfd0651fa4770fcc43cd1314661a311a
Summary:
Most of the work for this diff was achieved via:
```
find eden -type f | xargs sed -i -e 's#ThriftHgStatusCode#StatusCode#g'
find eden -type f | xargs sed -i -e 's#HgStatusCode#StatusCode#g'
```
Reviewed By: simpkins
Differential Revision: D4237975
fbshipit-source-id: 2ee04a89101291c8972ac7bd3ff6cca92cbb0799
Summary:
Add a helper function to create human-readable hashes in test code without
having to always specify a full 40-byte long string.
Reviewed By: wez
Differential Revision: D4336161
fbshipit-source-id: cf6af58dd788b5553a2a6daef56db43cddbbc04a
Summary:
This should make the common case of `hg add` or `hg add .` much more efficient
because it no longer performs a walk of the entire repository from the client
side.
Reviewed By: simpkins
Differential Revision: D4317871
fbshipit-source-id: 7061553fe0de0c4afa84b4f835316965088675e8
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.
This updates the main program to process path arguments using canonicalPath().
Reviewed By: bolinfest
Differential Revision: D4332273
fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
Summary:
This code compiled fine with clang, but gcc apparently requires explicitly
specifying "this->" when referring to members inside a lambda body.
Reviewed By: bolinfest
Differential Revision: D4332357
fbshipit-source-id: 3919dad4fca0b6676731f21781331be73fde8e55
Summary:
The default mode of detection assumes that the test runner par
will be the one from the eden dir.
In the watchman<->eden integration tests this is not the case, so we need
to provide a way to pass down the correct location.
The watchman tests (see D4323716) are using a TARGETS fragment like this:
```
eden_env = env.copy()
eden_env.update({
'EDENFS_CLI_PATH': '$(location @/eden/fs/cli:cli)',
'EDENFS_SERVER_PATH': '$(location @/eden/fs/service:edenfs)',
})
python_unittest(...
env=eden_env,
...)
```
We may want to cut eden over to a similar construction, but I've left
that alone for the moment.
Reviewed By: simpkins
Differential Revision: D4323695
fbshipit-source-id: f48afa6eef82a806c7b10e187d41e6253039d612
Summary:
Fix Dispatcher::symlink() to accept the symlink contents as a StringPiece
rather than a PathComponentPiece. symlink contents can be any arbitrary
string, and are not required to be a valid, normalized path name.
Reviewed By: wez
Differential Revision: D4325380
fbshipit-source-id: 88448bee50ea192c06442dc70042c7d17d49a12f
Summary:
Update EdenServiceHandler::getSHA1ForPath() to replace its own custom path
lookup code with EdenMount::getFileInode().
This also ended up fixing the error message to correctly include the path name
on EISDIR errors.
Reviewed By: wez
Differential Revision: D4325066
fbshipit-source-id: 9aa3932c71c33e6bc11d2c71cc8f1badb4c0dcb7
Summary:
InodeError is a subclass of std::system_error that accepts an InodePtr to the
inode that it refers to. This makes it easier to construct error objects that
retain information about the inode they refer to.
InodeError also avoids computing the inode path until the error message is
actually needed. This should make it less expensive in cases where errors are
thrown and handled internally without ever using the human-readable error
message. It is possible that the file may have been renamed or unlinked by the
time the error message is computed. However, this race condition might still
exist even if we computed the path at the time when the error is constructed.
getLogPath() will construct a usable human-readable string even if the file has
been unlinked.
Reviewed By: wez
Differential Revision: D4325043
fbshipit-source-id: c9683a80b022f281ca4583a9b7f73b15277335bb
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.
For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready. I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.
Reviewed By: bolinfest
Differential Revision: D4318055
fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
Summary:
This will make it easier to implement `hg add <directory>` such that
`<directory>` is expanded on the server rather than on the client.
Reviewed By: simpkins
Differential Revision: D4318735
fbshipit-source-id: cf0e89bd95eb58304cd23e70beb77bc7151f2c5c
Summary:
Previously, `.hg` entries were filtered when calling `hg status` on the client,
but this should be happening on the server. This updates the logic in
`Dirstate.cpp` to properly exclude `.hg` when traversing the overlay for
modified directories, so this will eliminate a bunch of unnecessary computation
and simplify the client.
I'm unsure of how best to implement the ownership relation for the set that
contains `.hg`. Please advise! I know that I could categorically exclude
`".hg"` in `getModifiedDirectoriesRecursive()`, but I haven't used it enough
scenarios yet to be sure that's the right thing to do. For example, if it were
a Git repo, arguably we should consider `.hg` and not `.git`, so I could also
require the set to be a parameter of `Dirstate`, but I want to make sure I get
the ownership stuff right.
Reviewed By: simpkins
Differential Revision: D4316531
fbshipit-source-id: a0f13ca1c3c620b686435c8aa6485ba4e850f043
Summary: These should have been added as part of D4270526.
Reviewed By: simpkins
Differential Revision: D4315382
fbshipit-source-id: 5920ff38f9cc63540e4813e8ab40f79ad46f9ec1
Summary: Follow-up on a comment that came out of the review for D4249214.
Reviewed By: simpkins
Differential Revision: D4314979
fbshipit-source-id: 76384474092e6fd48394f6faf8b84ba6220c556a
Summary:
Move the FakeObjectStoreTest class into fs/testharness, along with the
TestMount and TestBackingStore classes. This simply consolidates the test
utility code into a single location.
Reviewed By: bolinfest
Differential Revision: D4317517
fbshipit-source-id: 4e19590c5ffde88b66f2c8d4a964352ec349031c
Summary:
Hash objects are small enough (20 bytes) that it isn't worth allocating them on
the heap. This updates LocalStore::getSha1ForBlob() to return a
folly::Optional<Hash>, and ObjectStore::getSha1ForBlob() to return a plain
Hash.
Reviewed By: bolinfest
Differential Revision: D4298162
fbshipit-source-id: 9cf54f2997ba8c3b2346db315a2aca41e580b078
Summary:
Add comments in ObjectStore.h documenting the fact that the get* APIs all throw
std::domain_error when the specified ID does not exist, and never return
nullptr.
Also update the FakeObjectStore class used for testing to follow this behavior.
Reviewed By: bolinfest
Differential Revision: D4298160
fbshipit-source-id: c5509bb3aa2ed76619b06b733ad240aaa5f00862
Summary:
In addition to storing the SHA-1 of each file's contents, also store the size.
This will allow us to more quickly look up the file size, without having to
retreive the file size.
I haven't yet added an API to ObjectStore to retreive the full BlobMetadata
object; I will do that in a subsequent diff. One benefit for now is that this
does avoid double-computing the SHA-1 in ObjectStore::getSha1ForBlob() if we
had to load the blob.
Reviewed By: bolinfest
Differential Revision: D4298157
fbshipit-source-id: 4d83ebfa631c93fcef06ca1cd0ba0e1a70a2476d
Summary:
Define InodePtr, TreeInodePtr, and FileInodePtr as aliases for std::shared_ptr
of the underlying inode type. This also updates all of the code to use these
new type names.
This will make it easier swap out std::shared_ptr with a custom pointer type in
the future. (I believe we will need a custom type in the future so that we
can have more precise control of the reference counting so we can load and
unload Inode objects on demand. std::shared_ptr::unique() doesn't quite
provide the flexibility we need, and is also being deprecated in C++17.)
Reviewed By: bolinfest
Differential Revision: D4297791
fbshipit-source-id: 1080945649290e676f62689592159f1166159b20
Summary:
This updates the InodeBase code to track its location in the filesystem.
Since we do not support hard links, each inode has a single path where it
exists.
Tracking this data allows us to implement getPath() as a method of InodeBase.
This code is not really complete yet, but it seems worth getting the current
code in as-is. The location data is not updated properly on unlinks or
renames, but it looks like the existing InodeNameManager code does not get
updated either. I am working on some additional refactoring of inode object
management, and it will be easier to come back and fix the unlink and rename
handling after this refactoring is further along.
Reviewed By: bolinfest
Differential Revision: D4297591
fbshipit-source-id: 82ceb326e4f9c376f627b1d8f49bb7db3cfc2b0b
Summary:
In this revision, we override `committablectx.markcommitted()` to make a Thrift
call to Eden to record the new commit. For now, this defers the need for us to
implement `edendirstate.normal()`, though we expect we will have to provide a
proper implementation at some point.
Because `hg update` is not implemented yet, this puts us in a funny state where
we have to restart eden after `hg commit` to ensure all of our `TreeEntry` and
other in-memory data structures are in the correct state.
Reviewed By: simpkins
Differential Revision: D4249214
fbshipit-source-id: 8ec06dfee67070f008dd93a0ee6c810ce75d2faa
Summary:
This refines the initial support for `hg remove` by adding support for
directories.
Reviewed By: simpkins
Differential Revision: D4270546
fbshipit-source-id: c97dfea555ad489ddda01ad2587f1856b1953e02
Summary:
`getEntryForPath()` is helpful when we don't know (or care) whether the path
corresponds to a file or a directory.
Reviewed By: simpkins
Differential Revision: D4270526
fbshipit-source-id: 6dd2f76df1749e040dda788a74055d9da2156a4d
Summary:
This adds support for `hg remove <file>` in Eden and sets up some of the scaffolding
to support `hg remove <directory>`.
Note that the Thrift API for `scmRemove()` is slightly different than that of `scmAdd()`
in that it returns a list of error messages to display to the user rather than throwing
an exception. In practice, for batch operations, Mercurial will allow some operations
to succeed while others may fail, so it is possible to have multiple error messages to
return.
Unlike the current implementation of `hg add`, this does the directory traversal
on the server rather than on the client. Once we work out how to do this for
`hg remove`, we should figure out how to reuse the logic for `hg add`.
Reviewed By: simpkins
Differential Revision: D4263068
fbshipit-source-id: d084774d562c48c59664f313eba229d4197929fe
Summary:
The check on the type of an exception was inverted, which meant
`hg remove <path>` would throw an exception if the parent directory of `path`
did not exist. This is not correct because the user should be able to expect
to do:
```
mkdir -p /tmp/example
cd /tmp/example
hg init
mkdir mydir
touch mydir/a
hg add mydir/a
rm -rf mydir/a
hg rm mydir/a
```
In this scenario, `mydir` does not exist when `hg rm` is called, but the command
should succeed, making `mydir/a` no longer tracked.
Reviewed By: simpkins
Differential Revision: D4268451
fbshipit-source-id: 517d81252aa8e4b6bd1a32dece14776a9f7dd6f7
Summary:
A `TreeEntry` reports a `mode_t` whose "group" and "other" bits are set in a way
that reflects the "owner" bits (so they are non-zero). By comparison, the `mode`
of a `TreeInode::Entry` will reflect the permissions on disk in the overlay (if
the file is materialized). In general, the overlay bits will probably be the
same as those in the `TreeEntry` since we expect the user will rarely mess
with the "group" and "other" bits, but we're seeing a difference more often
right now because of t14953681.
Reviewed By: simpkins
Differential Revision: D4298214
fbshipit-source-id: 2919be94c6bba61135838ee86bbc68aa4031af7c
Summary:
Move the InodeBase class from the lower-level fusell code up to the
eden/fs/inodes layer, now that everything else that uses it is in
eden/fs/inodes.
I plan to start changing the ownership model of inode objects a bit, and this
will allow the InodeBase class to interact with EdenDispatcher and other
classes in eden/fs/inodes.
Reviewed By: bolinfest
Differential Revision: D4283392
fbshipit-source-id: 9e1d6fb81dc223f905847cbe8d165a40ad0aca4d
Summary:
`stdin`, `stdout` and `stderr` are macros that expand to function calls with the MSVC CRT implementation. This is also the case for musl-libc. This means that Subprocess simply cannot be compiled on those platforms without changing the API.
To solve that, we change the API and deprecate the old API.
For more fun, `stdin`, `stdout` and `stderr` are also macros in glibc, they just expand to other identifiers rather than a function call.
Reviewed By: yfeldblum
Differential Revision: D4229544
fbshipit-source-id: 97f1a3b228b83cfdcaffee56d729063ea235e608
Summary:
Update the TestMount APIs to use the term "TreeInode" instead of "DirInode" now
that the DirInode class is no more.
Reviewed By: bolinfest
Differential Revision: D4257171
fbshipit-source-id: 2407dfc25405f7161987260c0299f1723831e264
Summary:
Now that the EdenDispatcher class has been moved into eden/fs, we no longer
need the distinction between TreeInode and fusell::DirInode, and FileInode and
fusell::FileInode.
This diff deletes the fusell versions of these classes, and updates all of the
code to always directly use TreeInode and FileInode. This allows us to get rid
of the remaining dynamic_casts between these pairs of classes.
Reviewed By: bolinfest
Differential Revision: D4257165
fbshipit-source-id: e2b6f328b9605ca0e2882f5cf7a3983fb4470cdf
Summary:
Move the InodeDispatcher class out of the lower-level fusell namespace in
eden/fuse and into the higher-level eden code in eden/fs/inodes. I also
renamed it from InodeDispatcher to EdenDispatcher, in anticipation of it
getting more eden-specific functionality in the future.
The fusell::MountPoint class is now independent of the Dispatcher type, and can
work with any Dispatcher subclass. Previously the MountPoint class was
responsible for owning the InodeDispatcher object. Now its caller (EdenMount
in our case) is responsible for supplying a Dispatcher object that is owned
externally.
Several parts of EdenDispatcher had to be updated as a result of the namespace
move, but I tried to keep this change somewhat minimal. I did update it from
using fusell::DirInode and fusell::FileInode to eden's TreeInode and FileInode
classes directly. However, there still remains more clean-up work to do. I
will split remaining changes out into upcoming diffs.
Reviewed By: bolinfest
Differential Revision: D4257163
fbshipit-source-id: dc9c2526640798f9f924ae2531218ba2c45d1d0a
Summary:
Drop the MountPoint::setRootInode() method, and have EdenMount perform the
operation directly on the InodeDispatcher.
Reviewed By: bolinfest
Differential Revision: D4257158
fbshipit-source-id: af4a696de2d36979c658972104361f225f482338
Summary:
Update call sites in eden/fs to access the InodeNameManager through the
EdenMount object rather than the MountPoint.
It turns out that there was only one call site in TreeInode, and all other
callers in eden/fs get it indirectly via TreeInode::getNameMgr().
Reviewed By: bolinfest
Differential Revision: D4257156
fbshipit-source-id: 9f0212134b20c8dd8943827c17aa16ee7274bc36
Summary:
Move getInodeBaseForPath(), getDirInodeForPath(), and getFileInodeForPath()
entirely into the EdenMount class, and make sure all call sites are using the
EdenMount methods rather than the MountPoint methods.
Reviewed By: bolinfest
Differential Revision: D4257153
fbshipit-source-id: d528cfad174757c3c9f23e62a0616f8bf1976da7
Summary:
Update all code in eden/fs to call EdenMount::getDispatcher() instead of
getting the underlying MountPoint from the EdenMount and then calling
getDispatcher() on it. This will allow me to move the InodeDispatcher from
MountPoint to EdenMount in a subsequent diff. This also simplifies many of the
callers of this method.
Additionally, add an EdenMount::getRootInode() method, and update call sites to
use this rather than having to look up the InodeDispatcher and call
getRootInode() or getDirInode(FUSE_ROOT_ID) on it.
Reviewed By: bolinfest
Differential Revision: D4257152
fbshipit-source-id: 33e6f6b8853db2a88f4f2c221122eea50e796390
Summary:
This updates the Dirstate to also check if untracked files are ignored or not.
This is somewhat inefficient, as we have perform a separate check for each
untracked file we find. Ideally we should perform all of the dirstate
computation in a single tree walk, and check ignore status as we go. This
would allow us to skip ignored directories entirely, rather than potentially
having to check each file inside them. I intend to work on cleaning this up in
the future, but it will require refactoring some of the inode code first.
Reviewed By: bolinfest
Differential Revision: D4225308
fbshipit-source-id: 49e444c85cbb6ede11cc6e19052fdd16cf8aab9f
Summary:
Update the Dirstate to store a pointer to the EdenMount object that owns it,
rather than storing pointers to the lower-level MountPoint and ObjectStore
objects.
This change is necessary in order for me to move more functionality from
MountPoint to EdenMount. (In particular, I plan to move the InodeDispatcher to
the EdenMount.)
As part of this change I also started moving some APIs from MountPoint to
EdenMount. For now the EdenMount versions are just thin wrappers on top of the
MountPoint APIs. I will move the functionality directly into EdenMount in a
future diff.
Reviewed By: bolinfest
Differential Revision: D4255675
fbshipit-source-id: 93749c6516c3cea4b4ae93de4ca49ddf05f4d260
Summary:
Write the dirstate data using the new folly::writeFileAtomic() function.
This ensures that the dirstate file will always contain full, valid contents,
even if we crash or run out of disk space partway through writing out the data.
This diff also includes a couple other minor tweaks:
- Update Dirstate to store the DirstatePersistence object directly inline,
rather than allocating it separately on the heap.
- Update the DirstatePersistenceTest code to prefix temporary directories with
"eden_test". This just makes it easier to tell if the tests fail to clean up
after themselves for any reason.
Reviewed By: bolinfest
Differential Revision: D4254027
fbshipit-source-id: 6b6601b65aeacdee998a6c4260e972d5fb2426ac
Summary:
This cleans up construction of the EdenMount and Dirstate objects:
- The EdenMount constructor is now responsible for creating the Overlay and
Dirstate objects.
- The Dirstate constructor is now responsible for loading the
DirstatePersistence file.
- The EdenMount now takes ownership of the ClientConfig object, and stores it
for later use.
- The ClientConfig object now has a method to get the path to the
DirstatePersistence file.
- I added a ClientConfig::createTestConfig() method, so that the TestMount code
can now use the same EdenMount constructor as the normal code.
This simplifies the logic in EdenServiceHandler and TestMount, and makes some
of the initialization dependencies a little bit simpler.
This change is necessary in order for me to move some logic from
fusell::MountPoint into EdenMount. The Dirstate object will need a pointer
back to its EdenMount object, and this diff enables that.
Reviewed By: bolinfest
Differential Revision: D4249393
fbshipit-source-id: 439786accbf48c8696dbc6ca4fe77a4c6bdeab65
Summary:
Rename TreeEntryFileInode to FileInode, and TreeEntryFileHandle to FileHandle.
These class names were long and awkward.
It's slightly unfortunate that we now have classes named both
eden::fuse::FileInode and eden::fuse::fusell::FileInode, but I don't believe
this should cause any major problems. If we want to eliminate these name
collisions in the future I would advocate for renaming the fusell versions to
something like "FileInodeIface".
Reviewed By: bolinfest
Differential Revision: D4217909
fbshipit-source-id: 899672a318d7ae39595f2c18e171f8fd6cebedc6
Summary:
Previous to this commit, the `Dirstate` logic only worked correctly when the
changes occurred in the root directory. Obviously that is very limiting, so this
adds support for changes in arbitrary directories at arbitrary depths.
This also introduces support for things like a file being replaced by a
directory of same name or vice versa. The tests have been updated to verify
these cases.
One interesting design change that fell out of this was the addition of the
`removedDirectories` field to the `DirectoryDelta` struct. As you can see,
all entries in a removed directory need to be processed by the new
`addDeletedEntries()` function. These require special handling because deleted
directories do not show up in the traversal of modified directories.
In contrast, new directories do show up in the traversal, so they require a
different type of special handling. Specifically, this call will return `NULL`:
```
auto tree = getTreeForDirectory(directory, rootTree.get(), objectStore);
```
When this happens, we must pass an empty vector of tree entries to
`computeDelta()` rather than `&tree->getTreeEntries()`. Admittedly, the special
case of new directories is much simpler than the special case of deleted ones.
Reviewed By: simpkins
Differential Revision: D4219478
fbshipit-source-id: 4c805ba3d7688c4d12ab2ced003a7f5c19ca07eb
Summary:
This is a better fix for the quick fix introduced by D4198939.
It turns out that the `EdenMount` does not need to take ownership
of the `ClientConfig`, so removing the `std::move()` makes this code
much simpler because instead of declaring a bunch of variables
early in `mountImpl()` so that we can "hold on" to them before `EdenMount`
takes ownership of the `ClientConfig`, we can declare them closer to where they
are actually used.
Note that we may want `EdenMount` to actually take ownership of the
`ClientConfig` in the future, but we'll cross that bridge when we come to it.
Reviewed By: simpkins
Differential Revision: D4199000
fbshipit-source-id: 67411a9a5ef630a9d481aebc94631c79da4ab2c4
Summary:
This also introduces the change where the `EdenMount` creates
and takes ownership of the `Dirstate`.
To clean some of this up, I had to expose a `getEdenDir()` method on `EdenServer`
that returns an `AbsolutePathPiece`. This was previously stored internally as a
`std::string`, so I had to clean up a bunch of path construction that was using `edenDir_`.
Reviewed By: simpkins
Differential Revision: D4123763
fbshipit-source-id: 270b182521c1a84bb054832f4b5f92af849d67e4
Summary:
Previously, `Dirstate` took a `std::shared_ptr<EdenMount>`, but now it takes
pointers to a `MountPoint` and an `ObjectStore` because it does not need the
entire `EdenMount`. Ultimately, this will enable us to have `EdenMount` create
the `Dirstate` itself, but that will be done in a follow-up commit.
Fortunately, it was pretty easy to remove the references to `edenMount_` in
`Dirstate.cpp` and rewrite them in terms of `mountPoint_` or `objectStore_`.
The one thing that I also decided to move was `getModifiedDirectoriesForMount()`
because I already needed to create an `EdenMounts` file (admittedly not a
great name) to collect some utility functions that use members of an `EdenMount`
while not having access to the `EdenMount` itself.
As part of this change, all of the code in `eden/fs/model/hg` has been moved to
`eden/fs/inodes` so that it is alongside `EdenMount`. We are going to change
the `Dirstate` from an Hg-specific concept to a more general concept.
`LocalDirstatePersistence` is no longer one of two implementations of
`DirstatePersistence`. (The other was `FakeDirstatePersistence`.) Now there is
just one concrete implementation called `DirstatePersistence` that takes its
implementation from `LocalDirstatePersistence`. Because there is no longer a
`FakeDirstatePersistence`, `TestMount` must create a `DirstatePersistence` that
uses a `TemporaryFile`.
Because `TestMount` now takes responsibility for creating the `Dirstate`, it
must also give callers the ability to specify the user directives. To that end,
`TestMountBuilder` got an `addUserDirectives()` method while `TestMount` got a
`getDirstate()` method. Surprisingly, `TestMountTest` did not need to be updated
as part of this revision, but `DirstateTest` needed quite a few updates
(which were generally mechanical).
Reviewed By: simpkins
Differential Revision: D4230154
fbshipit-source-id: 9b8cb52b45ef5d75bc8f5e62a58fcd1cddc32bfa
Summary: This is an implementation of DirstatePersistence that persists data to a local file.
Reviewed By: simpkins
Differential Revision: D4181868
fbshipit-source-id: 7177b2ef67cd3aec56e5ad10f41169cc5ec69d81
Summary:
Update the gitignore handling code to perform pattern matching the same way git
does. Previously the code just called the standard fnmatch() function, which
does not handle "**" in patterns the same way git does.
This includes our own new implementation of glob pattern matching. I did
evaluate several other options before writing our own implementation here:
- The wildmatch() code used by git (and watchman, and rsync) has a few
downsides: it is not distributed by itself as a library anywhere else.
Therefore we would probably have to include a copy of this code in our
repository. Making another copy is unfortunate, and somewhat undesirable
from a legal and licensing perspective. This code also only works with
nul-terminated strings, and our code deals primarily with non-terminated
StringPiece objects.
- I did look at translating glob patterns in to regular expressions and using
re2 to perform matching. Unfortunately re2 turns out to be substantially
slower than wildmatch() for typical gitignore patterns.
This new implementation performs some preprocessing on the glob pattern, and
generates a pattern opcode buffer. Eden can perform this glob preprocessing
when it first loads a .gitignore file, and can then save and re-use this result
each time it needs to match a filename. Doing this preprocessing allows
matching to be done 50% to 100% faster than wildmatch() for typical glob
patterns.
Reviewed By: bolinfest
Differential Revision: D4194573
fbshipit-source-id: 46bc6a61b6d8066f4bbdb5d3e74265a3e72e42cc
Summary:
This adds some initial code for handling gitignore files.
I did check to see if there were APIs from libgit2 that we could leverage for
this, but it does not look like we can easily use their functionality. The
libgit2 ignore code seems to tightly coupled with their repository data
structures, and it requires that you actually have a git repository.
This code isn't quite 100% compatible with git's semantics yet. In particular:
- For now we are just using fnmatch() to do the matching. This is currently
inefficient as we have to do string allocations on each match attempt. This
also doesn't quite match git's behavior, particularly with regard to "**"
inside patterns.
- The code currently does not have a mechanism for indicating if a path refers
to a directory or not, so trailing slashes in the pattern are not honored
correctly.
We will probably need to implement our own fnmatch-like function in the future
to solve these issues.
Reviewed By: bolinfest
Differential Revision: D4156480
fbshipit-source-id: 8ceaefd3805358ae2edc29bfc316e5c8f2fb7d31
Summary:
For now, initial state is represented by a
`std::unordered_map<RelativePath, HgUserStatusDirective>`.
Reviewed By: simpkins
Differential Revision: D4123461
fbshipit-source-id: 83a99e1f504dd1efca1bc1ed33cbc3f116787a80
Summary:
This adds the logic to power `hg rm`. There are comprehensive tests that attempt to cover
all of the edge cases.
This evolved to become a complex change because I realized that I needed to change
my internal representation of the dirstate to implement it properly. Specifically, we now maintain
a map (`userDirectives`) of files that have been explicitly scheduled for change via `hg add` or `hg rm`.
To compute the result of `hg status`, we find the changes between the manifest/root tree
and the overlay and also consult `userDirectives`. `Dirstate::getStatus()` was updated
considerably as part of this commit due to the introduction of `userDirectives`.
As such, `Dirstate::remove()` must do several things:
* Defend the integrity of the dirstate by throwing appropriate exceptions for invalid inputs.
* Delete the specified file, if appropriate.
* Update `userDirectives`, if appropriate.
Although `Dirstate::add()` was not the focus of this commit, it also had to be updated to
match the pattern introduced by `Dirstate::remove()`.
Some important features that are still not supported are:
* Handling ignored files correctly.
* Storing copy/move information.
Reviewed By: simpkins
Differential Revision: D4104503
fbshipit-source-id: d5d45a279e16ded584c6cd4d528ba92d2c8e2993
Summary:
D4014598 changed this line but didn't change the test expectations.
Since it seems desirable for the mount point name to be used, I've reverted
back to the prior state for this line.
Reviewed By: bolinfest
Differential Revision: D4202265
fbshipit-source-id: bddc01436e0a5921a3b0b2c01c0fd2c32f5f1960
Summary:
Originally, D3858635 was going to introduce a scheme for hooks where the
repo type was included in the path:
/etc/eden/hooks/hg/post-clone <args...>
But over the course of the review, we decided to make the repo type a
parameter:
/etc/eden/hooks/post-clone hg <other-args...>
Unfortunately, `generate-hooks-dir` was not updated as part of that
change and it is not covered by unit tests. This error was particularly hard
to discover because of how `ENOENT` is handled, so I added a log statement for
that.
Reviewed By: simpkins
Differential Revision: D4200277
fbshipit-source-id: ffffd871cd78dcaeb717be8f1e01893ce9643a47
Summary:
This is a quick and dirty fix for this issue that was causing
and confusing bug where the memory for the `AbsolutePathPiece`
was getting reclaimed, so when it was later read as the value
for a path, it failed because it was binary garbage.
This is mainly caused by the `std::move(config)` that passes
the `ClientConfig` to the `EdenMount` constructor. I will do
some more general cleanup for that in a follow-up revision,
but I wanted to have this change in its own commit that makes
it clear where the failure/fix were coming from.
Reviewed By: simpkins
Differential Revision: D4198939
fbshipit-source-id: 19e0423a1bee924fa6cc2edc8bae534ef472c988
Summary:
Use new, less confusing names for mentioned thrift methods.
Codemod with 'Yes to All'. Reverted changes in thrift/
Reviewed By: yfeldblum
Differential Revision: D4076812
fbshipit-source-id: 4962ec0aead1f6a45efc1ac7fc2778f39c72e1d0
Summary:
This is the start of the C++ dirstate implementation. It's possible that this
commit does too many things at once:
* Introduces `Dirstate` type.
* Includes logic for serializing/deserializing the dirstate's data so that it persists across Eden restarts.
* Includes logic for basic `hg add` calls.
* Includes unit tests where we model Eden usage via the TestMount utility.
I'm backing this up in Phabricator with `--plan-changes` to start until I get
some basic `hg add` functionality working end-to-end. When that looks good, I'll
determine if/how this should be split into smaller commits.
Reviewed By: wez
Differential Revision: D4023232
fbshipit-source-id: 7fc931d547ccadb34f7caae93bc4eb8f91f6ceb8
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
Reviewed By: simpkins
Differential Revision: D4073653
fbshipit-source-id: dda1d8ea8d29aa071a31f8e2afab324f9109e9b2
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
As you can see, this helped uncover a bug in the way we were
inserting blobs into `LocalStore`.
Reviewed By: simpkins
Differential Revision: D4073039
fbshipit-source-id: 42683fd0bfdb0a1e77df9324fcaa79091f45e83d
Summary: This is a follow-up revision from a comment on D4013464.
Reviewed By: wez
Differential Revision: D4050278
fbshipit-source-id: 1e46526f58a07e1eedd8ace1a6d84a919240d899
Summary:
This is analogous to the existing `getEntryForFile()` helper function that we
have, and I was able to rewrite `getEntryForFile()` in terms of
`getTreeForDirectory()`, which simplifies the code considerably.
Also moved things from `eden/fs/model/hg/misc.h` to
`eden/fs/store/ObjectStores.h`, which is much more appropriate.
Reviewed By: wez
Differential Revision: D4032817
fbshipit-source-id: ff4d32120fb050f8b5c5c53b7f2e94b524781648
Summary:
This is not a one-liner and this is needed for the upcoming `Dirstate` class,
so moving this code to a place where it is more easily reusable.
Reviewed By: simpkins
Differential Revision: D4032001
fbshipit-source-id: 7d8d87802665ac2993ec0a3ac73c5f645fe4a1aa
Summary:
Performs a depth-first traversal of the overlay to find modified
directories and returns them in that order.
Reviewed By: simpkins
Differential Revision: D4025309
fbshipit-source-id: 09d8ed41b250dddbfb3fe545643ec3fd755a430e
Summary:
Now that I've done all this work, I'm not sure whether it is a good idea or even
necessary. I'll keep it in my back pocket.
Reviewed By: simpkins
Differential Revision: D4014598
fbshipit-source-id: 6ded3cc29838e964b56833ac24dff19e9de040f5
Summary:
These are new helper methods we need to create test scenarios.
They will be used in upcoming revisions.
Reviewed By: wez
Differential Revision: D4046981
fbshipit-source-id: 9c66c456be57006173e4a65eed603de4a426a438
Summary: This should be useful for my upcoming unit tests for the Hg dirstate.
Reviewed By: simpkins
Differential Revision: D4013464
fbshipit-source-id: 46460186abfa104aa026894068cd160e52c94729
Summary: This will make it easier to compare a `TreeEntry` with a `TreeInode::Entry`.
Reviewed By: simpkins
Differential Revision: D4034298
fbshipit-source-id: 29674e2902661bf46394ea71b81537b35bd4b107
Summary: This should make some of the upcoming test harness work a little easier.
Reviewed By: simpkins
Differential Revision: D4011747
fbshipit-source-id: 87ee80a6d641a29be9027b163b1adee496f4452f
Summary:
I need this for the upcoming test harness so I can avoid creating a
`ClientConfig`, which is currently a huge pain to do from a unit test.
Reviewed By: simpkins
Differential Revision: D4010842
fbshipit-source-id: 03d1e1de9c3047340a6f26202d4b432f4a8620b4
Summary:
This was reported by ASAN.
The major issue was that `FakeObjectStore` was returning a copy of a `Tree`,
so it was not the case that the `TreeEntry*` returned by `getEntryForFile()`
was guaranteed to be "owned by" the `Tree* root` that was passed in. To address
this, we change `getEntryForFile()` to now return a copy of the `TreeEntry*`
that it gets back from `getEntryPtr()`. It really comes down to this line:
```
auto entry = currentDirectory->getEntryPtr(piece.basename());
```
because we cannot guarantee that `currentDirectory` will live past the end of
`getEntryForFile()`, so we cannot guarantee that return return value of
`currentDirectory->getEntryPtr()` will, either.
Special thanks to meyering and yfeldblum for helping me debug this.
Reviewed By: simpkins
Differential Revision: D4024627
fbshipit-source-id: 6295e6f2b1d2f544271b2aebad27a4ad3ae04563
Summary:
Utility function that given a `Tree` and a `RelativePathPiece`, returns the
corresponding `TreeEntry` in the `ObjectStore`, if it exists.
Reviewed By: wez
Differential Revision: D3980261
fbshipit-source-id: 2808a4ca45be84e3a6bb91b0cf2db19a3bf88798
Summary:
In an upcoming revision, I am going to introduce a utility function that takes
an `ObjectStore` (well, now an `IObjectStore`) as a parameter and I want to be
able to test it. Having a `FakeObjectStore` should make this considerably easier
without having to resort to mocks.
Reviewed By: simpkins
Differential Revision: D3980580
fbshipit-source-id: 5886e2055c893e749cc898226e1baade776c3ea7
Summary:
Adds a very basic example of testing eden functionality with hypothesis.
We'll be building on this with stateful testing in a follow on diff tomorrow.
There's some prep/setup work in the base test class that can be removed when an updated version of hypothesis ships and is updated in our third-party repo.
Reviewed By: simpkins
Differential Revision: D3968250
fbshipit-source-id: 46382c3bf2d6a0edbd60ac2b048b1bae26ca2572
Summary: This is necessary so that it can be used as the key in an `unordered_map`.
Reviewed By: simpkins
Differential Revision: D3980575
fbshipit-source-id: d225a98f957f9aae2f2f50a6cc365011d953c92e
Summary:
Apparently we did not have an existing unit test for `Tree`, so this adds one.
The other methods should be tested, as well, but I'm about to use `getEntryPtr()`
elsewhere, which is why I just focused on this one for the moment.
Reviewed By: simpkins
Differential Revision: D3980150
fbshipit-source-id: 33456fd621a1894606605af4fee06ba42d124752
Summary:
We want to use these with Eden
Depends on D3961190
Depends on D3961193
Depends on D3961196
Depends on D3961208
Reviewed By: rhysparry
Differential Revision: D3961232
fbshipit-source-id: 56f5a1811625303514e4398a6d47ea90ba348724
Summary:
The getMaterializedEntries() would previously try to dereference a null pointer
if the input mount path did not refer to a valid mount piont.
Reviewed By: bolinfest, wez
Differential Revision: D3942600
fbshipit-source-id: 2a8c9aa87d2bd8175f7bc77f3d6293ad25e9c198
Summary:
Add some helper functions for constructing EdenError objects from a few
different types of arguments. Also update eden.thrift to indicate that most
functions can throw EdenErrors on failure.
Reviewed By: bolinfest, wez
Differential Revision: D3942588
fbshipit-source-id: 1b561c5310a8a218f88c38c70499e087fe47bbe0
Summary:
Python 2.x requires the current class name be passed into super().
Add arguments to super so that we can use this inside a mercurial extension.
(Mercurial only supports python 2.x.)
Reviewed By: bolinfest
Differential Revision: D3942573
fbshipit-source-id: 06df55f217631a398004c0d25448d3a612f772e9
Summary:
The keys in the config directory map are normalized, absolute paths to the
mount point. When trying to look up a mount point make sure we also always use
a normalized absolute path.
Reviewed By: bolinfest
Differential Revision: D3942565
fbshipit-source-id: 63db838ffc7139d779925adf07c50f849d73bcc5
Summary:
We were hitting an assertion in the case where we did a `mkdir`
followed by a `rename` followed by `getMaterializedEntries`.
The issue is that our in-memory representation has a boolean to indicate
whether a dir inode is materialized, but our serialization format does
not have this bit. When we loaded the data we were not setting the
field to true and this was caught by the DCHECK.
If we have serialized data for a dir then it is, by definition, materialized
and we should just set that field to true.
Reviewed By: bolinfest
Differential Revision: D3900795
fbshipit-source-id: 62d8281e7a1009056d274888c9aff87664d2e09f
Summary:
This design is inspired by that of Git hooks:
https://git-scm.com/docs/githooks
By default, `/etc/eden/hooks` should be the place where Eden looks for
hooks; however, this can be overridden in `~/.edenrc` on a per-`repository` basis.
This directory should be installed as part of installing Eden.
There is information in `eden/hooks/README.md` about this.
The first hook that is supported is for post-clone logic for a repository.
This change demonstrates the need for an `eden config --get <value>`
analogous to what Git has, as hooks should be able to leverage this in their
own scripts. There introduces a `TODO` in `post-clone.py` where such a
feature would be useful, so that I could add the following to my `~/.edenrc`
to develop the Eden extension for Hg:
```
[hooks]
hg.edenextension = /data/users/mbolin/fbsource/fbcode/eden/hg/eden
[repository fbsource]
path = /data/users/mbolin/fbsource
type = hg
hooks = /data/users/mbolin/eden-hooks
```
Note that this revision also introduces a `generate-hooks-dir` script that can be
used to generate the standard `/etc/eden/hooks` directory that we intend to
distribute with Eden. This is also useful in creating the basis for a custom `hooks`
directory that can be specified as shown above in an `~/.edenrc` file.
Reviewed By: simpkins
Differential Revision: D3858635
fbshipit-source-id: 215ca26379a4b3b0a07d50845fd645b4d9ccf0f2
Summary:
simpkins spotted this; we were passing the wrong path down to the overlay saving dir.
This adds a test to prove that the source and destination directory contents
are correct both immediately after performing the rename and after remounting,
where we just read the serialized data.
Reviewed By: simpkins
Differential Revision: D3888694
fbshipit-source-id: 7f5fb5be417db5c693ac8a07b85abbffdbfe0fff
Summary:
This is pretty straightforward; we just walk back until we hit the
boundary with the requested JournalPosition.sequenceNumber
Reviewed By: simpkins
Differential Revision: D3872970
fbshipit-source-id: 1405f05957346d7ac513070f0407a477548aff1d
Summary:
populate the position from the latest journal delta.
To facilitate this, we also define the mountGeneration value to be a
combination of the pid and the time at which we created the EdenMount object,
as well as a global counter that we bump for each mount.
The precise value and meaning of this bits really doesn't matter, just that we
are unlikely to pick the same value for this same mountPoint path again if we
were to remount in the future.
Since we are now in a position to report JournalPosition values to clients, now
is also a good time to fill out the `currentPosition` field for the
`getMaterializedEntries` thrift call, and to check that this value is
consistent with the value we return via `getCurrentJournalPosition`.
Reviewed By: simpkins
Differential Revision: D3872952
fbshipit-source-id: 2fbc25d2e9711035b66ab1bf5d746507b72de265
Summary:
This just populates the initial snapshot hash in the journal.
The `addDelta` method will propagate this into subsequent deltas if the delta
to be added has hash values that have not been set from the default 0-filled
hash values.
Reviewed By: simpkins
Differential Revision: D3872936
fbshipit-source-id: d0014ded40488a2be04d5a381e1d9815c7f0a638
Summary:
This diff adds a couple more things to our thrift interface:
1. Introduces JournalPosition
2. Adds methods to query the current JournalPosition and obtain a
delta since a given JournalPosition
3. Augments getMaterializedFiles to also return the current JournalPosition
4. Adds a method to evaluate a `glob` against Eden
5. Adds a method using thrift streaming to subscribe to realtime changes
Could probably finesse the naming a little bit.
The JournalPosition allows reasoning about changes to files that are not part
of an Eden snapshot. Internally the journal position is just the
SequenceNumber from the journal datastructures, but when we expose it to
clients we need to be able to distinguish between a sequence number from the
current instance of the eden service and a prior incarnation (eg: if the
process has been restarted, and we have no way to recreate the journal we need
to be able to indicate this to the client if they ask about changes in that
range). For the convenience of the client we also include the `toHash` (the
most recent hash from the journal entry) which is likely useful for the `hg`
dirstate operations; it is useful to know that the snapshot may have changed
since the last query about the dirstate.
The `getFileInformation` method returns the instantaneously available `stat()`
like information about the requested list of files. Since we simply don't
have historical data on how files in the overlay looked (only how they look
now), this method does not allow passing in a JournalPosition. When it comes
to comparing historical data, we will need to add an API that accepts two
snapshot hashes and generates the results from there. This particular method
is geared up to understanding the current state of the world; the obvious use
case is plugging in the file list from `getFilesChangedSince` into this
function to figure out what's what.
* Do we want a function that combines `getFilesChangedSince` + `getFileInformation` into a single RPC?
Why is there a glob method? It's to support a use-case in the watchman/buck
integration. I'm just sketching it out in the thrift interface at this stage.
In the future we also need to be able to express how to carry out a tree walk,
but that will require some query predicates that I don't want to get hung up on
specifying immediately.
Why is the streaming stuff in its own thrift file? We can't generate code for
it in java or perhaps also python. It's only needed to plumb data into
watchman so it's broken out into its own definition. Nothing depends on that
file yet, so it's probably not specified quite right. The important thing is
how the subscribe method looks: it's essentially the same as the method to
query a delta, but it keeps emitting deltas as they are produced. This is
another API that will benefit from query predicates when we get around to
specifying them.
I've added `JournalDelta::fromHash` and `JournalDelta::toHash` to hold the
appropriate snapshot ids in the journal entry; this will allow us to indicate
when we've checked out a new snapshot, or created a new snapshot. We have
no way to populate these yet; I commented on D3762646 about storing the
`snapshotID` that we have during `EdenServiceHandler::mountImpl` into either
the `EdenMount` or the proposed `RootInode` class. Once we have that we
can simply sample it and store it as we generate `JournalDelta`s.
Reviewed By: simpkins
Differential Revision: D3860804
fbshipit-source-id: 896c24c354e6f58328fb45c24b16915d9e937108
Summary:
This is pretty simplistic: we just wlock and add a delta for the set
of file(s) that were changed in a given fuse operation (this is typically 1
file, but rename affects 2).
To reduce boilerplate very slightly, I've added an initializer_list constructor
for JournalDelta that makes it less cumbersome to create a JournalDelta for a
list of files.
Reviewed By: simpkins
Differential Revision: D3866053
fbshipit-source-id: cd918e2c98c022d5ef79430cd8ab4aef88875239
Summary:
This implements a pretty simple change Journal and associated
JournalDelta.
The Journal is intended to be held in memory and not persisted to disk.
The idea is that we'll hold a `Synchronized<Journal>` along with the
other mount data and grab a `wlock` on it each time we want to add
a change record.
This diff doesn't change any other existing functionality.
Reviewed By: simpkins
Differential Revision: D3660162
fbshipit-source-id: a6b6fa28dd12e4d34718956167ee87f8cb2d89ca
Summary:
Adds a thrift call that returns the list of materialized entries from the whole tree.
This is intended to be plugged into the mercurial dirstate extension.
Reviewed By: simpkins
Differential Revision: D3851805
fbshipit-source-id: 8429fdb4eeccc32928e8abc154d4e6fd49343556
Summary:
Was chatting with simpkins the other day and he mentioned that our
instrumented hg wrappers are quite CPU intensive. This diff switches us to
running the underlying `hg.real` or `git.real` in our integration tests when we
find them in the path.
Reviewed By: bolinfest
Differential Revision: D3865996
fbshipit-source-id: d047749356f0c1c0662774e25801f3578f9f9243
Summary:
hg_import_helper.py imports mercurial python modules, which are GPLv2+, so this
code also needs to be licensed under GPLv2+, rather than the BSD-style license
used for the bulk of the eden code base.
Reviewed By: wez
Differential Revision: D3833508
fbshipit-source-id: eb2a8969a5a88c12444a3778875609f24e145e6b
Summary:
Annoying that gcc and clang behave differently here. The compilation
error is due to gcc not seeing the implicit this pointer for some of these
method calls, so we need to explicitly use it.
Reviewed By: simpkins
Differential Revision: D3846973
fbshipit-source-id: 3d5b8b8b8c9bbab1e7935cff0e65677f76d116fb
Summary:
Buck needs this API so that it knows which paths under a project
root it should exclude when deciding whether it can ask Eden for its
SHA-1 or if it must compute it on its own.
Reviewed By: simpkins
Differential Revision: D3840658
fbshipit-source-id: 5eddc0bef423d3b3ee165d2a4b0bbf193f94f61a
Summary:
we now serialize the overlay data for each directory independently.
When we mount, we try to load the root overlay data. The children are lazy
loaded as the inodes are instantiated.
Structural changes cause the overlay data for the impacted dirs to get saved out.
I need to make a pass over this to fixup comments and so on, I just wanted to get this diff out first.
I moved the overlay stuff from `eden/fs/overlay` -> `eden/fs/inodes` since most
of the overlay-ness is handled in `TreeInode` now; the `Overlay` class is
really just for carrying around the paths and providing the serialization
helpers.
Reviewed By: simpkins
Differential Revision: D3787108
fbshipit-source-id: f0e089a829defd953535b9d0a96b102ac729261b
Summary:
It was starting to get pretty complex to manage locking across the
inodes, filedata, overlay and soon the journal, so as a simplifying step, this
folds data that was tracked by the overlay into the TreeInode itself.
This is the first diff in a short series for this. This one:
1. Breaks the persistent overlay information, so shutting down eden and
bringing it back up will lose your changes (to be restored in the
following diff)
2. Allows deferring materialization of file data in more cases
3. Allows renaming dirs.
The approach here is now to keep just one source of information about the
directory contents; when we construct a TreeInode we import this data from the
Tree and then apply mutations to it locally.
Each inode can be mutated indepdently from others; we only need to lock the 1,
2 or 3 participating inodes in the various mutation operations.
I'll tackle persistence of the mutations in the following diff, but the high
level plan for that (to help understand this diff) is to always keep the
directory inodes for mutations alive as inode objects. We make use of the
canForget functionality introduced by D3774269 to ensure that these don't
get evicted early. On startup we'll load this information from the overlay
area.
This model simplifies some of the processing around reading dirs and looking up
children.
Since the overlay data now tracks the appropriate tree or content hash
we can be much more lazy at materializing data, especially in the rename
case. For example, renaming "fbcode" to "fbcod" doesn't require us to
recursively materialize the "fbcode" tree.
Depends on D3653706
Reviewed By: simpkins
Differential Revision: D3657894
fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
Summary:
We can't allow ~EdenServer to delete the memory until we're sure that
the other threads are done. To ensure that, we need to notify the condition
variable while the aux thread still holds the lock. This makes sure that the
thread destroying the EdenServer waits for the aux thread to release the lock
before we check the predicate and proceed to deleting the memory.
```
SUMMARY ThreadSanitizer: data race /
/common/concurrency/Event.cpp:107 in facebook::common::concurrency::Event::set() const
==================
I0909 14:51:18.543072 4147554 main.cpp:173] edenfs performing orderly shutdown
I0909 14:51:18.555794 4148654 Channel.cpp:177] session completed
I0909 14:51:18.556011 4148654 EdenServer.cpp:192] mount point "/tmp/eden_test.0ostuc90/mounts/main" stopped
==================
WARNING: ThreadSanitizer: data race (pid=4147554)
Write of size 8 at 0x7fff9e182d90 by main thread:
#0 pthread_cond_destroy <null> (edenfs+0x00000007671a)
#1 facebook::eden::EdenServer::~EdenServer() /
/eden/fs/service/EdenServer.cpp:93 (libeden_fs_service_server.so+0x0000000b96cd)
#2 main /
/eden/fs/service/main.cpp:176 (edenfs+0x000000018515)
Previous read of size 8 at 0x7fff9e182d90 by thread T73:
#0 pthread_cond_broadcast <null> (edenfs+0x0000000765b7)
#1 __gthread_cond_broadcast /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/include/x86_64-facebook-linux/bits/gthr-default.h:852 (libstdc++.so.6+0x0000000e14f8)
#2 std::condition_variable::notify_all() /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/condition_variable.cc:72 (libstdc
++.so.6+0x0000000e14f8)
#3 facebook::eden::EdenServer::mount(std::shared_ptr<facebook::eden::EdenMount>, std::unique_ptr<facebook::eden::ClientConfig, std::default_delete<facebook::eden::ClientConfig> >)::$_0::operator()() const /
/
/eden/fs/service/EdenServer.cpp:145 (libeden_fs_service_server.so+0x0000000bcdb5)
#4 std::_Function_handler<void (), facebook::eden::EdenServer::mount(std::shared_ptr<facebook::eden::EdenMount>, std::unique_ptr<facebook::eden::ClientConfig, std::default_delete<facebook::eden::ClientConfig>
>)::$_0>::_M_invoke(std::_Any_data const&) /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:2039 (libeden_fs_service_server.so+0x0000000bcab0)
#5 std::function<void ()>::operator()() const /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:2439 (libeden_fuse_fusell.so+0x00000020fbb9)
#6 facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0::operator()() const /
/eden/fuse/MountPoint.cpp:69 (libeden_fuse_fusell.so+0x000000237447
)
#7 void std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()>::_M_invoke<>(std::_Index_tuple<>) /
/third-party-buck/gcc-4.9-
glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:1699 (libeden_fuse_fusell.so+0x000000237048)
#8 std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()>::operator()() /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgc
c/include/c++/trunk/functional:1688 (libeden_fuse_fusell.so+0x000000236ff8)
#9 std:🧵:_Impl<std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()> >::_M_run() /
/third-party-buck/gcc-4.9-glibc-2.
20-fb/build/libgcc/include/c++/trunk/thread:115 (libeden_fuse_fusell.so+0x000000236d8c)
#10 execute_native_thread_routine /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:84 (libstdc++.so.6+0x0000000e6
ec0)
```
Reviewed By: simpkins
Differential Revision: D3844846
fbshipit-source-id: 545474bc1aff8621dbeb487dcd6b54c82828ff3b
Summary:
This did not work because the dict-like object read from
a `ConfigParser` was not JSON-serializable by Python.
I had to add some methods to the `Repository` that we use in our
integration test harness in order to verify everything I wanted to
in my new integration test. I implemented these methods in both
`HgRepository` and `GitRepository`.
Reviewed By: simpkins
Differential Revision: D3837879
fbshipit-source-id: e0bfb5f1bd3add192ef9bdf561591ac8e52bc002
Summary: I did what the linter told me to do.
Reviewed By: wez
Differential Revision: D3836659
fbshipit-source-id: a5d3fc8974cf6cb7c7e2d88a6215ac5c54479780
Summary: Open a file in a subdirectory in append mode and update it.
Reviewed By: wez
Differential Revision: D3802815
fbshipit-source-id: fe1726529d017345dfee530ae5ec84cfb7531602
Summary:
test_utime() seems to fail a reasonable amount of the time in my experience:
the atime returned by lstat() occasionally ends up being 1 or 2 milliseconds
earlier than the saved timestamp. With the existing behavior python and eden
are independently computing the current time, although I haven't investigated
why eden sometimes gets a timestamp slightly earlier than python did.
This fixes the test by asking eden to set an explicit timestamp value, rather
than letting eden independently get the current timestamp.
Reviewed By: wez
Differential Revision: D3802100
fbshipit-source-id: fecfd1d68b8c95d119099ef39f17004be13d3dff
Summary:
`buck-out` now has an extra level of depth due to its new CPU-specific directories,
so we have to change the place we look for `hg_import_helper.py` in development.
Reviewed By: simpkins
Differential Revision: D3807012
fbshipit-source-id: 24d1fc1fa22f3003580f59cfdd46b9b3917f6eeb
Summary: As noted in the comments, these dependencies were making this harder to use in Buck.
Reviewed By: simpkins
Differential Revision: D3805861
fbshipit-source-id: e04898d0e1a3ccc5e38a9629b1d30791853224a5
Summary:
This codemods `TARGETS` under `[a-d]*` directories in fbcode to make
the `headers` parameter explicitly refer to `AutoHeaders.RECURSIVE_GLOB`.
Reviewed By: yfeldblum
Differential Revision: D3801845
fbshipit-source-id: 715c753b6d4ca3a9779db1ff0a0e6632c56c0655
Summary:
Recent changes to our sudo policy seems to be dropping the SSH_AUTH_SOCK
environment variable for me, even when using "sudo -E". (We do explicitly
configure this to be dropped with sudo's env_delete setting.)
This updates the eden CLI to explicitly ask sudo to set environment variables
that we care about.
Reviewed By: bolinfest
Differential Revision: D3674124
fbshipit-source-id: ab14453056961f7b6a7f5d4122bc0c6f5caa6588
Summary:
Use fd 5 rather than stdout (fd 1) for communicating from the hg import helper
process. This way we won't have communication problems even if some of the
mercurial code does somehow ever end up printing to stdout. File descriptor 5
was picked rather arbitrarily. Anything greater than or equal to 3 should work
in practice.
Reviewed By: bolinfest
Differential Revision: D3673941
fbshipit-source-id: ed668542a17c585a5c54bcd3499174a6ed6fa138
Summary:
This is working around a noisy reporting issue in our CI system. Rather than
run and skip the integration tests in that environment, we swap out the base
class with one that won't report any tests to the python test case discovery
mechanism.
Reviewed By: simpkins
Differential Revision: D3634277
fbshipit-source-id: d118dca78a967bf068242790f7b874f865dddecc
Summary: Currently, all existing mount path are unmounted on 'eden shutdown' but are not remounted again after a subsequent 'eden daemon' call, though they appear as mounted when 'eden list' is called. These changes fix this behavior and have the daemon remount the paths that had been mounted before shutdown was called.
Reviewed By: simpkins
Differential Revision: D3580793
fbshipit-source-id: d03beafc20db4bd01662dd7f198a5ab8859b8e3d
Summary:
Load the config data when eden server is started so that it doesn't need to be re-loaded every time a mount is done. The normal use case for eden will not see that many changes to the config data (users adding repositories themselves is expected to be minimal) so this new logic will be more efficient overall.
Currently, the config data IS reloaded before use every time but this is because there is currently no way to reload the config data if any files are modified on disk. I am looking into how to do this now, and this feature will soon be updated to this diff so configData_ does not need to be constantly reloaded.
Reviewed By: simpkins
Differential Revision: D3580777
fbshipit-source-id: 5e23f51e4aab815e9812750617446dcb7e5483cb
Summary: Restructure the current logic used for loading the config data into a ClientConfig object. Rather than having loadFromClientDirectory iterate through all the config files and parse them to find the necessary information, abstract that logic out into a new method that compiles all of the relevant data so that all loadFromClientDirectory has to do is pull out the needed information. Since this change separates the two steps, this will make it easier to move the first step of compiling config information outside of ClientConfig - the goal here is to have the eden server load all of the config data at start up and cache it in memory so that it doesn't need to be done every time a ClientConfig object is created, and this change is an intermediate step.
Reviewed By: simpkins
Differential Revision: D3580757
fbshipit-source-id: c340a0fe715856066a554238249574f8177bc4d7
Summary: This should facilitate things by making the way to reference the path unambiguous.
Reviewed By: simpkins
Differential Revision: D3672311
fbshipit-source-id: b2a21e4dba60ea5ded28643f76aa402c4250d7ac
Summary:
If remotefilelog is unable to download a file from the server, it can end up
logging data through the UI which gets printed to stdout. Since we use stdout
to communicate with the main edenfs daemon this interferes with that
communication channel: edenfs ends up trying to parse the ASCII data as a
binary chunk header, sees a very large data length, and then ends up waiting
forever trying to read data that isn't coming.
This provides our own custom mercurial.ui.ui subclass to try and make sure that
we print all messages to stderr rather than stdout.
I will also send a separate diff so that we use a different pipe to communicate
with edenfs, rather than just using stdout/stdin. That should be a more
guaranteed way of making sure that nothing inadvertantly gets written to the
communication channel with edenfs.
Reviewed By: DurhamG
Differential Revision: D3673830
fbshipit-source-id: d64d492b10ee09ba63318b86d77f9181c4a0ce29
Summary: The repo_name could be cloned multiple times, which would cause a collision.
Reviewed By: simpkins
Differential Revision: D3672298
fbshipit-source-id: f27e1ec00ab773ed50f4fcf6f485461d01f8ba11
Summary:
Update the privhelper code to properly handle the error if it is asked to
unmount a path that isn't currently mounted.
This fixes the code to send an error response back to the main edenfs process.
Previously the exception propagated all the way up to the privhelper main loop,
causing the privhelper process to exit.
It would be nice to refactor the privhelper code a bit more to create a more
standardized API for implementing privhelper commands. This would provide a
more common place to catch exceptions, and make it harder for the command
implementation code to leak exceptions. However, I'll wait to do that in a
subsequent diff at some point.
Reviewed By: wez
Differential Revision: D3653727
fbshipit-source-id: e14c6ae974c8f99da3b426be250510c1f9b8017d
Summary:
Add back the "mount" CLI command.
While end-users may not need this command under normal circumstances, during
development it is sometimes necessary to re-mount unmounted clients. This adds
back the "mount" command which simply re-mounts an already configured client.
This also adds a "--no-forget" option to "unmount" which causes it to unmount
the client without forgetting its configuration. This allows it to be
remounted with "mount", rather than having to re-run the "clone" command and
specify the repository name again (and potentially restore the desired snapshot
hash).
Reviewed By: wez
Differential Revision: D3653706
fbshipit-source-id: 0e3f3fe1efdf72e9e54bdd1e0fb308ae67b7c058
Summary:
Use the default_permissions option when mounting. This tells the kernel to
enforce the normal file permissions semantics based on the file mode bits.
With this option enabled, the kernel will also implement the access() call for
us automatically.
Reviewed By: wez
Differential Revision: D3653198
fbshipit-source-id: e7227f455077a9ef012a5c102e4e22a976595681
Summary:
Include a configPath_ field for EdenServer that holds the path of the user ~/.edenrc config file. The server needs the data from this user config file in order to perform mounts and currently, the path to the home directory is passed via the CLI to the mount command as a field inside the MountInfo struct in order to get the file. As per discussion in D3498567, including the home directory inside the MountInfo struct is logically a bit disjointed, and this change would no longer require the home directory to be passed to the server via MountInfo.
This restructuring also sets up eden for a future change - having the server remount existing mount points on start-up is now possible from the inside. Before this change, mounting anything had to be done via the CLI since the home directory had to be passed in from the outside. This meant that remounting the existing mount points on start up could only be done if Eden was run in the background - running in the foreground would require manual remounting of all existing mount points. Now that the server has access to the config file's path, remounting can be done without any prompting from the CLI in both cases.
Reviewed By: simpkins
Differential Revision: D3580737
fbshipit-source-id: 46667ccd130b470a3a8a9e9aa08e5ec8e8b90336
Summary:
Previously we would simply report the raw pointer address to the kernel and
rely on it to return that same number to us as the file handle, and make sure
that it told us to shut down the handle when it was closed.
This meant that we had no real idea about which files were still open.
For our future live upgrade plans we need to be able to know this so that we
can transfer the appropriate information to our replacement process.
To facilitate this this diff implements a FileHandleMap class that will assign
file handle numbers and keep track of the instances. The number assignment
strategy is the same as it used to be in the common case: we take the address
of the newly created instance and use that 64-bit number as the file handle
number. However, in the future when we transfer the mapping to a new process,
we may experience a collision when subsequently opening a file handle. To deal
with that, we have a bounded number of attempts to assign a random file handle
number.
We don't yet offer a means to iterate the map, but it would be trivial to
expose such an accessor when we're ready to use it.
Since we now keep track of these things via shared_ptr this changes the
appropriate portions of the fuse interface from unique_ptr to shared_ptr.
Reviewed By: simpkins
Differential Revision: D3602364
fbshipit-source-id: dd996339c2838225a2caeee9da16ef99a06c1e2b
Summary:
When the EDEN_GDB environment variable is set, start eden under gdb when
running the integration tests. Have gdb automatically start the daemon,
and exit automatically if everything is successful. If the daemon crashes, gdb
will break and the user can interact with gdb normally.
This is generally only useful when manually running a single test. Trying to
use this in combination with "buck test" will probably cause problems if
multiple gdb instances all try to interact with the terminal together.
Reviewed By: wez
Differential Revision: D3459575
fbshipit-source-id: 1359735a391e93a332a4c673c6525ad44034cc73
Summary: This change updates LocalStore to perform serialization of trees and blobs internally so that its users don't need to be aware of the internal serialization format. Previously, the get and put APIs were asymmetric such that the get APIs returned deserialized Tree and Blob objects, while put required raw serialized bytes. After this change, put will also use deserialized Tree and Blob objects.
Reviewed By: simpkins
Differential Revision: D3589899
fbshipit-source-id: 2e572e6ec5af44d66206b178a03f7a9d619b2290
Summary:
Update the CLI to always close the thrift client socket, to avoid resource leak
warnings on exit.
I also updated the code to just monkey-patch a nicer EdenError.__str__()
method, rather than having to explicitly catch and modify this exception in
multiple different places.
Reviewed By: bolinfest
Differential Revision: D3560662
fbshipit-source-id: 900fe74c793ffd99f4a2c1f1ddd94b96e48f5eb7
Summary:
The "eden repository <name> <path>" command had a bug that it would duplicate
the existing edenrc contents when writing out a new config: it opened the file
in append mode rather than truncate mode when performing the write.
This addresses that issue, but also does a bigger overhaul of the config update
code. We now acquire a lock for the duration of the modification, so that
another eden CLI process cannot modify the file between when we read it and
when we write out our modifications. We also perform the write using an atomic
rename so the file contents are always valid at all points in time.
I also updated the CLI command to print the repositories in sorted order, and
to only catch expected exceptions, and show backtraces for unexpected errors.
Reviewed By: bolinfest
Differential Revision: D3554550
fbshipit-source-id: 5920ccb2447330673eac3f9956a8ae5d8a66a67e
Summary:
This feels awful, but eliminates an abort that shows up when running under ASAN.
Details in the comment.
Reviewed By: bolinfest
Differential Revision: D3533529
fbshipit-source-id: 1717384c716f565b74b1678c66883a35ee60c522
Summary:
For consistency with `git clone`, the destination path must be either:
* non-existent (but `mkdir -p`-able)
* an empty directory
Or else `clone` should fail.
In the process of writing the integration test, I cleaned up some of our test infra
and took advantage of the fact that we require Python 3 (I believe 3.5) now:
* We now use `subprocess.run()` instead of `subprocess.check_output()`.
* Removed the `client_name` argument to `EdenClient.init()` because that is no longer the right terminology and no one appeared to be overriding the default, anyway.
* `EdenClient.clone_cmd()` no longer calls `os.mkdir(self._mount_path)` because that was papering over the issue this revision is trying to solve.
Reviewed By: simpkins
Differential Revision: D3526909
fbshipit-source-id: 66b5799a1b47a51468faf5dc17d87a385926dc89
Summary: Move adding the mount path to the directory map as the final step of the clone command so that the path is only documented if the mount is actually successful. Previously, if 'eden clone' failed during the mounting, another call to 'eden clone' to try again would fail with 'Error: mount path NAME already exists.' This change ensures that a second call to 'eden clone' will actually try again if the first failed.
Reviewed By: simpkins
Differential Revision: D3543388
fbshipit-source-id: be6412ad6d5d009bd87ad6e0b4110b35e6300fcb
Summary: This change fixes exception messages so that they all match the same format and don't include 'error: ' twice
Reviewed By: simpkins
Differential Revision: D3544034
fbshipit-source-id: 32d600c712d1782d2a1a35331b66e06a842ed8e0
Summary:
[Folly] Move `IPAddress` definitions to source files.
And to internal header files. Keeping headers lightweight can help with build times.
Reviewed By: simpkins
Differential Revision: D3514455
fbshipit-source-id: de78f4ef9e70e7ddd7fb666348ed705c5228531c
Summary:
Add a decorator to more easily declare tests that should be run for both git
and mercurial repository types.
Reviewed By: wez
Differential Revision: D3458883
fbshipit-source-id: 8058873923f9519a6550a8dcb930a5edd02aec10
Summary:
- Refactor the EdenClient code to better reflect how the eden currently works.
This code was originally written when the edenfs daemon only supported
running a single mount point. This updates it to reflect the fact that it
manages a single edenfs daemon, but multiple repositories can be mounted.
- Refactor the EdenTestCase now that tests generally only need a single eden
daemon. EdenTestCase starts the eden daemon by default.
- Add EdenHgTest and EdenGitTest classes. These are subclasses of
EdenTestCase, and they create and mount an hg/git repository before starting
the test function.
- Update the tests to derive from EdenHgTest and EdenGitTest where appropriate.
Reviewed By: wez
Differential Revision: D3458842
fbshipit-source-id: 77349a60ff72a700a2c2526a27e7621b76f9eec2
Summary: Fix a bug with the eden repository command that let it add multiple repositories with the same name. If an entry for repository 'name' already exists in the home ~/.edenrc config file, the eden repository command will now raise an error message saying that the repository already exists. Note that if an entry for repository 'name' exists in any of the global /etc/eden/config.d/* files, the eden repository command will still allow the user to create the new repository and will add an entry for 'name' to the home ~/.edenrc config file (since entries in home config file overwrite those in global)
Reviewed By: wez
Differential Revision: D3529743
fbshipit-source-id: b7c7ceef6ef38fcd3b4ff9a92fbd23a4a0bd6951
Summary:
Mercurial maintains its manifest in sorted order, but since the manifest only tracks file names we can end up with the following sequence:
```
some/path-foo/bar
some/path/bar
```
This is because the `-` sorts ahead of the `/`.
This diff defers passing the entries to the tree serializer, buffering them up
into a temporary vector and using `std::lower_bound` to find the appropriate
insertion point.
Reviewed By: bolinfest
Differential Revision: D3529329
fbshipit-source-id: 395ed16a20c14d17717ec69192a38f0407b51e1d
Summary:
This check was already being done appropriately in our Python code, but we also need
to do it in our C++ code.
Reviewed By: wez
Differential Revision: D3526705
fbshipit-source-id: 3b28b88f63ae768113f363ace58d40a89a8f4b61
Summary: Move the _get_or_create_dir() method from config.py to util.py and rename as mkdir_p(). This change replaces all of the os.makedirs call sites with the new mkdir_p() method.
Reviewed By: bolinfest
Differential Revision: D3512570
fbshipit-source-id: a867049f9af22076934390061f09070bf9ee6397
Summary:
These changes restructure the eden directory so that 'client' directories are created during the `eden clone` command and are associated with a single mount path.
The new eden directory looks as follows:
~/.eden
config.json
clients/
abcd08d/
edenrc
SNAPSHOT
overlay/
efgh19i/
edenrc
SNAPSHOT
overlay/
...
Where the config.json file holds the mapping of mount paths to their respective client directory which is a hash, and the edenrc files in each client directory is an INI file which holds the name of repository associated with the mount path. This INI file follows the current format:
[repository]
name = fbsource
This restructuring required a couple other changes:
- unmount command now cleans up the client directory and removes the mapping of its mount path from config.json
- eden list command now lists all of the mount paths rather than the client names
Reviewed By: bolinfest
Differential Revision: D3506119
fbshipit-source-id: dc07a8baf1052be731ff335d9cf74a07ab8e661a
Summary: Change the ClientConfig class to parse client data via INI config file rather than json file. This class uses boost::property_tree::ini_parser and the ptree data structure to hold the parsed INI file contents. This change makes it possible for eden to no longer rely on json files for getting client data, and the json files will be completely taken out in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3498567
fbshipit-source-id: 3298047a014beda0c250475c0809a7a1ebd95b2b
Summary: Previously the repo snapshot id was collected and stored when the repository was initialized, but this diff moves the update to the clone command so now the snapshot is only stored right before a repository is mounted.
Reviewed By: bolinfest
Differential Revision: D3489550
fbshipit-source-id: 1271c1d7c7709ed332307a636c23e26e913483c9
Summary: Update the unmount command to use the mount point rather than the client name. This diff also removes the mount command completely since tests are no longer dependent on it.
Reviewed By: bolinfest
Differential Revision: D3489290
fbshipit-source-id: 9bfbcb78a78d1593f6d0950502bdd4a55e3e6ca4
Summary: Introduce new 'clone' command that takes in repo name and mount path. This command replaces the current mount command which uses the client name as an argument
Reviewed By: bolinfest
Differential Revision: D3489098
fbshipit-source-id: 719fb94387da1d4ccab770a0e659f717888a7dc6
Summary: Expand functionality of repository command to also create repositories. The current command line format is 'eden repository <name> <path>' and will soon replace the existing init command.
Reviewed By: bolinfest
Differential Revision: D3489064
fbshipit-source-id: 3e6946cbd80a64ad23b52383f3f14a92d4492a21
Summary: Add a simple eden repository command that takes no arguments. The behavior mimics that of git remote and lists all repositories by parsing both the global and local INI config files. The INI file currently needs to be manually added and updated to maintain information about existing repositories.
Reviewed By: bolinfest
Differential Revision: D3479543
fbshipit-source-id: b0d36f33c9b8e0c7b9fe20ec2f362ca15c0adeb8
Summary:
This is a bit ugly and longer term should probably be an
eden configuration that we deploy to our CI tier.
In our CI environment we don't allow unfettered password-less sudo
access and instead must launch privileged processes via a helper
process that is whitelisted. D3524299 has the definition of that
helper as it applies to our internal infra.
Reviewed By: bolinfest
Differential Revision: D3524373
fbshipit-source-id: f36bf29e6b455220c434b681a6d0d551bc21ecab
Summary:
We're seeing failures like this in our CI system:
```
stderr:
test_create (eden.fs.integration.basic_test.BasicTest) ...
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: no tty present and no askpass program specified
error: edenfs is not running
ERROR
ERROR
======================================================================
ERROR: test_create (eden.fs.integration.basic_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "eden/fs/integration/basic_test.py", line 68, in test_create
eden = self.init_git_eden()
File "eden/fs/integration/lib/testcase.py", line 104, in init_git_eden
return self.init_eden(repo_path)
File "eden/fs/integration/lib/testcase.py", line 54, in init_eden
inst.init(repo_path, *kwargs)
File "eden/fs/integration/lib/edenclient.py", line 160, in init
self.daemon_cmd(timeout)
File "eden/fs/integration/lib/edenclient.py", line 174, in daemon_cmd
self._wait_for_thrift(timeout)
File "eden/fs/integration/lib/edenclient.py", line 99, in _wait_for_thrift
raise Exception("edenfs didn't start within timeout of %s" % timeout)
Exception: edenfs didn't start within timeout of 10
```
This diff improves things by failing faster and printing a little more appropriate context.
Reviewed By: bolinfest
Differential Revision: D3523257
fbshipit-source-id: 14508a1b820d9d6c773713e2bb7c22955c5b2b66
Summary:
Implements basic rename(2) support. We only do this for files at this
time as directory renames require that we recursively materialize a tree and
are a bit more complex. I'll look at that in a follow-on diff; there's
potential for optimizing it to avoid eager materialization, but that depends on
the trie work that I'm tackling concurrently with this.
rename for files is the last piece needed to allow `sed -i` to operate correctly.
Reviewed By: bolinfest
Differential Revision: D3515361
fbshipit-source-id: 9c8cc5f9b8db6b5a9372ca9286336647f50490f8
Summary:
This enables O_EXCL to function by allowing the create routine to
move its folly::File instance down in to the underlying FileData instance.
Previously we would close and then re-open the file; this would discard
any of the natural gating for the open call that is performed by the kernel
for the underlying filesystem in the overlay.
Reviewed By: bolinfest
Differential Revision: D3513758
fbshipit-source-id: 85967a3b7affa1b1df46842be8ba21c8fbb843a6
Summary:
Despite being handy things for filesystems in general, these are
needed to support the `sed -i` integration test.
Reviewed By: bolinfest
Differential Revision: D3513754
fbshipit-source-id: 505b4bd58b254141e2ef996f01e3347fc1a77584
Summary:
This eliminates a linear scan from TreeInode and replaces it with a
binary search, exploiting the sorted order of the entries vector.
Two new methods are introduced: getEntryPtr which returns a pointer to the
entry with the matching name, and getEntryAt() which returns a reference
(throwing a range error if there is no such entry).
I wanted to use the PathMap class here, but that would cause us to duplicate
the name string as both the key and value in the map.
Reviewed By: bolinfest
Differential Revision: D3515723
fbshipit-source-id: 4ee0371f3ec08cbcf110cf28f5c1e1529b120fb6
Summary:
This removes use of the builtin Buck `thrift_library` support from
the macro library. It turns out that a lot of rules incorrectly
added deps onto the raw builtin `thrift_library`, rather than one
of the per-language rules. This is a noop, and removing the builtin
rule exposes this as missing target errors, so this diff removes them.
Reviewed By: Coneko
Differential Revision: D3512451
fbshipit-source-id: dd8beb148ed47a3ad7d3963fae600abd73d030d5
Summary:
setattr is a bit of a multi-purpose interface; depending on the flag
values, this is responsible for:
* ftruncate(2)
* fchmod(2)
* fchown(2)
* futimens(2)
In order to apply any of these things, we have to materialize the file. In
the future we may want to allow setting the utimes without materializing the file.
We don't allow chown to actually chown anything. We may want to relax that in
the future, but at the moment we will return an error if an attempt is made to
change the ownership of a file.
Reviewed By: bolinfest
Differential Revision: D3511011
fbshipit-source-id: 858d2c07686fcbe2dcdb60a07527f739a9726be3
Summary:
Make sure mount points are completely stopped before destroying the EdenServer
object. Previously the EdenServer was destroyed with the MountPoints still
running the fuse channels in background threads. When the privileged helper
process unmounted them, fuse requests from the kernel could arrive and access
memory that had already been destroyed.
Reviewed By: wez
Differential Revision: D3458898
fbshipit-source-id: 365bca716ff0f8315b66af92effeb8c6dc574ce1
Summary:
When running the integration tests when built with gcc, the tests would crash
in TreeInode::create(). It appears that the unique_ptr<FileHandle> object was
getting passed to the lambda before dereferencing it to call getattr().
Reviewed By: wez
Differential Revision: D3459605
fbshipit-source-id: 5e2ce98d268a85731acaf7d7f37f22c77fb571cf
Summary:
This fixes inode number handling problems in TreeInode and TreeEntryFileInode.
Previosly these classes each had an ino_ member variable, despite deriving from
InodeBase which has its own (private) ino_ member.
TreeEntryFileInode never actually initialized its own local ino_ variable.
This was causing problems for many applications which use inode numbers to
cache file data. TreeEntryFileInode returned garbage data in the inode field,
resulting in incorrect cache collisions.
This fixes the inode handling, and also fixes the stat data returned by
FileData. It now sets the uid, gid, and inode fields correctly. It 0s out
several other fields rather than leaving them uninitialized.
Reviewed By: bolinfest
Differential Revision: D3455126
fbshipit-source-id: 631276b01676733f96035bc153219ef84406dcc9
Summary:
Start adding utility code to make it easier for the integration tests to
manipulate git and mercurial repositories.
This adds an HgRepository class, but does not update the tests to use it yet.
I will do that in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3449328
fbshipit-source-id: caea5befb683a388cbad30a29017c566e22702af
Summary:
Move the eden/fs/importer/hg code to eden/fs/store/hg. This import code is
only used by the HgBackingStore, so logically it belongs together.
The eden/fs/importer/git directory was already moved into eden/fs/store/git by
D3448752, so this makes the mercurial code similar.
Reviewed By: bolinfest
Differential Revision: D3458128
fbshipit-source-id: 54b1f707197e41ff5f5f4a163a34d3d1ab432e21
Summary:
This moves all of the test library code into a lib/ subdirectory, just to help
distinguish tests from utility code.
This also changes the test so that we no longer pack the eden CLI and daemon
binaries into the python archives. This results in very large archives when
building in dbg and opt modes, and isn't really necessary. Instead
edenclient.py simply finds the CLI and daemon binaries relative to the test
binary. We pass in an EDENFS_SUFFIX variable to tell it which flavor of the
daemon to use.
Additionally, this changes the tests to run with python 3.
Reviewed By: bolinfest
Differential Revision: D3449013
fbshipit-source-id: 82533137090325766a52cd067aa97dd8391ae088
Summary:
This moves git import logic from the GitImporter class to GitBackingStore.
The logic is simpler now, since GitBackingStore only needs to import a single
Tree or Blob at a time.
Reviewed By: bolinfest
Differential Revision: D3448752
fbshipit-source-id: da2d59f953ada714d8512545ae83dd48e5d3e410
Summary:
Add a run_cmd() function that's slightly simpler to use compared to
_get_eden_args().
Reviewed By: bolinfest
Differential Revision: D3448844
fbshipit-source-id: 06b654b2f4bc773d67a4f9d1f6effdad2ce9aa2a
Summary:
- Update "eden daemon" to wait until edenfs is healthy before returning.
- Accept additional command line arguments and pass them through to edenfs.
- When starting edenfs as a background daemon, direct stdout and stderr to a
log file in the eden config directory.
- Update "eden shutdown" command to wait for edenfs to exit.
- Update "eden shutdown" command to print an error message if edenfs was not
running.
Reviewed By: bolinfest
Differential Revision: D3446403
fbshipit-source-id: 2ca1874256b7d124a85092b886548bea8c198327
Summary:
Look for hg_import_helper.py relative to the edenfs binary, rather than
relative to the current working directory. Also check in a couple places, so
we can work in both normal deployment locations as well as inside a source
repository.
Additionally, update hg_import_helper.py to emit a response chunk as soon as it
starts. This contains either message indicating that it has started
successfully, or an error message. This allows us to propagate error
information back to the thrift caller if something goes wrong starting
hg_import_helper.py for a given repository.
Reviewed By: bolinfest
Differential Revision: D3447027
fbshipit-source-id: 4bffd1d03ab6475c09cb3c8385145555c7fc7361
Summary:
[Folly] Wrappers for some of OpenSSL's crypto hash functions.
Wraps some of the OpenSSL crypto hash functions with variants that take `ByteRange` for input and `MutableByteRange` for output, and also variants that take `const IOBuf&` for input as well.
These are a bit nicer to use than passing pointers and lengths separately.
Reviewed By: ivmaykov
Differential Revision: D3434562
fbshipit-source-id: 3688ef11680a029b7664ac417a7781e70f9c6926
Summary:
This adds a --foreground flag to the "eden daemon" command, which causes the
daemon to run in the foreground, rather than daemonizing as a background
process. Additionally, this drops the --preserve-environment flag and instead
updates the CLI to always build a sane environment to run with.
This also updates the integration test code to use this flag, and wait for eden
to shut down during test clean up. Without this, the test case code could try
to clean up temporary directories before they were unmounted, which would fail.
This would leave many temporary directories behind after test runs.
This also re-organizes the temporary directory set up a bit, so that each test
uses a single top-level temporary directory. All other directories it creates
are put inside this one directory.
Reviewed By: bolinfest
Differential Revision: D3439232
fbshipit-source-id: 85305f7db60da6bae589d28c802ee260e42e4dea
Summary:
This changes the way that Eden is built and deployed.
* To build the binary that must be run as `root` (but quickly drops privileges), run `buck build eden-daemon`.
* To build the CLI that communicates with the daemon (and does not require privileges), run `buck build eden-cli`.
* To build both, run `buck build eden`.
There is an example of how to build the various parts of Eden using
Buck and how to package them up in the `install` script introduced by this revision.
While here, I also cleaned up some of our build files and changed them to be
parameterized between internal and external use. In both cases, the user gets the
"unadorned" version of their primary build targets. This ensures that shortcuts such as:
```
buck test eden/fs/integration
```
do the right thing by default.
Finally, I also made `find_default_config_dir()` and `find_default_daemon_binary()`
lazy whereas `find_default_config_dir()` was previously eager.
Reviewed By: simpkins
Differential Revision: D3436245
fbshipit-source-id: 4dfbd59ed0d198620324f0705c462334bb5a7daf
Summary:
If a directory is present in the overlay, we still need to check if a TreeEntry
exists from the source control data structures. Previously this was causing us
to incorrectly report directories as empty if they exist in the local overlay,
even when they had contents from the main Tree.
Reviewed By: wez
Differential Revision: D3434219
fbshipit-source-id: f872f90075602dfdc7b217f50eefcd7c248512e7
Summary:
Add some verbose logging about when trees and blobs are loaded in the object
store.
Reviewed By: bolinfest
Differential Revision: D3434182
fbshipit-source-id: 3e8d2617290604f119e6164d15d63324a4c9a2aa
Summary:
Update the HgImporter class to support retrieving file contents from mercurial.
This also includes simple code for storing the data in the LocalStore using
git's blob serialization format. In the future I think it would perhaps be
better to drop the "blob<length>" prefix, and instead just use a RocksDB column
family to separate blob data from other types of data. However, for now using
the git format is simplest for keeping compatibility with the getBlob() code.
Reviewed By: bolinfest
Differential Revision: D3416691
fbshipit-source-id: 268787533be2172b2dbedc3bf06464eabf3d2c5e
Summary:
Record mercurial commits that have been previously imported, and avoid
re-importing them when checking out a commit that was previously processed and
already has a tree present in the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3416654
fbshipit-source-id: 80f70fa2204d3a2961991b634ad934a623b42b82
Summary:
Add APIs for storing arbitrary (key, value) data.
This will allow BackingStore implementations to store additional metadata, such
as mapping mercurial commit IDs to the eden root tree ID.
Eventually we may want to use RocksDB column families to partition the
different types of data being put into the LocalStore. However, for now this
just uses a single key space. We can add column family support in a separate
diff, if desired.
Reviewed By: bolinfest
Differential Revision: D3409866
fbshipit-source-id: 19a1d340b65bff2081981bf5daf32d5ad15b60c4
Summary:
This adds an HgBackingStore implementation which can load tree data from a
mercurial repository. Blob loading is not implemented yet, but will come in a
separate diff.
This also adds a minimal GitBackingStore class. The GitBackingStore has nearly
no functionality, but is needed to keep the existing git functionality working.
Reviewed By: bolinfest
Differential Revision: D3409743
fbshipit-source-id: dbebf53e9de08bd1469e489baa48b84cbf889511
Summary:
Add the basic BackingStore interface, plus a NullBackingStore implementation
that always returns null. This updates the ObjectStore to query the
BackingStore if data is not found in the LocalStore.
Additionally, this updates EdenServer to manage the BackingStore objects. It
maintains a map of the BackingStore objects created for each known repository.
Reviewed By: bolinfest
Differential Revision: D3409602
fbshipit-source-id: 2920dc4c24ee1ec37efb542f058d0d121ceb5532
Summary:
The _get_hg_dir() function had an incorrect return statement caused "eden init"
to fail with mercurial repositories, unless the path pointed to a shared
working directory.
Also update the code to store the path to the mercurial repository, rather than
the path to the .hg directory. Unlike git, mercurial requires the repository
path itself, and refuses to work if given a path to the .hg directory.
Reviewed By: bolinfest
Differential Revision: D3409808
fbshipit-source-id: 22ee7748422b614f5af57aff07fcfa517af846d9
Summary:
This revision introduces two complementary changes:
* `eden daemon` no longer runs in the foreground.
* There is now an `eden shutdown` command to kill the daemon.
When `shutdown` is called, it tells the Thrift server to shutdown.
In turn, this causes `EdenServer::runThriftServer()` to exit,
which causes `EdenServer::run()` to exit.
Reviewed By: simpkins
Differential Revision: D3402347
fbshipit-source-id: 80032ba53eb69b3f69bef9d7cd169f93500c833c
Summary:
D3406773 included a change which compiles on clang and gcc-5.x, but fails to
build with gcc-4.9.
This looks like a bug in gcc-4.9's handling of list initialization. Overload
resolution for non-initializer-list constructors should be attempted if
no suitable initializer-list constructors are found, but gcc-4.9 does not
appear to do this.
Reviewed By: bolinfest
Differential Revision: D3410142
fbshipit-source-id: f34125000eb3fa949c2427aa4ffbd4ef92942cd7
Summary:
Update deserializeGitBlob() to work on an IOBuf, rather than an rvalue
reference to a string.
The ugliness about having to wrap a std::string in a managed IOBuf is now
hidden inside the StoreResult class, rather than being something that the
GitBlob code has to know about.
Reviewed By: bolinfest
Differential Revision: D3403977
fbshipit-source-id: 0c58c019557050d6e201c1a462fa051c2526674a
Summary:
Add a new StoreResult which wraps the std::string returned by RocksDB.
This replaces the std::unique<string> that LocalStore::get() used to return.
This lets us avoid a memory allocation. StoreResult can also represent a "not
found" result, so that this case can be processed efficiently without having to
throw an exception.
Additionally, StoreResult is move-only so we can't ever unintentionally copy
the string data, which is potentially expensive. It also provides APIs for
creating IOBuf wrappers, or moving the string to the heap so we can create an
managed IOBuf around it.
Reviewed By: bolinfest
Differential Revision: D3403958
fbshipit-source-id: ab0c304988a53eda50341ecc2f96ae5235e5260c
Summary:
Add a new ObjectStore class, which will eventually contain both a LocalStore
and a BackingStore. The LocalStore will be a cache of data loaded from the
authoritative BackingStore. The ObjectStore API will hide the work of querying
the BackingStore and updating the LocalStore when data is not already available
in the LocalStore.
For now ObjectStore only contains the LocalStore, but I will add BackingStore
functionality in subsequent diffs. This diff simply updates all call sites to
use the ObjectStore instead of directly accessing the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3403898
fbshipit-source-id: 47b8c51a7717a4c7c29911a7085b382521a8c0db
Summary:
Previously Hash objects could not be assigned to after they were created, since
they contained a const member. This makes the data non-const, so a Hash
variable can be replaced to contain new contents after it is created.
This also adds a default constructor, which zero-initializes the hash. The
default constructor makes it possible to declare a Hash with a 0-value at one
location, and then set it to the desired value at some later point.
Reviewed By: bolinfest
Differential Revision: D3406773
fbshipit-source-id: 41e2c7e3ad5bc4d14813be4adaa03866701380f6
Summary:
- Add a Sha1Key class that can more efficiently compute the key for
file content SHA-1 values, without having to copy it into a new std::string
object. (In practice fbstring would have avoided having to actually allocate
memory, but it was still an extra data copy.)
- The code was always converting the hash keys to hex on get and put
operations, just in case it needed it if an error occurred. This diff
changes the code to only compute the hex value if an error actually occurred.
Reviewed By: bolinfest
Differential Revision: D3403889
fbshipit-source-id: 5abd8ef202cb00677a84a03a82e2a3d21f16cd2f
Summary:
Update several places to use std::array rather than plain C arrays, using
folly::make_array() to automatically deduce the correct type when necessary.
Reviewed By: wez
Differential Revision: D3370445
fbshipit-source-id: b7642cf3a9b08eac817988bf95679bf5e584ef72
Summary:
Add an HgImporter class for importing mercurial data into the eden local store.
At the moment this only includes code for importing revision manifest data as
tree objects, and does not yet include code for importing file blob data.
Reviewed By: bolinfest
Differential Revision: D3367958
fbshipit-source-id: 58049bf1594b3c27d676c5ebe778917b4043fccf
Summary:
Update deserializeGitTree() to accept an IOBuf object. IOBuf objects can
easily wrap other buffers, so this can still easily support ByteRange objects
as well.
Being able to use IOBuf's Cursor class ended up simplifying the logic a bit as
well.
Note that using IOBuf does require copying the name and mode data out of the
buffer when we read it (using the readTerminatedString() API). This is
necessary since the data may not be stored contiguously in the IOBuf. However,
this shouldn't impact performance much: we already need to copy the name data
into a std::string anyway. For the mode, most modern platforms can avoid doing
a heap allocation for this small string.
Reviewed By: bolinfest
Differential Revision: D3357255
fbshipit-source-id: 5b6e1bc93199849327409a8039266d7dc4f3afdf
Summary: Add a GitTreeSerializer class for serializing git tree data.
Reviewed By: bolinfest
Differential Revision: D3356770
fbshipit-source-id: d04bc9788117272504c2faa335b3648e4ac93e81
Summary:
Previous to this change, if `~/local/.eden` did not exist when `eden daemon` was
run, then it would fail.
Now the logic to create `~/local/.eden` is encapsulated in `_ensure_dot_eden_folder_exists()`.
We do not call this for all subcommands because we want to make sure that running
`eden --help` does not have the side-effect of writing `~/local/.eden` if it does not exist.
Reviewed By: wez
Differential Revision: D3397057
fbshipit-source-id: a3f974f367058d9e4ebd515c78423e54edc179cc
Summary:
`eden health` will return with exit code 0 if both of the following are true:
* The Thrift client is up and running.
* The status of the client is either ALIVE or STARTING.
Reviewed By: wez
Differential Revision: D3395582
fbshipit-source-id: ba668d26acae73a51fbae8aca2b4979156c0c50f
Summary:
When running the CLI "daemon" command, we try killing the underlying eden
process group when we receive SIGTERM or SIGINT. (We really only want to kill
the main eden process, but we currently have to kill the entire process group
due to how sudo works.)
Since the privhelper process runs as root and is part of this process group, we
can get an EPERM error back. This was causing the CLI to fail with an
unhandled exception backtrace. This diff updates the code to ignore EPERM.
Reviewed By: bolinfest, wez
Differential Revision: D3384121
fbshipit-source-id: 39b2364d8c921b1d84a8902566fe9af2a370e4e5
Summary:
This modifies the iterator behavior to so the behavior is a bit cleaner
with respect to empty paths. It is valid to have an empty relative path,
and there are legitimate use cases where this is useful. For instance,
calling dirname() on a RelativePath with a single component will result in
an empty path. It is useful to use this empty path to refer to the parent
directory, to which the path is relative. Therefore it is also useful to
be able to include the empty path when iterating through the parent
directories of a path.
This removes RelativePath::begin() and RelativePath::end(), and replaces
them with a RelativePath::paths() function. paths() returns a struct with
a begin() and end() function, so it can be used in range-based for loops,
and has the same behavior that begin()/end() did. This also adds a
RelativePath::allPaths() function, which also includes the empty relative
path in the results.
Reviewed By: bolinfest
Differential Revision: D3366877
fbshipit-source-id: 3d92b600f07b993925f88d4f1e619b6c1705fb82
Summary:
PrivHelper serializes messages and sends it over to PrivHelperServer who verifies that mount point exists, cleans up bind mounts for the FUSE mount, and undoes FUSE mount.
Some repeated code in this diff since I was unsure on the protocol for that - let me know if/where I should generalize functions to avoid this.
Reviewed By: simpkins
Differential Revision: D3361955
fbshipit-source-id: a7324fb9660912d6c2b753e15b1fa6061c0d5261
Summary:
This avoids translation from string->Hash in the common case
where the file is unmodified and its hash is read directly from
the store rather than computed from the overlay.
I'm guessing I should use `unique_ptr` as the return value throughout?
Reviewed By: simpkins
Differential Revision: D3355773
fbshipit-source-id: 50dff879a78b3d6ff49f86b856866ca28808c4f7
Summary:
Other tools, such as Buck, will benefit from being able to get
the SHA-1 of a file without having to read the entire contents
of the file (or do the associated computation that is proportional
to the size of the contents of the file).
Reviewed By: simpkins
Differential Revision: D3345828
fbshipit-source-id: 360bb268793369af75f408208e8211d8b9db146d
Summary: Updated python CLI to include subparser for unmount command and added wrapper functions that hand over execution to privhelper process. Unmount currently requires client_name at the command line.
Reviewed By: simpkins
Differential Revision: D3359517
fbshipit-source-id: ff05e90bcdb96ecad63f37634c69dbeef429c90f
Summary:
Change Hash::getBytes() to return a folly::ByteRange rather than a
std::array<uint8_t, 20>. This makes Hash more convienent to use with existing
APIs that accept a ByteRange. (For instance, IOBuf.)
There were only 2 call sites using the existing getBytes() functionality,
and they only used the data() method on the returned std::array, so they don't
have to be updated at all to use a ByteRange.
Reviewed By: bolinfest
Differential Revision: D3354581
fbshipit-source-id: 8f2a3c196e59620fb5b0fb2caf4d1d7f26e1d2c4
Summary:
Add a function to compute the sha1 content hash for an overlay file.
We persist the computed hash in an extended attribute in the underlying overlay
file so that a subsequent read of the attribute doesn't require opening the
file to recompute it.
Each time the file is mutated, we blow the cached status of the file.
Each time the sha1 attribute is read, if the cache is blown, the content
hash will be recomputed and set in the overlay file.
Each time the file is flushed or sync'd, if the cache is blown, the content
hash will be computed and set in the overlay file.
Reviewed By: bolinfest
Differential Revision: D3302412
fbshipit-source-id: bd45c7a24b732bd0b7474b7f96e82936870b2117
Summary:
This is part 1 of 2 diffs. This one adds some plumbing to make it possible
to read the xattr attribute from an overlay.
It doesn't do anything to ensure that it is set; the next diff in this series will take care of that.
Reviewed By: bolinfest
Differential Revision: D3302410
fbshipit-source-id: 47406a9c75f29743691d396676c691bcb99c4760
Summary: This logic should be shared by the Eden CLI as well as unit tests.
Reviewed By: simpkins
Differential Revision: D3348300
fbshipit-source-id: c87b1f03f16560323f3d7685063bb6466c39efe2
Summary:
We look this up via the mount point or eden mount object instead.
I've also removed the mercurial library stuff that was added to support the now defunct lamehg fuse we had in the earlier days.
simpkins' new importer doesn't use these and it resolves our CI mismatch issue.
Reviewed By: bolinfest
Differential Revision: D3349698
fbshipit-source-id: 5f4ec16b76042959cd1e3184f46bb3526fbaf74c
Summary:
Update the CLI to support running "init" with a mercurial repository.
This is just some bare bones framework code at the moment. It doesn't actually
import any data from mercurial at the moment, and mounting doesn't work.
Reviewed By: bolinfest, wez
Differential Revision: D3345426
fbshipit-source-id: 72c31ac8d2aac2a16e0a7d6f0425eb4ca218d487
Summary:
Use the hex-encoded version of the key in the RocksException if _get() fails,
rather than the raw binary data.
Reviewed By: bolinfest
Differential Revision: D3345355
fbshipit-source-id: cd8dc644a56ca3d5f3b9a9a0f5cc789b142f0bda
Summary:
Buck is [currently] built with Java 7, so it can only use third-party dependencies
that are also Java 7.
Reviewed By: simpkins
Differential Revision: D3342367
fbshipit-source-id: 4370fd152e7d2055495e783de68a6bb59867bee5
Summary:
This adds a new API to `PrivHelper`: `privilegedBindMount()`.
Similar to `privilegedFuseMount()`, this sends a message to the privileged helper,
which is running as `root`, so it can set up the specified bind mount.
The changes in the `privhelper` directory parrot what was done to support `privilegedFuseMount()`.
Now, once the primary mount for a client is created, any bind mounts listed in the
config for the client are set up. This logic is introduced in `EdenServer.cpp`.
Reviewed By: simpkins
Differential Revision: D3296660
fbshipit-source-id: 61296f35e5c3a6f232a1c17e0f296dd5d3b5ec06
Summary:
Add a new class to serve as a single location where we can store all
information about a single eden mount point. Currently this contains the
MountPoint, LocalStore, and Overlay objects. This allows the TreeInode class
to just store a single pointer to the EdenMount, rather than having to track
these three objects separately.
In the future we could consider also keeping a copy of the ClientConfig in the
EdenMount object, but I haven't done that for now.
Reviewed By: bolinfest
Differential Revision: D3321355
fbshipit-source-id: 8a39bb49822ca8e90c88b2a834b59230d2f91435
Summary:
Enables mkdir in the overlay area.
I had to add some `lstat` calls in to the overlay dir reader because we depend
on knowing at least whether a node is a dir or not at the next level up.
When I run the test suite, the mounts are on my `/tmp` filesystem. When I run
eden manually, they are on my `/data` filesystem. The latter (xfs) does not
populate the type bits. This meant that the test suite passed but manual
testing did not.
Adding the `lstat` calls is a little unfortunate. On OS X there is a bulk
operation that combines `readdir` and `lstat` so that there are fewer syscalls.
We don't have an equivalent for Linux.
Reviewed By: bolinfest
Differential Revision: D3301532
fbshipit-source-id: e228f4a392f90aa491fec62e8b98471a8acecff2
Summary:
We still have naked pointers at the handoff to the kernel, but now
have a cleaner implementation at the level that we're going to be working at
day to day.
I also renamed `FileHandle::release` to `FileHandle::releasefile` so that it
isn't visually ambiguous with `std::unique_ptr::release` in the
`Dispatcher.cpp` code: `fh.release()` vs `fh->release()` look similar but are
dramatically different in behavior.
Reviewed By: bolinfest
Differential Revision: D3309455
fbshipit-source-id: f8cf055bcd51121048a20f0202988cf0aef1f085
Summary:
`creat(2)` and `open(2)` could decide to create a PassThru file handle.
This diff removes that usage. There is a TODO here around handling `O_EXCL` properly.
I'm punting this to a follow-up diff.
Reviewed By: bolinfest
Differential Revision: D3301387
fbshipit-source-id: d35104c536396e7fd064d786f3d5592ecfcbfecf
Summary:
Centralize and delegate most (all?) of the content sensitive portions of file accesses into the FileData class.
Add tests to show that we can write to the overlay file and that the stat data is consistent with the result.
Reviewed By: bolinfest
Differential Revision: D3301251
fbshipit-source-id: a09316ad61c6ef4c656bc5d6dbd43f906abb7932
Summary:
This is the workhorse for adjusting the state that we track for the file data.
It handles both overlay and Tree backed data cases and moving from the latter
to the former depending on the open flags provided when a file handle is
opened.
This diff handles more cases than we have tests for. Those will be covered later in this stack of diffs.
Reviewed By: bolinfest
Differential Revision: D3301213
fbshipit-source-id: c1dab40c0ad205ce6cee820043b70dd886e78431
Summary:
Another step towards making TreeEntryFileInode overlay aware, this diff implements:
- stat
- readlink (although it is not possible to create symlinks yet)
- open
Reviewed By: bolinfest
Differential Revision: D3255158
fbshipit-source-id: 3f90b624e629ef279d6cc32e1d82787ee24796eb
Summary:
This allows us to share the same state between multiple open files and
also helps to reduce the size of inode instances that are otherwise inactive;
when there are no outstanding references to the data, we can drop it and forget
it.
Reviewed By: bolinfest
Differential Revision: D3301198
fbshipit-source-id: f0b3fc73a666ec2033c7a22e9eb587d3212cf966
Summary: This should set us up to have `eden mount` perform the bind mounts.
Reviewed By: simpkins
Differential Revision: D3296370
fbshipit-source-id: 5d8c21308074b357bad3ace72cec157adb5f8b56
