Summary:
The temporary Hash object doesn't live long enough for the copy into
the string, so make an explicit temporary to extend its lifetime.
Reviewed By: simpkins
Differential Revision: D4900457
fbshipit-source-id: 948d919b929285c30ee1f9f0111cba20b60d4dc6
Summary:
When tree manifest data is available, we don't need to parse the full
manifest any more, which is great because we can simply and quickly load the
root manifest and be done with a checkout operation. We can then quickly load
the manifest entry for a given directory path on demand, making us closer to our
ideal of O(what-you-use).
This diff plumbs in the manifest code from mercurial so that we can reuse
the decoder implementation to get the manfiest data and translate it into
something that we can put into our LocalStore.
There's a little wrinkle when it comes to files; mercurial doesn't support
a means for getting the file contents based on just its hash, we have to
provide the filename and revision hash for that. We have existing code
to create proxy entries in the store to map from a proxy hash to a tuple
with the appropriate data needed to resolve the file entry.
We need to extend its use to the trees that we generate because we need
to be able to propagate that path down to a child tree when we generate
the tree entry for a file for the first time.
If a tree manifest is not available for a given revision, we fall back
to the existing implementation that parses the full flat manifest.
Note: the flat manifest import generates entries with different hashes
compared to the tree manifest hash version. While we could make the
serialization match up, there's little risk of this causing problems
for us in practice, so punting on that for now.
Reviewed By: simpkins
Differential Revision: D4873069
fbshipit-source-id: 865001996745700586f571ed67371aed1668a6a7
Summary:
This tweaks the main.cpp and EdenServer.cpp code so that we can avoid having
separate internal and open source builds of the eden/fs/service:server library.
Only one small function is different between the two builds. This updates the
code so that this function is only used from main.cpp, instead of being used
inside EdenServer.cpp.
Reviewed By: wez
Differential Revision: D4889721
fbshipit-source-id: 97abda92c36752516fb8a8e73b4cf50580db8d79
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.
Reviewed By: bolinfest, wez
Differential Revision: D4889053
fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
Summary:
Move the code for the command-line tool up one directory, out of eden/fs.
This better separates the code so that eden/fs contains code for the edenfs
daemon, while eden/cli contains code for the command line tool.
Reviewed By: bolinfest
Differential Revision: D4888633
fbshipit-source-id: 5041e292c5353d05122eefe5db3257289e31239a
Summary:
Fix a subtle crash during checkout when handling newly added entries that
already exist in the working directory: CheckoutAction passed the entry name to
checkoutUpdateEntry() as a PathComponentPiece. However, this
PathComponentPiece could refer to the entry name owned by newScmEntry_, and it
also passed newScmEntry_ into checkoutUpdateEntry() as an rvalue reference.
As a result, if the string data was stored invalidated by the move the name
would no longer be valid when checkoutUpdateEntry() tried to use it.
This bug is triggered by doing an "hg update --clean", where a file added in
the destination commit already exists on disk, and has an entry name of 23
characters or less. (The 23 character limit is fbstring's upper bound on
small string optimizations, where it will store the string data inline in the
object, causing it to be invalidated on move.)
This also fixes a crash in a VLOG() statement when the verbose log level for
TreeInode.cpp was set to 4 or greater.
Reviewed By: bolinfest
Differential Revision: D4882544
fbshipit-source-id: 917ede6eeae2224aaa0724b8b30324f3c3a5c924
Summary:
This is a bit too magical -- it's not clear that the thing produces an Options object. If you do know that you can chain further option setters off this thing, it's nice, but otherwise, the first impression is "what just happened?".
So, let's have one good way for doing things.
Reviewed By: yfeldblum
Differential Revision: D4863947
fbshipit-source-id: 3dfe83cfc077d47f604f47dcb21149fbaa2d2243
Summary:
Update the hg extension to implement dirstate.rebuild(). This is necessary for
the `hg reset` command. This also now implements dirstate.setparents() for
cases when there is only one parent.
Reviewed By: wez
Differential Revision: D4823780
fbshipit-source-id: 802de006e03860995095dc3af17acb2eb05f4e8b
Summary:
Update the checkOutRevision() thrift API to accept the commit ID as a
BinaryHash rather than as a hexadecimal ASCII string, to make it more
consistent with our other thrift APIs.
Reviewed By: wez
Differential Revision: D4823779
fbshipit-source-id: 6226f926533ef86d60f8b1d659bc5d912e2855e6
Summary:
The intent is to provide a way to locate the SNAPSHOT file
for tools that want to have a very fast way to figure out the commit
id without making any RPCs or subprocess invocations.
Reviewed By: simpkins
Differential Revision: D4824176
fbshipit-source-id: 5adca225d9984146852dad1e83de0d903848c1e5
Summary:
This simplifies the InodePtrImpl class by removing support for pointer-to-const
Inode objects. We don't currently use pointer-to-const Inode objects, and I
don't really anticipate that we will ever need this.
The primary motivating factor behind this diff is a recent change in clang that
breaks the existing code: clang PR31606 prevents children classes from
inheriting template constructor instantiations from their parent which take a
reference to the parent class type as the only argument.
While this could have been fixed by explicitly defining the necessary
constructors in the child class (or simply by overriding newPtrLocked() and
newPtrFromExisting() to return the subclass type), dropping support for
pointer-to-const allows us to simplify the code and resolve this issue.
Reviewed By: wez
Differential Revision: D4825526
fbshipit-source-id: 999de352788b13818b7f23788bb34686e193cd5d
Summary:
previously, the importer would read the entire manifest
and emit data to the store as it resolved complete directory entries.
The entire manifest data would be buffered and sent out to the store.
In the scenario where one subtree has been modified and a commit has
been made, only the parents of the subdirectory need to be hashed
and stored, but we would compute and try to store everything anyway.
While this diff can't avoid having to compute hashes for everything (we need
tree manifest data for that), by breaking the import into two passes we can
potentially avoid interrogating the LocalStore about every tree in the entire
manifest during an import; we only need to store the trees that are missing and
can simply cut out sub trees that are already present.
This saves us IO at the expense of buffering the manifest tree in memory
for the duration of the first pass; it is an acceptable trade off.
Reviewed By: simpkins
Differential Revision: D4832166
fbshipit-source-id: 0a40cb851c65393b407a8161db05c4b1795fb11a
Summary:
This avoids hard coding knowledge of the FB-specific `local` dir.
This is only suitable for use on already-mounted eden instances,
so the function accepts either the mounted path or the eden
configuration dir.
Reviewed By: bolinfest
Differential Revision: D4829166
fbshipit-source-id: df9daed6024cdec6222be90f5cb624dd9ca0da0b
Summary:
Folly is dropping support for GCC 4.8 now that the branch for last HHVM release that needs to support it has been cut.
This codemod's away all the trivially identifiable uses of `folly::make_unique` in fbcode outside of folly, mcrouter and hphp.
A future diff will kill it in those, and a diff after that will kill it entirely.
Reviewed By: djwatson
Differential Revision: D4498157
fbshipit-source-id: 5a2c05ffd29e5358298acb53dab189a699b022cc
Summary:
This diff fixes FileData::stat() so that we report reasonable timestamp values
on non-materialized files, rather than always leaving them as 0. We set the
timestamps to the time that we created the FileInode. This ensures that
timestamps are updated correctly when files are modified by a checkout
operation.
Note that for materialized files the code reports the timestamp of the overlay
file. This diff does not modify that behavior. However, this behavior is
incorrect, as the overlay file timestamps are not updated by a FUSE client
opening, modifying, then closing a file (since we keep the underyling overlay
file handle open, and don't close it).
In the future we'll need to implement our own tracking of atime, mtime, and
ctime values. We should probably store these in a header inside the overlay
file. For now, this diff is a stop-gap measure that ensures we at least update
non-materialized file timestamps correctly on checkouts.
Reviewed By: bolinfest
Differential Revision: D4765632
fbshipit-source-id: 478da6441e213cdfe830f1c5129212182ce4eeb0
Summary:
This diff adds a new "eden debug" command, with a few subcommands:
- "eden debug inode": Report information about currently loaded inodes
- "eden debug tree": Report data about source control Tree objects
- "eden debug blobmeta": Report metadata about source control Blob objects
- "eden debug blob": Report the contents of source control Blob objects
This diff also includes the thrift APIs in edenfs to support these commands.
Reviewed By: bolinfest
Differential Revision: D4760528
fbshipit-source-id: 7dc2bd6e0e952ba6fb61702c672fb9417d603ffa
Summary:
Previously EdenServer::getMount() returned a null pointer if the path did not
refer to a known mount point. However, most of the thrift APIs using this
method never checked if the return value was null, which could allow any thrift
client to crash edenfs.
This updates getMount() to throw if the path is not found, and never return
null.
Reviewed By: bolinfest
Differential Revision: D4760527
fbshipit-source-id: 6cd9a1e3bc0a0a220d7f5fa518d497d37f245eff
Summary:
This updates all of the references to gtest and gmock with googletest.
The change is mechanilcal, generated with the following one-liner:
```lang=bash
hg grep -lwE '(gtest|gmock)' 'glob:**/TARGETS' | grep -v '^third-party-buck' | xargs perl -pi -e '
$gt=qr!(["'"'"'])gtest\g1!;
(
s!$gt(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gtest\1\),!g or
s!$gt((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gt,?\s*\),?!\(\1googletest\1, None, \1gtest\1\),!g or
s!$gt,?!\(\1googletest\1, None, \1gtest\1\),!g
) unless /(name|type) *=/;
$gm=qr!(["'"'"'])gmock\g1!;
(
s!$gm(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gmock\1\),!g or
s!$gm((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gm,?\s*\),?!\(\1googletest\1, None, \1gmock\1\),!g or
s!$gm,?!\(\1googletest\1, None, \1gmock\1\),!g
) unless /(name|type) *=/;
'
```
Reviewed By: meyering
Differential Revision: D4643237
fbshipit-source-id: fda7f41760c7e44254231df87634631c343e6355
Summary:
The hg_import_helper script that eden uses to import data from mercurial keeps
a long-lived repository object open. This caches some data about the
repository, and if new commits are added after it was created, it can fail to
see them.
This updates hg_import_helper.py to catch errors that occur when trying to use
the repository objects. The code will invalidate the repository object and
then retry the operation once, in the hopes that it will now succeed after
invalidation.
Reviewed By: bolinfest
Differential Revision: D4752659
fbshipit-source-id: 1c75c84766d6bbda0710882a338eaa09e0cb0030
Summary:
Tweaks the stats stuff so that we can name the histograms and export them with p50, p90, p99 percentiles.
This is made a bit ugly because the fb303 stats code isn't yet open source, so
there's a moderately ugly preprocessor directive that we assume to be true for
our internal build. We'll need to force that to be 0 for the open source build
until we open the stats code and make it available.
Reviewed By: bolinfest
Differential Revision: D4801868
fbshipit-source-id: 643909e63bd4a74b2cfa580be131f65c5673bc94
Summary:
This explicitly uses a WriteBatch during importing. This results in a
5x perf gain for import operations.
I've also added in a read-before-write check to make sure that we're not
attempting to overwrite a key that already exists. Since this is a
content-addressed store, a given key should always have the same
value; it should be immutable. Overwriting an existing key is an
opportunity to change data that should never change and is also wasteful
of IO bandwidth.
Ideally we'd be able to exploit this to avoid descending into a directory;
I'd like to hook that up to the treemanifest stuff in mercurial in a
follow on diff.
Reviewed By: bolinfest
Differential Revision: D4783598
fbshipit-source-id: 24f95495d09a852859b346559fdf83d064de2b51
Summary:
this is the bare minimum to support creating unix domain sockets.
We only support using mknod to create a unix socket; other uses will yield an error.
I've added an rdev field as a sibling of the existing mode field that we track,
as that is the additional parameter that we need to track as part of the
special file node.
Special file nodes are tracked in the overlay as empty files.
Reviewed By: bolinfest
Differential Revision: D4774099
fbshipit-source-id: 0824b7e509063faa8bede7aff82a7c51930c4f83
Summary:
This is an initial pass at implementing "hg status" using the new EdenMount and
TreeInode diff() code.
More work still needs to be done to clean things up here, but this makes
"hg status" more correct in the face of modified files and directories that are
not modified, and vice-versa.
In particular, the following still needs to be cleaned up in future diffs:
- Update the "hg status" code to more correctly process user directives for
paths that did not appear in the diff output. This will probably be
simplified some by correctly updating the user directive list on checkout and
resetCommit() operations. It may also make things simpler if we store the
user directives in a hierarchical path map, rather than in a flat list.
- Update the addAll() code to also use the new diff logic, rather than the
getStatusForExistingDirectory() function.
- Clean up the locking some. I think it may be worth changing the code to use
a single lock to manage both the current commit ID for the mount and the
dirstate user directives. We probably should hold this lock for the duration
of a diff operation. The diff operation should also return the commit ID
that it is based against.
Reviewed By: wez
Differential Revision: D4752281
fbshipit-source-id: 6a6d7e2815321b09c75e95aa7460b0da41931dc0
Summary:
It's not really magic because we don't have a virtual directory
inode base any more. Instead, we mkdir and populate it at mount time.
What is slightly magical about it is that we give it some special powers:
* We know the inode number of the eden dir and prevent unlink operations
on it or inside it.
* The .eden dir is present in the contents of the root inode and will
show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
magic `.eden` inode number. This means that it is possible to `stat`
and consume the `.eden` directory from any directory inside the eden
mount, even though it won't show up in `readdir` for those child dirs.
The contents of the `.eden` dir are:
* `socket` - a symlink back to the unix domain socket that our thrift
server is listening on. This means that it is a simple
`readlink(".eden/socket")` operation to discover both whether a directory
is part of an eden mount and how to talk to the server.
* `root` - a symlink back to the root of this eden mount. This allows
using `readlink(".eden/root")` as a simple 1-step operation to find
the root of an eden mount, and avoids needing to walk up directory
by directory as is the common pattern for locating `.hg` or `.git`
dirs.
Reviewed By: simpkins
Differential Revision: D4637285
fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
Summary:
Update the EdenMount::diff() code to completely skip .hg and .eden directories.
This was implemented through the GitIgnore code, and adding a new HIDDEN status
The .hg and .eden directories are similar to ignored directories, but we never
want to report any information for them even when listIgnored is true.
This also changes the GitIgnore code to so that GitIgnoreStack computes the
basename for each path once, and passes it down into the underlying match code,
so we don't have to re-compute it for each pattern that needs the basename.
Reviewed By: wez
Differential Revision: D4751917
fbshipit-source-id: ec22d62f31a3ce9ef998a8ccb4413f72f1d7a487
Summary:
This updates the diff() logic to process gitignore rules.
The bulk of the existing TreeInode::diff() was moved into a
TreeInode::computeDiff() function. TreeInode::diff() now loads the gitignore
rules for this directory before calling computeDiff(). The argument list for
computeDiff() was now getting slightly unwieldy, so I created a new DiffContext
class to hold some of the arguments that are the same for the duration of the
entire diff operation. This resulted in some rather mechanical changes through
the computeDiff() and DeferredDiffEntry code to update the argument lists.
Reviewed By: wez
Differential Revision: D4744690
fbshipit-source-id: c4981d9f49878b522e0b3faf35de837739066f3c
Summary:
This tweaks the definition of the subscribe method in the
streamingeden thrift file and implements the server side of the
thrift service, and the client side to consume it in watchman.
The definition is now a bit simpler than it was previously; we're
now just sending a stream of the updated JournalPosition objects
to the client rather than computing and sending FileDelta objects.
This is cheaper for Eden to deal with because it is very cheap to take
the current journal position and pass that over the wire. This is
important because a burst of mutations will likely queue up a bunch
of these in quick succession.
In a future diff I'm considering adding a latency parameter for
the subscription so that we can constrain the number of updates
over a certain time period (likely in the 10's of milliseconds range).
For now I just want to prove that the concept works.
On the watchman side we just need to pull these off the wire as they are sent
by eden, then wait for the subscription stream to settle before internally
broadcasting to any connected subscribers.
Reviewed By: bolinfest
Differential Revision: D4647259
fbshipit-source-id: 03aa16e59a43195a2505a8d03bce1ccf88a8d42f
Summary:
Update the edenfs main function to check the exit code of the privhelper
process, and to exit with EX_SOFTWARE rather than EX_OK if the privhelper did
not exit with status 0.
This will ensure that our integration tests always catch crashes in the
privhelper process. Previously we could catch errors that happened before the
privhelper socket was closed, but if the privhelper server crashed during
cleanup we never noticed the error.
Reviewed By: bolinfest
Differential Revision: D4728006
fbshipit-source-id: 2c478612e5fabec90438b28b9c76a3343f561655
Summary:
Add a unit test that modifies a file and then does a forced checkout to revert
back to the original commit state.
This also includes some pretty-printers to make the tests print
CheckoutConflict objects nicely if any test assertions fail.
Reviewed By: bolinfest
Differential Revision: D4727779
fbshipit-source-id: b31dc90be28d17ed0f2a4076c96a08ba82588190
Summary:
This begins adding an EdenMount::diff() method, which walks through the inode
tree and reports differences from the current source control tree state.
My intent is to eventually update the Dirstate code to use this diff()
function, and to remove the existing getModifiedDirectories() logic. The
getModifiedDirectories() code is currently incorrect, as it reports
materialized directories, which is not the same as directories that are
modified from the current source control state.
The ignore processing logic is not currently implemented in EdenMount::diff(),
but it should be relatively straightforward to add in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D4726048
fbshipit-source-id: ad0bb3b5d72bb3830f60fc2b2e56a81217c35353
Summary:
Update the TreeInode and CheckoutAction to correctly be able to remove
directories during checkout operations. This handles both removing directories
that are newly empty as a result of the checkout, and directories that are
replaced with a file or symlink.
This also refactors the rmdir/unlink code a bit so we can share some of its
logic for doing directory removal during the checkout operation.
Reviewed By: bolinfest
Differential Revision: D4708607
fbshipit-source-id: 3b9dd9cc3536845dad0d9278e3c5d3ac1ed04570
Summary:
Synchronize access to the CheckoutContext conflicts_ list when recording
conflicts or errors during a checkout operation.
In theory it is possible for the checkout operation to be proceeding
simultaneously in multiple threads. At the moment this does not happen because
the BackingStore implementations currently block, and always return
immediately-ready Future objects in the same thread. However, once they are
updated to return Future objects that actually complete asynchronously in other
threads the checkout work will continue in these other threads.
Reviewed By: bolinfest
Differential Revision: D4708605
fbshipit-source-id: 885b0114d97dadcacf64652f795a7d3846412f11
Summary:
Add a test that exercises a checkout operation that creates a brand new
subdirectory.
Reviewed By: bolinfest
Differential Revision: D4693994
fbshipit-source-id: 70f43f67c2cdb10bd854f51d5ba2e61c45976975
Summary:
Unfortunately it looks like gcc won't support the [[nodiscard]] attribute until
the gcc-7.x release. (The functionality is available in gcc-4.8 and later, but
using the name [[gnu::warn_unused_result]] instead.)
This changes the code to use the FOLLY_WARN_UNUSED_RESULT macro from
folly/Portability.h instead. (This expands to older non-standard __attribute__
syntax for gcc and clang, and _Check_return_ on Windows.)
Reviewed By: bolinfest
Differential Revision: D4704184
fbshipit-source-id: d5c13630ea611a2f43080a501add42ce9fda6789
Summary:
Add new ASSIGN_PARENT_INODE and ASSIGN_INODE types to LoadBehavior, to exercise
additional code paths during checkout operations. This also cleans up the
overall handling of the LoadBehavior enum.
Reviewed By: bolinfest
Differential Revision: D4694548
fbshipit-source-id: e8359cfdadb1fabb1a0d07bc57a1a1610ed9ba57
Summary:
Refactor FileInode and FileData so that they no longer store the
TreeInode::Entry that refers to this file. The TreeInode::Entry is owned by
the parent TreeInode and is not safe to access without holding the TreeInode
contents_ lock, which the FileInode code never did. Additionally, once a
FileInode is unlocked the TreeInode::Entry is destroyed. Previously the
FileInode code would retain its pointer to this now invalid data, and could
still dereference it.
In the future I think it might be worth refactoring the FileInode/FileData
split a bit more, but this at least addresses the issues around invalid
accesses to entry_.
Reviewed By: bolinfest
Differential Revision: D4688058
fbshipit-source-id: 17d5d0c4e756afc6e3c4bb279cb1bb5231f3134f
Summary:
Remove the TreeInode::entry_ member variable. This was a pointer to the
TreeInode::Entry object representing this inode in the parent TreeInode.
However, it isn't safe for a child Inode to access the TreeInode::Entry owned
by its parent. This Entry object is supposed to be protected by the parent
TreeInode's contents_ lock, and the child inode never held this when accessing
the Entry.
This updates the TreeInode code to never use entry_ any more. I will work on
removing FileInode::entry_ in a subsequent diff.
This did require changing the overlay materialization code some. Previously
the checkout code directly modified entry_ to inform the parent TreeInode if
the child was materialized or not. Now we have to more explicitly tell the
parent TreeInode about changes to each child's materialization state. With
this change we now keep the overlay data on disk more consistent during the
checkout process. We should now always have overlay data for children inodes
on disk whenever the parent thinks they are materialized. We may end up
writing out TreeInode overlay data frequently during the checkout process,
however. We might be able to improve this in the future. For now it seemed
simpler and safer to just update the data frequently to make sure it stays
consistent.
Reviewed By: bolinfest
Differential Revision: D4688055
fbshipit-source-id: d08a14b9304ac49826f0897a0b53281177d9d9f4
Summary:
Update the code so that we no longer immediately remove overlay data about a
file or directory when the inode is unlinked. The file can continue being used
after it was unlinked if there are open file handles or other internal
references to it.
This updates the code to instead wait to remove the overlay data until the
InodeBase object is unloaded.
This does not fully fix all the outstanding issues with using inodes after they
have been unlinked, because the FileInode and TreeInode still use the entry_
pointer that now points to a bogus TreeInode::Entry.
Reviewed By: bolinfest
Differential Revision: D4688056
fbshipit-source-id: a9e1c1a9562590b8b81237057bfc4ef77795807f
Summary:
This changes a couple aspects of the materialization process:
- Write out child files in the overlay before their parent. If we throw an
exception or crash partway through this process, it is better to have the
child not marked materialized in its parent, rather than to have the parent
indicate that the child is materialized but to not actually have any overlay
data for the child. In the former case we will still load the correct child
data the next time we need to, but in the latter case we would fail to load
the child correctly.
- Hold the rename lock while materializing our parent directories, to ensure
that we materialize the correct parent, and it cannot change while we are
trying to perform the materialization.
- Make the parent responsible for modifying the child's TreeInode::Entry. This
data is owned by the parent and is protected by the parent's contents_ lock.
The child really shouldn't ever be directly accessing this data without
holding the parent's contents_ lock.
Reviewed By: bolinfest
Differential Revision: D4688057
fbshipit-source-id: 2662f79cb7d7febb086f4e0888a3d96a580c4bfa
Summary:
This fixes the checkout code to always recurse into directories with conflicts,
and accurately list all individual files with conflicts.
Previously the checkout code always avoided recursing into unloaded directory
inodes that were affected by the checkout. Since the inode was not loaded we
can easily replace it without recursing into it. Unfortunately this resulted
in an inaccurate conflict list--we would report the directory itself as a
conflict. This is insufficient for our mercurial extension, which needs to
know the specific list of files with conflicts, so it can ask mercurial to
perform conflict resolution on them.
Reviewed By: bolinfest
Differential Revision: D4672373
fbshipit-source-id: ff400a4a6e25b8b0754c66cb067a69d553a5c98b
Summary:
Loading the child inode is necessary to avoid race conditions with other
threads that may be currently trying to load the child inode in question.
This updates the code so that rmdir() and unlink() now share most of their
implementation.
Reviewed By: bolinfest
Differential Revision: D4664110
fbshipit-source-id: 871ca7b3c22819b24bc2639d9550bb45f6541f50
Summary:
Update the DEFS and .buckconfig files so that the C++ code can be built
successfully with buck in our github repository.
Reviewed By: wez
Differential Revision: D4633749
fbshipit-source-id: a07a6e664471237fcc2d13a8be8b92efa3a101bf
Summary:
Replace the one very basic checkout unit test with several tests that
more thoroughly test file modification changes. More tests for
directory changes and file type changes still need to be added, but this is a
first step for now. I will continue adding more tests in upcoming diffs.
Reviewed By: wez
Differential Revision: D4641595
fbshipit-source-id: 443a68e4e3c673fe687446f3469f035b4f48f1b3
Summary:
Add a method to reset the current commit without changing the working
directory state, similar to "hg reset" and "git reset --soft".
This also adds a new EXPECT_FILE_INODE() check for use in the unit tests,
and FileInode::getPermissions() and FileData::readFull() methods to support
this check.
Reviewed By: wez
Differential Revision: D4641476
fbshipit-source-id: 1e516774fe8e292a8d82cc2c354619374a3abe37
Summary:
The FakeTreeBuilder class in D4609587 provides a lot of the same functionality
as the TestMountBuilder class, but is not restricted to being used only at
mount initialization time. (It also provides more powerful functionality for
controlling the ordering of events when loading data from the ObjectStore.)
This switches all of the existing tests to use FakeTreeBuilder rather than
TestMountBuilder, and removes the TestMountBuilder class.
One difference is that FakeTreeBuilder adds data to the FakeBackingStore, while
TestMountBuilder previously put data directly into the LocalStore. This
shouldn't really make much difference for the behavior of the code. Putting
data in the FakeBackingStore gives us more control over when data is available
during load operations, and exercises slightly more of the normal ObjectStore
code path.
Reviewed By: wez
Differential Revision: D4641288
fbshipit-source-id: ca2b45bf69bd373848c12a7b739b286aabe6aa9b
Summary:
Add a FakeTreeBuilder class to make it easier to create Tree structures in the
FakeBackingStore using path names. Previously the test code had to manually
chain together Trees and Blobs. FakeTreeBuilder lets callers call setFile()
with a path name to define the file layout. The files that were defined then
get converted into Tree and Blob data when finalize() is called.
This is similar to the existing TestMountBuilder::addFile() API. However,
TestMountBuilder only populates the LocalStore. FakeTreeBuilder populates
the FakeBackingStore instead. The FakeBackingStore class allows tests to have
much more control about when objects appear ready from the backing store,
allowing control over the order in which Futures are fulfilled during the test.
Reviewed By: bolinfest
Differential Revision: D4609587
fbshipit-source-id: 9642d17daf0dc10f08901e51552bbb3c3e150b53
Summary:
Make sure the eden coded compiles cleanly with -Wshadow-compatible-local
Pretty much all of the warnings were issues with lambdas shadowing names from
their parent context (even though they didn't ask to capture those names from
the parent).
Reviewed By: wez
Differential Revision: D4644849
fbshipit-source-id: 66629cd98b5af4760f3fbb256e44c0bc47e52316
Summary:
Previously, we would only allow you to consume a symlink that was
checked into a revision, but not allow you to create a new one.
This diff adds support for making new symlinks.
It turns out that the code that we had for readlink had an unused code path for
reading an actual symlink out of the overlay. My first pass at this diff tried
to make the materialization code generate such a symlink, but it was breakin
some other assumptions in the rest of the code.
This version of the code just stores the symlink target in the file contents.
This means that we can simplify readlink to just calling `readAll` on the
underlying file data, and that will load the contents out of the storage layer
if the file wasn't materialized. This helps simplify things a bit more.
Reviewed By: bolinfest, simpkins
Differential Revision: D4604016
fbshipit-source-id: 3138d0f9880639d2bbeaf4bb03bef3f021c3ecb3
Summary:
In order to send realtime subscription information to watchman
we need to use the thrift streaming data functionality. Since this
isn't supported in java we originally broke this out into its own
thrift file.
It's now time to hook this up; this diff adds some plumbing so that
our thrift server knows to implement this interface.
This was made a bit more complicated because the inheritance in
the thrift definitions causes some ambiguities in the header maps
that buck uses to manage the includes.
I also had to rename the file from `eden.stream.thrift` to `streamingeden.thrift`
to avoid a problem with the period in the middle of the identifier.
For the server side of things, I've removed the `:thrift-cpp2` target
and replaced it with `thrift-streaming` instead.
This diff doesn't implement the endpoint, it is just setting things
up for the follow-on diff that does.
Reviewed By: bolinfest
Differential Revision: D4640639
fbshipit-source-id: b3625b0ff33a42945bf523944beed050f549a18c
Summary:
I originally added this to facilitate `hg status` but we didn't end up
using it. Rather than keeping it around and updating the tests in the `.eden`
dir diffs that follow, let's just remove it.
Reviewed By: simpkins
Differential Revision: D4610768
fbshipit-source-id: 158c0207f88980e86aeeddf75e6fd49763d2a402
Summary:
Remove AutoHeaders.RECURSIVE_GLOB from all of our TARGETS files. This is a
Facebook-internal directive that was added as part of a codemod. The vanilla
open source buck does not know about this directive, and can't parse TARGETS
files that contain it.
All of the affected rules are cpp_unittest() and cpp_binary() rules that don't
actually have any header files, so these parameters didn't actually have any
effect.
Reviewed By: andrewjcg
Differential Revision: D4633748
fbshipit-source-id: 6227fe9b7f2ea838a0c72408ca195685bcae9f5a
Summary:
In TreeInode::computeCheckoutActions() the code incremented the wrong index
when we reached the end of the new tree before the old tree.
panda-facepalm
Reviewed By: bolinfest
Differential Revision: D4609906
fbshipit-source-id: 61ca6baabe4eebb033c48b742562558466a45104
Summary:
Invalidate directory entries as they are modified during the checkout
operation. We never need to invalidate inode contents, since we always update
files by replacing the file with a new inode.
Reviewed By: wez
Differential Revision: D4577942
fbshipit-source-id: afde3040960b076d2cd384301a89cd7bdfce5a6a
Summary:
When the "eden daemon" CLI command has to run edenfs via sudo, also explicitly
pass through the KRB5CCNAME environment variable. This will allow edenfs and
it's hg import helper to find the correct Kerberos credentials if it needs to
connect to a remote server.
Reviewed By: wez, bolinfest
Differential Revision: D4584132
fbshipit-source-id: 6c72e4eca76d81b3d4b53a9e72a0e8388da30a7d
Summary:
This updates CheckoutAction to make the old TreeEntry object optional.
This is necessary to support situations where the entry did not exist in the
old source control tree, but does exist in the new tree and also exists locally
on the file system. This situation is always a conflict, so it only applies
when a force checkout is being done.
This diff also refactors the TreeInode::computeCheckoutActions() logic to
combine the processNewEntry/processRemovedEntry/processEntry lambdas into a
single function, as suggested in the code review for D4538516.
Reviewed By: wez
Differential Revision: D4577924
fbshipit-source-id: 9435205b541a12c75a8b9bd2b2b599a339a2f6c8
Summary:
Add a thrift call to get the ID of the snapshot currently checked out in a
given eden mount.
Reviewed By: bolinfest
Differential Revision: D4570729
fbshipit-source-id: 208ceb309aacfb07e30ddb8fdfc0950a3d7d748b
Summary:
Fix the FileHandle code that was calling getPathBuggy() when writing journal
entries. This fixes a crash when writing to an unlinked file.
Also remove an unused call to getPathBuggy() in TreeInodeDirHandle.
These were the last places calling getPathBuggy(), so I have now deleted this
method entirely.
Reviewed By: bolinfest
Differential Revision: D4570606
fbshipit-source-id: a1c4aab10df96e2aeee78fca6d5808db28ad0cbf
Summary:
This is the initial code for implementing checkout.
This isn't quite 100% implemented yet, but I think it's worth checking in this
code as-is, and getting the remaining functionality done in separate diffs.
In particular, a few operations aren't implemented:
- Removing a directory that was deleted in the new revision
- Replacing a directory that was replaced with a file or symlink in the new
revision
- When doing a forced update, replacing a file or directory that did not exist
in the old revision, but that was created locally in the working directory,
and also exists in the new revision.
Reviewed By: wez
Differential Revision: D4538516
fbshipit-source-id: 5bb4889b02f23ab2048fcae2c8b7614340181aa6
Summary:
Update PrivHelperServer::bindUnmount() to give up after 2 seconds instead of
possibly looping forever.
Also fix the unmount code in the eden process to rethrow the original exception
as-is. Previously it was re-throwing it as a std::exception, losing the
exception details.
I think there are still other bugs here in this unmount code--it repeatedly
hangs for me on unmount. (I believe it is trying to unmount things that aren't
actually bind mounts.) However, I'll look into fixing those in a separate
diff.
Reviewed By: wez
Differential Revision: D4547711
fbshipit-source-id: af6bce0262e8314361450874ef998ebf7ad590e3
Summary:
Refactor the Overlay code to store data using inode numbers rather than the
affected file's path in the repository. This simplifies the TreeInode code a
bit, as we no longer have to rename overlay files to stay in sync with the file
paths. This also eliminates some crashes when trying to update overlay files
for inodes that have been unlinked (and hence no longer have a path). This
also includes a few fixes to avoid writing journal entries for unlinked files
too. Additionally this contains a few fixes to how mode bits are stored in the
overlay, and fixes a bug where create() was ignoring the mode argument.
Reviewed By: wez
Differential Revision: D4517578
fbshipit-source-id: c1e31497dcf62c322b0deff72b0a02675b0509ab
Summary:
InodeBase::getNodeId() already returns the inode number, so there is no need
for the TreeInode subclass to provide an alternatively-named method that does
the same thing.
Reviewed By: bolinfest
Differential Revision: D4500215
fbshipit-source-id: fbc595cf9e8f43efe44ed352a97e3a4974b5aed0
Summary:
It turns out that the manifest class provides an iterentries() method to
iterate through the (path, hash, flags) values. Update the code to use this
rather than iterating through the paths and then having to look up the hash and
flags using the path.
This dramatically speeds up importing a commit with 1M files: the import time
drops from around 8 seconds to under 1.5 seconds. (Of the 1.5 seconds, about
half is spent reading the manifest data, and half is spent formatting and
writing the chunks.)
Reviewed By: wez
Differential Revision: D4512538
fbshipit-source-id: ccfff162082d327d525224b2d7dc87f67bda5e22
Summary:
This updates the TreeInode::rename() code to handle concurrency better.
In particular:
- The code now ensures that both the source inode being renamed and destination
inode (if it exists) are loaded. This simplifies issues when an inode is
being loaded at the same time a rename is in progress. This ensures that any
pending load is processed before the rename takes place. (All promises for
the load might not be fulfilled before the rename completes, but the relevant
TreeInode and InodeMap data structures are updated before the rename occurs.)
This does mean that the rename code potentially might have to retry several
times if the inode it began loading is no longer the affected source or
destination or child once the load completes. However, this seems like a
reasonable trade-off, compared to dealing with the complications that would
arise with the load code having to handle renames occuring before load
completion.
- The code now implements proper lock ordering, to avoid acquiring locks in
conflicting orders that might cause deadlock with other threads also trying
to acquire the same locks. The InodeLocks.md document has been updated to
clarify the TreeInode lock ordering requirements.
Reviewed By: wez
Differential Revision: D4493526
fbshipit-source-id: 627393fafad90eb551aea62be7762d59ed043abe
Summary:
Move the integration tests from eden/fs/integration up one directory, to
eden/integration.
The main benefit is that this makes it easy to run just the edenfs unit tests
by running "buck test eden/fs/...". These unit tests complete much more
quickly than the full set of integration tests, providing a faster test suite
to re-run repeatedly during development. The integration tests can be run with
"buck test eden/integration/...", and the full set of tests can still be run
with "buck test eden/..."
Reviewed By: wez
Differential Revision: D4490247
fbshipit-source-id: 5ceb5a19526f56e1cb926f352fa30ad2f1212c05
Summary:
Most of the FileData logic assumes that file_ is open if the file is
materialized. Fix the FileData() constructor to actually open file_ if the
entry is marked materialized, so that this assumption is actually true.
Reviewed By: wez
Differential Revision: D4471365
fbshipit-source-id: b399e6e5e25eaf12f4204f5835db01baa5a6cd78
Summary:
Update the code to release the parent TreeInode's contents lock when processing
its children. This makes sure we do not hold the root lock for the entire
duration of this function.
Reviewed By: wez
Differential Revision: D4471289
fbshipit-source-id: dc4b50bd06fcceb7a7df3d7dd04cfc0d41285556
Summary:
This is to facilitate the watchman integration and draws on the
watchman glob implementation; the approach is to split the glob strings into
path components and evaluate the components step by step as the tree is walked.
Components that do not include any glob special characters can be handled as
a direct lookup from the directory contents (O(1) rather than O(num-entries)).
The glob method returns a set of filenames that match a list of
of glob patterns.
Recursive globs are supported. It is worth noting that a glob like "**/*" will
return a list of every entry in the filesystem. This is potentially expensive
and should be avoided. simpkins is in favor of disallowing this as a forcing
function to encourage tool-makers to adopt patterns that don't rely on a
complete listing of the filesystem.
For now I'd like to get this in without such a restriction; it's also worth
noting that running `find .` in the root of the mount point has a similar
effect and we can't prevent that from happening, so the effect of the overly
broad glob is something that we need to be able to withstand in any case.
Unrestricted recursive globs will make it easier to connect certain watchman
queries in the interim, until we have a more expressive thrift API for walking
and filtering the list of files.
Note: I've removed the wildmatch flags that I'd put in the API when I stubbed
it out originally. Since this is built on top of our GlobMatcher code and that
doesn't have those flags, I thought it would be simplest to just remove them.
If we find that we need them, we can figure out how to add them later.
Also Note: the evaluation of the glob is parallel-ready but currently limited
to 1 at a time by constraining the folly::window call to 1. We could make this
larger but would need a more intelligent constraint. For example, a recursive
glob could initiate N concurrent futures per level where N is the number of
sub-dirs at a given level. Using a custom Executor for these futures may be a
better option to set an upper bound on the number of concurrent jobs allowed
for a given glob call.
Depends on D4361197
Reviewed By: simpkins
Differential Revision: D4371934
fbshipit-source-id: 444735600bc16d2c2185f2277ddc5b51f672600a
Summary:
As part of the mount process, make sure we load InodeBase objects for all files
in the mount that are already materialized.
Other parts of the code assume that InodeBase objects are always loaded for
materialized files. We never unload materialized inodes, but if the mount
point was unmounted then remounted we previously did not ensure to load the
materialized InodeBase objects. This diff makes sure we load all materialized
inodes before starting the mount.
Reviewed By: bolinfest
Differential Revision: D4461193
fbshipit-source-id: 70d06fd01e2df333ce2816d5d7a392b0a5d6e1e6
Summary:
Fix two issues with inode load completion:
1) Make sure we release the parent's TreeInode lock before fulfilling promises
waiting on the inode load to complete. Otherwise this can result in deadlock
if the promises have callbacks that then try to acquire the TreeInode lock.
2) Use unique_ptr<InodeBase> in the load completion code before we have
inserted the Inode into the InodeMap, rather than using InodePtr. The new
InodePtr class relies on the InodeMap to handle ownership, and it cannot
destroy Inode objects unless they are in the InodeMap. Using a unique_ptr
instead ensures that the InodeBase will be destroyed properly if an error
occurs between when we construct the InodeBase and when we insert it into the
map.
Reviewed By: bolinfest
Differential Revision: D4427889
fbshipit-source-id: 2627a4956735f4acebe5c259abd8de0430ff6177
Summary:
Rename the current EdenMount::getInodeBase() function to
EdenMount::getInodeBaseBlocking() and add a new getInodeBase() function that
performs the lookup asynchronously and returns a Future<InodePtr>.
This is implemented with a TreeInode::getChildRecursive() function that allows
asynchronous recursive lookups at any point in the tree.
Reviewed By: bolinfest
Differential Revision: D4427855
fbshipit-source-id: aca55e681a48c848b085b7fc6a13efe6cf0a4e3a
Summary:
Add methods to InodeBase to get the parent of the current Inode, and remove the
existing TreeInode::getParent() function. The old TreeInode parent_ member was
never updated properly after rename or unlink operations, so it could return
incorrect information.
Reviewed By: bolinfest
Differential Revision: D4427103
fbshipit-source-id: cb66a014f745b4af31033e0a3c5405e29791b869
Summary:
Thesis: Since we don't yet have `hg update` implemented, these attributes can be cached forever.
When we implement `hg update` we will explicitly inform the kernel to invalidate anyway.
... and it turns out that the `scmRemove` thrift call is making changes that
aren't visible to the kernel already so we need to hook up some basic
invalidation.
This diff includes the bare minimum code to facilitate this; I've added a
helper method to the fusell::RequestData class to test whether we are in the
context of a FUSE request.
In the EdenDispatcher::unlink method we need to explicitly invalidate if we are
not being called via FUSE.
I have mixed feelings about putting this code in here. Given the initial
mental model I had in the prototype, this is the right place to put it, but it
does mean that all tree mutations must have via the Dispatcher in order for
invalidations to be routed to the correct places. Do we want to move away from
this?
It wouldn't be the end of the world to expand this diff a bit to add
similar/appropriate invalidation calls to all the mutation methods in our
dispatcher implementations while we figure out if we want to keep this.
Reviewed By: simpkins
Differential Revision: D4452961
fbshipit-source-id: 9471f145242fce0620c6872b74b02c56c8d78af1
Summary:
This mirrors the capability added in D4444058.
I don't think the `getenv` calls are awesome, but also don't think that we're
likely to want to override these outside of the environment either, so it
doesn't warrant plumbing these all the way through the eden cli, edenfs command
line flags and the layers to get to instantiate this object.
Reviewed By: simpkins
Differential Revision: D4446320
fbshipit-source-id: d5661e4f3e8dee82617eb6edddbcb9da5f4296d2
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures. These were caused by the integration tests picking up the
default post-clone hook configuration.
This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.
Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.
The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.
Reviewed By: bolinfest
Differential Revision: D4446321
fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
Summary:
It's been bothering me that we claim to run and complete running
the globally installed hooks when they aren't there, so tidy up the logging.
Reviewed By: bolinfest
Differential Revision: D4446318
fbshipit-source-id: acda223e1df2302602ad9967b1aa1b334b8477c7
Summary:
This adopts the new InterpolatedPropertyTree class as
the configuration storage.
It doesn't introduce any actual configuration of the interpolation
replacements; that will be in a follow-on diff.
Reviewed By: bolinfest
Differential Revision: D4444157
fbshipit-source-id: 18ead8e9074d23b1154e81f012f0c90efced1350
Summary:
This makes it possible to deploy via packaging (we'll put the par file
in the same dir as the extension) and eliminates a recursive buck build call
that was slowing down our integration tests; we can now just look up the path
from the environment that we set up in the TARGETS file to locate the par files
in the integration tests.
Reviewed By: bolinfest
Differential Revision: D4444096
fbshipit-source-id: 2ad6b6aa783563dc54a1cfd347a509c9919bb17a
Summary:
I don't know that I'm happy with the name; it started out as
just a helper wrapper to apply interpolation, but grew a bit to better
encapsulate how we access the config from the eden server.
This is a utility class that I'd like to use in place of the raw boost
property tree in our ClientConfig code.
It encapsulates the underlying property tree and provides a couple of simple
methods; one for loading data from a config file and one for looking up
a configuration value.
The configuration value lookup will replace tokens of the form `${KEY}`
with the value of `KEY` from its internal replacements dictionary.
This mirrors the interpolation handling that we do in the equivalent python
code for the eden client.
Reviewed By: bolinfest
Differential Revision: D4444075
fbshipit-source-id: 5d46d63f87caad4f409fbb981aa83165fcd6596d
Summary:
I'm going to follow up with some changes to substitute
some core env vars; this is just some prep.
Reviewed By: bolinfest
Differential Revision: D4444030
fbshipit-source-id: ab5d0e39aba14cdba39bb4867ec24955665723c6
Summary:
`eden config` dumps all parsed config values.
`eden config --get SECTION.OPTION` prints the value of the corresponding key.
Reviewed By: bolinfest
Differential Revision: D4434084
fbshipit-source-id: 63d9f72aa8794371e89a6d9e527bacaab17540de
Summary:
This makes it easier to reason about where these things can be found,
and in particular, allows passing the path through to the post-clone hook
script. Although post-clone accepts some command line configuration options,
our hooks scheme doesn't seem to allow setting that in the eden configuration
file.
Reviewed By: bolinfest
Differential Revision: D4443872
fbshipit-source-id: ed92dbac3b91b7f7dc86a39a85b465907f290b47
Summary:
We implement some sanity checking in wrappers, preventing some unsafe
consequences of using raw Buck UI in fbcode.
Reviewed By: Gownta
Differential Revision: D4453147
fbshipit-source-id: 3069898069c7e89223b133224f7c87d1a6b5886a
Summary:
Fix issues flagged by running "arc lint" on all eden files. There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.
Reviewed By: bolinfest
Differential Revision: D4446615
fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
Summary:
10 seconds isn't long enough for the tests to pass consistently on my devserver (I have a windows VM running on here, so there is more background crap running).
Raising this up to 30 gave me a successful test run.
Reviewed By: bolinfest
Differential Revision: D4442300
fbshipit-source-id: 5c3ae1fc0d0acb21c5bb44e36dae88d5180749d1
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Previously we'd emit an AttributeError. This diff
fixes things up so that we show the help when no arguments are provided.
Reviewed By: simpkins
Differential Revision: D4434060
fbshipit-source-id: 838b1cab47118d8517e4dc0c5fde17fc69752b13
Summary:
This lock protects all operations that change the location of existing inodes
(rename(), unlink(), and rmdir()). This is needed to avoid race conditions
during rename operations.
I also included some FIXME comments about properly updating location
information for unloaded inodes. These should be taken care of in a subsequent
diff.
Reviewed By: wez
Differential Revision: D4361197
fbshipit-source-id: 5aaa1b8250f196b23207be8f3c0fd7f2603e6ae8
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.
The inode lookup itself still happens serially at the moment. We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.
Reviewed By: wez
Differential Revision: D4361091
fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
Summary:
This updates the InodePtr and InodeBase code to actually implement Inode
unloading and destruction.
At the moment we keep Inode objects as long as possible, and only unload them
during shutdown, or when the last reference to an unlinked inode goes away.
However, it should be straightforward to add on-demand unloading in the future
for Inodes that have not been accessed in a while. The
TreeInode::unloadChildrenNow() function provides a template for what this would
look like (it would simply need to be changed to check an access time when
doing on-demand unloading).
Reviewed By: wez
Differential Revision: D4360765
fbshipit-source-id: a46b355f0ac0c25f873a156e62af5184317de735
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function. This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.
This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.
Reviewed By: wez
Differential Revision: D4360558
fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
Summary:
This defines our own custom smart pointer type for Inode objects. This will
provide us with more control over Inode lifetime, allowing us to decide if we
want to unload them immediately when they become unreferenced, or keep them
around for a while.
This will also allow us to fix some memory management issues around EdenMount
destruction. Currently we destroy the EdenMount immediately when it is
unmounted. This can cause issues if other parts of the code are still holding
references to Inode objects from this EdenMount. Our custom InodePtr class
will also allow us to delay destroying the EdenMount until all of its Inodes
have been destroyed.
This diff adds the new pointer types and updates the code to use them, but does
not actually implement destroying unreferenced inodes yet. The logic for that
has proven to be slightly subtle; I will split it out into its own separate
diff.
Reviewed By: wez
Differential Revision: D4351072
fbshipit-source-id: 7a9d81cbd226c9662a79a2f2ceda82fe2651f312
Summary: Folly will soon be dropping support for GCC 4.8, which means that `folly::make_unique` will be going away, so codemod away as much as possible before then.
Reviewed By: yfeldblum
Differential Revision: D4411221
fbshipit-source-id: 31d61425f6595d0750ea2e4c95c7d42bb5a5d955
Summary: I fixed an unrelated bug in HTE that made some more paths lintable - and some more move-abuses visible :-(
Reviewed By: meyering
Differential Revision: D4397732
fbshipit-source-id: 560db05f64ecf160e8198caf80dbd988e0dad75d
Summary:
I ported the C++ test I wrote for `hg add` in D4317871 as an integration test.
Note that this caught a real bug in the Hg extension!
Reviewed By: wez
Differential Revision: D4385970
fbshipit-source-id: 97ee555d463f22af54791f334f9349d88861f45a
Summary:
This constructor is unsafe. The check it uses before doing the comparison isn't a safe enough check to see if the cast is valid. For example, this is broken in the presence of multiple inheritance because it doesn't adjust the pointer offset to the correct vtable.
e.g.
struct A {
virtual ~A() {};
virtual void doSomething() = 0;
};
struct B {
virtual ~B() {}
virtual void doSomethingElse() = 0;
};
struct C : public B, public A {
virtual ~C() {}
void doSomething() override {
std::cout << "Something!" << std::endl;
}
void doSomethingElse() override {
std::cout << "Something Else!" << std::endl;
}
};
int main (int argc, char **argv) {
auto c = folly::makeFuture<std::shared_ptr<C>>(std::make_shared<C>());
folly::Future<std::shared_ptr<A>> a = std::move(c);
a.get()->doSomething();
return 0;
}
This code will print "Something else!" when run.
Reviewed By: siyengar
Differential Revision: D3679673
fbshipit-source-id: dcbf40ca82d458f17ee11191591f8b8daf58c919
Summary:
If the EDEN_DAEMON_ARGS environment variable is set when running integration
tests, append it to the edenfs daemon command line.
For instance, this makes it easy to pass in a "--vmodule" flag to turn up
verbose logging in particular parts of the code.
Reviewed By: bolinfest
Differential Revision: D4359306
fbshipit-source-id: 6b549ac7220f6c8d7a1dfad3827921d6462d0f77
Summary:
Add VLOG() statements for every FUSE entry point. Turning up the VLOG level in
EdenDispatcher now allows us to have a record of all FUSE calls made using
inode numbers. (Calls involving file handles or directory handles currently go
directly to the handle in question and do not pass through the EdenDispatcher.
We should probably add log statements for those too, but that can wait for a
later diff.)
Reviewed By: bolinfest
Differential Revision: D4359066
fbshipit-source-id: b378d829c39fdb61faf63d2d400f3ff556c376e8
Summary:
This member variable was not updated properly when files were renamed.
InodeBase now tracks our parent properly, so we don't need our own copy.
This does still call getParentBuggy() (which does not perform proper locking)
in a couple places for performing overlay operations. We'll need to fix this
later when addressing other overlay concurrency handling issues.
Reviewed By: bolinfest
Differential Revision: D4348481
fbshipit-source-id: 19c1ffced6f63e1ff041d0bab2363fecdb93d5a3
Summary:
Have FileData objects store a pointer to the FileInode that owns them, rather
than just to the EdenMount.
FileData objects still store a direct pointer to their FileInode's mutex_ and
entry_. It's potentially worth just accessing these through inode_ in the
future.
Reviewed By: bolinfest
Differential Revision: D4348103
fbshipit-source-id: 1f8497979bfc89c6a192ca0195209335db0d911c
Summary:
Add an EdenMount* member variable to InodeBase. Previously each TreeInode kept
a pointer to its EdenMount, and this moves that into the InodeBase class.
This is needed to support upcoming diffs that will change InodeBase memory
management. InodeBase objects will be responsible for notifying the InodeMap
when they become unreferenced.
Reviewed By: bolinfest
Differential Revision: D4348079
fbshipit-source-id: 6bf7ea908c6096aa2bca5b21290c09cbd58d5af7
Summary:
This updates all of the eden code to use the new InodeMap class. This replaces
the InodeNameManager class and the unordered_map previously stored in the
EdenDispatcher.
Reviewed By: bolinfest
Differential Revision: D4325750
fbshipit-source-id: d80ae7581ba79ca2b63155e184995a3e83e85dc1
Summary:
This diff starts adding a new InodeMap class. This class will eventually
consolidate the functionality of InodeNameMap plus the inode map stored in
EdenDispatcher.
This new class will bring several new benefits:
- All inode mapping logic consolidated into a single class, with a single lock
protecting the maps.
- A well-defined model for loaded vs unloaded inodes. InodeMap explicitly
tracks inodes that have InodeBase objects created for them vs inodes that
have an inode number allocated (for FUSE) but do not have an InodeBase object
in memory. This will make it possible to unload Inode objects on demand to
reduce memory usage.
- Tracking of pending loads, and de-duplicating load requests. This ensures
that only one Inode object ever exists for a given inode number / path. If a
second request to load an Inode arrives when a previous load request is still
in progress, InodeMap deals with this situation properly.
- Better support for using Inode objects without FUSE. With the old code,
attempts to interact with Inode objects without going through the FUSE
dispatch (say, when processing a thrift call) could result in inconsistent
state. New inodes created would not be put into the EdenDispatcher map,
which could result in problems.
- More convenient child inode access from TreeInode. With this change, the
TreeInode class can easily tell which of its children are loaded. This makes
it easier to do tasks which only need to operate on existing loaded inode
state (for instance, dirstate computation).
- Support for saving and loading state, to implement graceful restart..
InodeMap provides a central place to write out inode state on shutdown and
restoring it on startup. Saved inodes can easily be restored to an
"unloaded" state on startup. This code is not implemented yet as part of
this diff, but it should be straightforward to add in future diffs.
Reviewed By: bolinfest
Differential Revision: D4318060
fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
Summary:
This introduces an `HgExtensionTestBase` class that can be subclassed
to create tests for Mercurial integration with Eden. It also introduces
`status_test.py`, which includes some basic integration tests for `hg status`
in Eden. Although our existing `.cpp` tests give us a lot of coverage, they
do not verify the logic in the Python code for our Mercurial extension,
so this will help me sleep better at night.
Note that `HgExtensionTestBase` is a subclass of `testcase.EdenHgTest`
rather than using the `testcase.eden_repo_test` decorator because these
tests will only be used in the Mercurial case for now, so there is no point in
trying to run them when creating the mount from a Git repo, as well.
Further note that this also required a small change to `eden/fs/integration/lib/testcase.py`
because the Eden directory was not being created at `$HOME/local/.eden`, which
is something that the Mercurial extension relies on.
Reviewed By: simpkins
Differential Revision: D4337660
fbshipit-source-id: ea825134420ae6eb794d37b6c75da19a6aadaae5
Summary:
This preserves the existing behavior while making `REPO_ROOT` and `BUCK_OUT`
available to other test code.
Reviewed By: simpkins
Differential Revision: D4358875
fbshipit-source-id: a7b2cafe7e56673831beac59a85fcab436e637a5
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.
This revision was initially created by running:
```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```
And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.
But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.
Reviewed By: simpkins
Differential Revision: D4356724
fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
Summary:
Rename the existing TestBackingStore class to FakeBackingStore, and fill it out
with an implementation that allows test code to control the store.
The test code can populate the store with Trees and Blobs to return, and can
control when the Futures returned by the store are fulfilled.
Reviewed By: bolinfest
Differential Revision: D4338577
fbshipit-source-id: 79221b04d844bd6011078b799e55182de4ccdfdc
Summary:
Upstream mercurial commit d83ca854fa21bef17bf3f097087c691c7d07bb5b changed the
mercurial.ui API, breaking existing scripts using it.
ui objects now have to be created using the ui.load() classmethod, rather than
by constructing them directly. (Creating a ui object directly no longer loads
the global or user configs.)
Reviewed By: bolinfest
Differential Revision: D4354629
fbshipit-source-id: 650db3abab9f2eebaefa2b4d8029c169ccc30217
Summary:
This is a first diff to remove the "easiest" unused includes in fbcode.
* For safety, we only touch .cpp files without #if and #define,
* We do not try to remove redundant systems headers (aka. "packing").
The diff was generated as follows:
```
foundation/scripts/ls-cpp-dirs | grep -v '^\(\.\.\|external/\|.*/external\)' | xargs ffmr -o /tmp/ffmr-diff-1 codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
cat /tmp/ffmr-diff-1/*.diff | patch -p2
hg commit -m something
arc diff --prepare --nolint --nounit --less-context --excuse refactoring
```
Note: `grep -v` is just an optimization. The actual configuration is in these two files:
diffusion/FBS/browse/master/fbcode/codegraph/analysis/config.py
diffusion/FBS/browse/master/fbcode/codegraph/scripts/ffmr/analyze_includes_no_headers_no_packing_skipping_ifdefs.sh
See the task for more context, and the recent "safety" improvements on the tool.
depends on D4317825 for very few cases where `nolint` had to be manually added.
Reviewed By: igorsugak
Differential Revision: D4312617
fbshipit-source-id: ecc1f0addfd0651fa4770fcc43cd1314661a311a
Summary:
Most of the work for this diff was achieved via:
```
find eden -type f | xargs sed -i -e 's#ThriftHgStatusCode#StatusCode#g'
find eden -type f | xargs sed -i -e 's#HgStatusCode#StatusCode#g'
```
Reviewed By: simpkins
Differential Revision: D4237975
fbshipit-source-id: 2ee04a89101291c8972ac7bd3ff6cca92cbb0799
Summary:
Add a helper function to create human-readable hashes in test code without
having to always specify a full 40-byte long string.
Reviewed By: wez
Differential Revision: D4336161
fbshipit-source-id: cf6af58dd788b5553a2a6daef56db43cddbbc04a
Summary:
This should make the common case of `hg add` or `hg add .` much more efficient
because it no longer performs a walk of the entire repository from the client
side.
Reviewed By: simpkins
Differential Revision: D4317871
fbshipit-source-id: 7061553fe0de0c4afa84b4f835316965088675e8
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.
This updates the main program to process path arguments using canonicalPath().
Reviewed By: bolinfest
Differential Revision: D4332273
fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
Summary:
This code compiled fine with clang, but gcc apparently requires explicitly
specifying "this->" when referring to members inside a lambda body.
Reviewed By: bolinfest
Differential Revision: D4332357
fbshipit-source-id: 3919dad4fca0b6676731f21781331be73fde8e55
Summary:
The default mode of detection assumes that the test runner par
will be the one from the eden dir.
In the watchman<->eden integration tests this is not the case, so we need
to provide a way to pass down the correct location.
The watchman tests (see D4323716) are using a TARGETS fragment like this:
```
eden_env = env.copy()
eden_env.update({
'EDENFS_CLI_PATH': '$(location @/eden/fs/cli:cli)',
'EDENFS_SERVER_PATH': '$(location @/eden/fs/service:edenfs)',
})
python_unittest(...
env=eden_env,
...)
```
We may want to cut eden over to a similar construction, but I've left
that alone for the moment.
Reviewed By: simpkins
Differential Revision: D4323695
fbshipit-source-id: f48afa6eef82a806c7b10e187d41e6253039d612
Summary:
Fix Dispatcher::symlink() to accept the symlink contents as a StringPiece
rather than a PathComponentPiece. symlink contents can be any arbitrary
string, and are not required to be a valid, normalized path name.
Reviewed By: wez
Differential Revision: D4325380
fbshipit-source-id: 88448bee50ea192c06442dc70042c7d17d49a12f
Summary:
Update EdenServiceHandler::getSHA1ForPath() to replace its own custom path
lookup code with EdenMount::getFileInode().
This also ended up fixing the error message to correctly include the path name
on EISDIR errors.
Reviewed By: wez
Differential Revision: D4325066
fbshipit-source-id: 9aa3932c71c33e6bc11d2c71cc8f1badb4c0dcb7
Summary:
InodeError is a subclass of std::system_error that accepts an InodePtr to the
inode that it refers to. This makes it easier to construct error objects that
retain information about the inode they refer to.
InodeError also avoids computing the inode path until the error message is
actually needed. This should make it less expensive in cases where errors are
thrown and handled internally without ever using the human-readable error
message. It is possible that the file may have been renamed or unlinked by the
time the error message is computed. However, this race condition might still
exist even if we computed the path at the time when the error is constructed.
getLogPath() will construct a usable human-readable string even if the file has
been unlinked.
Reviewed By: wez
Differential Revision: D4325043
fbshipit-source-id: c9683a80b022f281ca4583a9b7f73b15277335bb
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.
For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready. I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.
Reviewed By: bolinfest
Differential Revision: D4318055
fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
Summary:
This will make it easier to implement `hg add <directory>` such that
`<directory>` is expanded on the server rather than on the client.
Reviewed By: simpkins
Differential Revision: D4318735
fbshipit-source-id: cf0e89bd95eb58304cd23e70beb77bc7151f2c5c
Summary:
Previously, `.hg` entries were filtered when calling `hg status` on the client,
but this should be happening on the server. This updates the logic in
`Dirstate.cpp` to properly exclude `.hg` when traversing the overlay for
modified directories, so this will eliminate a bunch of unnecessary computation
and simplify the client.
I'm unsure of how best to implement the ownership relation for the set that
contains `.hg`. Please advise! I know that I could categorically exclude
`".hg"` in `getModifiedDirectoriesRecursive()`, but I haven't used it enough
scenarios yet to be sure that's the right thing to do. For example, if it were
a Git repo, arguably we should consider `.hg` and not `.git`, so I could also
require the set to be a parameter of `Dirstate`, but I want to make sure I get
the ownership stuff right.
Reviewed By: simpkins
Differential Revision: D4316531
fbshipit-source-id: a0f13ca1c3c620b686435c8aa6485ba4e850f043
Summary: These should have been added as part of D4270526.
Reviewed By: simpkins
Differential Revision: D4315382
fbshipit-source-id: 5920ff38f9cc63540e4813e8ab40f79ad46f9ec1
Summary: Follow-up on a comment that came out of the review for D4249214.
Reviewed By: simpkins
Differential Revision: D4314979
fbshipit-source-id: 76384474092e6fd48394f6faf8b84ba6220c556a
Summary:
Move the FakeObjectStoreTest class into fs/testharness, along with the
TestMount and TestBackingStore classes. This simply consolidates the test
utility code into a single location.
Reviewed By: bolinfest
Differential Revision: D4317517
fbshipit-source-id: 4e19590c5ffde88b66f2c8d4a964352ec349031c
Summary:
Hash objects are small enough (20 bytes) that it isn't worth allocating them on
the heap. This updates LocalStore::getSha1ForBlob() to return a
folly::Optional<Hash>, and ObjectStore::getSha1ForBlob() to return a plain
Hash.
Reviewed By: bolinfest
Differential Revision: D4298162
fbshipit-source-id: 9cf54f2997ba8c3b2346db315a2aca41e580b078
Summary:
Add comments in ObjectStore.h documenting the fact that the get* APIs all throw
std::domain_error when the specified ID does not exist, and never return
nullptr.
Also update the FakeObjectStore class used for testing to follow this behavior.
Reviewed By: bolinfest
Differential Revision: D4298160
fbshipit-source-id: c5509bb3aa2ed76619b06b733ad240aaa5f00862
Summary:
In addition to storing the SHA-1 of each file's contents, also store the size.
This will allow us to more quickly look up the file size, without having to
retreive the file size.
I haven't yet added an API to ObjectStore to retreive the full BlobMetadata
object; I will do that in a subsequent diff. One benefit for now is that this
does avoid double-computing the SHA-1 in ObjectStore::getSha1ForBlob() if we
had to load the blob.
Reviewed By: bolinfest
Differential Revision: D4298157
fbshipit-source-id: 4d83ebfa631c93fcef06ca1cd0ba0e1a70a2476d
Summary:
Define InodePtr, TreeInodePtr, and FileInodePtr as aliases for std::shared_ptr
of the underlying inode type. This also updates all of the code to use these
new type names.
This will make it easier swap out std::shared_ptr with a custom pointer type in
the future. (I believe we will need a custom type in the future so that we
can have more precise control of the reference counting so we can load and
unload Inode objects on demand. std::shared_ptr::unique() doesn't quite
provide the flexibility we need, and is also being deprecated in C++17.)
Reviewed By: bolinfest
Differential Revision: D4297791
fbshipit-source-id: 1080945649290e676f62689592159f1166159b20
Summary:
This updates the InodeBase code to track its location in the filesystem.
Since we do not support hard links, each inode has a single path where it
exists.
Tracking this data allows us to implement getPath() as a method of InodeBase.
This code is not really complete yet, but it seems worth getting the current
code in as-is. The location data is not updated properly on unlinks or
renames, but it looks like the existing InodeNameManager code does not get
updated either. I am working on some additional refactoring of inode object
management, and it will be easier to come back and fix the unlink and rename
handling after this refactoring is further along.
Reviewed By: bolinfest
Differential Revision: D4297591
fbshipit-source-id: 82ceb326e4f9c376f627b1d8f49bb7db3cfc2b0b
Summary:
In this revision, we override `committablectx.markcommitted()` to make a Thrift
call to Eden to record the new commit. For now, this defers the need for us to
implement `edendirstate.normal()`, though we expect we will have to provide a
proper implementation at some point.
Because `hg update` is not implemented yet, this puts us in a funny state where
we have to restart eden after `hg commit` to ensure all of our `TreeEntry` and
other in-memory data structures are in the correct state.
Reviewed By: simpkins
Differential Revision: D4249214
fbshipit-source-id: 8ec06dfee67070f008dd93a0ee6c810ce75d2faa
Summary:
This refines the initial support for `hg remove` by adding support for
directories.
Reviewed By: simpkins
Differential Revision: D4270546
fbshipit-source-id: c97dfea555ad489ddda01ad2587f1856b1953e02
Summary:
`getEntryForPath()` is helpful when we don't know (or care) whether the path
corresponds to a file or a directory.
Reviewed By: simpkins
Differential Revision: D4270526
fbshipit-source-id: 6dd2f76df1749e040dda788a74055d9da2156a4d
Summary:
This adds support for `hg remove <file>` in Eden and sets up some of the scaffolding
to support `hg remove <directory>`.
Note that the Thrift API for `scmRemove()` is slightly different than that of `scmAdd()`
in that it returns a list of error messages to display to the user rather than throwing
an exception. In practice, for batch operations, Mercurial will allow some operations
to succeed while others may fail, so it is possible to have multiple error messages to
return.
Unlike the current implementation of `hg add`, this does the directory traversal
on the server rather than on the client. Once we work out how to do this for
`hg remove`, we should figure out how to reuse the logic for `hg add`.
Reviewed By: simpkins
Differential Revision: D4263068
fbshipit-source-id: d084774d562c48c59664f313eba229d4197929fe
Summary:
The check on the type of an exception was inverted, which meant
`hg remove <path>` would throw an exception if the parent directory of `path`
did not exist. This is not correct because the user should be able to expect
to do:
```
mkdir -p /tmp/example
cd /tmp/example
hg init
mkdir mydir
touch mydir/a
hg add mydir/a
rm -rf mydir/a
hg rm mydir/a
```
In this scenario, `mydir` does not exist when `hg rm` is called, but the command
should succeed, making `mydir/a` no longer tracked.
Reviewed By: simpkins
Differential Revision: D4268451
fbshipit-source-id: 517d81252aa8e4b6bd1a32dece14776a9f7dd6f7
Summary:
A `TreeEntry` reports a `mode_t` whose "group" and "other" bits are set in a way
that reflects the "owner" bits (so they are non-zero). By comparison, the `mode`
of a `TreeInode::Entry` will reflect the permissions on disk in the overlay (if
the file is materialized). In general, the overlay bits will probably be the
same as those in the `TreeEntry` since we expect the user will rarely mess
with the "group" and "other" bits, but we're seeing a difference more often
right now because of t14953681.
Reviewed By: simpkins
Differential Revision: D4298214
fbshipit-source-id: 2919be94c6bba61135838ee86bbc68aa4031af7c
Summary:
Move the InodeBase class from the lower-level fusell code up to the
eden/fs/inodes layer, now that everything else that uses it is in
eden/fs/inodes.
I plan to start changing the ownership model of inode objects a bit, and this
will allow the InodeBase class to interact with EdenDispatcher and other
classes in eden/fs/inodes.
Reviewed By: bolinfest
Differential Revision: D4283392
fbshipit-source-id: 9e1d6fb81dc223f905847cbe8d165a40ad0aca4d
Summary:
`stdin`, `stdout` and `stderr` are macros that expand to function calls with the MSVC CRT implementation. This is also the case for musl-libc. This means that Subprocess simply cannot be compiled on those platforms without changing the API.
To solve that, we change the API and deprecate the old API.
For more fun, `stdin`, `stdout` and `stderr` are also macros in glibc, they just expand to other identifiers rather than a function call.
Reviewed By: yfeldblum
Differential Revision: D4229544
fbshipit-source-id: 97f1a3b228b83cfdcaffee56d729063ea235e608
Summary:
Update the TestMount APIs to use the term "TreeInode" instead of "DirInode" now
that the DirInode class is no more.
Reviewed By: bolinfest
Differential Revision: D4257171
fbshipit-source-id: 2407dfc25405f7161987260c0299f1723831e264
Summary:
Now that the EdenDispatcher class has been moved into eden/fs, we no longer
need the distinction between TreeInode and fusell::DirInode, and FileInode and
fusell::FileInode.
This diff deletes the fusell versions of these classes, and updates all of the
code to always directly use TreeInode and FileInode. This allows us to get rid
of the remaining dynamic_casts between these pairs of classes.
Reviewed By: bolinfest
Differential Revision: D4257165
fbshipit-source-id: e2b6f328b9605ca0e2882f5cf7a3983fb4470cdf
Summary:
Move the InodeDispatcher class out of the lower-level fusell namespace in
eden/fuse and into the higher-level eden code in eden/fs/inodes. I also
renamed it from InodeDispatcher to EdenDispatcher, in anticipation of it
getting more eden-specific functionality in the future.
The fusell::MountPoint class is now independent of the Dispatcher type, and can
work with any Dispatcher subclass. Previously the MountPoint class was
responsible for owning the InodeDispatcher object. Now its caller (EdenMount
in our case) is responsible for supplying a Dispatcher object that is owned
externally.
Several parts of EdenDispatcher had to be updated as a result of the namespace
move, but I tried to keep this change somewhat minimal. I did update it from
using fusell::DirInode and fusell::FileInode to eden's TreeInode and FileInode
classes directly. However, there still remains more clean-up work to do. I
will split remaining changes out into upcoming diffs.
Reviewed By: bolinfest
Differential Revision: D4257163
fbshipit-source-id: dc9c2526640798f9f924ae2531218ba2c45d1d0a
Summary:
Drop the MountPoint::setRootInode() method, and have EdenMount perform the
operation directly on the InodeDispatcher.
Reviewed By: bolinfest
Differential Revision: D4257158
fbshipit-source-id: af4a696de2d36979c658972104361f225f482338
Summary:
Update call sites in eden/fs to access the InodeNameManager through the
EdenMount object rather than the MountPoint.
It turns out that there was only one call site in TreeInode, and all other
callers in eden/fs get it indirectly via TreeInode::getNameMgr().
Reviewed By: bolinfest
Differential Revision: D4257156
fbshipit-source-id: 9f0212134b20c8dd8943827c17aa16ee7274bc36
Summary:
Move getInodeBaseForPath(), getDirInodeForPath(), and getFileInodeForPath()
entirely into the EdenMount class, and make sure all call sites are using the
EdenMount methods rather than the MountPoint methods.
Reviewed By: bolinfest
Differential Revision: D4257153
fbshipit-source-id: d528cfad174757c3c9f23e62a0616f8bf1976da7
Summary:
Update all code in eden/fs to call EdenMount::getDispatcher() instead of
getting the underlying MountPoint from the EdenMount and then calling
getDispatcher() on it. This will allow me to move the InodeDispatcher from
MountPoint to EdenMount in a subsequent diff. This also simplifies many of the
callers of this method.
Additionally, add an EdenMount::getRootInode() method, and update call sites to
use this rather than having to look up the InodeDispatcher and call
getRootInode() or getDirInode(FUSE_ROOT_ID) on it.
Reviewed By: bolinfest
Differential Revision: D4257152
fbshipit-source-id: 33e6f6b8853db2a88f4f2c221122eea50e796390
Summary:
This updates the Dirstate to also check if untracked files are ignored or not.
This is somewhat inefficient, as we have perform a separate check for each
untracked file we find. Ideally we should perform all of the dirstate
computation in a single tree walk, and check ignore status as we go. This
would allow us to skip ignored directories entirely, rather than potentially
having to check each file inside them. I intend to work on cleaning this up in
the future, but it will require refactoring some of the inode code first.
Reviewed By: bolinfest
Differential Revision: D4225308
fbshipit-source-id: 49e444c85cbb6ede11cc6e19052fdd16cf8aab9f
Summary:
Update the Dirstate to store a pointer to the EdenMount object that owns it,
rather than storing pointers to the lower-level MountPoint and ObjectStore
objects.
This change is necessary in order for me to move more functionality from
MountPoint to EdenMount. (In particular, I plan to move the InodeDispatcher to
the EdenMount.)
As part of this change I also started moving some APIs from MountPoint to
EdenMount. For now the EdenMount versions are just thin wrappers on top of the
MountPoint APIs. I will move the functionality directly into EdenMount in a
future diff.
Reviewed By: bolinfest
Differential Revision: D4255675
fbshipit-source-id: 93749c6516c3cea4b4ae93de4ca49ddf05f4d260
Summary:
Write the dirstate data using the new folly::writeFileAtomic() function.
This ensures that the dirstate file will always contain full, valid contents,
even if we crash or run out of disk space partway through writing out the data.
This diff also includes a couple other minor tweaks:
- Update Dirstate to store the DirstatePersistence object directly inline,
rather than allocating it separately on the heap.
- Update the DirstatePersistenceTest code to prefix temporary directories with
"eden_test". This just makes it easier to tell if the tests fail to clean up
after themselves for any reason.
Reviewed By: bolinfest
Differential Revision: D4254027
fbshipit-source-id: 6b6601b65aeacdee998a6c4260e972d5fb2426ac
Summary:
This cleans up construction of the EdenMount and Dirstate objects:
- The EdenMount constructor is now responsible for creating the Overlay and
Dirstate objects.
- The Dirstate constructor is now responsible for loading the
DirstatePersistence file.
- The EdenMount now takes ownership of the ClientConfig object, and stores it
for later use.
- The ClientConfig object now has a method to get the path to the
DirstatePersistence file.
- I added a ClientConfig::createTestConfig() method, so that the TestMount code
can now use the same EdenMount constructor as the normal code.
This simplifies the logic in EdenServiceHandler and TestMount, and makes some
of the initialization dependencies a little bit simpler.
This change is necessary in order for me to move some logic from
fusell::MountPoint into EdenMount. The Dirstate object will need a pointer
back to its EdenMount object, and this diff enables that.
Reviewed By: bolinfest
Differential Revision: D4249393
fbshipit-source-id: 439786accbf48c8696dbc6ca4fe77a4c6bdeab65
Summary:
Rename TreeEntryFileInode to FileInode, and TreeEntryFileHandle to FileHandle.
These class names were long and awkward.
It's slightly unfortunate that we now have classes named both
eden::fuse::FileInode and eden::fuse::fusell::FileInode, but I don't believe
this should cause any major problems. If we want to eliminate these name
collisions in the future I would advocate for renaming the fusell versions to
something like "FileInodeIface".
Reviewed By: bolinfest
Differential Revision: D4217909
fbshipit-source-id: 899672a318d7ae39595f2c18e171f8fd6cebedc6
Summary:
Previous to this commit, the `Dirstate` logic only worked correctly when the
changes occurred in the root directory. Obviously that is very limiting, so this
adds support for changes in arbitrary directories at arbitrary depths.
This also introduces support for things like a file being replaced by a
directory of same name or vice versa. The tests have been updated to verify
these cases.
One interesting design change that fell out of this was the addition of the
`removedDirectories` field to the `DirectoryDelta` struct. As you can see,
all entries in a removed directory need to be processed by the new
`addDeletedEntries()` function. These require special handling because deleted
directories do not show up in the traversal of modified directories.
In contrast, new directories do show up in the traversal, so they require a
different type of special handling. Specifically, this call will return `NULL`:
```
auto tree = getTreeForDirectory(directory, rootTree.get(), objectStore);
```
When this happens, we must pass an empty vector of tree entries to
`computeDelta()` rather than `&tree->getTreeEntries()`. Admittedly, the special
case of new directories is much simpler than the special case of deleted ones.
Reviewed By: simpkins
Differential Revision: D4219478
fbshipit-source-id: 4c805ba3d7688c4d12ab2ced003a7f5c19ca07eb
Summary:
This is a better fix for the quick fix introduced by D4198939.
It turns out that the `EdenMount` does not need to take ownership
of the `ClientConfig`, so removing the `std::move()` makes this code
much simpler because instead of declaring a bunch of variables
early in `mountImpl()` so that we can "hold on" to them before `EdenMount`
takes ownership of the `ClientConfig`, we can declare them closer to where they
are actually used.
Note that we may want `EdenMount` to actually take ownership of the
`ClientConfig` in the future, but we'll cross that bridge when we come to it.
Reviewed By: simpkins
Differential Revision: D4199000
fbshipit-source-id: 67411a9a5ef630a9d481aebc94631c79da4ab2c4
Summary:
This also introduces the change where the `EdenMount` creates
and takes ownership of the `Dirstate`.
To clean some of this up, I had to expose a `getEdenDir()` method on `EdenServer`
that returns an `AbsolutePathPiece`. This was previously stored internally as a
`std::string`, so I had to clean up a bunch of path construction that was using `edenDir_`.
Reviewed By: simpkins
Differential Revision: D4123763
fbshipit-source-id: 270b182521c1a84bb054832f4b5f92af849d67e4
Summary:
Previously, `Dirstate` took a `std::shared_ptr<EdenMount>`, but now it takes
pointers to a `MountPoint` and an `ObjectStore` because it does not need the
entire `EdenMount`. Ultimately, this will enable us to have `EdenMount` create
the `Dirstate` itself, but that will be done in a follow-up commit.
Fortunately, it was pretty easy to remove the references to `edenMount_` in
`Dirstate.cpp` and rewrite them in terms of `mountPoint_` or `objectStore_`.
The one thing that I also decided to move was `getModifiedDirectoriesForMount()`
because I already needed to create an `EdenMounts` file (admittedly not a
great name) to collect some utility functions that use members of an `EdenMount`
while not having access to the `EdenMount` itself.
As part of this change, all of the code in `eden/fs/model/hg` has been moved to
`eden/fs/inodes` so that it is alongside `EdenMount`. We are going to change
the `Dirstate` from an Hg-specific concept to a more general concept.
`LocalDirstatePersistence` is no longer one of two implementations of
`DirstatePersistence`. (The other was `FakeDirstatePersistence`.) Now there is
just one concrete implementation called `DirstatePersistence` that takes its
implementation from `LocalDirstatePersistence`. Because there is no longer a
`FakeDirstatePersistence`, `TestMount` must create a `DirstatePersistence` that
uses a `TemporaryFile`.
Because `TestMount` now takes responsibility for creating the `Dirstate`, it
must also give callers the ability to specify the user directives. To that end,
`TestMountBuilder` got an `addUserDirectives()` method while `TestMount` got a
`getDirstate()` method. Surprisingly, `TestMountTest` did not need to be updated
as part of this revision, but `DirstateTest` needed quite a few updates
(which were generally mechanical).
Reviewed By: simpkins
Differential Revision: D4230154
fbshipit-source-id: 9b8cb52b45ef5d75bc8f5e62a58fcd1cddc32bfa
Summary: This is an implementation of DirstatePersistence that persists data to a local file.
Reviewed By: simpkins
Differential Revision: D4181868
fbshipit-source-id: 7177b2ef67cd3aec56e5ad10f41169cc5ec69d81
Summary:
Update the gitignore handling code to perform pattern matching the same way git
does. Previously the code just called the standard fnmatch() function, which
does not handle "**" in patterns the same way git does.
This includes our own new implementation of glob pattern matching. I did
evaluate several other options before writing our own implementation here:
- The wildmatch() code used by git (and watchman, and rsync) has a few
downsides: it is not distributed by itself as a library anywhere else.
Therefore we would probably have to include a copy of this code in our
repository. Making another copy is unfortunate, and somewhat undesirable
from a legal and licensing perspective. This code also only works with
nul-terminated strings, and our code deals primarily with non-terminated
StringPiece objects.
- I did look at translating glob patterns in to regular expressions and using
re2 to perform matching. Unfortunately re2 turns out to be substantially
slower than wildmatch() for typical gitignore patterns.
This new implementation performs some preprocessing on the glob pattern, and
generates a pattern opcode buffer. Eden can perform this glob preprocessing
when it first loads a .gitignore file, and can then save and re-use this result
each time it needs to match a filename. Doing this preprocessing allows
matching to be done 50% to 100% faster than wildmatch() for typical glob
patterns.
Reviewed By: bolinfest
Differential Revision: D4194573
fbshipit-source-id: 46bc6a61b6d8066f4bbdb5d3e74265a3e72e42cc
Summary:
This adds some initial code for handling gitignore files.
I did check to see if there were APIs from libgit2 that we could leverage for
this, but it does not look like we can easily use their functionality. The
libgit2 ignore code seems to tightly coupled with their repository data
structures, and it requires that you actually have a git repository.
This code isn't quite 100% compatible with git's semantics yet. In particular:
- For now we are just using fnmatch() to do the matching. This is currently
inefficient as we have to do string allocations on each match attempt. This
also doesn't quite match git's behavior, particularly with regard to "**"
inside patterns.
- The code currently does not have a mechanism for indicating if a path refers
to a directory or not, so trailing slashes in the pattern are not honored
correctly.
We will probably need to implement our own fnmatch-like function in the future
to solve these issues.
Reviewed By: bolinfest
Differential Revision: D4156480
fbshipit-source-id: 8ceaefd3805358ae2edc29bfc316e5c8f2fb7d31
Summary:
For now, initial state is represented by a
`std::unordered_map<RelativePath, HgUserStatusDirective>`.
Reviewed By: simpkins
Differential Revision: D4123461
fbshipit-source-id: 83a99e1f504dd1efca1bc1ed33cbc3f116787a80
Summary:
This adds the logic to power `hg rm`. There are comprehensive tests that attempt to cover
all of the edge cases.
This evolved to become a complex change because I realized that I needed to change
my internal representation of the dirstate to implement it properly. Specifically, we now maintain
a map (`userDirectives`) of files that have been explicitly scheduled for change via `hg add` or `hg rm`.
To compute the result of `hg status`, we find the changes between the manifest/root tree
and the overlay and also consult `userDirectives`. `Dirstate::getStatus()` was updated
considerably as part of this commit due to the introduction of `userDirectives`.
As such, `Dirstate::remove()` must do several things:
* Defend the integrity of the dirstate by throwing appropriate exceptions for invalid inputs.
* Delete the specified file, if appropriate.
* Update `userDirectives`, if appropriate.
Although `Dirstate::add()` was not the focus of this commit, it also had to be updated to
match the pattern introduced by `Dirstate::remove()`.
Some important features that are still not supported are:
* Handling ignored files correctly.
* Storing copy/move information.
Reviewed By: simpkins
Differential Revision: D4104503
fbshipit-source-id: d5d45a279e16ded584c6cd4d528ba92d2c8e2993
Summary:
D4014598 changed this line but didn't change the test expectations.
Since it seems desirable for the mount point name to be used, I've reverted
back to the prior state for this line.
Reviewed By: bolinfest
Differential Revision: D4202265
fbshipit-source-id: bddc01436e0a5921a3b0b2c01c0fd2c32f5f1960
Summary:
Originally, D3858635 was going to introduce a scheme for hooks where the
repo type was included in the path:
/etc/eden/hooks/hg/post-clone <args...>
But over the course of the review, we decided to make the repo type a
parameter:
/etc/eden/hooks/post-clone hg <other-args...>
Unfortunately, `generate-hooks-dir` was not updated as part of that
change and it is not covered by unit tests. This error was particularly hard
to discover because of how `ENOENT` is handled, so I added a log statement for
that.
Reviewed By: simpkins
Differential Revision: D4200277
fbshipit-source-id: ffffd871cd78dcaeb717be8f1e01893ce9643a47
Summary:
This is a quick and dirty fix for this issue that was causing
and confusing bug where the memory for the `AbsolutePathPiece`
was getting reclaimed, so when it was later read as the value
for a path, it failed because it was binary garbage.
This is mainly caused by the `std::move(config)` that passes
the `ClientConfig` to the `EdenMount` constructor. I will do
some more general cleanup for that in a follow-up revision,
but I wanted to have this change in its own commit that makes
it clear where the failure/fix were coming from.
Reviewed By: simpkins
Differential Revision: D4198939
fbshipit-source-id: 19e0423a1bee924fa6cc2edc8bae534ef472c988
Summary:
Use new, less confusing names for mentioned thrift methods.
Codemod with 'Yes to All'. Reverted changes in thrift/
Reviewed By: yfeldblum
Differential Revision: D4076812
fbshipit-source-id: 4962ec0aead1f6a45efc1ac7fc2778f39c72e1d0
Summary:
This is the start of the C++ dirstate implementation. It's possible that this
commit does too many things at once:
* Introduces `Dirstate` type.
* Includes logic for serializing/deserializing the dirstate's data so that it persists across Eden restarts.
* Includes logic for basic `hg add` calls.
* Includes unit tests where we model Eden usage via the TestMount utility.
I'm backing this up in Phabricator with `--plan-changes` to start until I get
some basic `hg add` functionality working end-to-end. When that looks good, I'll
determine if/how this should be split into smaller commits.
Reviewed By: wez
Differential Revision: D4023232
fbshipit-source-id: 7fc931d547ccadb34f7caae93bc4eb8f91f6ceb8
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
Reviewed By: simpkins
Differential Revision: D4073653
fbshipit-source-id: dda1d8ea8d29aa071a31f8e2afab324f9109e9b2
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
As you can see, this helped uncover a bug in the way we were
inserting blobs into `LocalStore`.
Reviewed By: simpkins
Differential Revision: D4073039
fbshipit-source-id: 42683fd0bfdb0a1e77df9324fcaa79091f45e83d
Summary: This is a follow-up revision from a comment on D4013464.
Reviewed By: wez
Differential Revision: D4050278
fbshipit-source-id: 1e46526f58a07e1eedd8ace1a6d84a919240d899
Summary:
This is analogous to the existing `getEntryForFile()` helper function that we
have, and I was able to rewrite `getEntryForFile()` in terms of
`getTreeForDirectory()`, which simplifies the code considerably.
Also moved things from `eden/fs/model/hg/misc.h` to
`eden/fs/store/ObjectStores.h`, which is much more appropriate.
Reviewed By: wez
Differential Revision: D4032817
fbshipit-source-id: ff4d32120fb050f8b5c5c53b7f2e94b524781648
Summary:
This is not a one-liner and this is needed for the upcoming `Dirstate` class,
so moving this code to a place where it is more easily reusable.
Reviewed By: simpkins
Differential Revision: D4032001
fbshipit-source-id: 7d8d87802665ac2993ec0a3ac73c5f645fe4a1aa
Summary:
Performs a depth-first traversal of the overlay to find modified
directories and returns them in that order.
Reviewed By: simpkins
Differential Revision: D4025309
fbshipit-source-id: 09d8ed41b250dddbfb3fe545643ec3fd755a430e
Summary:
Now that I've done all this work, I'm not sure whether it is a good idea or even
necessary. I'll keep it in my back pocket.
Reviewed By: simpkins
Differential Revision: D4014598
fbshipit-source-id: 6ded3cc29838e964b56833ac24dff19e9de040f5
Summary:
These are new helper methods we need to create test scenarios.
They will be used in upcoming revisions.
Reviewed By: wez
Differential Revision: D4046981
fbshipit-source-id: 9c66c456be57006173e4a65eed603de4a426a438
Summary: This should be useful for my upcoming unit tests for the Hg dirstate.
Reviewed By: simpkins
Differential Revision: D4013464
fbshipit-source-id: 46460186abfa104aa026894068cd160e52c94729
Summary: This will make it easier to compare a `TreeEntry` with a `TreeInode::Entry`.
Reviewed By: simpkins
Differential Revision: D4034298
fbshipit-source-id: 29674e2902661bf46394ea71b81537b35bd4b107
Summary: This should make some of the upcoming test harness work a little easier.
Reviewed By: simpkins
Differential Revision: D4011747
fbshipit-source-id: 87ee80a6d641a29be9027b163b1adee496f4452f
Summary:
I need this for the upcoming test harness so I can avoid creating a
`ClientConfig`, which is currently a huge pain to do from a unit test.
Reviewed By: simpkins
Differential Revision: D4010842
fbshipit-source-id: 03d1e1de9c3047340a6f26202d4b432f4a8620b4
Summary:
This was reported by ASAN.
The major issue was that `FakeObjectStore` was returning a copy of a `Tree`,
so it was not the case that the `TreeEntry*` returned by `getEntryForFile()`
was guaranteed to be "owned by" the `Tree* root` that was passed in. To address
this, we change `getEntryForFile()` to now return a copy of the `TreeEntry*`
that it gets back from `getEntryPtr()`. It really comes down to this line:
```
auto entry = currentDirectory->getEntryPtr(piece.basename());
```
because we cannot guarantee that `currentDirectory` will live past the end of
`getEntryForFile()`, so we cannot guarantee that return return value of
`currentDirectory->getEntryPtr()` will, either.
Special thanks to meyering and yfeldblum for helping me debug this.
Reviewed By: simpkins
Differential Revision: D4024627
fbshipit-source-id: 6295e6f2b1d2f544271b2aebad27a4ad3ae04563
Summary:
Utility function that given a `Tree` and a `RelativePathPiece`, returns the
corresponding `TreeEntry` in the `ObjectStore`, if it exists.
Reviewed By: wez
Differential Revision: D3980261
fbshipit-source-id: 2808a4ca45be84e3a6bb91b0cf2db19a3bf88798
Summary:
In an upcoming revision, I am going to introduce a utility function that takes
an `ObjectStore` (well, now an `IObjectStore`) as a parameter and I want to be
able to test it. Having a `FakeObjectStore` should make this considerably easier
without having to resort to mocks.
Reviewed By: simpkins
Differential Revision: D3980580
fbshipit-source-id: 5886e2055c893e749cc898226e1baade776c3ea7
Summary:
Adds a very basic example of testing eden functionality with hypothesis.
We'll be building on this with stateful testing in a follow on diff tomorrow.
There's some prep/setup work in the base test class that can be removed when an updated version of hypothesis ships and is updated in our third-party repo.
Reviewed By: simpkins
Differential Revision: D3968250
fbshipit-source-id: 46382c3bf2d6a0edbd60ac2b048b1bae26ca2572
Summary: This is necessary so that it can be used as the key in an `unordered_map`.
Reviewed By: simpkins
Differential Revision: D3980575
fbshipit-source-id: d225a98f957f9aae2f2f50a6cc365011d953c92e
Summary:
Apparently we did not have an existing unit test for `Tree`, so this adds one.
The other methods should be tested, as well, but I'm about to use `getEntryPtr()`
elsewhere, which is why I just focused on this one for the moment.
Reviewed By: simpkins
Differential Revision: D3980150
fbshipit-source-id: 33456fd621a1894606605af4fee06ba42d124752
Summary:
We want to use these with Eden
Depends on D3961190
Depends on D3961193
Depends on D3961196
Depends on D3961208
Reviewed By: rhysparry
Differential Revision: D3961232
fbshipit-source-id: 56f5a1811625303514e4398a6d47ea90ba348724
Summary:
The getMaterializedEntries() would previously try to dereference a null pointer
if the input mount path did not refer to a valid mount piont.
Reviewed By: bolinfest, wez
Differential Revision: D3942600
fbshipit-source-id: 2a8c9aa87d2bd8175f7bc77f3d6293ad25e9c198
Summary:
Add some helper functions for constructing EdenError objects from a few
different types of arguments. Also update eden.thrift to indicate that most
functions can throw EdenErrors on failure.
Reviewed By: bolinfest, wez
Differential Revision: D3942588
fbshipit-source-id: 1b561c5310a8a218f88c38c70499e087fe47bbe0
Summary:
Python 2.x requires the current class name be passed into super().
Add arguments to super so that we can use this inside a mercurial extension.
(Mercurial only supports python 2.x.)
Reviewed By: bolinfest
Differential Revision: D3942573
fbshipit-source-id: 06df55f217631a398004c0d25448d3a612f772e9
Summary:
The keys in the config directory map are normalized, absolute paths to the
mount point. When trying to look up a mount point make sure we also always use
a normalized absolute path.
Reviewed By: bolinfest
Differential Revision: D3942565
fbshipit-source-id: 63db838ffc7139d779925adf07c50f849d73bcc5
Summary:
We were hitting an assertion in the case where we did a `mkdir`
followed by a `rename` followed by `getMaterializedEntries`.
The issue is that our in-memory representation has a boolean to indicate
whether a dir inode is materialized, but our serialization format does
not have this bit. When we loaded the data we were not setting the
field to true and this was caught by the DCHECK.
If we have serialized data for a dir then it is, by definition, materialized
and we should just set that field to true.
Reviewed By: bolinfest
Differential Revision: D3900795
fbshipit-source-id: 62d8281e7a1009056d274888c9aff87664d2e09f
Summary:
This design is inspired by that of Git hooks:
https://git-scm.com/docs/githooks
By default, `/etc/eden/hooks` should be the place where Eden looks for
hooks; however, this can be overridden in `~/.edenrc` on a per-`repository` basis.
This directory should be installed as part of installing Eden.
There is information in `eden/hooks/README.md` about this.
The first hook that is supported is for post-clone logic for a repository.
This change demonstrates the need for an `eden config --get <value>`
analogous to what Git has, as hooks should be able to leverage this in their
own scripts. There introduces a `TODO` in `post-clone.py` where such a
feature would be useful, so that I could add the following to my `~/.edenrc`
to develop the Eden extension for Hg:
```
[hooks]
hg.edenextension = /data/users/mbolin/fbsource/fbcode/eden/hg/eden
[repository fbsource]
path = /data/users/mbolin/fbsource
type = hg
hooks = /data/users/mbolin/eden-hooks
```
Note that this revision also introduces a `generate-hooks-dir` script that can be
used to generate the standard `/etc/eden/hooks` directory that we intend to
distribute with Eden. This is also useful in creating the basis for a custom `hooks`
directory that can be specified as shown above in an `~/.edenrc` file.
Reviewed By: simpkins
Differential Revision: D3858635
fbshipit-source-id: 215ca26379a4b3b0a07d50845fd645b4d9ccf0f2
Summary:
simpkins spotted this; we were passing the wrong path down to the overlay saving dir.
This adds a test to prove that the source and destination directory contents
are correct both immediately after performing the rename and after remounting,
where we just read the serialized data.
Reviewed By: simpkins
Differential Revision: D3888694
fbshipit-source-id: 7f5fb5be417db5c693ac8a07b85abbffdbfe0fff
Summary:
This is pretty straightforward; we just walk back until we hit the
boundary with the requested JournalPosition.sequenceNumber
Reviewed By: simpkins
Differential Revision: D3872970
fbshipit-source-id: 1405f05957346d7ac513070f0407a477548aff1d
Summary:
populate the position from the latest journal delta.
To facilitate this, we also define the mountGeneration value to be a
combination of the pid and the time at which we created the EdenMount object,
as well as a global counter that we bump for each mount.
The precise value and meaning of this bits really doesn't matter, just that we
are unlikely to pick the same value for this same mountPoint path again if we
were to remount in the future.
Since we are now in a position to report JournalPosition values to clients, now
is also a good time to fill out the `currentPosition` field for the
`getMaterializedEntries` thrift call, and to check that this value is
consistent with the value we return via `getCurrentJournalPosition`.
Reviewed By: simpkins
Differential Revision: D3872952
fbshipit-source-id: 2fbc25d2e9711035b66ab1bf5d746507b72de265
Summary:
This just populates the initial snapshot hash in the journal.
The `addDelta` method will propagate this into subsequent deltas if the delta
to be added has hash values that have not been set from the default 0-filled
hash values.
Reviewed By: simpkins
Differential Revision: D3872936
fbshipit-source-id: d0014ded40488a2be04d5a381e1d9815c7f0a638
Summary:
This diff adds a couple more things to our thrift interface:
1. Introduces JournalPosition
2. Adds methods to query the current JournalPosition and obtain a
delta since a given JournalPosition
3. Augments getMaterializedFiles to also return the current JournalPosition
4. Adds a method to evaluate a `glob` against Eden
5. Adds a method using thrift streaming to subscribe to realtime changes
Could probably finesse the naming a little bit.
The JournalPosition allows reasoning about changes to files that are not part
of an Eden snapshot. Internally the journal position is just the
SequenceNumber from the journal datastructures, but when we expose it to
clients we need to be able to distinguish between a sequence number from the
current instance of the eden service and a prior incarnation (eg: if the
process has been restarted, and we have no way to recreate the journal we need
to be able to indicate this to the client if they ask about changes in that
range). For the convenience of the client we also include the `toHash` (the
most recent hash from the journal entry) which is likely useful for the `hg`
dirstate operations; it is useful to know that the snapshot may have changed
since the last query about the dirstate.
The `getFileInformation` method returns the instantaneously available `stat()`
like information about the requested list of files. Since we simply don't
have historical data on how files in the overlay looked (only how they look
now), this method does not allow passing in a JournalPosition. When it comes
to comparing historical data, we will need to add an API that accepts two
snapshot hashes and generates the results from there. This particular method
is geared up to understanding the current state of the world; the obvious use
case is plugging in the file list from `getFilesChangedSince` into this
function to figure out what's what.
* Do we want a function that combines `getFilesChangedSince` + `getFileInformation` into a single RPC?
Why is there a glob method? It's to support a use-case in the watchman/buck
integration. I'm just sketching it out in the thrift interface at this stage.
In the future we also need to be able to express how to carry out a tree walk,
but that will require some query predicates that I don't want to get hung up on
specifying immediately.
Why is the streaming stuff in its own thrift file? We can't generate code for
it in java or perhaps also python. It's only needed to plumb data into
watchman so it's broken out into its own definition. Nothing depends on that
file yet, so it's probably not specified quite right. The important thing is
how the subscribe method looks: it's essentially the same as the method to
query a delta, but it keeps emitting deltas as they are produced. This is
another API that will benefit from query predicates when we get around to
specifying them.
I've added `JournalDelta::fromHash` and `JournalDelta::toHash` to hold the
appropriate snapshot ids in the journal entry; this will allow us to indicate
when we've checked out a new snapshot, or created a new snapshot. We have
no way to populate these yet; I commented on D3762646 about storing the
`snapshotID` that we have during `EdenServiceHandler::mountImpl` into either
the `EdenMount` or the proposed `RootInode` class. Once we have that we
can simply sample it and store it as we generate `JournalDelta`s.
Reviewed By: simpkins
Differential Revision: D3860804
fbshipit-source-id: 896c24c354e6f58328fb45c24b16915d9e937108
Summary:
This is pretty simplistic: we just wlock and add a delta for the set
of file(s) that were changed in a given fuse operation (this is typically 1
file, but rename affects 2).
To reduce boilerplate very slightly, I've added an initializer_list constructor
for JournalDelta that makes it less cumbersome to create a JournalDelta for a
list of files.
Reviewed By: simpkins
Differential Revision: D3866053
fbshipit-source-id: cd918e2c98c022d5ef79430cd8ab4aef88875239
Summary:
This implements a pretty simple change Journal and associated
JournalDelta.
The Journal is intended to be held in memory and not persisted to disk.
The idea is that we'll hold a `Synchronized<Journal>` along with the
other mount data and grab a `wlock` on it each time we want to add
a change record.
This diff doesn't change any other existing functionality.
Reviewed By: simpkins
Differential Revision: D3660162
fbshipit-source-id: a6b6fa28dd12e4d34718956167ee87f8cb2d89ca
Summary:
Adds a thrift call that returns the list of materialized entries from the whole tree.
This is intended to be plugged into the mercurial dirstate extension.
Reviewed By: simpkins
Differential Revision: D3851805
fbshipit-source-id: 8429fdb4eeccc32928e8abc154d4e6fd49343556
Summary:
Was chatting with simpkins the other day and he mentioned that our
instrumented hg wrappers are quite CPU intensive. This diff switches us to
running the underlying `hg.real` or `git.real` in our integration tests when we
find them in the path.
Reviewed By: bolinfest
Differential Revision: D3865996
fbshipit-source-id: d047749356f0c1c0662774e25801f3578f9f9243
Summary:
hg_import_helper.py imports mercurial python modules, which are GPLv2+, so this
code also needs to be licensed under GPLv2+, rather than the BSD-style license
used for the bulk of the eden code base.
Reviewed By: wez
Differential Revision: D3833508
fbshipit-source-id: eb2a8969a5a88c12444a3778875609f24e145e6b
Summary:
Annoying that gcc and clang behave differently here. The compilation
error is due to gcc not seeing the implicit this pointer for some of these
method calls, so we need to explicitly use it.
Reviewed By: simpkins
Differential Revision: D3846973
fbshipit-source-id: 3d5b8b8b8c9bbab1e7935cff0e65677f76d116fb
Summary:
Buck needs this API so that it knows which paths under a project
root it should exclude when deciding whether it can ask Eden for its
SHA-1 or if it must compute it on its own.
Reviewed By: simpkins
Differential Revision: D3840658
fbshipit-source-id: 5eddc0bef423d3b3ee165d2a4b0bbf193f94f61a
Summary:
we now serialize the overlay data for each directory independently.
When we mount, we try to load the root overlay data. The children are lazy
loaded as the inodes are instantiated.
Structural changes cause the overlay data for the impacted dirs to get saved out.
I need to make a pass over this to fixup comments and so on, I just wanted to get this diff out first.
I moved the overlay stuff from `eden/fs/overlay` -> `eden/fs/inodes` since most
of the overlay-ness is handled in `TreeInode` now; the `Overlay` class is
really just for carrying around the paths and providing the serialization
helpers.
Reviewed By: simpkins
Differential Revision: D3787108
fbshipit-source-id: f0e089a829defd953535b9d0a96b102ac729261b
Summary:
It was starting to get pretty complex to manage locking across the
inodes, filedata, overlay and soon the journal, so as a simplifying step, this
folds data that was tracked by the overlay into the TreeInode itself.
This is the first diff in a short series for this. This one:
1. Breaks the persistent overlay information, so shutting down eden and
bringing it back up will lose your changes (to be restored in the
following diff)
2. Allows deferring materialization of file data in more cases
3. Allows renaming dirs.
The approach here is now to keep just one source of information about the
directory contents; when we construct a TreeInode we import this data from the
Tree and then apply mutations to it locally.
Each inode can be mutated indepdently from others; we only need to lock the 1,
2 or 3 participating inodes in the various mutation operations.
I'll tackle persistence of the mutations in the following diff, but the high
level plan for that (to help understand this diff) is to always keep the
directory inodes for mutations alive as inode objects. We make use of the
canForget functionality introduced by D3774269 to ensure that these don't
get evicted early. On startup we'll load this information from the overlay
area.
This model simplifies some of the processing around reading dirs and looking up
children.
Since the overlay data now tracks the appropriate tree or content hash
we can be much more lazy at materializing data, especially in the rename
case. For example, renaming "fbcode" to "fbcod" doesn't require us to
recursively materialize the "fbcode" tree.
Depends on D3653706
Reviewed By: simpkins
Differential Revision: D3657894
fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
Summary:
We can't allow ~EdenServer to delete the memory until we're sure that
the other threads are done. To ensure that, we need to notify the condition
variable while the aux thread still holds the lock. This makes sure that the
thread destroying the EdenServer waits for the aux thread to release the lock
before we check the predicate and proceed to deleting the memory.
```
SUMMARY ThreadSanitizer: data race /
/common/concurrency/Event.cpp:107 in facebook::common::concurrency::Event::set() const
==================
I0909 14:51:18.543072 4147554 main.cpp:173] edenfs performing orderly shutdown
I0909 14:51:18.555794 4148654 Channel.cpp:177] session completed
I0909 14:51:18.556011 4148654 EdenServer.cpp:192] mount point "/tmp/eden_test.0ostuc90/mounts/main" stopped
==================
WARNING: ThreadSanitizer: data race (pid=4147554)
Write of size 8 at 0x7fff9e182d90 by main thread:
#0 pthread_cond_destroy <null> (edenfs+0x00000007671a)
#1 facebook::eden::EdenServer::~EdenServer() /
/eden/fs/service/EdenServer.cpp:93 (libeden_fs_service_server.so+0x0000000b96cd)
#2 main /
/eden/fs/service/main.cpp:176 (edenfs+0x000000018515)
Previous read of size 8 at 0x7fff9e182d90 by thread T73:
#0 pthread_cond_broadcast <null> (edenfs+0x0000000765b7)
#1 __gthread_cond_broadcast /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/include/x86_64-facebook-linux/bits/gthr-default.h:852 (libstdc++.so.6+0x0000000e14f8)
#2 std::condition_variable::notify_all() /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/condition_variable.cc:72 (libstdc
++.so.6+0x0000000e14f8)
#3 facebook::eden::EdenServer::mount(std::shared_ptr<facebook::eden::EdenMount>, std::unique_ptr<facebook::eden::ClientConfig, std::default_delete<facebook::eden::ClientConfig> >)::$_0::operator()() const /
/
/eden/fs/service/EdenServer.cpp:145 (libeden_fs_service_server.so+0x0000000bcdb5)
#4 std::_Function_handler<void (), facebook::eden::EdenServer::mount(std::shared_ptr<facebook::eden::EdenMount>, std::unique_ptr<facebook::eden::ClientConfig, std::default_delete<facebook::eden::ClientConfig>
>)::$_0>::_M_invoke(std::_Any_data const&) /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:2039 (libeden_fs_service_server.so+0x0000000bcab0)
#5 std::function<void ()>::operator()() const /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:2439 (libeden_fuse_fusell.so+0x00000020fbb9)
#6 facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0::operator()() const /
/eden/fuse/MountPoint.cpp:69 (libeden_fuse_fusell.so+0x000000237447
)
#7 void std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()>::_M_invoke<>(std::_Index_tuple<>) /
/third-party-buck/gcc-4.9-
glibc-2.20-fb/build/libgcc/include/c++/trunk/functional:1699 (libeden_fuse_fusell.so+0x000000237048)
#8 std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()>::operator()() /
/third-party-buck/gcc-4.9-glibc-2.20-fb/build/libgc
c/include/c++/trunk/functional:1688 (libeden_fuse_fusell.so+0x000000236ff8)
#9 std:🧵:_Impl<std::_Bind_simple<facebook::eden::fusell::MountPoint::start(bool, std::function<void ()> const&)::$_0 ()> >::_M_run() /
/third-party-buck/gcc-4.9-glibc-2.
20-fb/build/libgcc/include/c++/trunk/thread:115 (libeden_fuse_fusell.so+0x000000236d8c)
#10 execute_native_thread_routine /home/engshare/third-party2/libgcc/4.9.x/src/gcc-4_9/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:84 (libstdc++.so.6+0x0000000e6
ec0)
```
Reviewed By: simpkins
Differential Revision: D3844846
fbshipit-source-id: 545474bc1aff8621dbeb487dcd6b54c82828ff3b
Summary:
This did not work because the dict-like object read from
a `ConfigParser` was not JSON-serializable by Python.
I had to add some methods to the `Repository` that we use in our
integration test harness in order to verify everything I wanted to
in my new integration test. I implemented these methods in both
`HgRepository` and `GitRepository`.
Reviewed By: simpkins
Differential Revision: D3837879
fbshipit-source-id: e0bfb5f1bd3add192ef9bdf561591ac8e52bc002
Summary: I did what the linter told me to do.
Reviewed By: wez
Differential Revision: D3836659
fbshipit-source-id: a5d3fc8974cf6cb7c7e2d88a6215ac5c54479780
Summary: Open a file in a subdirectory in append mode and update it.
Reviewed By: wez
Differential Revision: D3802815
fbshipit-source-id: fe1726529d017345dfee530ae5ec84cfb7531602
Summary:
test_utime() seems to fail a reasonable amount of the time in my experience:
the atime returned by lstat() occasionally ends up being 1 or 2 milliseconds
earlier than the saved timestamp. With the existing behavior python and eden
are independently computing the current time, although I haven't investigated
why eden sometimes gets a timestamp slightly earlier than python did.
This fixes the test by asking eden to set an explicit timestamp value, rather
than letting eden independently get the current timestamp.
Reviewed By: wez
Differential Revision: D3802100
fbshipit-source-id: fecfd1d68b8c95d119099ef39f17004be13d3dff
Summary:
`buck-out` now has an extra level of depth due to its new CPU-specific directories,
so we have to change the place we look for `hg_import_helper.py` in development.
Reviewed By: simpkins
Differential Revision: D3807012
fbshipit-source-id: 24d1fc1fa22f3003580f59cfdd46b9b3917f6eeb
Summary: As noted in the comments, these dependencies were making this harder to use in Buck.
Reviewed By: simpkins
Differential Revision: D3805861
fbshipit-source-id: e04898d0e1a3ccc5e38a9629b1d30791853224a5
Summary:
This codemods `TARGETS` under `[a-d]*` directories in fbcode to make
the `headers` parameter explicitly refer to `AutoHeaders.RECURSIVE_GLOB`.
Reviewed By: yfeldblum
Differential Revision: D3801845
fbshipit-source-id: 715c753b6d4ca3a9779db1ff0a0e6632c56c0655
Summary:
Recent changes to our sudo policy seems to be dropping the SSH_AUTH_SOCK
environment variable for me, even when using "sudo -E". (We do explicitly
configure this to be dropped with sudo's env_delete setting.)
This updates the eden CLI to explicitly ask sudo to set environment variables
that we care about.
Reviewed By: bolinfest
Differential Revision: D3674124
fbshipit-source-id: ab14453056961f7b6a7f5d4122bc0c6f5caa6588
Summary:
Use fd 5 rather than stdout (fd 1) for communicating from the hg import helper
process. This way we won't have communication problems even if some of the
mercurial code does somehow ever end up printing to stdout. File descriptor 5
was picked rather arbitrarily. Anything greater than or equal to 3 should work
in practice.
Reviewed By: bolinfest
Differential Revision: D3673941
fbshipit-source-id: ed668542a17c585a5c54bcd3499174a6ed6fa138
Summary:
This is working around a noisy reporting issue in our CI system. Rather than
run and skip the integration tests in that environment, we swap out the base
class with one that won't report any tests to the python test case discovery
mechanism.
Reviewed By: simpkins
Differential Revision: D3634277
fbshipit-source-id: d118dca78a967bf068242790f7b874f865dddecc
Summary: Currently, all existing mount path are unmounted on 'eden shutdown' but are not remounted again after a subsequent 'eden daemon' call, though they appear as mounted when 'eden list' is called. These changes fix this behavior and have the daemon remount the paths that had been mounted before shutdown was called.
Reviewed By: simpkins
Differential Revision: D3580793
fbshipit-source-id: d03beafc20db4bd01662dd7f198a5ab8859b8e3d
Summary:
Load the config data when eden server is started so that it doesn't need to be re-loaded every time a mount is done. The normal use case for eden will not see that many changes to the config data (users adding repositories themselves is expected to be minimal) so this new logic will be more efficient overall.
Currently, the config data IS reloaded before use every time but this is because there is currently no way to reload the config data if any files are modified on disk. I am looking into how to do this now, and this feature will soon be updated to this diff so configData_ does not need to be constantly reloaded.
Reviewed By: simpkins
Differential Revision: D3580777
fbshipit-source-id: 5e23f51e4aab815e9812750617446dcb7e5483cb
Summary: Restructure the current logic used for loading the config data into a ClientConfig object. Rather than having loadFromClientDirectory iterate through all the config files and parse them to find the necessary information, abstract that logic out into a new method that compiles all of the relevant data so that all loadFromClientDirectory has to do is pull out the needed information. Since this change separates the two steps, this will make it easier to move the first step of compiling config information outside of ClientConfig - the goal here is to have the eden server load all of the config data at start up and cache it in memory so that it doesn't need to be done every time a ClientConfig object is created, and this change is an intermediate step.
Reviewed By: simpkins
Differential Revision: D3580757
fbshipit-source-id: c340a0fe715856066a554238249574f8177bc4d7
Summary: This should facilitate things by making the way to reference the path unambiguous.
Reviewed By: simpkins
Differential Revision: D3672311
fbshipit-source-id: b2a21e4dba60ea5ded28643f76aa402c4250d7ac
Summary:
If remotefilelog is unable to download a file from the server, it can end up
logging data through the UI which gets printed to stdout. Since we use stdout
to communicate with the main edenfs daemon this interferes with that
communication channel: edenfs ends up trying to parse the ASCII data as a
binary chunk header, sees a very large data length, and then ends up waiting
forever trying to read data that isn't coming.
This provides our own custom mercurial.ui.ui subclass to try and make sure that
we print all messages to stderr rather than stdout.
I will also send a separate diff so that we use a different pipe to communicate
with edenfs, rather than just using stdout/stdin. That should be a more
guaranteed way of making sure that nothing inadvertantly gets written to the
communication channel with edenfs.
Reviewed By: DurhamG
Differential Revision: D3673830
fbshipit-source-id: d64d492b10ee09ba63318b86d77f9181c4a0ce29
Summary: The repo_name could be cloned multiple times, which would cause a collision.
Reviewed By: simpkins
Differential Revision: D3672298
fbshipit-source-id: f27e1ec00ab773ed50f4fcf6f485461d01f8ba11
Summary:
Update the privhelper code to properly handle the error if it is asked to
unmount a path that isn't currently mounted.
This fixes the code to send an error response back to the main edenfs process.
Previously the exception propagated all the way up to the privhelper main loop,
causing the privhelper process to exit.
It would be nice to refactor the privhelper code a bit more to create a more
standardized API for implementing privhelper commands. This would provide a
more common place to catch exceptions, and make it harder for the command
implementation code to leak exceptions. However, I'll wait to do that in a
subsequent diff at some point.
Reviewed By: wez
Differential Revision: D3653727
fbshipit-source-id: e14c6ae974c8f99da3b426be250510c1f9b8017d
Summary:
Add back the "mount" CLI command.
While end-users may not need this command under normal circumstances, during
development it is sometimes necessary to re-mount unmounted clients. This adds
back the "mount" command which simply re-mounts an already configured client.
This also adds a "--no-forget" option to "unmount" which causes it to unmount
the client without forgetting its configuration. This allows it to be
remounted with "mount", rather than having to re-run the "clone" command and
specify the repository name again (and potentially restore the desired snapshot
hash).
Reviewed By: wez
Differential Revision: D3653706
fbshipit-source-id: 0e3f3fe1efdf72e9e54bdd1e0fb308ae67b7c058
Summary:
Use the default_permissions option when mounting. This tells the kernel to
enforce the normal file permissions semantics based on the file mode bits.
With this option enabled, the kernel will also implement the access() call for
us automatically.
Reviewed By: wez
Differential Revision: D3653198
fbshipit-source-id: e7227f455077a9ef012a5c102e4e22a976595681
Summary:
Include a configPath_ field for EdenServer that holds the path of the user ~/.edenrc config file. The server needs the data from this user config file in order to perform mounts and currently, the path to the home directory is passed via the CLI to the mount command as a field inside the MountInfo struct in order to get the file. As per discussion in D3498567, including the home directory inside the MountInfo struct is logically a bit disjointed, and this change would no longer require the home directory to be passed to the server via MountInfo.
This restructuring also sets up eden for a future change - having the server remount existing mount points on start-up is now possible from the inside. Before this change, mounting anything had to be done via the CLI since the home directory had to be passed in from the outside. This meant that remounting the existing mount points on start up could only be done if Eden was run in the background - running in the foreground would require manual remounting of all existing mount points. Now that the server has access to the config file's path, remounting can be done without any prompting from the CLI in both cases.
Reviewed By: simpkins
Differential Revision: D3580737
fbshipit-source-id: 46667ccd130b470a3a8a9e9aa08e5ec8e8b90336
Summary:
Previously we would simply report the raw pointer address to the kernel and
rely on it to return that same number to us as the file handle, and make sure
that it told us to shut down the handle when it was closed.
This meant that we had no real idea about which files were still open.
For our future live upgrade plans we need to be able to know this so that we
can transfer the appropriate information to our replacement process.
To facilitate this this diff implements a FileHandleMap class that will assign
file handle numbers and keep track of the instances. The number assignment
strategy is the same as it used to be in the common case: we take the address
of the newly created instance and use that 64-bit number as the file handle
number. However, in the future when we transfer the mapping to a new process,
we may experience a collision when subsequently opening a file handle. To deal
with that, we have a bounded number of attempts to assign a random file handle
number.
We don't yet offer a means to iterate the map, but it would be trivial to
expose such an accessor when we're ready to use it.
Since we now keep track of these things via shared_ptr this changes the
appropriate portions of the fuse interface from unique_ptr to shared_ptr.
Reviewed By: simpkins
Differential Revision: D3602364
fbshipit-source-id: dd996339c2838225a2caeee9da16ef99a06c1e2b
Summary:
When the EDEN_GDB environment variable is set, start eden under gdb when
running the integration tests. Have gdb automatically start the daemon,
and exit automatically if everything is successful. If the daemon crashes, gdb
will break and the user can interact with gdb normally.
This is generally only useful when manually running a single test. Trying to
use this in combination with "buck test" will probably cause problems if
multiple gdb instances all try to interact with the terminal together.
Reviewed By: wez
Differential Revision: D3459575
fbshipit-source-id: 1359735a391e93a332a4c673c6525ad44034cc73
Summary: This change updates LocalStore to perform serialization of trees and blobs internally so that its users don't need to be aware of the internal serialization format. Previously, the get and put APIs were asymmetric such that the get APIs returned deserialized Tree and Blob objects, while put required raw serialized bytes. After this change, put will also use deserialized Tree and Blob objects.
Reviewed By: simpkins
Differential Revision: D3589899
fbshipit-source-id: 2e572e6ec5af44d66206b178a03f7a9d619b2290
Summary:
Update the CLI to always close the thrift client socket, to avoid resource leak
warnings on exit.
I also updated the code to just monkey-patch a nicer EdenError.__str__()
method, rather than having to explicitly catch and modify this exception in
multiple different places.
Reviewed By: bolinfest
Differential Revision: D3560662
fbshipit-source-id: 900fe74c793ffd99f4a2c1f1ddd94b96e48f5eb7
Summary:
The "eden repository <name> <path>" command had a bug that it would duplicate
the existing edenrc contents when writing out a new config: it opened the file
in append mode rather than truncate mode when performing the write.
This addresses that issue, but also does a bigger overhaul of the config update
code. We now acquire a lock for the duration of the modification, so that
another eden CLI process cannot modify the file between when we read it and
when we write out our modifications. We also perform the write using an atomic
rename so the file contents are always valid at all points in time.
I also updated the CLI command to print the repositories in sorted order, and
to only catch expected exceptions, and show backtraces for unexpected errors.
Reviewed By: bolinfest
Differential Revision: D3554550
fbshipit-source-id: 5920ccb2447330673eac3f9956a8ae5d8a66a67e
Summary:
This feels awful, but eliminates an abort that shows up when running under ASAN.
Details in the comment.
Reviewed By: bolinfest
Differential Revision: D3533529
fbshipit-source-id: 1717384c716f565b74b1678c66883a35ee60c522
Summary:
For consistency with `git clone`, the destination path must be either:
* non-existent (but `mkdir -p`-able)
* an empty directory
Or else `clone` should fail.
In the process of writing the integration test, I cleaned up some of our test infra
and took advantage of the fact that we require Python 3 (I believe 3.5) now:
* We now use `subprocess.run()` instead of `subprocess.check_output()`.
* Removed the `client_name` argument to `EdenClient.init()` because that is no longer the right terminology and no one appeared to be overriding the default, anyway.
* `EdenClient.clone_cmd()` no longer calls `os.mkdir(self._mount_path)` because that was papering over the issue this revision is trying to solve.
Reviewed By: simpkins
Differential Revision: D3526909
fbshipit-source-id: 66b5799a1b47a51468faf5dc17d87a385926dc89
Summary: Move adding the mount path to the directory map as the final step of the clone command so that the path is only documented if the mount is actually successful. Previously, if 'eden clone' failed during the mounting, another call to 'eden clone' to try again would fail with 'Error: mount path NAME already exists.' This change ensures that a second call to 'eden clone' will actually try again if the first failed.
Reviewed By: simpkins
Differential Revision: D3543388
fbshipit-source-id: be6412ad6d5d009bd87ad6e0b4110b35e6300fcb
Summary: This change fixes exception messages so that they all match the same format and don't include 'error: ' twice
Reviewed By: simpkins
Differential Revision: D3544034
fbshipit-source-id: 32d600c712d1782d2a1a35331b66e06a842ed8e0
Summary:
[Folly] Move `IPAddress` definitions to source files.
And to internal header files. Keeping headers lightweight can help with build times.
Reviewed By: simpkins
Differential Revision: D3514455
fbshipit-source-id: de78f4ef9e70e7ddd7fb666348ed705c5228531c
Summary:
Add a decorator to more easily declare tests that should be run for both git
and mercurial repository types.
Reviewed By: wez
Differential Revision: D3458883
fbshipit-source-id: 8058873923f9519a6550a8dcb930a5edd02aec10
Summary:
- Refactor the EdenClient code to better reflect how the eden currently works.
This code was originally written when the edenfs daemon only supported
running a single mount point. This updates it to reflect the fact that it
manages a single edenfs daemon, but multiple repositories can be mounted.
- Refactor the EdenTestCase now that tests generally only need a single eden
daemon. EdenTestCase starts the eden daemon by default.
- Add EdenHgTest and EdenGitTest classes. These are subclasses of
EdenTestCase, and they create and mount an hg/git repository before starting
the test function.
- Update the tests to derive from EdenHgTest and EdenGitTest where appropriate.
Reviewed By: wez
Differential Revision: D3458842
fbshipit-source-id: 77349a60ff72a700a2c2526a27e7621b76f9eec2
Summary: Fix a bug with the eden repository command that let it add multiple repositories with the same name. If an entry for repository 'name' already exists in the home ~/.edenrc config file, the eden repository command will now raise an error message saying that the repository already exists. Note that if an entry for repository 'name' exists in any of the global /etc/eden/config.d/* files, the eden repository command will still allow the user to create the new repository and will add an entry for 'name' to the home ~/.edenrc config file (since entries in home config file overwrite those in global)
Reviewed By: wez
Differential Revision: D3529743
fbshipit-source-id: b7c7ceef6ef38fcd3b4ff9a92fbd23a4a0bd6951
Summary:
Mercurial maintains its manifest in sorted order, but since the manifest only tracks file names we can end up with the following sequence:
```
some/path-foo/bar
some/path/bar
```
This is because the `-` sorts ahead of the `/`.
This diff defers passing the entries to the tree serializer, buffering them up
into a temporary vector and using `std::lower_bound` to find the appropriate
insertion point.
Reviewed By: bolinfest
Differential Revision: D3529329
fbshipit-source-id: 395ed16a20c14d17717ec69192a38f0407b51e1d
Summary:
This check was already being done appropriately in our Python code, but we also need
to do it in our C++ code.
Reviewed By: wez
Differential Revision: D3526705
fbshipit-source-id: 3b28b88f63ae768113f363ace58d40a89a8f4b61
Summary: Move the _get_or_create_dir() method from config.py to util.py and rename as mkdir_p(). This change replaces all of the os.makedirs call sites with the new mkdir_p() method.
Reviewed By: bolinfest
Differential Revision: D3512570
fbshipit-source-id: a867049f9af22076934390061f09070bf9ee6397
Summary:
These changes restructure the eden directory so that 'client' directories are created during the `eden clone` command and are associated with a single mount path.
The new eden directory looks as follows:
~/.eden
config.json
clients/
abcd08d/
edenrc
SNAPSHOT
overlay/
efgh19i/
edenrc
SNAPSHOT
overlay/
...
Where the config.json file holds the mapping of mount paths to their respective client directory which is a hash, and the edenrc files in each client directory is an INI file which holds the name of repository associated with the mount path. This INI file follows the current format:
[repository]
name = fbsource
This restructuring required a couple other changes:
- unmount command now cleans up the client directory and removes the mapping of its mount path from config.json
- eden list command now lists all of the mount paths rather than the client names
Reviewed By: bolinfest
Differential Revision: D3506119
fbshipit-source-id: dc07a8baf1052be731ff335d9cf74a07ab8e661a
Summary: Change the ClientConfig class to parse client data via INI config file rather than json file. This class uses boost::property_tree::ini_parser and the ptree data structure to hold the parsed INI file contents. This change makes it possible for eden to no longer rely on json files for getting client data, and the json files will be completely taken out in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3498567
fbshipit-source-id: 3298047a014beda0c250475c0809a7a1ebd95b2b
Summary: Previously the repo snapshot id was collected and stored when the repository was initialized, but this diff moves the update to the clone command so now the snapshot is only stored right before a repository is mounted.
Reviewed By: bolinfest
Differential Revision: D3489550
fbshipit-source-id: 1271c1d7c7709ed332307a636c23e26e913483c9
Summary: Update the unmount command to use the mount point rather than the client name. This diff also removes the mount command completely since tests are no longer dependent on it.
Reviewed By: bolinfest
Differential Revision: D3489290
fbshipit-source-id: 9bfbcb78a78d1593f6d0950502bdd4a55e3e6ca4
Summary: Introduce new 'clone' command that takes in repo name and mount path. This command replaces the current mount command which uses the client name as an argument
Reviewed By: bolinfest
Differential Revision: D3489098
fbshipit-source-id: 719fb94387da1d4ccab770a0e659f717888a7dc6
Summary: Expand functionality of repository command to also create repositories. The current command line format is 'eden repository <name> <path>' and will soon replace the existing init command.
Reviewed By: bolinfest
Differential Revision: D3489064
fbshipit-source-id: 3e6946cbd80a64ad23b52383f3f14a92d4492a21
Summary: Add a simple eden repository command that takes no arguments. The behavior mimics that of git remote and lists all repositories by parsing both the global and local INI config files. The INI file currently needs to be manually added and updated to maintain information about existing repositories.
Reviewed By: bolinfest
Differential Revision: D3479543
fbshipit-source-id: b0d36f33c9b8e0c7b9fe20ec2f362ca15c0adeb8
Summary:
This is a bit ugly and longer term should probably be an
eden configuration that we deploy to our CI tier.
In our CI environment we don't allow unfettered password-less sudo
access and instead must launch privileged processes via a helper
process that is whitelisted. D3524299 has the definition of that
helper as it applies to our internal infra.
Reviewed By: bolinfest
Differential Revision: D3524373
fbshipit-source-id: f36bf29e6b455220c434b681a6d0d551bc21ecab
Summary:
We're seeing failures like this in our CI system:
```
stderr:
test_create (eden.fs.integration.basic_test.BasicTest) ...
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: no tty present and no askpass program specified
error: edenfs is not running
ERROR
ERROR
======================================================================
ERROR: test_create (eden.fs.integration.basic_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "eden/fs/integration/basic_test.py", line 68, in test_create
eden = self.init_git_eden()
File "eden/fs/integration/lib/testcase.py", line 104, in init_git_eden
return self.init_eden(repo_path)
File "eden/fs/integration/lib/testcase.py", line 54, in init_eden
inst.init(repo_path, *kwargs)
File "eden/fs/integration/lib/edenclient.py", line 160, in init
self.daemon_cmd(timeout)
File "eden/fs/integration/lib/edenclient.py", line 174, in daemon_cmd
self._wait_for_thrift(timeout)
File "eden/fs/integration/lib/edenclient.py", line 99, in _wait_for_thrift
raise Exception("edenfs didn't start within timeout of %s" % timeout)
Exception: edenfs didn't start within timeout of 10
```
This diff improves things by failing faster and printing a little more appropriate context.
Reviewed By: bolinfest
Differential Revision: D3523257
fbshipit-source-id: 14508a1b820d9d6c773713e2bb7c22955c5b2b66
Summary:
Implements basic rename(2) support. We only do this for files at this
time as directory renames require that we recursively materialize a tree and
are a bit more complex. I'll look at that in a follow-on diff; there's
potential for optimizing it to avoid eager materialization, but that depends on
the trie work that I'm tackling concurrently with this.
rename for files is the last piece needed to allow `sed -i` to operate correctly.
Reviewed By: bolinfest
Differential Revision: D3515361
fbshipit-source-id: 9c8cc5f9b8db6b5a9372ca9286336647f50490f8
Summary:
This enables O_EXCL to function by allowing the create routine to
move its folly::File instance down in to the underlying FileData instance.
Previously we would close and then re-open the file; this would discard
any of the natural gating for the open call that is performed by the kernel
for the underlying filesystem in the overlay.
Reviewed By: bolinfest
Differential Revision: D3513758
fbshipit-source-id: 85967a3b7affa1b1df46842be8ba21c8fbb843a6
Summary:
Despite being handy things for filesystems in general, these are
needed to support the `sed -i` integration test.
Reviewed By: bolinfest
Differential Revision: D3513754
fbshipit-source-id: 505b4bd58b254141e2ef996f01e3347fc1a77584
Summary:
This eliminates a linear scan from TreeInode and replaces it with a
binary search, exploiting the sorted order of the entries vector.
Two new methods are introduced: getEntryPtr which returns a pointer to the
entry with the matching name, and getEntryAt() which returns a reference
(throwing a range error if there is no such entry).
I wanted to use the PathMap class here, but that would cause us to duplicate
the name string as both the key and value in the map.
Reviewed By: bolinfest
Differential Revision: D3515723
fbshipit-source-id: 4ee0371f3ec08cbcf110cf28f5c1e1529b120fb6
Summary:
This removes use of the builtin Buck `thrift_library` support from
the macro library. It turns out that a lot of rules incorrectly
added deps onto the raw builtin `thrift_library`, rather than one
of the per-language rules. This is a noop, and removing the builtin
rule exposes this as missing target errors, so this diff removes them.
Reviewed By: Coneko
Differential Revision: D3512451
fbshipit-source-id: dd8beb148ed47a3ad7d3963fae600abd73d030d5
Summary:
setattr is a bit of a multi-purpose interface; depending on the flag
values, this is responsible for:
* ftruncate(2)
* fchmod(2)
* fchown(2)
* futimens(2)
In order to apply any of these things, we have to materialize the file. In
the future we may want to allow setting the utimes without materializing the file.
We don't allow chown to actually chown anything. We may want to relax that in
the future, but at the moment we will return an error if an attempt is made to
change the ownership of a file.
Reviewed By: bolinfest
Differential Revision: D3511011
fbshipit-source-id: 858d2c07686fcbe2dcdb60a07527f739a9726be3
Summary:
Make sure mount points are completely stopped before destroying the EdenServer
object. Previously the EdenServer was destroyed with the MountPoints still
running the fuse channels in background threads. When the privileged helper
process unmounted them, fuse requests from the kernel could arrive and access
memory that had already been destroyed.
Reviewed By: wez
Differential Revision: D3458898
fbshipit-source-id: 365bca716ff0f8315b66af92effeb8c6dc574ce1
Summary:
When running the integration tests when built with gcc, the tests would crash
in TreeInode::create(). It appears that the unique_ptr<FileHandle> object was
getting passed to the lambda before dereferencing it to call getattr().
Reviewed By: wez
Differential Revision: D3459605
fbshipit-source-id: 5e2ce98d268a85731acaf7d7f37f22c77fb571cf
Summary:
This fixes inode number handling problems in TreeInode and TreeEntryFileInode.
Previosly these classes each had an ino_ member variable, despite deriving from
InodeBase which has its own (private) ino_ member.
TreeEntryFileInode never actually initialized its own local ino_ variable.
This was causing problems for many applications which use inode numbers to
cache file data. TreeEntryFileInode returned garbage data in the inode field,
resulting in incorrect cache collisions.
This fixes the inode handling, and also fixes the stat data returned by
FileData. It now sets the uid, gid, and inode fields correctly. It 0s out
several other fields rather than leaving them uninitialized.
Reviewed By: bolinfest
Differential Revision: D3455126
fbshipit-source-id: 631276b01676733f96035bc153219ef84406dcc9
Summary:
Start adding utility code to make it easier for the integration tests to
manipulate git and mercurial repositories.
This adds an HgRepository class, but does not update the tests to use it yet.
I will do that in a separate diff.
Reviewed By: bolinfest
Differential Revision: D3449328
fbshipit-source-id: caea5befb683a388cbad30a29017c566e22702af
Summary:
Move the eden/fs/importer/hg code to eden/fs/store/hg. This import code is
only used by the HgBackingStore, so logically it belongs together.
The eden/fs/importer/git directory was already moved into eden/fs/store/git by
D3448752, so this makes the mercurial code similar.
Reviewed By: bolinfest
Differential Revision: D3458128
fbshipit-source-id: 54b1f707197e41ff5f5f4a163a34d3d1ab432e21
Summary:
This moves all of the test library code into a lib/ subdirectory, just to help
distinguish tests from utility code.
This also changes the test so that we no longer pack the eden CLI and daemon
binaries into the python archives. This results in very large archives when
building in dbg and opt modes, and isn't really necessary. Instead
edenclient.py simply finds the CLI and daemon binaries relative to the test
binary. We pass in an EDENFS_SUFFIX variable to tell it which flavor of the
daemon to use.
Additionally, this changes the tests to run with python 3.
Reviewed By: bolinfest
Differential Revision: D3449013
fbshipit-source-id: 82533137090325766a52cd067aa97dd8391ae088
Summary:
This moves git import logic from the GitImporter class to GitBackingStore.
The logic is simpler now, since GitBackingStore only needs to import a single
Tree or Blob at a time.
Reviewed By: bolinfest
Differential Revision: D3448752
fbshipit-source-id: da2d59f953ada714d8512545ae83dd48e5d3e410
Summary:
Add a run_cmd() function that's slightly simpler to use compared to
_get_eden_args().
Reviewed By: bolinfest
Differential Revision: D3448844
fbshipit-source-id: 06b654b2f4bc773d67a4f9d1f6effdad2ce9aa2a
Summary:
- Update "eden daemon" to wait until edenfs is healthy before returning.
- Accept additional command line arguments and pass them through to edenfs.
- When starting edenfs as a background daemon, direct stdout and stderr to a
log file in the eden config directory.
- Update "eden shutdown" command to wait for edenfs to exit.
- Update "eden shutdown" command to print an error message if edenfs was not
running.
Reviewed By: bolinfest
Differential Revision: D3446403
fbshipit-source-id: 2ca1874256b7d124a85092b886548bea8c198327
Summary:
Look for hg_import_helper.py relative to the edenfs binary, rather than
relative to the current working directory. Also check in a couple places, so
we can work in both normal deployment locations as well as inside a source
repository.
Additionally, update hg_import_helper.py to emit a response chunk as soon as it
starts. This contains either message indicating that it has started
successfully, or an error message. This allows us to propagate error
information back to the thrift caller if something goes wrong starting
hg_import_helper.py for a given repository.
Reviewed By: bolinfest
Differential Revision: D3447027
fbshipit-source-id: 4bffd1d03ab6475c09cb3c8385145555c7fc7361
Summary:
[Folly] Wrappers for some of OpenSSL's crypto hash functions.
Wraps some of the OpenSSL crypto hash functions with variants that take `ByteRange` for input and `MutableByteRange` for output, and also variants that take `const IOBuf&` for input as well.
These are a bit nicer to use than passing pointers and lengths separately.
Reviewed By: ivmaykov
Differential Revision: D3434562
fbshipit-source-id: 3688ef11680a029b7664ac417a7781e70f9c6926
Summary:
This adds a --foreground flag to the "eden daemon" command, which causes the
daemon to run in the foreground, rather than daemonizing as a background
process. Additionally, this drops the --preserve-environment flag and instead
updates the CLI to always build a sane environment to run with.
This also updates the integration test code to use this flag, and wait for eden
to shut down during test clean up. Without this, the test case code could try
to clean up temporary directories before they were unmounted, which would fail.
This would leave many temporary directories behind after test runs.
This also re-organizes the temporary directory set up a bit, so that each test
uses a single top-level temporary directory. All other directories it creates
are put inside this one directory.
Reviewed By: bolinfest
Differential Revision: D3439232
fbshipit-source-id: 85305f7db60da6bae589d28c802ee260e42e4dea
Summary:
This changes the way that Eden is built and deployed.
* To build the binary that must be run as `root` (but quickly drops privileges), run `buck build eden-daemon`.
* To build the CLI that communicates with the daemon (and does not require privileges), run `buck build eden-cli`.
* To build both, run `buck build eden`.
There is an example of how to build the various parts of Eden using
Buck and how to package them up in the `install` script introduced by this revision.
While here, I also cleaned up some of our build files and changed them to be
parameterized between internal and external use. In both cases, the user gets the
"unadorned" version of their primary build targets. This ensures that shortcuts such as:
```
buck test eden/fs/integration
```
do the right thing by default.
Finally, I also made `find_default_config_dir()` and `find_default_daemon_binary()`
lazy whereas `find_default_config_dir()` was previously eager.
Reviewed By: simpkins
Differential Revision: D3436245
fbshipit-source-id: 4dfbd59ed0d198620324f0705c462334bb5a7daf
Summary:
If a directory is present in the overlay, we still need to check if a TreeEntry
exists from the source control data structures. Previously this was causing us
to incorrectly report directories as empty if they exist in the local overlay,
even when they had contents from the main Tree.
Reviewed By: wez
Differential Revision: D3434219
fbshipit-source-id: f872f90075602dfdc7b217f50eefcd7c248512e7
Summary:
Add some verbose logging about when trees and blobs are loaded in the object
store.
Reviewed By: bolinfest
Differential Revision: D3434182
fbshipit-source-id: 3e8d2617290604f119e6164d15d63324a4c9a2aa
Summary:
Update the HgImporter class to support retrieving file contents from mercurial.
This also includes simple code for storing the data in the LocalStore using
git's blob serialization format. In the future I think it would perhaps be
better to drop the "blob<length>" prefix, and instead just use a RocksDB column
family to separate blob data from other types of data. However, for now using
the git format is simplest for keeping compatibility with the getBlob() code.
Reviewed By: bolinfest
Differential Revision: D3416691
fbshipit-source-id: 268787533be2172b2dbedc3bf06464eabf3d2c5e
Summary:
Record mercurial commits that have been previously imported, and avoid
re-importing them when checking out a commit that was previously processed and
already has a tree present in the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3416654
fbshipit-source-id: 80f70fa2204d3a2961991b634ad934a623b42b82
Summary:
Add APIs for storing arbitrary (key, value) data.
This will allow BackingStore implementations to store additional metadata, such
as mapping mercurial commit IDs to the eden root tree ID.
Eventually we may want to use RocksDB column families to partition the
different types of data being put into the LocalStore. However, for now this
just uses a single key space. We can add column family support in a separate
diff, if desired.
Reviewed By: bolinfest
Differential Revision: D3409866
fbshipit-source-id: 19a1d340b65bff2081981bf5daf32d5ad15b60c4
Summary:
This adds an HgBackingStore implementation which can load tree data from a
mercurial repository. Blob loading is not implemented yet, but will come in a
separate diff.
This also adds a minimal GitBackingStore class. The GitBackingStore has nearly
no functionality, but is needed to keep the existing git functionality working.
Reviewed By: bolinfest
Differential Revision: D3409743
fbshipit-source-id: dbebf53e9de08bd1469e489baa48b84cbf889511
Summary:
Add the basic BackingStore interface, plus a NullBackingStore implementation
that always returns null. This updates the ObjectStore to query the
BackingStore if data is not found in the LocalStore.
Additionally, this updates EdenServer to manage the BackingStore objects. It
maintains a map of the BackingStore objects created for each known repository.
Reviewed By: bolinfest
Differential Revision: D3409602
fbshipit-source-id: 2920dc4c24ee1ec37efb542f058d0d121ceb5532
Summary:
The _get_hg_dir() function had an incorrect return statement caused "eden init"
to fail with mercurial repositories, unless the path pointed to a shared
working directory.
Also update the code to store the path to the mercurial repository, rather than
the path to the .hg directory. Unlike git, mercurial requires the repository
path itself, and refuses to work if given a path to the .hg directory.
Reviewed By: bolinfest
Differential Revision: D3409808
fbshipit-source-id: 22ee7748422b614f5af57aff07fcfa517af846d9
Summary:
This revision introduces two complementary changes:
* `eden daemon` no longer runs in the foreground.
* There is now an `eden shutdown` command to kill the daemon.
When `shutdown` is called, it tells the Thrift server to shutdown.
In turn, this causes `EdenServer::runThriftServer()` to exit,
which causes `EdenServer::run()` to exit.
Reviewed By: simpkins
Differential Revision: D3402347
fbshipit-source-id: 80032ba53eb69b3f69bef9d7cd169f93500c833c
Summary:
D3406773 included a change which compiles on clang and gcc-5.x, but fails to
build with gcc-4.9.
This looks like a bug in gcc-4.9's handling of list initialization. Overload
resolution for non-initializer-list constructors should be attempted if
no suitable initializer-list constructors are found, but gcc-4.9 does not
appear to do this.
Reviewed By: bolinfest
Differential Revision: D3410142
fbshipit-source-id: f34125000eb3fa949c2427aa4ffbd4ef92942cd7
Summary:
Update deserializeGitBlob() to work on an IOBuf, rather than an rvalue
reference to a string.
The ugliness about having to wrap a std::string in a managed IOBuf is now
hidden inside the StoreResult class, rather than being something that the
GitBlob code has to know about.
Reviewed By: bolinfest
Differential Revision: D3403977
fbshipit-source-id: 0c58c019557050d6e201c1a462fa051c2526674a
Summary:
Add a new StoreResult which wraps the std::string returned by RocksDB.
This replaces the std::unique<string> that LocalStore::get() used to return.
This lets us avoid a memory allocation. StoreResult can also represent a "not
found" result, so that this case can be processed efficiently without having to
throw an exception.
Additionally, StoreResult is move-only so we can't ever unintentionally copy
the string data, which is potentially expensive. It also provides APIs for
creating IOBuf wrappers, or moving the string to the heap so we can create an
managed IOBuf around it.
Reviewed By: bolinfest
Differential Revision: D3403958
fbshipit-source-id: ab0c304988a53eda50341ecc2f96ae5235e5260c
Summary:
Add a new ObjectStore class, which will eventually contain both a LocalStore
and a BackingStore. The LocalStore will be a cache of data loaded from the
authoritative BackingStore. The ObjectStore API will hide the work of querying
the BackingStore and updating the LocalStore when data is not already available
in the LocalStore.
For now ObjectStore only contains the LocalStore, but I will add BackingStore
functionality in subsequent diffs. This diff simply updates all call sites to
use the ObjectStore instead of directly accessing the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3403898
fbshipit-source-id: 47b8c51a7717a4c7c29911a7085b382521a8c0db
Summary:
Previously Hash objects could not be assigned to after they were created, since
they contained a const member. This makes the data non-const, so a Hash
variable can be replaced to contain new contents after it is created.
This also adds a default constructor, which zero-initializes the hash. The
default constructor makes it possible to declare a Hash with a 0-value at one
location, and then set it to the desired value at some later point.
Reviewed By: bolinfest
Differential Revision: D3406773
fbshipit-source-id: 41e2c7e3ad5bc4d14813be4adaa03866701380f6
Summary:
- Add a Sha1Key class that can more efficiently compute the key for
file content SHA-1 values, without having to copy it into a new std::string
object. (In practice fbstring would have avoided having to actually allocate
memory, but it was still an extra data copy.)
- The code was always converting the hash keys to hex on get and put
operations, just in case it needed it if an error occurred. This diff
changes the code to only compute the hex value if an error actually occurred.
Reviewed By: bolinfest
Differential Revision: D3403889
fbshipit-source-id: 5abd8ef202cb00677a84a03a82e2a3d21f16cd2f
Summary:
Update several places to use std::array rather than plain C arrays, using
folly::make_array() to automatically deduce the correct type when necessary.
Reviewed By: wez
Differential Revision: D3370445
fbshipit-source-id: b7642cf3a9b08eac817988bf95679bf5e584ef72
Summary:
Add an HgImporter class for importing mercurial data into the eden local store.
At the moment this only includes code for importing revision manifest data as
tree objects, and does not yet include code for importing file blob data.
Reviewed By: bolinfest
Differential Revision: D3367958
fbshipit-source-id: 58049bf1594b3c27d676c5ebe778917b4043fccf
Summary:
Update deserializeGitTree() to accept an IOBuf object. IOBuf objects can
easily wrap other buffers, so this can still easily support ByteRange objects
as well.
Being able to use IOBuf's Cursor class ended up simplifying the logic a bit as
well.
Note that using IOBuf does require copying the name and mode data out of the
buffer when we read it (using the readTerminatedString() API). This is
necessary since the data may not be stored contiguously in the IOBuf. However,
this shouldn't impact performance much: we already need to copy the name data
into a std::string anyway. For the mode, most modern platforms can avoid doing
a heap allocation for this small string.
Reviewed By: bolinfest
Differential Revision: D3357255
fbshipit-source-id: 5b6e1bc93199849327409a8039266d7dc4f3afdf
Summary: Add a GitTreeSerializer class for serializing git tree data.
Reviewed By: bolinfest
Differential Revision: D3356770
fbshipit-source-id: d04bc9788117272504c2faa335b3648e4ac93e81
Summary:
Previous to this change, if `~/local/.eden` did not exist when `eden daemon` was
run, then it would fail.
Now the logic to create `~/local/.eden` is encapsulated in `_ensure_dot_eden_folder_exists()`.
We do not call this for all subcommands because we want to make sure that running
`eden --help` does not have the side-effect of writing `~/local/.eden` if it does not exist.
Reviewed By: wez
Differential Revision: D3397057
fbshipit-source-id: a3f974f367058d9e4ebd515c78423e54edc179cc
Summary:
`eden health` will return with exit code 0 if both of the following are true:
* The Thrift client is up and running.
* The status of the client is either ALIVE or STARTING.
Reviewed By: wez
Differential Revision: D3395582
fbshipit-source-id: ba668d26acae73a51fbae8aca2b4979156c0c50f
Summary:
When running the CLI "daemon" command, we try killing the underlying eden
process group when we receive SIGTERM or SIGINT. (We really only want to kill
the main eden process, but we currently have to kill the entire process group
due to how sudo works.)
Since the privhelper process runs as root and is part of this process group, we
can get an EPERM error back. This was causing the CLI to fail with an
unhandled exception backtrace. This diff updates the code to ignore EPERM.
Reviewed By: bolinfest, wez
Differential Revision: D3384121
fbshipit-source-id: 39b2364d8c921b1d84a8902566fe9af2a370e4e5
Summary:
This modifies the iterator behavior to so the behavior is a bit cleaner
with respect to empty paths. It is valid to have an empty relative path,
and there are legitimate use cases where this is useful. For instance,
calling dirname() on a RelativePath with a single component will result in
an empty path. It is useful to use this empty path to refer to the parent
directory, to which the path is relative. Therefore it is also useful to
be able to include the empty path when iterating through the parent
directories of a path.
This removes RelativePath::begin() and RelativePath::end(), and replaces
them with a RelativePath::paths() function. paths() returns a struct with
a begin() and end() function, so it can be used in range-based for loops,
and has the same behavior that begin()/end() did. This also adds a
RelativePath::allPaths() function, which also includes the empty relative
path in the results.
Reviewed By: bolinfest
Differential Revision: D3366877
fbshipit-source-id: 3d92b600f07b993925f88d4f1e619b6c1705fb82
Summary:
PrivHelper serializes messages and sends it over to PrivHelperServer who verifies that mount point exists, cleans up bind mounts for the FUSE mount, and undoes FUSE mount.
Some repeated code in this diff since I was unsure on the protocol for that - let me know if/where I should generalize functions to avoid this.
Reviewed By: simpkins
Differential Revision: D3361955
fbshipit-source-id: a7324fb9660912d6c2b753e15b1fa6061c0d5261
Summary:
This avoids translation from string->Hash in the common case
where the file is unmodified and its hash is read directly from
the store rather than computed from the overlay.
I'm guessing I should use `unique_ptr` as the return value throughout?
Reviewed By: simpkins
Differential Revision: D3355773
fbshipit-source-id: 50dff879a78b3d6ff49f86b856866ca28808c4f7
Summary:
Other tools, such as Buck, will benefit from being able to get
the SHA-1 of a file without having to read the entire contents
of the file (or do the associated computation that is proportional
to the size of the contents of the file).
Reviewed By: simpkins
Differential Revision: D3345828
fbshipit-source-id: 360bb268793369af75f408208e8211d8b9db146d
Summary: Updated python CLI to include subparser for unmount command and added wrapper functions that hand over execution to privhelper process. Unmount currently requires client_name at the command line.
Reviewed By: simpkins
Differential Revision: D3359517
fbshipit-source-id: ff05e90bcdb96ecad63f37634c69dbeef429c90f
Summary:
Change Hash::getBytes() to return a folly::ByteRange rather than a
std::array<uint8_t, 20>. This makes Hash more convienent to use with existing
APIs that accept a ByteRange. (For instance, IOBuf.)
There were only 2 call sites using the existing getBytes() functionality,
and they only used the data() method on the returned std::array, so they don't
have to be updated at all to use a ByteRange.
Reviewed By: bolinfest
Differential Revision: D3354581
fbshipit-source-id: 8f2a3c196e59620fb5b0fb2caf4d1d7f26e1d2c4
Summary:
Add a function to compute the sha1 content hash for an overlay file.
We persist the computed hash in an extended attribute in the underlying overlay
file so that a subsequent read of the attribute doesn't require opening the
file to recompute it.
Each time the file is mutated, we blow the cached status of the file.
Each time the sha1 attribute is read, if the cache is blown, the content
hash will be recomputed and set in the overlay file.
Each time the file is flushed or sync'd, if the cache is blown, the content
hash will be computed and set in the overlay file.
Reviewed By: bolinfest
Differential Revision: D3302412
fbshipit-source-id: bd45c7a24b732bd0b7474b7f96e82936870b2117
Summary:
This is part 1 of 2 diffs. This one adds some plumbing to make it possible
to read the xattr attribute from an overlay.
It doesn't do anything to ensure that it is set; the next diff in this series will take care of that.
Reviewed By: bolinfest
Differential Revision: D3302410
fbshipit-source-id: 47406a9c75f29743691d396676c691bcb99c4760
Summary: This logic should be shared by the Eden CLI as well as unit tests.
Reviewed By: simpkins
Differential Revision: D3348300
fbshipit-source-id: c87b1f03f16560323f3d7685063bb6466c39efe2
Summary:
We look this up via the mount point or eden mount object instead.
I've also removed the mercurial library stuff that was added to support the now defunct lamehg fuse we had in the earlier days.
simpkins' new importer doesn't use these and it resolves our CI mismatch issue.
Reviewed By: bolinfest
Differential Revision: D3349698
fbshipit-source-id: 5f4ec16b76042959cd1e3184f46bb3526fbaf74c
Summary:
Update the CLI to support running "init" with a mercurial repository.
This is just some bare bones framework code at the moment. It doesn't actually
import any data from mercurial at the moment, and mounting doesn't work.
Reviewed By: bolinfest, wez
Differential Revision: D3345426
fbshipit-source-id: 72c31ac8d2aac2a16e0a7d6f0425eb4ca218d487
Summary:
Use the hex-encoded version of the key in the RocksException if _get() fails,
rather than the raw binary data.
Reviewed By: bolinfest
Differential Revision: D3345355
fbshipit-source-id: cd8dc644a56ca3d5f3b9a9a0f5cc789b142f0bda
Summary:
Buck is [currently] built with Java 7, so it can only use third-party dependencies
that are also Java 7.
Reviewed By: simpkins
Differential Revision: D3342367
fbshipit-source-id: 4370fd152e7d2055495e783de68a6bb59867bee5
Summary:
This adds a new API to `PrivHelper`: `privilegedBindMount()`.
Similar to `privilegedFuseMount()`, this sends a message to the privileged helper,
which is running as `root`, so it can set up the specified bind mount.
The changes in the `privhelper` directory parrot what was done to support `privilegedFuseMount()`.
Now, once the primary mount for a client is created, any bind mounts listed in the
config for the client are set up. This logic is introduced in `EdenServer.cpp`.
Reviewed By: simpkins
Differential Revision: D3296660
fbshipit-source-id: 61296f35e5c3a6f232a1c17e0f296dd5d3b5ec06
Summary:
Add a new class to serve as a single location where we can store all
information about a single eden mount point. Currently this contains the
MountPoint, LocalStore, and Overlay objects. This allows the TreeInode class
to just store a single pointer to the EdenMount, rather than having to track
these three objects separately.
In the future we could consider also keeping a copy of the ClientConfig in the
EdenMount object, but I haven't done that for now.
Reviewed By: bolinfest
Differential Revision: D3321355
fbshipit-source-id: 8a39bb49822ca8e90c88b2a834b59230d2f91435
Summary:
Enables mkdir in the overlay area.
I had to add some `lstat` calls in to the overlay dir reader because we depend
on knowing at least whether a node is a dir or not at the next level up.
When I run the test suite, the mounts are on my `/tmp` filesystem. When I run
eden manually, they are on my `/data` filesystem. The latter (xfs) does not
populate the type bits. This meant that the test suite passed but manual
testing did not.
Adding the `lstat` calls is a little unfortunate. On OS X there is a bulk
operation that combines `readdir` and `lstat` so that there are fewer syscalls.
We don't have an equivalent for Linux.
Reviewed By: bolinfest
Differential Revision: D3301532
fbshipit-source-id: e228f4a392f90aa491fec62e8b98471a8acecff2
Summary:
We still have naked pointers at the handoff to the kernel, but now
have a cleaner implementation at the level that we're going to be working at
day to day.
I also renamed `FileHandle::release` to `FileHandle::releasefile` so that it
isn't visually ambiguous with `std::unique_ptr::release` in the
`Dispatcher.cpp` code: `fh.release()` vs `fh->release()` look similar but are
dramatically different in behavior.
Reviewed By: bolinfest
Differential Revision: D3309455
fbshipit-source-id: f8cf055bcd51121048a20f0202988cf0aef1f085
Summary:
`creat(2)` and `open(2)` could decide to create a PassThru file handle.
This diff removes that usage. There is a TODO here around handling `O_EXCL` properly.
I'm punting this to a follow-up diff.
Reviewed By: bolinfest
Differential Revision: D3301387
fbshipit-source-id: d35104c536396e7fd064d786f3d5592ecfcbfecf
Summary:
Centralize and delegate most (all?) of the content sensitive portions of file accesses into the FileData class.
Add tests to show that we can write to the overlay file and that the stat data is consistent with the result.
Reviewed By: bolinfest
Differential Revision: D3301251
fbshipit-source-id: a09316ad61c6ef4c656bc5d6dbd43f906abb7932
Summary:
This is the workhorse for adjusting the state that we track for the file data.
It handles both overlay and Tree backed data cases and moving from the latter
to the former depending on the open flags provided when a file handle is
opened.
This diff handles more cases than we have tests for. Those will be covered later in this stack of diffs.
Reviewed By: bolinfest
Differential Revision: D3301213
fbshipit-source-id: c1dab40c0ad205ce6cee820043b70dd886e78431
Summary:
Another step towards making TreeEntryFileInode overlay aware, this diff implements:
- stat
- readlink (although it is not possible to create symlinks yet)
- open
Reviewed By: bolinfest
Differential Revision: D3255158
fbshipit-source-id: 3f90b624e629ef279d6cc32e1d82787ee24796eb
Summary:
This allows us to share the same state between multiple open files and
also helps to reduce the size of inode instances that are otherwise inactive;
when there are no outstanding references to the data, we can drop it and forget
it.
Reviewed By: bolinfest
Differential Revision: D3301198
fbshipit-source-id: f0b3fc73a666ec2033c7a22e9eb587d3212cf966
Summary: This should set us up to have `eden mount` perform the bind mounts.
Reviewed By: simpkins
Differential Revision: D3296370
fbshipit-source-id: 5d8c21308074b357bad3ace72cec157adb5f8b56
