Commit Graph

518 Commits

Author SHA1 Message Date
Adam Simpkins
49adc9c96b fix the behavior of "hg update --clean ."
Summary:
Previously the eden hg extension short-circuited the checkout operation if the
destination commit was the same as the one currently checked out.  This was
incorrect if --clean was specified, since we do need to reset the working
directory state in this case.

This updates the extension code to always make the thrift checkout() call when
doing a force checkout.

This also avoids calling applyupdates() to resolve conflicts when force=True.
When doing a force checkout, eden reports files with conflicts that it
overwrote, but these do not need to be resolved by mercurial.

Additionally, this also updates a couple other APIs that have recently been
changed in upstream mercurial: merge.update() now takes an updatecheck
argument, and repo.join() should now be written as repo.vfs.join().

Reviewed By: bolinfest

Differential Revision: D4752510

fbshipit-source-id: e1ee92d086315e35a1378f674e668876a667c0ce
2017-03-31 11:39:48 -07:00
Wez Furlong
66ec00f294 connect the fuse stats to fb303
Summary:
Tweaks the stats stuff so that we can name the histograms and export them with p50, p90, p99 percentiles.

This is made a bit ugly because the fb303 stats code isn't yet open source, so
there's a moderately ugly preprocessor directive that we assume to be true for
our internal build.   We'll need to force that to be 0 for the open source build
until we open the stats code and make it available.

Reviewed By: bolinfest

Differential Revision: D4801868

fbshipit-source-id: 643909e63bd4a74b2cfa580be131f65c5673bc94
2017-03-31 11:39:48 -07:00
Wez Furlong
d808e1d46c improve batching during importing
Summary:
This explicitly uses a WriteBatch during importing.  This results in a
5x perf gain for import operations.

I've also added in a read-before-write check to make sure that we're not
attempting to overwrite a key that already exists.  Since this is a
content-addressed store, a given key should always have the same
value; it should be immutable.  Overwriting an existing key is an
opportunity to change data that should never change and is also wasteful
of IO bandwidth.

Ideally we'd be able to exploit this to avoid descending into a directory;
I'd like to hook that up to the treemanifest stuff in mercurial in a
follow on diff.

Reviewed By: bolinfest

Differential Revision: D4783598

fbshipit-source-id: 24f95495d09a852859b346559fdf83d064de2b51
2017-03-31 11:20:14 -07:00
Wez Furlong
7c6f9608c2 add basic mknod support
Summary:
this is the bare minimum to support creating unix domain sockets.

We only support using mknod to create a unix socket; other uses will yield an error.

I've added an rdev field as a sibling of the existing mode field that we track,
as that is the additional parameter that we need to track as part of the
special file node.

Special file nodes are tracked in the overlay as empty files.

Reviewed By: bolinfest

Differential Revision: D4774099

fbshipit-source-id: 0824b7e509063faa8bede7aff82a7c51930c4f83
2017-03-30 23:53:05 -07:00
Adam Simpkins
6cced7c449 implement "hg status" using EdenMount::diff()
Summary:
This is an initial pass at implementing "hg status" using the new EdenMount and
TreeInode diff() code.

More work still needs to be done to clean things up here, but this makes
"hg status" more correct in the face of modified files and directories that are
not modified, and vice-versa.

In particular, the following still needs to be cleaned up in future diffs:
- Update the "hg status" code to more correctly process user directives for
  paths that did not appear in the diff output.  This will probably be
  simplified some by correctly updating the user directive list on checkout and
  resetCommit() operations.  It may also make things simpler if we store the
  user directives in a hierarchical path map, rather than in a flat list.
- Update the addAll() code to also use the new diff logic, rather than the
  getStatusForExistingDirectory() function.
- Clean up the locking some.  I think it may be worth changing the code to use
  a single lock to manage both the current commit ID for the mount and the
  dirstate user directives.  We probably should hold this lock for the duration
  of a diff operation.  The diff operation should also return the commit ID
  that it is based against.

Reviewed By: wez

Differential Revision: D4752281

fbshipit-source-id: 6a6d7e2815321b09c75e95aa7460b0da41931dc0
2017-03-30 21:35:00 -07:00
Wez Furlong
d7cd07e0eb add CommandCollector utility for monitoring purposes
Summary:
This is a utility that is intended to be consumed by our
fbagent monitoring infrastructure.

The typical pattern at FB is that fbagent is configured via monitoring config
to pull data from the hosts that belong to a thrift tier and then push that
into ODS.

Since our model is a bit different we don't fit that pattern; eden runs as a
process per user on a host, and is accessible only via unix domain socket.

That means that we need to use a CommandCollector to sample data from our eden
servers, and that's where this tool comes in.

The intent is to use configerator to set up the command collector to run the tool
using a command line that looks something like this:

```
eden-fb303-collector /data/users/*/.eden/socket --tierName %%TIER_NAME%% --counterRegexes '[".*mount.*.rate.60"]'
```

Note that we rely on the shell to expand the glob to find plausible eden sockets.

The `counterRegexes` parameter is a JSON encoded array listing the pcre regexes
that should be used to select the counters that we want to place into ODS.

I chose to pass these as JSON encoded parameters because it will allow the list
of counters to be configured easily in configerator (python) code alongside the
CommandCollector definition and pass that down to the collector.  That makes it
a no-brainer to `json.dumps()` and pass the list down.

This collector emits three entities for each key:

* user@hostname - allows us to distinguish different users on the same host.
* hostname - aggregated by fbagent for all instances on the same host.
* TIER - aggregated by fbagent for all instances on the same tier.

I could add more levels of aggregation to this; it is common to track by region
and cluster, but I'm not sure how many of these we really want; any strong preferences?

The intent is that we'll deploy this alongside the edenfs binary that we
package with packman and then have a separate configerator diff to add the
CommandCollector to enable collection for the dev tiers.

D4797021 and D4797101 are the related diffs to hook this up in the right places

Reviewed By: bolinfest

Differential Revision: D4789717

fbshipit-source-id: 4a1d58c50847a8ca99043542cd2f0415da783ad7
2017-03-29 12:50:22 -07:00
Wez Furlong
13b492f9cc leave getlk and setlk as nullptr to implement locking
Summary:
Leaving these as nullptr causes the kernel to handle advisory locking for us.

There's some plumbing in libfuse that handles negotiating this for us.

Reviewed By: simpkins

Differential Revision: D4774540

fbshipit-source-id: 5495963b980d95f06ededa72f4d483cc74fd8ec3
2017-03-24 23:07:42 -07:00
Wez Furlong
4235784907 add .eden "magic" dir
Summary:
It's not really magic because we don't have a virtual directory
inode base any more.  Instead, we mkdir and populate it at mount time.

What is slightly magical about it is that we give it some special powers:

* We know the inode number of the eden dir and prevent unlink operations
  on it or inside it.
* The .eden dir is present in the contents of the root inode and will
  show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
  magic `.eden` inode number.  This means that it is possible to `stat`
  and consume the `.eden` directory from any directory inside the eden
  mount, even though it won't show up in `readdir` for those child dirs.

The contents of the `.eden` dir are:

* `socket` - a symlink back to the unix domain socket that our thrift
  server is listening on.  This means that it is a simple
  `readlink(".eden/socket")` operation to discover both whether a directory
  is part of an eden mount and how to talk to the server.

* `root` - a symlink back to the root of this eden mount.  This allows
  using `readlink(".eden/root")` as a simple 1-step operation to find
  the root of an eden mount, and avoids needing to walk up directory
  by directory as is the common pattern for locating `.hg` or `.git`
  dirs.

Reviewed By: simpkins

Differential Revision: D4637285

fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
2017-03-24 23:07:42 -07:00
Adam Simpkins
0a6ea6bbfe fix EdenMount::diff() to skip .hg and .eden directories
Summary:
Update the EdenMount::diff() code to completely skip .hg and .eden directories.
This was implemented through the GitIgnore code, and adding a new HIDDEN status
The .hg and .eden directories are similar to ignored directories, but we never
want to report any information for them even when listIgnored is true.

This also changes the GitIgnore code to so that GitIgnoreStack computes the
basename for each path once, and passes it down into the underlying match code,
so we don't have to re-compute it for each pattern that needs the basename.

Reviewed By: wez

Differential Revision: D4751917

fbshipit-source-id: ec22d62f31a3ce9ef998a8ccb4413f72f1d7a487
2017-03-24 13:50:09 -07:00
Adam Simpkins
b6fe1a5111 update EdenMount::diff() to process .gitignore files
Summary:
This updates the diff() logic to process gitignore rules.

The bulk of the existing TreeInode::diff() was moved into a
TreeInode::computeDiff() function.  TreeInode::diff() now loads the gitignore
rules for this directory before calling computeDiff().  The argument list for
computeDiff() was now getting slightly unwieldy, so I created a new DiffContext
class to hold some of the arguments that are the same for the duration of the
entire diff operation.  This resulted in some rather mechanical changes through
the computeDiff() and DeferredDiffEntry code to update the argument lists.

Reviewed By: wez

Differential Revision: D4744690

fbshipit-source-id: c4981d9f49878b522e0b3faf35de837739066f3c
2017-03-24 13:50:09 -07:00
Andrew Gallagher
315330eb6f buckification: remove various non-determinism from build files
Reviewed By: yfeldblum

Differential Revision: D4761474

fbshipit-source-id: 140f0bcb1413365b93c50819bedb5039770efeb8
2017-03-23 11:09:28 -07:00
Adam Simpkins
f9f90ec116 add suffixes() and rsuffixes() iteration to composed path classes
Summary:
Add suffixes() and rsuffixes() methods to the RelativePath and AbsolutePath
classes.

I am planning to use this functionality when performing gitignore matching.
(Each .gitignore file matches against relative path suffixes from the directory
containing the .gitignore file.)

Reviewed By: wez

Differential Revision: D4744612

fbshipit-source-id: 1fd3bdb4a66193fbd73aa514442c51b5024a30e3
2017-03-21 21:06:28 -07:00
Wez Furlong
5ca8bcdc60 implement eden->watchman subscriptions
Summary:
This tweaks the definition of the subscribe method in the
streamingeden thrift file and implements the server side of the
thrift service, and the client side to consume it in watchman.

The definition is now a bit simpler than it was previously; we're
now just sending a stream of the updated JournalPosition objects
to the client rather than computing and sending FileDelta objects.

This is cheaper for Eden to deal with because it is very cheap to take
the current journal position and pass that over the wire.  This is
important because a burst of mutations will likely queue up a bunch
of these in quick succession.

In a future diff I'm considering adding a latency parameter for
the subscription so that we can constrain the number of updates
over a certain time period (likely in the 10's of milliseconds range).

For now I just want to prove that the concept works.

On the watchman side we just need to pull these off the wire as they are sent
by eden, then wait for the subscription stream to settle before internally
broadcasting to any connected subscribers.

Reviewed By: bolinfest

Differential Revision: D4647259

fbshipit-source-id: 03aa16e59a43195a2505a8d03bce1ccf88a8d42f
2017-03-21 13:35:20 -07:00
Adam Simpkins
b48fad3547 make sure edenfs does not exit successfully if privhelper crashes
Summary:
Update the edenfs main function to check the exit code of the privhelper
process, and to exit with EX_SOFTWARE rather than EX_OK if the privhelper did
not exit with status 0.

This will ensure that our integration tests always catch crashes in the
privhelper process.  Previously we could catch errors that happened before the
privhelper socket was closed, but if the privhelper server crashed during
cleanup we never noticed the error.

Reviewed By: bolinfest

Differential Revision: D4728006

fbshipit-source-id: 2c478612e5fabec90438b28b9c76a3343f561655
2017-03-21 13:05:57 -07:00
Adam Simpkins
a4d599c4c1 add one more checkout unit test
Summary:
Add a unit test that modifies a file and then does a forced checkout to revert
back to the original commit state.

This also includes some pretty-printers to make the tests print
CheckoutConflict objects nicely if any test assertions fail.

Reviewed By: bolinfest

Differential Revision: D4727779

fbshipit-source-id: b31dc90be28d17ed0f2a4076c96a08ba82588190
2017-03-21 12:53:26 -07:00
Adam Simpkins
d101f404dc add an EdenMount::diff() method
Summary:
This begins adding an EdenMount::diff() method, which walks through the inode
tree and reports differences from the current source control tree state.

My intent is to eventually update the Dirstate code to use this diff()
function, and to remove the existing getModifiedDirectories() logic.  The
getModifiedDirectories() code is currently incorrect, as it reports
materialized directories, which is not the same as directories that are
modified from the current source control state.

The ignore processing logic is not currently implemented in EdenMount::diff(),
but it should be relatively straightforward to add in a subsequent diff.

Reviewed By: bolinfest

Differential Revision: D4726048

fbshipit-source-id: ad0bb3b5d72bb3830f60fc2b2e56a81217c35353
2017-03-21 12:06:44 -07:00
Adam Simpkins
38bbe966ab fix a bug in bind mount shutdown
Summary:
The privhelper code was erasing elements from a std::unordered_map before it
was done using the iterator pointing to that element.  This causes memory
corruption issues.

Between this and some of my other recent unmount fixes (D4548030, D4547938)
this makes the bind-mount-related integration tests work.

Reviewed By: bolinfest

Differential Revision: D4727850

fbshipit-source-id: 6d1fda3f89cb91c89d0020921b1805fc10e65785
2017-03-20 22:11:19 -07:00
Adam Simpkins
c8ea0d5ba6 implement removing directories during checkout
Summary:
Update the TreeInode and CheckoutAction to correctly be able to remove
directories during checkout operations.  This handles both removing directories
that are newly empty as a result of the checkout, and directories that are
replaced with a file or symlink.

This also refactors the rmdir/unlink code a bit so we can share some of its
logic for doing directory removal during the checkout operation.

Reviewed By: bolinfest

Differential Revision: D4708607

fbshipit-source-id: 3b9dd9cc3536845dad0d9278e3c5d3ac1ed04570
2017-03-17 17:13:20 -07:00
Adam Simpkins
7202883bf8 synchronize access to the CheckoutContext
Summary:
Synchronize access to the CheckoutContext conflicts_ list when recording
conflicts or errors during a checkout operation.

In theory it is possible for the checkout operation to be proceeding
simultaneously in multiple threads.  At the moment this does not happen because
the BackingStore implementations currently block, and always return
immediately-ready Future objects in the same thread.  However, once they are
updated to return Future objects that actually complete asynchronously in other
threads the checkout work will continue in these other threads.

Reviewed By: bolinfest

Differential Revision: D4708605

fbshipit-source-id: 885b0114d97dadcacf64652f795a7d3846412f11
2017-03-17 17:13:20 -07:00
Saif Hasan
a03676f7fb Add ifName field to BinaryAddress
Summary:
We are using `BinaryAddress` heavily within fboss codebase to represent a
nexthop as well as an IPAddres in wire format. Nexthops will need to have
a scoping identifier (iface name) for link-local address inorder to resolve
it properly.

We do this in OpenR and well tested on Arista. This seems bit hacky but it
saves a lot of time. In future we would like to have separate network util for
fboss and openr and we can get rid of this hack at that time.

Differential Revision: D4710768

fbshipit-source-id: 72092935f3738b3fb88a422462d1c995cc0acf18
2017-03-17 17:13:20 -07:00
Adam Simpkins
7f58caffe4 Add fb303 stubs in common/ to support building in the opensource repo
Summary:
This adds a common/ directory with stub files required for building
eden.  These stubs are the same ones as used in the fboss repository:
https://github.com/facebook/fboss/

Our goal is to eventually create an opensource fb303 repository with
complete implementations of these files, but for now these basic stubs
allow building eden.  (Most of the underlying stats code that supports
fb303 is already available in folly/stats/ in the folly repository.)

Test Plan:
Tested building eden on Ubuntu 16.04
2017-03-17 17:08:05 -07:00
Adam Simpkins
7de5d1d90c add a test for creating a new subdirectory via checkout()
Summary:
Add a test that exercises a checkout operation that creates a brand new
subdirectory.

Reviewed By: bolinfest

Differential Revision: D4693994

fbshipit-source-id: 70f43f67c2cdb10bd854f51d5ba2e61c45976975
2017-03-14 16:49:43 -07:00
Adam Simpkins
1add48d20b fix the build with gcc
Summary:
Unfortunately it looks like gcc won't support the [[nodiscard]] attribute until
the gcc-7.x release.  (The functionality is available in gcc-4.8 and later, but
using the name [[gnu::warn_unused_result]] instead.)

This changes the code to use the FOLLY_WARN_UNUSED_RESULT macro from
folly/Portability.h instead.  (This expands to older non-standard __attribute__
syntax for gcc and clang, and _Check_return_ on Windows.)

Reviewed By: bolinfest

Differential Revision: D4704184

fbshipit-source-id: d5c13630ea611a2f43080a501add42ce9fda6789
2017-03-14 14:31:54 -07:00
Adam Simpkins
bda49af5a3 add additional LoadBehavior types during checkout tests
Summary:
Add new ASSIGN_PARENT_INODE and ASSIGN_INODE types to LoadBehavior, to exercise
additional code paths during checkout operations.  This also cleans up the
overall handling of the LoadBehavior enum.

Reviewed By: bolinfest

Differential Revision: D4694548

fbshipit-source-id: e8359cfdadb1fabb1a0d07bc57a1a1610ed9ba57
2017-03-13 21:02:30 -07:00
Adam Simpkins
d1e1e4c621 refactor FileInode to avoid using its parent's TreeInode::Entry
Summary:
Refactor FileInode and FileData so that they no longer store the
TreeInode::Entry that refers to this file.  The TreeInode::Entry is owned by
the parent TreeInode and is not safe to access without holding the TreeInode
contents_ lock, which the FileInode code never did.  Additionally, once a
FileInode is unlocked the TreeInode::Entry is destroyed.  Previously the
FileInode code would retain its pointer to this now invalid data, and could
still dereference it.

In the future I think it might be worth refactoring the FileInode/FileData
split a bit more, but this at least addresses the issues around invalid
accesses to entry_.

Reviewed By: bolinfest

Differential Revision: D4688058

fbshipit-source-id: 17d5d0c4e756afc6e3c4bb279cb1bb5231f3134f
2017-03-10 18:29:29 -08:00
Adam Simpkins
90b44c1ec2 remove TreeInode::entry_
Summary:
Remove the TreeInode::entry_ member variable.  This was a pointer to the
TreeInode::Entry object representing this inode in the parent TreeInode.

However, it isn't safe for a child Inode to access the TreeInode::Entry owned
by its parent.  This Entry object is supposed to be protected by the parent
TreeInode's contents_ lock, and the child inode never held this when accessing
the Entry.

This updates the TreeInode code to never use entry_ any more.  I will work on
removing FileInode::entry_ in a subsequent diff.

This did require changing the overlay materialization code some.  Previously
the checkout code directly modified entry_ to inform the parent TreeInode if
the child was materialized or not.  Now we have to more explicitly tell the
parent TreeInode about changes to each child's materialization state.  With
this change we now keep the overlay data on disk more consistent during the
checkout process.  We should now always have overlay data for children inodes
on disk whenever the parent thinks they are materialized.  We may end up
writing out TreeInode overlay data frequently during the checkout process,
however.  We might be able to improve this in the future.  For now it seemed
simpler and safer to just update the data frequently to make sure it stays
consistent.

Reviewed By: bolinfest

Differential Revision: D4688055

fbshipit-source-id: d08a14b9304ac49826f0897a0b53281177d9d9f4
2017-03-10 18:29:29 -08:00
Adam Simpkins
a13d55796c delay removing overlay data until inodes are completely unreferenced
Summary:
Update the code so that we no longer immediately remove overlay data about a
file or directory when the inode is unlinked.  The file can continue being used
after it was unlinked if there are open file handles or other internal
references to it.

This updates the code to instead wait to remove the overlay data until the
InodeBase object is unloaded.

This does not fully fix all the outstanding issues with using inodes after they
have been unlinked, because the FileInode and TreeInode still use the entry_
pointer that now points to a bogus TreeInode::Entry.

Reviewed By: bolinfest

Differential Revision: D4688056

fbshipit-source-id: a9e1c1a9562590b8b81237057bfc4ef77795807f
2017-03-10 18:29:29 -08:00
Adam Simpkins
2369f6f810 address some issues with materializing inodes
Summary:
This changes a couple aspects of the materialization process:

- Write out child files in the overlay before their parent.  If we throw an
  exception or crash partway through this process, it is better to have the
  child not marked materialized in its parent, rather than to have the parent
  indicate that the child is materialized but to not actually have any overlay
  data for the child.  In the former case we will still load the correct child
  data the next time we need to, but in the latter case we would fail to load
  the child correctly.

- Hold the rename lock while materializing our parent directories, to ensure
  that we materialize the correct parent, and it cannot change while we are
  trying to perform the materialization.

- Make the parent responsible for modifying the child's TreeInode::Entry.  This
  data is owned by the parent and is protected by the parent's contents_ lock.
  The child really shouldn't ever be directly accessing this data without
  holding the parent's contents_ lock.

Reviewed By: bolinfest

Differential Revision: D4688057

fbshipit-source-id: 2662f79cb7d7febb086f4e0888a3d96a580c4bfa
2017-03-10 18:29:29 -08:00
Adam Simpkins
9980907d1e fix checkout to always report the specific list of files with conflicts
Summary:
This fixes the checkout code to always recurse into directories with conflicts,
and accurately list all individual files with conflicts.

Previously the checkout code always avoided recursing into unloaded directory
inodes that were affected by the checkout.  Since the inode was not loaded we
can easily replace it without recursing into it.  Unfortunately this resulted
in an inaccurate conflict list--we would report the directory itself as a
conflict.  This is insufficient for our mercurial extension, which needs to
know the specific list of files with conflicts, so it can ask mercurial to
perform conflict resolution on them.

Reviewed By: bolinfest

Differential Revision: D4672373

fbshipit-source-id: ff400a4a6e25b8b0754c66cb067a69d553a5c98b
2017-03-09 20:57:27 -08:00
Igor Sugak
ab1943d4fc use gflags namespace instead of google
Summary:
`google` namespace is deprecated in gflags. Replacing it with `gflags` namespace.

gflags was generated from this diff: P57170122
```
% echo $gflags
google::(RegisterFlagValidator|CommandLineFlagInfo|GetAllFlags|ShowUsageWithFlags|ShowUsageWithFlagsRestrict|\
DescribeOneFlag|SetArgv|GetArgvs|GetArgv|GetArgv0|GetArgvSum|ProgramInvocationName|ProgramInvocationShortName|\
ProgramUsage|VersionString|GetCommandLineOption|GetCommandLineFlagInfo|GetCommandLineFlagInfoOrDie|]
FlagSettingMode|SET_FLAGS_VALUE|SET_FLAG_IF_DEFAULT|SET_FLAGS_DEFAULT|SetCommandLineOption|\
SetCommandLineOptionWithMode|FlagSaver|CommandlineFlagsIntoString|ReadFlagsFromString|AppendFlagsIntoFile|\
ReadFromFlagsFile|BoolFromEnv|Int32FromEnv|Uint32FromEnv|Int64FromEnv|Uint64FromEnv|DoubleFromEnv|\
StringFromEnv|SetUsageMessage|SetVersionString|ParseCommandLineNonHelpFlags|HandleCommandLineHelpFlags|\
AllowCommandLineReparsing|ReparseCommandLineNonHelpFlags|ShutDownCommandLineFlags|FlagRegisterer)

% hg grep -wlE "$gflags" 're:fbcode.*\.(cc|cpp|h)' | xargs perl -pi -e 's,\bgoogle::,gflags::,g if /'"$gflags"'/'
```

Reviewed By: meyering

Differential Revision: D4669201

fbshipit-source-id: 8053ba6fba9acf6eaf6796f0f297a9e07784973f
2017-03-08 22:14:27 -08:00
Adam Simpkins
aec52ad41d update unlink() to load the child inode before removing it
Summary:
Loading the child inode is necessary to avoid race conditions with other
threads that may be currently trying to load the child inode in question.

This updates the code so that rmdir() and unlink() now share most of their
implementation.

Reviewed By: bolinfest

Differential Revision: D4664110

fbshipit-source-id: 871ca7b3c22819b24bc2639d9550bb45f6541f50
2017-03-08 17:09:36 -08:00
Adam Simpkins
2c4dde3afb replace the getdeps.sh script with a python version
Summary:
Replace the hacky getdeps.sh shell script added in D4633747 with a slightly
less hacky python version.  In the future we could potentially split this up
into a generic library module and an eden-specific piece, so that it could be
more easily shared with other projects (such as fboss).

Reviewed By: bolinfest

Differential Revision: D4665160

fbshipit-source-id: 627d34d10ca1119c302a67974324784122b67c8b
2017-03-08 16:38:36 -08:00
Adam Simpkins
f9b54814b3 add a simple getdeps.sh script
Summary:
This adds a somewhat hacky shell script for downloading and building the
dependencies necessary to build eden from github.  This was based on the
getdeps.sh script from the fboss repository.  Ideally I really should clean
this up in the future so that it can share more code with the fboss version,
instead of copying much of the logic.  For now this is a quick hack that
suffices to get the open source eden code building.

It fetches the necessary external libraries from github, and builds them.

If "pkg" is specified as the first argument it first runs apt-get to install
the necessary package dependencies on Ubuntu.  Some of the external
repositories that are downloaded from github are available as Ubuntu packages,
such as zstd, but eden requires more recent versions of these libraries.

Reviewed By: bolinfest, wez

Differential Revision: D4633747

fbshipit-source-id: 618a98c8803cbbb4f45fdf76058d25617d0117ff
2017-03-06 20:37:08 -08:00
Adam Simpkins
097bc477de get the open source build working
Summary:
Update the DEFS and .buckconfig files so that the C++ code can be built
successfully with buck in our github repository.

Reviewed By: wez

Differential Revision: D4633749

fbshipit-source-id: a07a6e664471237fcc2d13a8be8b92efa3a101bf
2017-03-06 20:28:48 -08:00
Adam Simpkins
02664c185f add more unit tests for checkout
Summary:
Replace the one very basic checkout unit test with several tests that
more thoroughly test file modification changes.  More tests for
directory changes and file type changes still need to be added, but this is a
first step for now.  I will continue adding more tests in upcoming diffs.

Reviewed By: wez

Differential Revision: D4641595

fbshipit-source-id: 443a68e4e3c673fe687446f3469f035b4f48f1b3
2017-03-02 14:24:10 -08:00
Adam Simpkins
dcf8e6a3a2 add an EdenMount::resetCommit() method
Summary:
Add a method to reset the current commit without changing the working
directory state, similar to "hg reset" and "git reset --soft".

This also adds a new EXPECT_FILE_INODE() check for use in the unit tests,
and FileInode::getPermissions() and FileData::readFull() methods to support
this check.

Reviewed By: wez

Differential Revision: D4641476

fbshipit-source-id: 1e516774fe8e292a8d82cc2c354619374a3abe37
2017-03-02 14:24:10 -08:00
Adam Simpkins
7100eecdaa additional refactoring of unit test initialization code
Summary:
The FakeTreeBuilder class in D4609587 provides a lot of the same functionality
as the TestMountBuilder class, but is not restricted to being used only at
mount initialization time.  (It also provides more powerful functionality for
controlling the ordering of events when loading data from the ObjectStore.)

This switches all of the existing tests to use FakeTreeBuilder rather than
TestMountBuilder, and removes the TestMountBuilder class.

One difference is that FakeTreeBuilder adds data to the FakeBackingStore, while
TestMountBuilder previously put data directly into the LocalStore.  This
shouldn't really make much difference for the behavior of the code.  Putting
data in the FakeBackingStore gives us more control over when data is available
during load operations, and exercises slightly more of the normal ObjectStore
code path.

Reviewed By: wez

Differential Revision: D4641288

fbshipit-source-id: ca2b45bf69bd373848c12a7b739b286aabe6aa9b
2017-03-02 14:24:10 -08:00
Adam Simpkins
4ac4d94cb8 add a FakeTreeBuilder class to make writing tests easier
Summary:
Add a FakeTreeBuilder class to make it easier to create Tree structures in the
FakeBackingStore using path names.  Previously the test code had to manually
chain together Trees and Blobs.  FakeTreeBuilder lets callers call setFile()
with a path name to define the file layout.  The files that were defined then
get converted into Tree and Blob data when finalize() is called.

This is similar to the existing TestMountBuilder::addFile() API.  However,
TestMountBuilder only populates the LocalStore.  FakeTreeBuilder populates
the FakeBackingStore instead.  The FakeBackingStore class allows tests to have
much more control about when objects appear ready from the backing store,
allowing control over the order in which Futures are fulfilled during the test.

Reviewed By: bolinfest

Differential Revision: D4609587

fbshipit-source-id: 9642d17daf0dc10f08901e51552bbb3c3e150b53
2017-03-02 14:24:10 -08:00
Adam Simpkins
ec85eb7ec7 fix warnings triggered with -Wshadow-compatible-local
Summary:
Make sure the eden coded compiles cleanly with -Wshadow-compatible-local

Pretty much all of the warnings were issues with lambdas shadowing names from
their parent context (even though they didn't ask to capture those names from
the parent).

Reviewed By: wez

Differential Revision: D4644849

fbshipit-source-id: 66629cd98b5af4760f3fbb256e44c0bc47e52316
2017-03-02 13:32:51 -08:00
Wez Furlong
c543f89404 add symlink support to the overlay
Summary:
Previously, we would only allow you to consume a symlink that was
checked into a revision, but not allow you to create a new one.

This diff adds support for making new symlinks.

It turns out that the code that we had for readlink had an unused code path for
reading an actual symlink out of the overlay.  My first pass at this diff tried
to make the materialization code generate such a symlink, but it was breakin
some other assumptions in the rest of the code.

This version of the code just stores the symlink target in the file contents.
This means that we can simplify readlink to just calling `readAll` on the
underlying file data, and that will load the contents out of the storage layer
if the file wasn't materialized.  This helps simplify things a bit more.

Reviewed By: bolinfest, simpkins

Differential Revision: D4604016

fbshipit-source-id: 3138d0f9880639d2bbeaf4bb03bef3f021c3ecb3
2017-03-02 08:18:45 -08:00
Wez Furlong
2fc9fc155c enable StreamingEdenService
Summary:
In order to send realtime subscription information to watchman
we need to use the thrift streaming data functionality.  Since this
isn't supported in java we originally broke this out into its own
thrift file.

It's now time to hook this up; this diff adds some plumbing so that
our thrift server knows to implement this interface.

This was made a bit more complicated because the inheritance in
the thrift definitions causes some ambiguities in the header maps
that buck uses to manage the includes.

I also had to rename the file from `eden.stream.thrift` to `streamingeden.thrift`
to avoid a problem with the period in the middle of the identifier.

For the server side of things, I've removed the `:thrift-cpp2` target
and replaced it with `thrift-streaming` instead.

This diff doesn't implement the endpoint, it is just setting things
up for the follow-on diff that does.

Reviewed By: bolinfest

Differential Revision: D4640639

fbshipit-source-id: b3625b0ff33a42945bf523944beed050f549a18c
2017-03-01 23:12:50 -08:00
Wez Furlong
82612ba4d6 remove getMaterializedEntries thrift API
Summary:
I originally added this to facilitate `hg status` but we didn't end up
using it.  Rather than keeping it around and updating the tests in the `.eden`
dir diffs that follow, let's just remove it.

Reviewed By: simpkins

Differential Revision: D4610768

fbshipit-source-id: 158c0207f88980e86aeeddf75e6fd49763d2a402
2017-03-01 08:19:29 -08:00
Adam Simpkins
09006d0e4a Drop AutoHeaders.RECURSIVE_GLOB from TARGETS files
Summary:
Remove AutoHeaders.RECURSIVE_GLOB from all of our TARGETS files.  This is a
Facebook-internal directive that was added as part of a codemod.  The vanilla
open source buck does not know about this directive, and can't parse TARGETS
files that contain it.

All of the affected rules are cpp_unittest() and cpp_binary() rules that don't
actually have any header files, so these parameters didn't actually have any
effect.

Reviewed By: andrewjcg

Differential Revision: D4633748

fbshipit-source-id: 6227fe9b7f2ea838a0c72408ca195685bcae9f5a
2017-02-28 21:46:39 -08:00
Michael Bolin
22994cb1e8 Make oss install script less hacky using --out argument to buck build.
Summary:
The `--out` option was not available when this script was originally written,
but this is exactly the sort of hacky shell-scripting it is desgined to
replace.

Reviewed By: mzlee

Differential Revision: D4625460

fbshipit-source-id: 9c884d0274bb5ebda8c59a7c9c6b2be22efd6265
2017-02-28 12:12:35 -08:00
Adam Simpkins
c764deb101 implement path component iteration
Summary:
Add components() and rcomponents() methods to ComposedPathBase.
These functions return a pair of iterators allowing you to iterate through the
components in the path.

This is similar to the existing paths() and rpaths() method which iterate
through path prefixes.

Reviewed By: wez

Differential Revision: D4609907

fbshipit-source-id: 698e5d58db897bbc760f2a2de90ef430fcb03a52
2017-02-23 20:22:21 -08:00
Adam Simpkins
0515c8c6a4 fix a bug incrementing the wrong index during TreeInode checkout
Summary:
In TreeInode::computeCheckoutActions() the code incremented the wrong index
when we reached the end of the new tree before the old tree.

panda-facepalm

Reviewed By: bolinfest

Differential Revision: D4609906

fbshipit-source-id: 61ca6baabe4eebb033c48b742562558466a45104
2017-02-23 18:29:41 -08:00
Adam Simpkins
b766aba11c invalidate the FUSE cache properly during checkout
Summary:
Invalidate directory entries as they are modified during the checkout
operation.  We never need to invalidate inode contents, since we always update
files by replacing the file with a new inode.

Reviewed By: wez

Differential Revision: D4577942

fbshipit-source-id: afde3040960b076d2cd384301a89cd7bdfce5a6a
2017-02-22 18:37:42 -08:00
Adam Simpkins
645b6fca0a update "eden daemon" to also pass through KRB5CCNAME
Summary:
When the "eden daemon" CLI command has to run edenfs via sudo, also explicitly
pass through the KRB5CCNAME environment variable.  This will allow edenfs and
it's hg import helper to find the correct Kerberos credentials if it needs to
connect to a remote server.

Reviewed By: wez, bolinfest

Differential Revision: D4584132

fbshipit-source-id: 6c72e4eca76d81b3d4b53a9e72a0e8388da30a7d
2017-02-22 18:37:42 -08:00
Adam Simpkins
82d8d57ea7 handle forced checkout of newly added files with conflicts
Summary:
This updates CheckoutAction to make the old TreeEntry object optional.
This is necessary to support situations where the entry did not exist in the
old source control tree, but does exist in the new tree and also exists locally
on the file system.  This situation is always a conflict, so it only applies
when a force checkout is being done.

This diff also refactors the TreeInode::computeCheckoutActions() logic to
combine the processNewEntry/processRemovedEntry/processEntry lambdas into a
single function, as suggested in the code review for D4538516.

Reviewed By: wez

Differential Revision: D4577924

fbshipit-source-id: 9435205b541a12c75a8b9bd2b2b599a339a2f6c8
2017-02-22 12:21:48 -08:00
Adam Simpkins
84bd24cf96 add a getCurrentSnapshot() thrift call
Summary:
Add a thrift call to get the ID of the snapshot currently checked out in a
given eden mount.

Reviewed By: bolinfest

Differential Revision: D4570729

fbshipit-source-id: 208ceb309aacfb07e30ddb8fdfc0950a3d7d748b
2017-02-22 12:21:48 -08:00