Commit Graph

126 Commits

Author SHA1 Message Date
Eamonn Kent
27c7865526 Update eden mount point to be created asyncronously
Summary:
- Updated makeShared() to return a folly::Future<std::shared_ptr<EdenMount>> instead of just a std::shared_ptr<EdenMount>.
- Updated callers to use the returned future (get() for now)
- Refactor makeShared() and the EdenMount constructor to avoid blocking.

Reviewed By: simpkins

Differential Revision: D5424088

fbshipit-source-id: f026a3a3e4abb3593bafda76673e12c55da26322
2017-07-17 13:06:52 -07:00
Adam Simpkins
dbe029d83d exit gracefully on SIGTERM and SIGINT
Summary:
Update EdenServer to catch SIGTERM and SIGINT, and to stop gracefully on these
signals.

Reviewed By: wez

Differential Revision: D5315323

fbshipit-source-id: 16611190c8d522a78cf6b624d97d4f7ecc478f96
2017-06-26 19:34:55 -07:00
Christopher Dykes
3c2bb6b1dd Cleanup the logging targets
Summary: This includes adjusting them to conform to Folly's style for single file sources and headers, making targets as small as possible, not providing aggregate targets, and using auto-deps.

Reviewed By: yfeldblum

Differential Revision: D5312078

fbshipit-source-id: 1c7f5aa04da3bad236ffa23000c7bda47b82e3ef
2017-06-23 15:58:11 -07:00
Adam Simpkins
8893b0d694 use AbsolutePathPiece for repository path arguments
Summary:
Update HgBackingStore and GitBackingStore to accept the repository path as an
AbsolutePathPiece rather than as a plain StringPiece.

Reviewed By: bolinfest

Differential Revision: D5309172

fbshipit-source-id: aca4818c3add11d07ece796298f6175ca4fb1448
2017-06-23 15:23:32 -07:00
Jyothsna Konisa
299f7d1373 Integration Test for UnnloadFreeInodes
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.

Reviewed By: simpkins

Differential Revision: D5277870

fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
2017-06-22 22:38:47 -07:00
Adam Simpkins
b0d1e57975 update logging statements to use folly logging APIs
Summary:
Update eden to log via the new folly logging APIs rather than with glog.

This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else.  (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)

Reviewed By: wez

Differential Revision: D5290783

fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
2017-06-22 13:50:13 -07:00
Adam Simpkins
6b62f186c7 drop Future from the name of ObjectStore APIs
Summary:
Now that the non-future versions of these APIs have been removed, rename
getBlobFuture() to getBlob(), and getTreeFuture() to getTree()

Reviewed By: wez

Differential Revision: D5295690

fbshipit-source-id: 30dcb88854b23160692b9cd83a632f863e07b491
2017-06-21 17:20:50 -07:00
Jyothsna Konisa
b8be1633bc Cli command to unload free inodes of a under a given directory.
Summary:
1.Added a new thrift method to unload free inodes of a directory and its sub directories.
2.Added a new debug sub command 'eden debug unload <path>' to cli tools to unload free inodes.

Reviewed By: simpkins

Differential Revision: D5261038

fbshipit-source-id: 85b4c5ae18c0ae24c666a44ac9892765e753397f
2017-06-16 16:22:32 -07:00
Andrew Gallagher
03bdaff954 codemod: format TARGETS with buildifier [4/5] (D5092623)
Reviewed By: igorsugak

fbshipit-source-id: 277a9d2bdc1d7e3ff3075bfe2d7307502fd0a507
2017-06-01 17:52:40 -07:00
Michael Bolin
57f5d72a27 Reimplement dirstate used by Eden's Hg extension as a subclass of Hg's dirstate.
Summary:
This is a major change to Eden's Hg extension.

Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.

In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.

However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.

Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).

In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)

That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.

In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.

One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).

In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!

I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.

For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.

Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)

The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.

Reviewed By: simpkins

Differential Revision: D5071778

fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
2017-05-26 12:05:29 -07:00
Wez Furlong
b06681a4da remove use of folly::window
Summary:
We're using `window` to constrain resources and maintain an upper
bound on the number of futures that we spawn.  Ironically, for a largish number
of already completed futures, this results in a recursive chain of `spawn`
calls in the `window` implementation and results in a stack overflow (the kind
that crashes a program, not the kind that answers programming questions).

This diff removes the use of `window` and replaces it with `collect`.
An earlier iteration of this diff just made all of these calls serially,
blocking the main thread.  This at least does things in parallel; both
can torture the system for pathological walks, but at least the parallel
one should complete faster.

Reviewed By: simpkins

Differential Revision: D4932774

fbshipit-source-id: 40d39a85029f38ff69a530070efb879a81950451
2017-05-18 12:55:17 -07:00
Eddie Elizondo
dd1cc7ca9b Codemod enums to add explicit values [2/6]
Summary: This modifies thrift files to make enum values explicit

Reviewed By: yfeldblum

Differential Revision: D5014605

fbshipit-source-id: 6822928b6a6bb8e34e89b7bbb2621c66cd670cf0
2017-05-08 13:55:13 -07:00
Adam Simpkins
a9b2c41328 update the hg extension to support multiple parents
Summary:
Update the eden thrift APIs to support multiple thrift parents, and update the
hg extension to use the new thrift APIs.

Reviewed By: bolinfest

Differential Revision: D4949139

fbshipit-source-id: 98a7bd17ccad7be6a429df34ecfaf3fe7ae0b39a
2017-04-27 17:37:03 -07:00
Adam Simpkins
f5e924af94 support storing two parent commits
Summary:
This updates the ClientConfig and EdenMount code to support storing two parent
commits.

This changes the on-disk SNAPSHOT file contents add an 8-byte header that
includes a file identifier and a file format version number, followed by up to
two commit hashes.  The code currently can read either the old or new format
from the SNAPSHOT file.  We should be able to drop the code for reading the old
format fairly soon if we want, though.

This diff only updates the ClientConfig and EdenMount code, and does not yet
update the thrift APIs or the eden mercurial extension yet.  I will update the
rest of the code in a subsequent diff.

Reviewed By: bolinfest, wez

Differential Revision: D4943917

fbshipit-source-id: cf456e67b845aa0cf8b45c822985cb932df107b4
2017-04-27 11:50:13 -07:00
Adam Simpkins
87cbfe142b update the in-memory snapshot correctly after a commit
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout().  The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.

This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().

Reviewed By: bolinfest

Differential Revision: D4935943

fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
2017-04-24 18:06:59 -07:00
Adam Simpkins
67a756de93 avoid multiple builds of the eden server library
Summary:
This tweaks the main.cpp and EdenServer.cpp code so that we can avoid having
separate internal and open source builds of the eden/fs/service:server library.

Only one small function is different between the two builds.  This updates the
code so that this function is only used from main.cpp, instead of being used
inside EdenServer.cpp.

Reviewed By: wez

Differential Revision: D4889721

fbshipit-source-id: 97abda92c36752516fb8a8e73b4cf50580db8d79
2017-04-14 11:39:02 -07:00
Adam Simpkins
a6ae3edab9 move eden/utils and eden/fuse into eden/fs
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.

Reviewed By: bolinfest, wez

Differential Revision: D4889053

fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
2017-04-14 11:39:02 -07:00
Alexey Spiridonov
8d14740d20 Replace Subprocess::pipe* syntax sugar with Subprocess::Options().pipe*
Summary:
This is a bit too magical -- it's not clear that the thing produces an Options object. If you do know that you can chain further option setters off this thing, it's nice, but otherwise, the first impression is "what just happened?".

So, let's have one good way for doing things.

Reviewed By: yfeldblum

Differential Revision: D4863947

fbshipit-source-id: 3dfe83cfc077d47f604f47dcb21149fbaa2d2243
2017-04-12 14:50:22 -07:00
Adam Simpkins
8fcaefe6b2 implement hg reset
Summary:
Update the hg extension to implement dirstate.rebuild().  This is necessary for
the `hg reset` command.  This also now implements dirstate.setparents() for
cases when there is only one parent.

Reviewed By: wez

Differential Revision: D4823780

fbshipit-source-id: 802de006e03860995095dc3af17acb2eb05f4e8b
2017-04-06 17:50:43 -07:00
Adam Simpkins
114361df88 update the checkOutRevision() API to accept a BinaryHash
Summary:
Update the checkOutRevision() thrift API to accept the commit ID as a
BinaryHash rather than as a hexadecimal ASCII string, to make it more
consistent with our other thrift APIs.

Reviewed By: wez

Differential Revision: D4823779

fbshipit-source-id: 6226f926533ef86d60f8b1d659bc5d912e2855e6
2017-04-06 17:50:43 -07:00
Wez Furlong
fd073f1961 use .eden/socket in hg extensions
Summary:
This avoids hard coding knowledge of the FB-specific `local` dir.
This is only suitable for use on already-mounted eden instances,
so the function accepts either the mounted path or the eden
configuration dir.

Reviewed By: bolinfest

Differential Revision: D4829166

fbshipit-source-id: df9daed6024cdec6222be90f5cb624dd9ca0da0b
2017-04-06 10:52:06 -07:00
Christopher Dykes
de4bb63e49 Codemod folly::make_unique to std::make_unique
Summary:
Folly is dropping support for GCC 4.8 now that the branch for last HHVM release that needs to support it has been cut.
This codemod's away all the trivially identifiable uses of `folly::make_unique` in fbcode outside of folly, mcrouter and hphp.
A future diff will kill it in those, and a diff after that will kill it entirely.

Reviewed By: djwatson

Differential Revision: D4498157

fbshipit-source-id: 5a2c05ffd29e5358298acb53dab189a699b022cc
2017-04-05 12:51:32 -07:00
Adam Simpkins
6d4d187deb an "eden debug" CLI command and thrift APIs to support it
Summary:
This diff adds a new "eden debug" command, with a few subcommands:

- "eden debug inode": Report information about currently loaded inodes
- "eden debug tree": Report data about source control Tree objects
- "eden debug blobmeta": Report metadata about source control Blob objects
- "eden debug blob": Report the contents of source control Blob objects

This diff also includes the thrift APIs in edenfs to support these commands.

Reviewed By: bolinfest

Differential Revision: D4760528

fbshipit-source-id: 7dc2bd6e0e952ba6fb61702c672fb9417d603ffa
2017-04-03 15:50:32 -07:00
Adam Simpkins
5eb692d345 update EdenServer::getMount() to throw on error
Summary:
Previously EdenServer::getMount() returned a null pointer if the path did not
refer to a known mount point.  However, most of the thrift APIs using this
method never checked if the return value was null, which could allow any thrift
client to crash edenfs.

This updates getMount() to throw if the path is not found, and never return
null.

Reviewed By: bolinfest

Differential Revision: D4760527

fbshipit-source-id: 6cd9a1e3bc0a0a220d7f5fa518d497d37f245eff
2017-04-03 15:50:32 -07:00
Wez Furlong
66ec00f294 connect the fuse stats to fb303
Summary:
Tweaks the stats stuff so that we can name the histograms and export them with p50, p90, p99 percentiles.

This is made a bit ugly because the fb303 stats code isn't yet open source, so
there's a moderately ugly preprocessor directive that we assume to be true for
our internal build.   We'll need to force that to be 0 for the open source build
until we open the stats code and make it available.

Reviewed By: bolinfest

Differential Revision: D4801868

fbshipit-source-id: 643909e63bd4a74b2cfa580be131f65c5673bc94
2017-03-31 11:39:48 -07:00
Adam Simpkins
6cced7c449 implement "hg status" using EdenMount::diff()
Summary:
This is an initial pass at implementing "hg status" using the new EdenMount and
TreeInode diff() code.

More work still needs to be done to clean things up here, but this makes
"hg status" more correct in the face of modified files and directories that are
not modified, and vice-versa.

In particular, the following still needs to be cleaned up in future diffs:
- Update the "hg status" code to more correctly process user directives for
  paths that did not appear in the diff output.  This will probably be
  simplified some by correctly updating the user directive list on checkout and
  resetCommit() operations.  It may also make things simpler if we store the
  user directives in a hierarchical path map, rather than in a flat list.
- Update the addAll() code to also use the new diff logic, rather than the
  getStatusForExistingDirectory() function.
- Clean up the locking some.  I think it may be worth changing the code to use
  a single lock to manage both the current commit ID for the mount and the
  dirstate user directives.  We probably should hold this lock for the duration
  of a diff operation.  The diff operation should also return the commit ID
  that it is based against.

Reviewed By: wez

Differential Revision: D4752281

fbshipit-source-id: 6a6d7e2815321b09c75e95aa7460b0da41931dc0
2017-03-30 21:35:00 -07:00
Wez Furlong
4235784907 add .eden "magic" dir
Summary:
It's not really magic because we don't have a virtual directory
inode base any more.  Instead, we mkdir and populate it at mount time.

What is slightly magical about it is that we give it some special powers:

* We know the inode number of the eden dir and prevent unlink operations
  on it or inside it.
* The .eden dir is present in the contents of the root inode and will
  show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
  magic `.eden` inode number.  This means that it is possible to `stat`
  and consume the `.eden` directory from any directory inside the eden
  mount, even though it won't show up in `readdir` for those child dirs.

The contents of the `.eden` dir are:

* `socket` - a symlink back to the unix domain socket that our thrift
  server is listening on.  This means that it is a simple
  `readlink(".eden/socket")` operation to discover both whether a directory
  is part of an eden mount and how to talk to the server.

* `root` - a symlink back to the root of this eden mount.  This allows
  using `readlink(".eden/root")` as a simple 1-step operation to find
  the root of an eden mount, and avoids needing to walk up directory
  by directory as is the common pattern for locating `.hg` or `.git`
  dirs.

Reviewed By: simpkins

Differential Revision: D4637285

fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
2017-03-24 23:07:42 -07:00
Wez Furlong
5ca8bcdc60 implement eden->watchman subscriptions
Summary:
This tweaks the definition of the subscribe method in the
streamingeden thrift file and implements the server side of the
thrift service, and the client side to consume it in watchman.

The definition is now a bit simpler than it was previously; we're
now just sending a stream of the updated JournalPosition objects
to the client rather than computing and sending FileDelta objects.

This is cheaper for Eden to deal with because it is very cheap to take
the current journal position and pass that over the wire.  This is
important because a burst of mutations will likely queue up a bunch
of these in quick succession.

In a future diff I'm considering adding a latency parameter for
the subscription so that we can constrain the number of updates
over a certain time period (likely in the 10's of milliseconds range).

For now I just want to prove that the concept works.

On the watchman side we just need to pull these off the wire as they are sent
by eden, then wait for the subscription stream to settle before internally
broadcasting to any connected subscribers.

Reviewed By: bolinfest

Differential Revision: D4647259

fbshipit-source-id: 03aa16e59a43195a2505a8d03bce1ccf88a8d42f
2017-03-21 13:35:20 -07:00
Adam Simpkins
b48fad3547 make sure edenfs does not exit successfully if privhelper crashes
Summary:
Update the edenfs main function to check the exit code of the privhelper
process, and to exit with EX_SOFTWARE rather than EX_OK if the privhelper did
not exit with status 0.

This will ensure that our integration tests always catch crashes in the
privhelper process.  Previously we could catch errors that happened before the
privhelper socket was closed, but if the privhelper server crashed during
cleanup we never noticed the error.

Reviewed By: bolinfest

Differential Revision: D4728006

fbshipit-source-id: 2c478612e5fabec90438b28b9c76a3343f561655
2017-03-21 13:05:57 -07:00
Adam Simpkins
a4d599c4c1 add one more checkout unit test
Summary:
Add a unit test that modifies a file and then does a forced checkout to revert
back to the original commit state.

This also includes some pretty-printers to make the tests print
CheckoutConflict objects nicely if any test assertions fail.

Reviewed By: bolinfest

Differential Revision: D4727779

fbshipit-source-id: b31dc90be28d17ed0f2a4076c96a08ba82588190
2017-03-21 12:53:26 -07:00
Adam Simpkins
c8ea0d5ba6 implement removing directories during checkout
Summary:
Update the TreeInode and CheckoutAction to correctly be able to remove
directories during checkout operations.  This handles both removing directories
that are newly empty as a result of the checkout, and directories that are
replaced with a file or symlink.

This also refactors the rmdir/unlink code a bit so we can share some of its
logic for doing directory removal during the checkout operation.

Reviewed By: bolinfest

Differential Revision: D4708607

fbshipit-source-id: 3b9dd9cc3536845dad0d9278e3c5d3ac1ed04570
2017-03-17 17:13:20 -07:00
Adam Simpkins
d1e1e4c621 refactor FileInode to avoid using its parent's TreeInode::Entry
Summary:
Refactor FileInode and FileData so that they no longer store the
TreeInode::Entry that refers to this file.  The TreeInode::Entry is owned by
the parent TreeInode and is not safe to access without holding the TreeInode
contents_ lock, which the FileInode code never did.  Additionally, once a
FileInode is unlocked the TreeInode::Entry is destroyed.  Previously the
FileInode code would retain its pointer to this now invalid data, and could
still dereference it.

In the future I think it might be worth refactoring the FileInode/FileData
split a bit more, but this at least addresses the issues around invalid
accesses to entry_.

Reviewed By: bolinfest

Differential Revision: D4688058

fbshipit-source-id: 17d5d0c4e756afc6e3c4bb279cb1bb5231f3134f
2017-03-10 18:29:29 -08:00
Igor Sugak
ab1943d4fc use gflags namespace instead of google
Summary:
`google` namespace is deprecated in gflags. Replacing it with `gflags` namespace.

gflags was generated from this diff: P57170122
```
% echo $gflags
google::(RegisterFlagValidator|CommandLineFlagInfo|GetAllFlags|ShowUsageWithFlags|ShowUsageWithFlagsRestrict|\
DescribeOneFlag|SetArgv|GetArgvs|GetArgv|GetArgv0|GetArgvSum|ProgramInvocationName|ProgramInvocationShortName|\
ProgramUsage|VersionString|GetCommandLineOption|GetCommandLineFlagInfo|GetCommandLineFlagInfoOrDie|]
FlagSettingMode|SET_FLAGS_VALUE|SET_FLAG_IF_DEFAULT|SET_FLAGS_DEFAULT|SetCommandLineOption|\
SetCommandLineOptionWithMode|FlagSaver|CommandlineFlagsIntoString|ReadFlagsFromString|AppendFlagsIntoFile|\
ReadFromFlagsFile|BoolFromEnv|Int32FromEnv|Uint32FromEnv|Int64FromEnv|Uint64FromEnv|DoubleFromEnv|\
StringFromEnv|SetUsageMessage|SetVersionString|ParseCommandLineNonHelpFlags|HandleCommandLineHelpFlags|\
AllowCommandLineReparsing|ReparseCommandLineNonHelpFlags|ShutDownCommandLineFlags|FlagRegisterer)

% hg grep -wlE "$gflags" 're:fbcode.*\.(cc|cpp|h)' | xargs perl -pi -e 's,\bgoogle::,gflags::,g if /'"$gflags"'/'
```

Reviewed By: meyering

Differential Revision: D4669201

fbshipit-source-id: 8053ba6fba9acf6eaf6796f0f297a9e07784973f
2017-03-08 22:14:27 -08:00
Adam Simpkins
097bc477de get the open source build working
Summary:
Update the DEFS and .buckconfig files so that the C++ code can be built
successfully with buck in our github repository.

Reviewed By: wez

Differential Revision: D4633749

fbshipit-source-id: a07a6e664471237fcc2d13a8be8b92efa3a101bf
2017-03-06 20:28:48 -08:00
Wez Furlong
2fc9fc155c enable StreamingEdenService
Summary:
In order to send realtime subscription information to watchman
we need to use the thrift streaming data functionality.  Since this
isn't supported in java we originally broke this out into its own
thrift file.

It's now time to hook this up; this diff adds some plumbing so that
our thrift server knows to implement this interface.

This was made a bit more complicated because the inheritance in
the thrift definitions causes some ambiguities in the header maps
that buck uses to manage the includes.

I also had to rename the file from `eden.stream.thrift` to `streamingeden.thrift`
to avoid a problem with the period in the middle of the identifier.

For the server side of things, I've removed the `:thrift-cpp2` target
and replaced it with `thrift-streaming` instead.

This diff doesn't implement the endpoint, it is just setting things
up for the follow-on diff that does.

Reviewed By: bolinfest

Differential Revision: D4640639

fbshipit-source-id: b3625b0ff33a42945bf523944beed050f549a18c
2017-03-01 23:12:50 -08:00
Wez Furlong
82612ba4d6 remove getMaterializedEntries thrift API
Summary:
I originally added this to facilitate `hg status` but we didn't end up
using it.  Rather than keeping it around and updating the tests in the `.eden`
dir diffs that follow, let's just remove it.

Reviewed By: simpkins

Differential Revision: D4610768

fbshipit-source-id: 158c0207f88980e86aeeddf75e6fd49763d2a402
2017-03-01 08:19:29 -08:00
Adam Simpkins
84bd24cf96 add a getCurrentSnapshot() thrift call
Summary:
Add a thrift call to get the ID of the snapshot currently checked out in a
given eden mount.

Reviewed By: bolinfest

Differential Revision: D4570729

fbshipit-source-id: 208ceb309aacfb07e30ddb8fdfc0950a3d7d748b
2017-02-22 12:21:48 -08:00
Adam Simpkins
0687431924 implement EdenMount::checkout()
Summary:
This is the initial code for implementing checkout.

This isn't quite 100% implemented yet, but I think it's worth checking in this
code as-is, and getting the remaining functionality done in separate diffs.
In particular, a few operations aren't implemented:
- Removing a directory that was deleted in the new revision
- Replacing a directory that was replaced with a file or symlink in the new
  revision
- When doing a forced update, replacing a file or directory that did not exist
  in the old revision, but that was created locally in the working directory,
  and also exists in the new revision.

Reviewed By: wez

Differential Revision: D4538516

fbshipit-source-id: 5bb4889b02f23ab2048fcae2c8b7614340181aa6
2017-02-15 20:33:31 -08:00
Adam Simpkins
ff7b05831f don't hang the privhelper forever waiting for bind unmounts
Summary:
Update PrivHelperServer::bindUnmount() to give up after 2 seconds instead of
possibly looping forever.

Also fix the unmount code in the eden process to rethrow the original exception
as-is.  Previously it was re-throwing it as a std::exception, losing the
exception details.

I think there are still other bugs here in this unmount code--it repeatedly
hangs for me on unmount.  (I believe it is trying to unmount things that aren't
actually bind mounts.)  However, I'll look into fixing those in a separate
diff.

Reviewed By: wez

Differential Revision: D4547711

fbshipit-source-id: af6bce0262e8314361450874ef998ebf7ad590e3
2017-02-15 16:56:29 -08:00
Adam Simpkins
fed342da30 store overlay files using inode numbers instead of paths
Summary:
Refactor the Overlay code to store data using inode numbers rather than the
affected file's path in the repository.  This simplifies the TreeInode code a
bit, as we no longer have to rename overlay files to stay in sync with the file
paths.  This also eliminates some crashes when trying to update overlay files
for inodes that have been unlinked (and hence no longer have a path).  This
also includes a few fixes to avoid writing journal entries for unlinked files
too.  Additionally this contains a few fixes to how mode bits are stored in the
overlay, and fixes a bug where create() was ignoring the mode argument.

Reviewed By: wez

Differential Revision: D4517578

fbshipit-source-id: c1e31497dcf62c322b0deff72b0a02675b0509ab
2017-02-10 14:17:52 -08:00
Adam Simpkins
2c16e33026 do not hold tree locks recursively in getMaterializedEntriesRecursive()
Summary:
Update the code to release the parent TreeInode's contents lock when processing
its children.  This makes sure we do not hold the root lock for the entire
duration of this function.

Reviewed By: wez

Differential Revision: D4471289

fbshipit-source-id: dc4b50bd06fcceb7a7df3d7dd04cfc0d41285556
2017-01-27 16:47:57 -08:00
Wez Furlong
932ef52a55 implement glob thrift method
Summary:
This is to facilitate the watchman integration and draws on the
watchman glob implementation; the approach is to split the glob strings into
path components and evaluate the components step by step as the tree is walked.
Components that do not include any glob special characters can be handled as
a direct lookup from the directory contents (O(1) rather than O(num-entries)).

The glob method returns a set of filenames that match a list of
of glob patterns.

Recursive globs are supported.  It is worth noting that a glob like "**/*" will
return a list of every entry in the filesystem.  This is potentially expensive
and should be avoided.  simpkins is in favor of disallowing this as a forcing
function to encourage tool-makers to adopt patterns that don't rely on a
complete listing of the filesystem.

For now I'd like to get this in without such a restriction; it's also worth
noting that running `find .` in the root of the mount point has a similar
effect and we can't prevent that from happening, so the effect of the overly
broad glob is something that we need to be able to withstand in any case.

Unrestricted recursive globs will make it easier to connect certain watchman
queries in the interim, until we have a more expressive thrift API for walking
and filtering the list of files.

Note: I've removed the wildmatch flags that I'd put in the API when I stubbed
it out originally.  Since this is built on top of our GlobMatcher code and that
doesn't have those flags, I thought it would be simplest to just remove them.
If we find that we need them, we can figure out how to add them later.

Also Note: the evaluation of the glob is parallel-ready but currently limited
to 1 at a time by constraining the folly::window call to 1.  We could make this
larger but would need a more intelligent constraint.  For example, a recursive
glob could initiate N concurrent futures per level where N is the number of
sub-dirs at a given level.  Using a custom Executor for these futures may be a
better option to set an upper bound on the number of concurrent jobs allowed
for a given glob call.

Depends on D4361197

Reviewed By: simpkins

Differential Revision: D4371934

fbshipit-source-id: 444735600bc16d2c2185f2277ddc5b51f672600a
2017-01-26 12:47:05 -08:00
Adam Simpkins
4a6aaa5de4 load all materialized inodes when starting a mount point
Summary:
As part of the mount process, make sure we load InodeBase objects for all files
in the mount that are already materialized.

Other parts of the code assume that InodeBase objects are always loaded for
materialized files.  We never unload materialized inodes, but if the mount
point was unmounted then remounted we previously did not ensure to load the
materialized InodeBase objects.  This diff makes sure we load all materialized
inodes before starting the mount.

Reviewed By: bolinfest

Differential Revision: D4461193

fbshipit-source-id: 70d06fd01e2df333ce2816d5d7a392b0a5d6e1e6
2017-01-25 16:56:12 -08:00
Adam Simpkins
d1556c3601 implement Future-based recursive Inode lookup
Summary:
Rename the current EdenMount::getInodeBase() function to
EdenMount::getInodeBaseBlocking() and add a new getInodeBase() function that
performs the lookup asynchronously and returns a Future<InodePtr>.

This is implemented with a TreeInode::getChildRecursive() function that allows
asynchronous recursive lookups at any point in the tree.

Reviewed By: bolinfest

Differential Revision: D4427855

fbshipit-source-id: aca55e681a48c848b085b7fc6a13efe6cf0a4e3a
2017-01-25 16:56:12 -08:00
Wez Furlong
4829dfe319 change systemConfigDir to etcEdenDir
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures.  These were caused by the integration tests picking up the
default post-clone hook configuration.

This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.

Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.

The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.

Reviewed By: bolinfest

Differential Revision: D4446321

fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
2017-01-23 23:54:38 -08:00
Wez Furlong
220a2d4e1e clarify logging around running post-clone when it doesn't exist
Summary:
It's been bothering me that we claim to run and complete running
the globally installed hooks when they aren't there, so tidy up the logging.

Reviewed By: bolinfest

Differential Revision: D4446318

fbshipit-source-id: acda223e1df2302602ad9967b1aa1b334b8477c7
2017-01-23 23:54:38 -08:00
Wez Furlong
2e4610d35d cut ClientConfig over to InterpolatedPropertyTree
Summary:
This adopts the new InterpolatedPropertyTree class as
the configuration storage.

It doesn't introduce any actual configuration of the interpolation
replacements; that will be in a follow-on diff.

Reviewed By: bolinfest

Differential Revision: D4444157

fbshipit-source-id: 18ead8e9074d23b1154e81f012f0c90efced1350
2017-01-23 23:54:38 -08:00
Adam Simpkins
e31889d295 fix a variety of lint issues
Summary:
Fix issues flagged by running "arc lint" on all eden files.  There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.

Reviewed By: bolinfest

Differential Revision: D4446615

fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
2017-01-23 18:03:09 -08:00
Adam Simpkins
251da81f36 update all copyright statements to "2016-present"
Summary:
Update copyright statements to "2016-present".  This makes our updated lint
rules happy and complies with the recommended license header statement.

Reviewed By: wez, bolinfest

Differential Revision: D4433594

fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
2017-01-20 22:03:02 -08:00
Adam Simpkins
a3af576c5a parallelize getSHA1()
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.

The inode lookup itself still happens serially at the moment.  We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.

Reviewed By: wez

Differential Revision: D4361091

fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
2017-01-17 15:03:20 -08:00