Commit Graph

82 Commits

Author SHA1 Message Date
Wez Furlong
4829dfe319 change systemConfigDir to etcEdenDir
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures.  These were caused by the integration tests picking up the
default post-clone hook configuration.

This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.

Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.

The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.

Reviewed By: bolinfest

Differential Revision: D4446321

fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
2017-01-23 23:54:38 -08:00
Wez Furlong
220a2d4e1e clarify logging around running post-clone when it doesn't exist
Summary:
It's been bothering me that we claim to run and complete running
the globally installed hooks when they aren't there, so tidy up the logging.

Reviewed By: bolinfest

Differential Revision: D4446318

fbshipit-source-id: acda223e1df2302602ad9967b1aa1b334b8477c7
2017-01-23 23:54:38 -08:00
Wez Furlong
2e4610d35d cut ClientConfig over to InterpolatedPropertyTree
Summary:
This adopts the new InterpolatedPropertyTree class as
the configuration storage.

It doesn't introduce any actual configuration of the interpolation
replacements; that will be in a follow-on diff.

Reviewed By: bolinfest

Differential Revision: D4444157

fbshipit-source-id: 18ead8e9074d23b1154e81f012f0c90efced1350
2017-01-23 23:54:38 -08:00
Adam Simpkins
e31889d295 fix a variety of lint issues
Summary:
Fix issues flagged by running "arc lint" on all eden files.  There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.

Reviewed By: bolinfest

Differential Revision: D4446615

fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
2017-01-23 18:03:09 -08:00
Adam Simpkins
251da81f36 update all copyright statements to "2016-present"
Summary:
Update copyright statements to "2016-present".  This makes our updated lint
rules happy and complies with the recommended license header statement.

Reviewed By: wez, bolinfest

Differential Revision: D4433594

fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
2017-01-20 22:03:02 -08:00
Adam Simpkins
a3af576c5a parallelize getSHA1()
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.

The inode lookup itself still happens serially at the moment.  We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.

Reviewed By: wez

Differential Revision: D4361091

fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
2017-01-17 15:03:20 -08:00
Adam Simpkins
0237a0c56d add an EdenMount::destroy() function
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function.  This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.

This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.

Reviewed By: wez

Differential Revision: D4360558

fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
2017-01-17 15:03:20 -08:00
Andrew Gallagher
26c2526080 Fixup more rule names in TARGETS
Summary: Manually fixup rule names which the codemod scripting didn't catch.

Reviewed By: igorsugak

Differential Revision: D4420657

fbshipit-source-id: 20794b17732ca5c37686bd22edc1538837098022
2017-01-16 11:40:07 -08:00
Christopher Dykes
138bc12add Codemod folly::make_unique to std::make_unique
Summary: Folly will soon be dropping support for GCC 4.8, which means that `folly::make_unique` will be going away, so codemod away as much as possible before then.

Reviewed By: yfeldblum

Differential Revision: D4411221

fbshipit-source-id: 31d61425f6595d0750ea2e4c95c7d42bb5a5d955
2017-01-13 14:22:04 -08:00
Adam Simpkins
5b346bbff2 update code to use InodeMap
Summary:
This updates all of the eden code to use the new InodeMap class.  This replaces
the InodeNameManager class and the unordered_map previously stored in the
EdenDispatcher.

Reviewed By: bolinfest

Differential Revision: D4325750

fbshipit-source-id: d80ae7581ba79ca2b63155e184995a3e83e85dc1
2016-12-22 15:36:29 -08:00
Michael Bolin
3f2f22d8fa Normalize build targets to always start with //.
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.

This revision was initially created by running:

```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```

And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.

But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.

Reviewed By: simpkins

Differential Revision: D4356724

fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
2016-12-21 16:28:02 -08:00
Michael Bolin
1ea4af26c5 Use the StatusCode from Thrift as the canonical representation.
Summary:
Most of the work for this diff was achieved via:

```
find eden -type f | xargs sed -i -e 's#ThriftHgStatusCode#StatusCode#g'
find eden -type f | xargs sed -i -e 's#HgStatusCode#StatusCode#g'
```

Reviewed By: simpkins

Differential Revision: D4237975

fbshipit-source-id: 2ee04a89101291c8972ac7bd3ff6cca92cbb0799
2016-12-16 17:49:05 -08:00
Michael Bolin
dfd1634170 Move batch processing of hg add <directory> to the server.
Summary:
This should make the common case of `hg add` or `hg add .` much more efficient
because it no longer performs a walk of the entire repository from the client
side.

Reviewed By: simpkins

Differential Revision: D4317871

fbshipit-source-id: 7061553fe0de0c4afa84b4f835316965088675e8
2016-12-15 13:02:38 -08:00
Adam Simpkins
970159aa5c normalize path arguments
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.

This updates the main program to process path arguments using canonicalPath().

Reviewed By: bolinfest

Differential Revision: D4332273

fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
2016-12-15 13:02:38 -08:00
Adam Simpkins
4cb1fa6379 use EdenMount::getFileInode() getSHA1ForPath()
Summary:
Update EdenServiceHandler::getSHA1ForPath() to replace its own custom path
lookup code with EdenMount::getFileInode().

This also ended up fixing the error message to correctly include the path name
on EISDIR errors.

Reviewed By: wez

Differential Revision: D4325066

fbshipit-source-id: 9aa3932c71c33e6bc11d2c71cc8f1badb4c0dcb7
2016-12-14 15:36:11 -08:00
Adam Simpkins
fc202f81e5 add new Future-based APIs to ObjectStore
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.

For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready.  I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.

Reviewed By: bolinfest

Differential Revision: D4318055

fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
2016-12-13 18:12:21 -08:00
Adam Simpkins
89fb0f811b add InodePtr, TreeInodePtr, and FileInodePtr type names
Summary:
Define InodePtr, TreeInodePtr, and FileInodePtr as aliases for std::shared_ptr
of the underlying inode type.  This also updates all of the code to use these
new type names.

This will make it easier swap out std::shared_ptr with a custom pointer type in
the future.  (I believe we will need a custom type in the future so that we
can have more precise control of the reference counting so we can load and
unload Inode objects on demand.  std::shared_ptr::unique() doesn't quite
provide the flexibility we need, and is also being deprecated in C++17.)

Reviewed By: bolinfest

Differential Revision: D4297791

fbshipit-source-id: 1080945649290e676f62689592159f1166159b20
2016-12-12 17:50:35 -08:00
Michael Bolin
309d0da769 Make it possible to make a commit from Eden.
Summary:
In this revision, we override `committablectx.markcommitted()` to make a Thrift
call to Eden to record the new commit. For now, this defers the need for us to
implement `edendirstate.normal()`, though we expect we will have to provide a
proper implementation at some point.

Because `hg update` is not implemented yet, this puts us in a funny state where
we have to restart eden after `hg commit` to ensure all of our `TreeEntry` and
other in-memory data structures are in the correct state.

Reviewed By: simpkins

Differential Revision: D4249214

fbshipit-source-id: 8ec06dfee67070f008dd93a0ee6c810ce75d2faa
2016-12-10 01:07:06 -08:00
Michael Bolin
017b35bca6 Add support for hg remove for an individual file.
Summary:
This adds support for `hg remove <file>` in Eden and sets up some of the scaffolding
to support `hg remove <directory>`.

Note that the Thrift API for `scmRemove()` is slightly different than that of `scmAdd()`
in that it returns a list of error messages to display to the user rather than throwing
an exception. In practice, for batch operations, Mercurial will allow some operations
to succeed while others may fail, so it is possible to have multiple error messages to
return.

Unlike the current implementation of `hg add`, this does the directory traversal
on the server rather than on the client. Once we work out how to do this for
`hg remove`, we should figure out how to reuse the logic for `hg add`.

Reviewed By: simpkins

Differential Revision: D4263068

fbshipit-source-id: d084774d562c48c59664f313eba229d4197929fe
2016-12-09 19:36:17 -08:00
Adam Simpkins
da04640287 move InodeBase from eden/fuse to eden/fs/inodes
Summary:
Move the InodeBase class from the lower-level fusell code up to the
eden/fs/inodes layer, now that everything else that uses it is in
eden/fs/inodes.

I plan to start changing the ownership model of inode objects a bit, and this
will allow the InodeBase class to interact with EdenDispatcher and other
classes in eden/fs/inodes.

Reviewed By: bolinfest

Differential Revision: D4283392

fbshipit-source-id: 9e1d6fb81dc223f905847cbe8d165a40ad0aca4d
2016-12-07 20:05:20 -08:00
Adam Simpkins
f9a99e0ba1 remove fusell::DirInode and fusell::FileInode
Summary:
Now that the EdenDispatcher class has been moved into eden/fs, we no longer
need the distinction between TreeInode and fusell::DirInode, and FileInode and
fusell::FileInode.

This diff deletes the fusell versions of these classes, and updates all of the
code to always directly use TreeInode and FileInode.  This allows us to get rid
of the remaining dynamic_casts between these pairs of classes.

Reviewed By: bolinfest

Differential Revision: D4257165

fbshipit-source-id: e2b6f328b9605ca0e2882f5cf7a3983fb4470cdf
2016-12-01 17:52:31 -08:00
Adam Simpkins
2a08798f88 move InodeDispatcher from eden/fuse to eden/fs
Summary:
Move the InodeDispatcher class out of the lower-level fusell namespace in
eden/fuse and into the higher-level eden code in eden/fs/inodes.  I also
renamed it from InodeDispatcher to EdenDispatcher, in anticipation of it
getting more eden-specific functionality in the future.

The fusell::MountPoint class is now independent of the Dispatcher type, and can
work with any Dispatcher subclass.  Previously the MountPoint class was
responsible for owning the InodeDispatcher object.  Now its caller (EdenMount
in our case) is responsible for supplying a Dispatcher object that is owned
externally.

Several parts of EdenDispatcher had to be updated as a result of the namespace
move, but I tried to keep this change somewhat minimal.  I did update it from
using fusell::DirInode and fusell::FileInode to eden's TreeInode and FileInode
classes directly.  However, there still remains more clean-up work to do.  I
will split remaining changes out into upcoming diffs.

Reviewed By: bolinfest

Differential Revision: D4257163

fbshipit-source-id: dc9c2526640798f9f924ae2531218ba2c45d1d0a
2016-12-01 17:52:31 -08:00
Adam Simpkins
0af5d187c2 move MountPoint::getInode* methods into EdenMount
Summary:
Move getInodeBaseForPath(), getDirInodeForPath(), and getFileInodeForPath()
entirely into the EdenMount class, and make sure all call sites are using the
EdenMount methods rather than the MountPoint methods.

Reviewed By: bolinfest

Differential Revision: D4257153

fbshipit-source-id: d528cfad174757c3c9f23e62a0616f8bf1976da7
2016-12-01 17:52:31 -08:00
Adam Simpkins
f0082e9178 call EdenMount::getDispatcher() and EdenMount::getRootInode()
Summary:
Update all code in eden/fs to call EdenMount::getDispatcher() instead of
getting the underlying MountPoint from the EdenMount and then calling
getDispatcher() on it.  This will allow me to move the InodeDispatcher from
MountPoint to EdenMount in a subsequent diff.  This also simplifies many of the
callers of this method.

Additionally, add an EdenMount::getRootInode() method, and update call sites to
use this rather than having to look up the InodeDispatcher and call
getRootInode() or getDirInode(FUSE_ROOT_ID) on it.

Reviewed By: bolinfest

Differential Revision: D4257152

fbshipit-source-id: 33e6f6b8853db2a88f4f2c221122eea50e796390
2016-12-01 17:52:31 -08:00
Adam Simpkins
aaa3332644 simplify EdenMount and Dirstate construction
Summary:
This cleans up construction of the EdenMount and Dirstate objects:

- The EdenMount constructor is now responsible for creating the Overlay and
  Dirstate objects.
- The Dirstate constructor is now responsible for loading the
  DirstatePersistence file.
- The EdenMount now takes ownership of the ClientConfig object, and stores it
  for later use.
- The ClientConfig object now has a method to get the path to the
  DirstatePersistence file.
- I added a ClientConfig::createTestConfig() method, so that the TestMount code
  can now use the same EdenMount constructor as the normal code.

This simplifies the logic in EdenServiceHandler and TestMount, and makes some
of the initialization dependencies a little bit simpler.

This change is necessary in order for me to move some logic from
fusell::MountPoint into EdenMount.  The Dirstate object will need a pointer
back to its EdenMount object, and this diff enables that.

Reviewed By: bolinfest

Differential Revision: D4249393

fbshipit-source-id: 439786accbf48c8696dbc6ca4fe77a4c6bdeab65
2016-12-01 17:52:30 -08:00
Adam Simpkins
0e613bd96b rename TreeEntryFileInode to FileInode
Summary:
Rename TreeEntryFileInode to FileInode, and TreeEntryFileHandle to FileHandle.
These class names were long and awkward.

It's slightly unfortunate that we now have classes named both
eden::fuse::FileInode and eden::fuse::fusell::FileInode, but I don't believe
this should cause any major problems.  If we want to eliminate these name
collisions in the future I would advocate for renaming the fusell versions to
something like "FileInodeIface".

Reviewed By: bolinfest

Differential Revision: D4217909

fbshipit-source-id: 899672a318d7ae39595f2c18e171f8fd6cebedc6
2016-11-30 15:49:13 -08:00
Michael Bolin
9156794f06 Pass ClientConfig as a raw pointer rather than transferring ownership.
Summary:
This is a better fix for the quick fix introduced by D4198939.
It turns out that the `EdenMount` does not need to take ownership
of the `ClientConfig`, so removing the `std::move()` makes this code
much simpler because instead of declaring a bunch of variables
early in `mountImpl()` so that we can "hold on" to them before `EdenMount`
takes ownership of the `ClientConfig`, we can declare them closer to where they
are actually used.

Note that we may want `EdenMount` to actually take ownership of the
`ClientConfig` in the future, but we'll cross that bridge when we come to it.

Reviewed By: simpkins

Differential Revision: D4199000

fbshipit-source-id: 67411a9a5ef630a9d481aebc94631c79da4ab2c4
2016-11-26 12:01:41 -08:00
Michael Bolin
b078392a9c Add Thrift endpoints for Hg dirstate.
Summary:
This also introduces the change where the `EdenMount` creates
and takes ownership of the `Dirstate`.

To clean some of this up, I had to expose a `getEdenDir()` method on `EdenServer`
that returns an `AbsolutePathPiece`. This was previously stored internally as a
`std::string`, so I had to clean up a bunch of path construction that was using `edenDir_`.

Reviewed By: simpkins

Differential Revision: D4123763

fbshipit-source-id: 270b182521c1a84bb054832f4b5f92af849d67e4
2016-11-26 12:01:41 -08:00
Michael Bolin
0f834ea809 Flip Dirstate -> EdenMount dependency.
Summary:
Previously, `Dirstate` took a `std::shared_ptr<EdenMount>`, but now it takes
pointers to a `MountPoint` and an `ObjectStore` because it does not need the
entire `EdenMount`. Ultimately, this will enable us to have `EdenMount` create
the `Dirstate` itself, but that will be done in a follow-up commit.

Fortunately, it was pretty easy to remove the references to `edenMount_` in
`Dirstate.cpp` and rewrite them in terms of `mountPoint_` or `objectStore_`.
The one thing that I also decided to move was `getModifiedDirectoriesForMount()`
because I already needed to create an `EdenMounts` file (admittedly not a
great name) to collect some utility functions that use members of an `EdenMount`
while not having access to the `EdenMount` itself.

As part of this change, all of the code in `eden/fs/model/hg` has been moved to
`eden/fs/inodes` so that it is alongside `EdenMount`. We are going to change
the `Dirstate` from an Hg-specific concept to a more general concept.

`LocalDirstatePersistence` is no longer one of two implementations of
`DirstatePersistence`. (The other was `FakeDirstatePersistence`.) Now there is
just one concrete implementation called `DirstatePersistence` that takes its
implementation from `LocalDirstatePersistence`. Because there is no longer a
`FakeDirstatePersistence`, `TestMount` must create a `DirstatePersistence` that
uses a `TemporaryFile`.

Because `TestMount` now takes responsibility for creating the `Dirstate`, it
must also give callers the ability to specify the user directives. To that end,
`TestMountBuilder` got an `addUserDirectives()` method while `TestMount` got a
`getDirstate()` method. Surprisingly, `TestMountTest` did not need to be updated
as part of this revision, but `DirstateTest` needed quite a few updates
(which were generally mechanical).

Reviewed By: simpkins

Differential Revision: D4230154

fbshipit-source-id: 9b8cb52b45ef5d75bc8f5e62a58fcd1cddc32bfa
2016-11-26 12:01:41 -08:00
Wez Furlong
96e19f1fe5 fix test_bad_mount_path
Summary:
D4014598 changed this line but didn't change the test expectations.
Since it seems desirable for the mount point name to be used, I've reverted
back to the prior state for this line.

Reviewed By: bolinfest

Differential Revision: D4202265

fbshipit-source-id: bddc01436e0a5921a3b0b2c01c0fd2c32f5f1960
2016-11-17 18:44:58 -08:00
Michael Bolin
61ff7492db Fix a bug where generate-hooks-dir was creating the wrong structure.
Summary:
Originally, D3858635 was going to introduce a scheme for hooks where the
repo type was included in the path:

    /etc/eden/hooks/hg/post-clone <args...>

But over the course of the review, we decided to make the repo type a
parameter:

    /etc/eden/hooks/post-clone hg <other-args...>

Unfortunately, `generate-hooks-dir` was not updated as part of that
change and it is not covered by unit tests. This error was particularly hard
to discover because of how `ENOENT` is handled, so I added a log statement for
that.

Reviewed By: simpkins

Differential Revision: D4200277

fbshipit-source-id: ffffd871cd78dcaeb717be8f1e01893ce9643a47
2016-11-17 14:39:06 -08:00
Michael Bolin
9f42f9a5f0 Fix a bug where an AbsolutePathPiece was getting collected.
Summary:
This is a quick and dirty fix for this issue that was causing
and confusing bug where the memory for the `AbsolutePathPiece`
was getting reclaimed, so when it was later read as the value
for a path, it failed because it was binary garbage.

This is mainly caused by the `std::move(config)` that passes
the `ClientConfig` to the `EdenMount` constructor. I will do
some more general cleanup for that in a follow-up revision,
but I wanted to have this change in its own commit that makes
it clear where the failure/fix were coming from.

Reviewed By: simpkins

Differential Revision: D4198939

fbshipit-source-id: 19e0423a1bee924fa6cc2edc8bae534ef472c988
2016-11-17 13:22:06 -08:00
Andrii Korotkov
8f94770d20 Rename thrift methods from NWorker/NPoolThreads to NumIOWorker/NumCPUWorkerThreads for all remaining directories
Summary:
Use new, less confusing names for mentioned thrift methods.

Codemod with 'Yes to All'. Reverted changes in thrift/

Reviewed By: yfeldblum

Differential Revision: D4076812

fbshipit-source-id: 4962ec0aead1f6a45efc1ac7fc2778f39c72e1d0
2016-11-04 15:31:04 -07:00
Michael Bolin
a23cb5d8a2 Introduce getModifiedDirectoriesForMount().
Summary:
Performs a depth-first traversal of the overlay to find modified
directories and returns them in that order.

Reviewed By: simpkins

Differential Revision: D4025309

fbshipit-source-id: 09d8ed41b250dddbfb3fe545643ec3fd755a430e
2016-10-21 13:32:02 -07:00
Michael Bolin
83996c4630 Move some logic out of EdenServiceHandler so it can be reused by the dirstate and test harness.
Summary:
Now that I've done all this work, I'm not sure whether it is a good idea or even
necessary. I'll keep it in my back pocket.

Reviewed By: simpkins

Differential Revision: D4014598

fbshipit-source-id: 6ded3cc29838e964b56833ac24dff19e9de040f5
2016-10-21 13:32:02 -07:00
Michael Bolin
6c8512bbad Extract logic into MountPoint.getInodeBaseForPath() method.
Summary: This should make some of the upcoming test harness work a little easier.

Reviewed By: simpkins

Differential Revision: D4011747

fbshipit-source-id: 87ee80a6d641a29be9027b163b1adee496f4452f
2016-10-18 12:19:32 -07:00
Adam Simpkins
5b2e9cc7dc don't crash if getMaterializedEntries() is called with a bad mount point
Summary:
The getMaterializedEntries() would previously try to dereference a null pointer
if the input mount path did not refer to a valid mount piont.

Reviewed By: bolinfest, wez

Differential Revision: D3942600

fbshipit-source-id: 2a8c9aa87d2bd8175f7bc77f3d6293ad25e9c198
2016-10-04 11:04:18 -07:00
Adam Simpkins
17cf69d3b2 add some helper functions for constructing EdenErrors
Summary:
Add some helper functions for constructing EdenError objects from a few
different types of arguments.  Also update eden.thrift to indicate that most
functions can throw EdenErrors on failure.

Reviewed By: bolinfest, wez

Differential Revision: D3942588

fbshipit-source-id: 1b561c5310a8a218f88c38c70499e087fe47bbe0
2016-10-04 11:04:17 -07:00
Adam Simpkins
859a4c265b update the python client library to be python 2.x compatible
Summary:
Python 2.x requires the current class name be passed into super().
Add arguments to super so that we can use this inside a mercurial extension.
(Mercurial only supports python 2.x.)

Reviewed By: bolinfest

Differential Revision: D3942573

fbshipit-source-id: 06df55f217631a398004c0d25448d3a612f772e9
2016-09-30 19:13:13 -07:00
Michael Bolin
634e96872e Add initial support for hooks akin to Git hooks for Eden.
Summary:
This design is inspired by that of Git hooks:
https://git-scm.com/docs/githooks

By default, `/etc/eden/hooks` should be the place where Eden looks for
hooks; however, this can be overridden in `~/.edenrc` on a per-`repository` basis.
This directory should be installed as part of installing Eden.
There is information in `eden/hooks/README.md` about this.

The first hook that is supported is for post-clone logic for a repository.

This change demonstrates the need for an `eden config --get <value>`
analogous to what Git has, as hooks should be able to leverage this in their
own scripts. There introduces a `TODO` in `post-clone.py` where such a
feature would be useful, so that I could add the following to my `~/.edenrc`
to develop the Eden extension for Hg:

```
[hooks]
hg.edenextension = /data/users/mbolin/fbsource/fbcode/eden/hg/eden

[repository fbsource]
path = /data/users/mbolin/fbsource
type = hg
hooks = /data/users/mbolin/eden-hooks
```

Note that this revision also introduces a `generate-hooks-dir` script that can be
used to generate the standard `/etc/eden/hooks` directory that we intend to
distribute with Eden. This is also useful in creating the basis for a custom `hooks`
directory that can be specified as shown above in an `~/.edenrc` file.

Reviewed By: simpkins

Differential Revision: D3858635

fbshipit-source-id: 215ca26379a4b3b0a07d50845fd645b4d9ccf0f2
2016-09-26 13:53:05 -07:00
Wez Furlong
734fbf0c59 fix build in @mode/opt
Summary: CI didn't catch this, annoying!

Reviewed By: simpkins

fbshipit-source-id: 8217cfdc75365a5c1a5d2962792805d35b31d1b9
2016-09-26 13:52:25 -07:00
Wez Furlong
aef0c5b279 implement getFileInformation
Summary: fairly straight forward.

Reviewed By: simpkins

Differential Revision: D3872989

fbshipit-source-id: def2dfc624a6aa08ad089f19bd3a8438e26f0bbd
2016-09-26 13:52:25 -07:00
Wez Furlong
8f4373571b implement getFilesChangedSince
Summary:
This is pretty straightforward; we just walk back until we hit the
boundary with the requested JournalPosition.sequenceNumber

Reviewed By: simpkins

Differential Revision: D3872970

fbshipit-source-id: 1405f05957346d7ac513070f0407a477548aff1d
2016-09-26 13:52:25 -07:00
Wez Furlong
82c57b2bf8 implement getCurrentJournalPosition thrift API
Summary:
populate the position from the latest journal delta.

To facilitate this, we also define the mountGeneration value to be a
combination of the pid and the time at which we created the EdenMount object,
as well as a global counter that we bump for each mount.

The precise value and meaning of this bits really doesn't matter, just that we
are unlikely to pick the same value for this same mountPoint path again if we
were to remount in the future.

Since we are now in a position to report JournalPosition values to clients, now
is also a good time to fill out the `currentPosition` field for the
`getMaterializedEntries` thrift call, and to check that this value is
consistent with the value we return via `getCurrentJournalPosition`.

Reviewed By: simpkins

Differential Revision: D3872952

fbshipit-source-id: 2fbc25d2e9711035b66ab1bf5d746507b72de265
2016-09-26 13:52:25 -07:00
Wez Furlong
8b41b90108 sample the snapshot id in the journal at mount time
Summary:
This just populates the initial snapshot hash in the journal.

The `addDelta` method will propagate this into subsequent deltas if the delta
to be added has hash values that have not been set from the default 0-filled
hash values.

Reviewed By: simpkins

Differential Revision: D3872936

fbshipit-source-id: d0014ded40488a2be04d5a381e1d9815c7f0a638
2016-09-26 13:52:25 -07:00
Wez Furlong
07df3d8fbc additional query API for our thrift interface
Summary:
This diff adds a couple more things to our thrift interface:

1. Introduces JournalPosition
2. Adds methods to query the current JournalPosition and obtain a
   delta since a given JournalPosition
3. Augments getMaterializedFiles to also return the current JournalPosition
4. Adds a method to evaluate a `glob` against Eden
5. Adds a method using thrift streaming to subscribe to realtime changes

Could probably finesse the naming a little bit.

The JournalPosition allows reasoning about changes to files that are not part
of an Eden snapshot.  Internally the journal position is just the
SequenceNumber from the journal datastructures, but when we expose it to
clients we need to be able to distinguish between a sequence number from the
current instance of the eden service and a prior incarnation (eg: if the
process has been restarted, and we have no way to recreate the journal we need
to be able to indicate this to the client if they ask about changes in that
range).   For the convenience of the client we also include the `toHash` (the
most recent hash from the journal entry) which is likely useful for the `hg`
dirstate operations; it is useful to know that the snapshot may have changed
since the last query about the dirstate.

The `getFileInformation` method returns the instantaneously available `stat()`
like information about the requested list of files.   Since we simply don't
have historical data on how files in the overlay looked (only how they look
now), this method does not allow passing in a JournalPosition.  When it comes
to comparing historical data, we will need to add an API that accepts two
snapshot hashes and generates the results from there.  This particular method
is geared up to understanding the current state of the world; the obvious use
case is plugging in the file list from `getFilesChangedSince` into this
function to figure out what's what.

* Do we want a function that combines `getFilesChangedSince` + `getFileInformation` into a single RPC?

Why is there a glob method?  It's to support a use-case in the watchman/buck
integration.  I'm just sketching it out in the thrift interface at this stage.
In the future we also need to be able to express how to carry out a tree walk,
but that will require some query predicates that I don't want to get hung up on
specifying immediately.

Why is the streaming stuff in its own thrift file?  We can't generate code for
it in java or perhaps also python.  It's only needed to plumb data into
watchman so it's broken out into its own definition.  Nothing depends on that
file yet, so it's probably not specified quite right.  The important thing is
how the subscribe method looks: it's essentially the same as the method to
query a delta, but it keeps emitting deltas as they are produced.  This is
another API that will benefit from query predicates when we get around to
specifying them.

I've added `JournalDelta::fromHash` and `JournalDelta::toHash` to hold the
appropriate snapshot ids in the journal entry; this will allow us to indicate
when we've checked out a new snapshot, or created a new snapshot.  We have
no way to populate these yet; I commented on D3762646 about storing the
`snapshotID` that we have during `EdenServiceHandler::mountImpl` into either
the `EdenMount` or the proposed `RootInode` class.  Once we have that we
can simply sample it and store it as we generate `JournalDelta`s.

Reviewed By: simpkins

Differential Revision: D3860804

fbshipit-source-id: 896c24c354e6f58328fb45c24b16915d9e937108
2016-09-26 13:52:25 -07:00
Wez Furlong
e54df2e422 add getMaterializedEntries thrift call
Summary:
Adds a thrift call that returns the list of materialized entries from the whole tree.

This is intended to be plugged into the mercurial dirstate extension.

Reviewed By: simpkins

Differential Revision: D3851805

fbshipit-source-id: 8429fdb4eeccc32928e8abc154d4e6fd49343556
2016-09-26 13:52:24 -07:00
Kirill Sazonov
af13d51a22 Fix fbcode projects that depend on fbthrift
Summary:
Previous diff in a stack moves following buck targets:
`//thrift/lib/java/src:thrift` -> `//thrift/lib/java:thrift`
`//thrift/lib/java/src/com/facebook/thrift/direct_server:DirectServer` -> `//thrift/lib/java:DirectServer`
`//thrift/lib/java/src/com/facebook/thrift/direct_server:TDirectServer` -> `//thrift/lib/java:TDirectServer`

This diff fixes fbcode projects to reflect that change via following script:
  find . -name TARGETS -type f -exec sed -i "s#'//thrift/lib/java/src:thrift'#'//thrift/lib/java:thrift'#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#'@/thrift/lib/java/src:thrift'#'@/thrift/lib/java:thrift'#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#\"//thrift/lib/java/src:thrift\"#\"//thrift/lib/java:thrift\"#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#\"@/thrift/lib/java/src:thrift\"#\"@/thrift/lib/java:thrift\"#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#'//thrift/lib/java/src/com/facebook/thrift/direct_server:DirectServer'#'//thrift/lib/java:DirectServer'#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#'@/thrift/lib/java/src/com/facebook/thrift/direct_server:DirectServer'#'@/thrift/lib/java:DirectServer'#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#\"/thrift/lib/java/src/com/facebook/thrift/direct_server:DirectServer\"#\"//thrift/lib/java:DirectServer\"#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#\"@/thrift/lib/java/src/com/facebook/thrift/direct_server:DirectServer\"#\"@/thrift/lib/java:DirectServer\"#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#'//thrift/lib/java/src/com/facebook/thrift/direct_server:TDirectServer'#'//thrift/lib/java:TDirectServer'#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#'@/thrift/lib/java/src/com/facebook/thrift/direct_server:TDirectServer'#'@/thrift/lib/java:TDirectServer'#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#\"/thrift/lib/java/src/com/facebook/thrift/direct_server:TDirectServer\"#\"//thrift/lib/java:TDirectServer\"#g" {} \;
  find . -name TARGETS -type f -exec sed -i "s#\"@/thrift/lib/java/src/com/facebook/thrift/direct_server:TDirectServer\"#\"@/thrift/lib/java:TDirectServer\"#g" {} \;

After that, following files are fixed manually after found via BigGrep:
  - .buckconfig
  - adsatlas/rex/pom.xml
  - facer/engine-java-client/.classpath
  - facer/vizi_java_utils/.classpath
  - java-foundations/scripts/tasks-csv/tasks.2016.java8.csv
  - mobile/device_categorization/service/copy-libs.sh
  - nettools/collection/job_manager/java/src/com/facebook/nettools/collection/job_manager/CachedModel.java
  - nettools/collection/job_manager/java/src/com/facebook/nettools/collection/job_manager/Util.java
  - stampede/example/buckconfig
  - titan/.hgsparse-fbsource
  - tools/build/buck/macro_lib/convert/thrift_library.py

Reviewed By: ryandm

Differential Revision: D3873877

fbshipit-source-id: fb008c13f75d663016b15c6fcfcd563033e7fbb8
2016-09-26 13:52:24 -07:00
Michael Bolin
7a05213f34 New Thrift endpoint: getBindMounts(mountPoint).
Summary:
Buck needs this API so that it knows which paths under a project
root it should exclude when deciding whether it can ask Eden for its
SHA-1 or if it must compute it on its own.

Reviewed By: simpkins

Differential Revision: D3840658

fbshipit-source-id: 5eddc0bef423d3b3ee165d2a4b0bbf193f94f61a
2016-09-12 18:29:15 -07:00
Wez Furlong
9c79b74456 eden: re-do overlay serialization
Summary:
we now serialize the overlay data for each directory independently.

When we mount, we try to load the root overlay data.  The children are lazy
loaded as the inodes are instantiated.

Structural changes cause the overlay data for the impacted dirs to get saved out.

I need to make a pass over this to fixup comments and so on, I just wanted to get this diff out first.

I moved the overlay stuff from `eden/fs/overlay` -> `eden/fs/inodes` since most
of the overlay-ness is handled in `TreeInode` now; the `Overlay` class is
really just for carrying around the paths and providing the serialization
helpers.

Reviewed By: simpkins

Differential Revision: D3787108

fbshipit-source-id: f0e089a829defd953535b9d0a96b102ac729261b
2016-09-09 16:57:58 -07:00