Commit Graph

244 Commits

Author SHA1 Message Date
Wez Furlong
9bb3db9ee4 stop journal subscribers prior to shutting down thrift
Summary:
This avoids the long delay and the `Cpp2Worker.cpp:281] Failed to join
outstanding requests.` message from showing up in the eden logs during shutdown
and graceful restart.

Reviewed By: chadaustin

Differential Revision: D6693117

fbshipit-source-id: 3c56314f288a596264cddae0d8bbab66ab19e9fe
2018-01-10 13:13:50 -08:00
Wez Furlong
530346887d split EdenMount::{create,initialize} -> safer takeover mounting
Summary:
Previously, `EdenMount::create` would implicitly call
`EdenMount::initialize` which would load the root inode and the `.eden` magical
directory.  That's fine for the fresh mount case, but in the case of the
graceful restart we need to take the opportunity to apply the `InodeMap`
from the old process before we start muddying its state.

This diff breaks out the `initialize` method from the `create` method and
makes the mount code call it after potentially loading the `InodeMap` from
the takeover data.

In addition, this diff removes the the `root->loadMaterializedChildren()`
call from the mount initialization code.  It is no longer required to do
this eagerly and it makes things simpler and our memory profile a little
smaller to defer this (I haven't measured how much that impacts things).

Reviewed By: simpkins

Differential Revision: D6691182

fbshipit-source-id: 52033a6d64105b658314a919f69dbfcd4eea242b
2018-01-10 13:13:50 -08:00
Wez Furlong
7daecf237d refactor mount/takeover code into a single flow
Summary:
This removes the duplicated logic and makes it a little
bit easier to follow the initialization sequence.

It doesn't change the behavior, just moves some code around.

Reviewed By: simpkins

Differential Revision: D6691180

fbshipit-source-id: 2068dbe56ebe9a6136d69689997aec8dedd32be0
2018-01-10 13:13:50 -08:00
Wez Furlong
c82616c564 refactor takeoverAll() -> unmountAll()
Summary:
This is the "simple" threading through of a `doTakeover`
flag and the return of `Optional<TakeoverData>` in the unmount code.

Reviewed By: simpkins

Differential Revision: D6691181

fbshipit-source-id: 4a384787783c16085f2e9964964023ba07cefca3
2018-01-09 22:23:12 -08:00
Wez Furlong
014789b4ca open file handles now survive graceful restart
Summary:
I'm so-so on a bit of the implementation here, but it works!

I had to change the `takeoverPromise` from the `pair<fuseDevice, connInfo>`
to a new helper struct because we now have three distinct pieces of data
to pass out of EdenMount to build up the overall TakeoverData.

The key change in this diff is that we have to release all of the file handles
we're maintaining in the `FileHandleMap` prior to shutting down the `InodeMap`,
otherwise the `InodeMap` will never complete (it waits for all inodes to be
unreferenced, and that cannot happen while there are open file handles).  I've
made the `FileHandleMap` serialization and clearing contingent on performing a
takeover shutdown because that feels like the safest thing to do wrt. not
losing any pending writes.

Reviewed By: simpkins

Differential Revision: D6672437

fbshipit-source-id: 7b1f0f8e7ff09dbed850c7737383ecdf1e5ff0c7
2018-01-09 22:23:11 -08:00
Wez Furlong
490e6adcf5 inform PrivHelper of takeover; graceful restarts now work!
Summary:
This is the key portion that makes the graceful restart
function.  This diff connects almost all of the moving pieces together;
it informs the priv helper about the takeover mount and transfers
the InodeMap information into the new generation of the eden server.

It doesn't yet load the fileHandleMap (will tackle that in a follow up diff)

Reviewed By: simpkins

Differential Revision: D6670903

fbshipit-source-id: 1770d99eb1477440a6c1deed83b0da55b9c1bbe4
2018-01-09 22:23:10 -08:00
Wez Furlong
ef214c6c4f serialize the InodeMap
Summary:
this isn't how we really want to do this long term, it's
just the most expedient short term implementation.

This diff provides an implementation of the `InodeMap::save()` which
was previously a stub method; the new implementation returns a thrift
structure that encompasses the unloaded inodes in the map, and adds
a corresponding load() method that performs the reverse transformation.

The struct is serialized into the Takeover data.

This diff doesn't hook up the real serialized data to EdenServer; that will happen
in a follow-on diff.

The way that we actually want to handle this longer term is to store the
`numFuseReferences` field into the overlay file on disk, but to do so we
will need to add a mountGeneration field alongside it and ensure that we
always write out the correct information at the correct times.  In addition,
we'd need to add equivalent data to TreeInode::Entry and add accessors that
safely return the correct values in the correct situations.

In the interest of getting something working, I've just dumped this code in
here.

I've also placed the thrift structure for this in `fuse/handlemap.thrift`;
this is a slight layering violation but one that feels "OK" in light of
the imminent refactor of the Takeover data struct to be its own thrift
struct anyway.

Reviewed By: simpkins

Differential Revision: D6670904

fbshipit-source-id: 11a0918954c741935c587e46fcb0e38849010de1
2018-01-09 22:23:10 -08:00
Wez Furlong
65b2d3c4b1 serialize FileHandleMap into TakeoverData
Summary:
This puts the data into the takeover information during takeover
shutdown, but doesn't do anything to pull it out again (that will be in a follow on diff).

The serialization stuff could be done a little bit more efficiently (since we
will perform an extra thrift serialization step just to compute the length, and
repeat it again later), but we're planning on replacing this with thrift
serializing soon, once simpkins diff stack lands, so I'm not losing sleep over
it.

Reviewed By: simpkins

Differential Revision: D6668846

fbshipit-source-id: e6d01428bd506a9e93b427db499770fce0a0983a
2018-01-09 22:23:08 -08:00
Wez Furlong
1c2a4c3f93 add client configuration path to TakeoverData
Summary:
This fulfils a TODO but doesn't do anything useful
with that data at this time.

Reviewed By: simpkins

Differential Revision: D6552750

fbshipit-source-id: 0c441fd0c2ab43785b4d98c4ca6ff643a20629e0
2018-01-09 22:23:07 -08:00
Wez Furlong
8689b37221 connect the fuse device information to the takeover data
Summary:
This adds some plumbing to thread the fuse device descriptor and
negotiated capabilities through to the takeover code.

I initially wanted to just make the
unmount future yield the device descriptor, but since that uses
`SharedPromise` it is not compatible with a move-only type like
`folly:File`, so I added an optional promise to deal with just that.

I'm also populating the takeover mount information (path, bind mounts)
for each mount point.

Reviewed By: simpkins

Differential Revision: D6494509

fbshipit-source-id: a90684292dc1d8e06ce2c0721eadd8d393377f33
2018-01-09 22:23:04 -08:00
Chad Austin
b0fac29b0b enable python typechecking from buck
Summary: Turn on check_types for CLI's python_binary.

Reviewed By: simpkins

Differential Revision: D6668636

fbshipit-source-id: abde1e1cedf2a5104cdaa5433377b1d2adc372fd
2018-01-08 12:10:19 -08:00
Wez Furlong
6ff492d11c remove dep on libfuse
Summary:
This serves a few purposes:

1. We can avoid some conditional code inside eden if we know that
   we have a specific fuse_kernel.h header implementation.
2. We don't have to figure out a way to propagate the kernel
   capabilities through the graceful restart process.
3. libfuse3 removed the channel/session hooks that we've been
   using thus far to interject ourselves for mounting and
   graceful restarting, so we were already effectively the
   walking dead here.
4. We're now able to take advtange of the latest aspects of
   the fuse kernel interface without being tied to the implementation
   of libfuse2 or libfuse3.  We're interested in the readdirplus
   functionality and will look at enabling that in a future diff.

This may make some things slightly harder for the more immediate
macOS port but I belive that we're in a much better place overall.

This diff is relatively mechanical and sadly is (unavoidably) large.

The main aspects of this diff are:

1. The `fuse_ino_t` type was provided by libfuse so we needed to
   replace it with our own definition.  This has decent penetration
   throughout the codebase.
2. The confusing `fuse_file_info` type that was multi-purpose and
   had fields that were sometimes *in* parameters and sometimes *out*
   parameters has been removed and replaced with a simpler *flags*
   parameter that corresponds to the `open(2)` flags parameter.
   The *out* portions are subsumed by existing file handle metadata
   methods.
3. The fuse parameters returned from variations of the `LOOKUP` opcode
   now return the fuse kernel type for this directly.  I suspect
   that we may need to introduce a compatibility type when we revisit
   the macOS port, but this at least makes this diff slightly simpler.
   You'll notice that some field and symbol name prefixes vary as
   a result of this.
4. Similarly for `setattr`, libfuse separated the kernel data into
   two parameters that were a little awkward to use; we're now just
   passing the kernel data through and this, IMO, makes the interface
   slightly more understandable.
5. The bulk of the code from `Dispatcher.cpp` that shimmed the
   libfuse callbacks into the C++ virtual methods has been removed
   and replaced by a `switch` statement based dispatcher in
   `FuseChannel`.   I'm not married to this being `switch` based
   and may revise this to be driven by an `unordered_map` of
   opcode -> dispatcher method defined in `FuseChannel`.  Regardless,
   `Dispatcher.cpp` is now much slimmer and should be easier to
   replace by rolling it together into `EdenDispatcher.cpp` in
   the future should we desire to do so.
6. This diff disables dispatching `poll` and `ioctl` calls.  We
   didn't make use of them and their interfaces are a bit fiddly.
7. `INTERRUPT` is also disabled here.  I will re-enable it in
   a follow-up diff where I can also revise how we track outstanding
   requests for graceful shutdown.
8. I've imported `fuse_kernel.h` from libfuse.  This is included
   under the permissive 2-clause BSD license that it allows for
   exactly this integration purpose.

Reviewed By: simpkins

Differential Revision: D6576472

fbshipit-source-id: 7cb088af5e06fe27bf22a1bed295c18c17d8006c
2018-01-02 16:36:16 -08:00
Philip Jameson
8604b8f5b0 Migrate TARGETS files from @/ to //
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros

The only thing left of the old format should be @/third-party:foo:bar

drop-conflicts

Reviewed By: ttsugriy

Differential Revision: D6605465

fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
2017-12-20 16:57:41 -08:00
Chad Austin
9a3fa8bd60 replace the system memory info in eden stats with process memory
Summary:
`eden stats` used to show system memory usage which was not very
interesting (and can be gleaned from top).  Instead read the contents
of /proc/self/smaps and sum the Private_Dirty fields to get a number
that more accurately reflects impact on the rest of the system.

Reviewed By: wez

Differential Revision: D6575595

fbshipit-source-id: 9badc5cd5a1b56d3ccb27edd1a2d20ee74ec34ae
2017-12-18 12:00:58 -08:00
Adam Simpkins
dc10a595ed attempt to resolve symlinks in the eden directory path
Summary:
During edenfs startup, use `realpath()` or `normalizeBestEffort()` to resolve
symlinks in the input configuration paths if possible.

Reviewed By: chadaustin

Differential Revision: D6527494

fbshipit-source-id: 4377099e8c65217fd128c06de77d50f4316f4fc7
2017-12-14 12:41:47 -08:00
Adam Simpkins
f42c51c6eb remove the --rocksPath command line flag
Summary:
The RocksDB location is relative to the .eden directory.  Given that the
location of the .eden directory can be controlled from the command line it
doesn't seem worthwhile to make the RocksDB location independently controllable
too.

Reviewed By: chadaustin

Differential Revision: D6527495

fbshipit-source-id: dab8c22f3f1a74de908ea33d0b20c4115c28ce31
2017-12-14 12:41:47 -08:00
Chad Austin
2f22a9f534 introduce a type representing the unbounded eden cpu pool
Summary:
Add EdenCPUThreadPool and UnboundedQueueThreadPool types to make it clearer
that it's always okay for prefetch, deferred diff entry, and hg import to
shuttle work back to the main thread pool.

This diff changes no behavior - it just makes some invariants explicit.

Reviewed By: wez, simpkins

Differential Revision: D6504117

fbshipit-source-id: 3400ad55c00b3719ecba31807fd992442f622cd9
2017-12-12 12:35:35 -08:00
Chad Austin
5eaddd7921 Make sure shutdown() errors propagate to unmount() waiters
Summary: Use SharedPromise::setFuture to make sure errors propagate.

Reviewed By: simpkins

Differential Revision: D6515818

fbshipit-source-id: dee95bccdc086a040ead08e60373cbc7daa4db3f
2017-12-11 18:16:05 -08:00
Christopher Dykes
0b74d03fbd Ensure explicit dependencies on everything < 500 references
Summary: Ensure everything remaining in dependencies of `folly:folly` that has < 500 references is explicit referenced.

Reviewed By: yfeldblum

Differential Revision: D6540137

fbshipit-source-id: 0a2ae5cf775278eedcccdb914688890acd12dab7
2017-12-11 17:51:23 -08:00
Christopher Dykes
c19afa8152 Shift folly:format out of folly:base
Summary: And then, there was 1, left all alone, preventing the whole thing from collapsing in on itself.

Reviewed By: yfeldblum

Differential Revision: D6469584

fbshipit-source-id: 4ea1fbf97ad466bc34f2e682394d328c97e539ba
2017-12-10 15:24:40 -08:00
Christopher Dykes
5056668373 Shift folly:range out of folly:base
Summary: And then there were... 2.

Reviewed By: yfeldblum

Differential Revision: D6469788

fbshipit-source-id: 8a2d2f01f2d1fdbac05922701ba60c494061c8b9
2017-12-10 11:09:38 -08:00
Christopher Dykes
522690a6d0 Shift folly:executor, folly/lang:bits, folly:scope_guard and folly:rw_spin_lock out of folly:base
Summary:
```
foundation/depenendency_management/ensure-explicit-dependencies.sh folly:executor
foundation/depenendency_management/ensure-explicit-dependencies.sh folly:bits
foundation/depenendency_management/ensure-explicit-dependencies.sh folly/lang:bits
foundation/depenendency_management/ensure-explicit-dependencies.sh folly:scope_guard
foundation/depenendency_management/ensure-explicit-dependencies.sh folly:rw_spin_lock
```

Reviewed By: yfeldblum

Differential Revision: D6529842

fbshipit-source-id: fc13ef398402a0323ce4f6cf61d12b30c196ce43
2017-12-09 22:07:31 -08:00
Christopher Dykes
561453dee9 Shift folly:file, folly:exception_wrapper and folly/synchronization:baton out of folly:folly
Summary:
```
foundation/dependency_management/ensure-explicit-deps.sh folly:file
foundation/dependency_management/ensure-explicit-deps.sh folly:exception_wrapper
foundation/dependency_management/ensure-explicit-deps.sh folly/synchronization:baton
```

Reviewed By: yfeldblum

Differential Revision: D6529590

fbshipit-source-id: a08f8bdd01eda3fb28410af55c4dec6b0c5f9399
2017-12-09 17:20:17 -08:00
Chad Austin
e19939cf15 fix crash when too many tasks are put on the eden thread pool
Summary:
Use an unbounded queue for edenfs's main thread pool.  This fixes a
crash where DeferredDiffEntry multigets a batch of trees and pushes
the completion callbacks back onto the server thread pool.  If the
server thread pool is bounded and throws when the queue is full, then
the import fails.

There is a slight performance hit relative to LifoSemMPMCQueue but
hopefully it isn't a big deal.  An unbounded lock-free queue would be
nicer.

Reviewed By: simpkins

Differential Revision: D6490979

fbshipit-source-id: bc55dd6526f0ceb9d8b5e43a1a275250a9838aca
2017-12-05 17:52:04 -08:00
Chad Austin
f91ae520b2 introduce a Clock seam for testability
Summary:
Introduce a Clock seam.  This will allow us to write tests around
ctime, mtime, and atime logic.

Reviewed By: wez

Differential Revision: D6392543

fbshipit-source-id: 1721d76d2364b135b4ef5c078ef60f7f8526259e
2017-12-05 10:06:48 -08:00
Adam Simpkins
1b86627d43 logging: update initialization code to use the new LogConfig logic
Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6342086

fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242
2017-12-01 17:07:56 -08:00
Michael Bolin
5e2afa735f Change how the UNTRACKED_ADDED conflict and merges are handled.
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.

Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:

```
if updatecheck == 'noconflict':
    for f, (m, args, msg) in actionbyfile.iteritems():
        if m not in ('g', 'k', 'e', 'r', 'pr'):
            msg = _("conflicting changes")
            hint = _("commit or update --clean to discard changes")
            raise error.Abort(msg, hint=hint)
```

However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.

To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.

To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.

One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.

Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:

```
[commands]
update.check = noconflict
```

Though the original name for this setting was:

```
[experimental]
updatecheck = noconflict
```

Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.

Reviewed By: simpkins

Differential Revision: D6366007

fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
2017-11-29 21:50:34 -08:00
Andrew Gallagher
7ad41d0ec4 codemod: add undefined_symbols to rules with improper deps
Reviewed By: yfeldblum

Differential Revision: D6427879

fbshipit-source-id: b91de83b287e999167ca32968f710f1e4fcb4ac8
2017-11-29 17:36:09 -08:00
Wez Furlong
28e74f1ba6 add scmGetStatusBetweenRevisions thrift call
Summary:
The goal is to provide a fast path for watchman to flesh
out the total set of changed files when it needs relay that information
on to consumers.

We choose not to include the full list in the Journal when checking out
between revisions because it will not always be needed and may be an
expensive `O(repo)` operation to compute.  This means that watchman
needs to expand that information for itself, and that is currently
a fairly slow query to invoke through mercurial.

Since watchman is responding to journal events from eden we know that
we have tree data for the old and new hashes and thus we should be
able to efficiently compute that diff.

This implementation is slightly awful because it will instantiate an
unlinked TreeInode object for one side of the query, and will in
turn populate any children that differ as it walks down the tree.
A follow on diff will look at making a flavor of the diff code that
can diff raw Tree objects instead.

Reviewed By: bolinfest

Differential Revision: D6305844

fbshipit-source-id: 7506c9ba1f4febebcdc283c414261810a3951588
2017-11-28 19:36:32 -08:00
Christopher Dykes
b665a1a012 Shift folly:string out of folly:base
Summary:
```
foundation/dependency_management/ensure-explicit-dependencies.sh folly:string
```

Reviewed By: yfeldblum

Differential Revision: D6411357

fbshipit-source-id: 97afac97e8c1bfb15af5a53fadfb8d6a098a0d37
2017-11-26 16:22:54 -08:00
Christopher Dykes
4408aa7c86 Shift folly:optional out of folly:base
Summary:
```
foundation/dependency_management/ensure-explicit-dependencies.sh folly:optional
```

Reviewed By: yfeldblum

Differential Revision: D6411007

fbshipit-source-id: 57dc0ab0ba21fc1a14e0132216ec35533472441f
2017-11-26 02:19:56 -08:00
Michael Bolin
736e720764 Set up arc lint to run autodeps automatically for Eden.
Reviewed By: andrewjcg

Differential Revision: D6327798

fbshipit-source-id: de98268b45291ed3aeb73e074a7d9dd8420d3e99
2017-11-21 18:23:41 -08:00
Michael Bolin
99e29ed185 Remove getParentCommits() Thrift API.
Summary: It is currently unused. Let's bring it back if/when we need it.

Reviewed By: chadaustin

Differential Revision: D6368867

fbshipit-source-id: 096015ba597a6e04f544273ba9773576429e39ce
2017-11-20 15:56:46 -08:00
Adam Simpkins
7d98d9cfc4 define constants for several files in the .eden directory
Summary:
Define constants at the top of EdenServer.cpp for the names of the main lock
file, the thrift socket, and the takeover socket.

Reviewed By: bolinfest

Differential Revision: D6295040

fbshipit-source-id: 8605840a50c84bc89b798123d1063bbb11ff2502
2017-11-20 11:35:50 -08:00
Adam Simpkins
e64baf16db add a --takeover flag to allow graceful takeover
Summary:
This begins implementing the "client-side" portion of graceful takeover in
edenfs.  When the --takeover flag is specified, if edenfs finds that another
edenfs process is already running it will attempt to gracefully take over its
mount points rather than exiting with an error.

This does not yet actually take over the mount points themselves--it still
sends dummy mount information during shutdown, and does not use this data
during startup.  However, we do perform takeover of the eden lock file and the
thrift server socket.

Reviewed By: bolinfest

Differential Revision: D6038944

fbshipit-source-id: 42406a0559367cec79af088b4ca84c22de3f3ef3
2017-11-20 11:35:49 -08:00
Adam Simpkins
649b4e3ecb send the thrift server socket during graceful takeover
Summary:
Update the TakeoverData to also include the thrift server socket.
Also update EdenServer to set this field when performing a takeover
shutdown.

Reviewed By: bolinfest

Differential Revision: D6038945

fbshipit-source-id: 725faa431b3b55d617ef645c8a7eae080e4fe066
2017-11-19 15:47:30 -08:00
Adam Simpkins
532e42cb87 update EdenServer to implement TakeoverHandler
Summary:
Update EdenServer to implement the TakeoverHandler API, and to exit after
sending the mount point takeover data.  The actual shutdown logic itself is not
implemented yet--this just sends dummy data for now.  However, this does serve
as a proof of concept that the TakeoverServer and TakeoverClient code functions
as desired.

Reviewed By: bolinfest

Differential Revision: D6018180

fbshipit-source-id: c19581928926a46b767f1ee5c1761381e5055fa9
2017-11-19 15:47:25 -08:00
Rafit Izhak-Ratzin
77421f2abc Add thirsd party slf4j-api to the TARGETS files' configuration
Summary: remove deps from thrift build target

Reviewed By: ryandm

Differential Revision: D6221258

fbshipit-source-id: 5838cf1ca9873201b02d94db410a57a46bca90da
2017-11-16 21:54:42 -08:00
Michael Bolin
019f456fab Change the contents and format for the edenrc file under ~/local/.eden.
Summary:
The headline changes of this revision are:

- Changes the format of the config file from INI to TOML
  (the `edenrc` file under `~/local/.eden` has been replaced
  with `config.toml`). This revision includes logic for automatically
  performing the migration when Eden is restarted.
- Inlines data from `/etc/eden/config.d` into the TOML file.

Historically, the `edenrc` file for a client would contain the
name of the "configuration alias" defined in a config file like
`~/.edenrc` or `/etc/eden/config.d/00-defaults`. When Eden
loaded a client, it would have to first read the `edenrc` and
then reconstitute the rest of the client configuration by
looking up the alias in the set of config files that were used to
create the client in the first place.

This changes things so that all of the data that was being
cross-referenced is now inlined in the client's config file.
This makes loading a config considerably simpler at the cost
of no longer being able to change the config for multiple clients
that were cloned from the same configuration alias in one place.
It was questionable whether being able to modify a client from
a foreign config after it was created was a safe thing to do, anyway.

Eliminating the need for a historic link to the configuration alias
will make it easier to support running `eden clone` on an arbitrary
local Hg or Git repo. So long as `eden clone` can extract enough
information from the local repo to create an appropriate config file
for the new Eden client, there is no need for a configuration alias
to exist a priori.

Since we were already changing the data in the config file, this
seemed like an appropriate time to make the switch from INI to
TOML, as this was something we wanted to do, anyway.
In testing, I discovered a discrepancy between how boost's
`boost::property_tree::ptree` and Python's `ConfigParser` handled
the following section heading:

```
[repository ZtmpZsillyZeden-clone.LIkh32]
```

Apparently `hasSection("repository ZtmpZsillyZeden-clone.LIkh32")`
in boost would fail to find this section. Because
[[https://stackoverflow.com/questions/13109506/are-hyphens-allowed-in-section-definitions-in-ini-files | there is no spec for INI]],
it is not that surprising that boost and `ConfigParser` do not 100% agree
on what they accept. Moving to TOML means we have a configuration
language with the following desirable properties:

- It has a formal spec, unlike INI. This is important because there are parsers
  in a wide range of programming languages that, in theory, accept a consistent
  input language.
- It is reasonable for humans to write, as it supports comments, unlike JSON.
- It supports nested structures, like maps and arrays, without going crazy
  on the input language it supports, unlike YAML.

Eden now depends on the following third-party TOML parsers:
* C++ https://github.com/skystrife/cpptoml
* Python https://github.com/uiri/toml

This revision also changes the organization of `~/local/.eden` slightly. For now,
there is still a `config.json` file, but the values are no longer hashes of the realpath
of the mount. Instead, we take the basename of the realpath and use that as the
name of the directory under `~/local/.eden/clients`. If there is a naming collision, we
add the first available integral suffix. Using the basename makes it easier to
navigate the `~/local/.eden/clients` directory.

Although the `edenrc` file under `~/local/.eden/clients` has been switched from INI
to TOML, the other Eden config files (`~/.edenrc` and `/etc/eden/config.d/*`) still use
INI. Migrating those to TOML will be done in a future revision.

Note this revision allowed us to eliminate `facebook::eden::InterpolatedPropertyTree`
as well as a number of uses of boost due to the elimination of
`ClientConfig::loadConfigData()` in the C++ code. Because `ClientConfig`
no longer does interpolation, a bit of `ClientConfigTest` was deleted as part of
this revision because it is no longer relevant.

Reviewed By: wez

Differential Revision: D6310325

fbshipit-source-id: 2548149c064cdf8e78a3b3ce6fe667ff70f94f84
2017-11-16 13:23:27 -08:00
Chad Austin
385a94a69f Use a pool of HgImporter workers
Summary:
Have HgBackingStore hold multiple HgImporters in its own thread pool. Incoming
requests are processed by threads in the thread pool.

Reviewed By: wez

Differential Revision: D6265043

fbshipit-source-id: b2d4f345b772f296c5335a7fbcadfce1d93245fd
2017-11-15 09:49:48 -08:00
Michael Bolin
e8d25ee211 Move the post-clone step from C++ to Python.
Summary:
Moving the post-clone step out of C++ makes it so that
`ClientConfig` in C++ no longer knows about `hooks` and requires
`RepoConfig` to know about `hooks`. This helps us reduce
`ClientConfig`'s dependency on `ConfigData`, as my next step
is to move the remaining information that `ClientConfig` gets from
`ConfigData` directly into the client's `edenrc` file under
`~/local/.eden`.

Reviewed By: chadaustin

Differential Revision: D6310544

fbshipit-source-id: dec7a21281ab49e0416b8872757970a4eff2d943
2017-11-14 14:06:25 -08:00
Chad Austin
17661a573c Grab __func__ in a constexpr StringPiece again
Summary:
This is another, more efficient way to fix the mode/opt build issues
seen in D6272580.  When C++17 is released, it could be updated to
std::size.

Reviewed By: yfeldblum

Differential Revision: D6274205

fbshipit-source-id: 07f25a1a0f6575b80b7d1a8af7b7c723f0f9e4f0
2017-11-10 23:54:04 -08:00
Chad Austin
587d0ad1ee Move Journal locking inside of Journal itself
Summary:
This is follow-up to the lock ordering issues in
StreamingSubscriber.  The Journal locks are now finer-grained and no
locks are held while the subscribers are invoked.  This change
prevents future deadlocks.

Reviewed By: wez

Differential Revision: D6281410

fbshipit-source-id: 797c164395831752f61cc15928b6d4ce4dab1b68
2017-11-10 19:57:49 -08:00
Adam Simpkins
21dd075fd9 tweak log levels for thrift calls
Summary:
Tweak the `INSTRUMENT_THRIFT_CALL()` log levels in EdenServiceHandler.

For the most part this changes the code so that modifying calls (e.g.,
`mount`, `checkOutRevision`, etc) are logged at `DBG2` and higher, while
read-only calls (`getSHA1`, `getParentCommits`) are logged at `DBG3` and
lower.  Since we log all eden messages at `DBG2` by default this means that
modifying calls will be enabled by default but read-only calls will be
disabled.

Some important read-only calls such as `getScmStatus` and
`getFilesChangedSince` do still log at the `DBG2` level.

The main motivation for this is that `buck build` often triggers many thousands
of separate `getSHA1` calls.  It doesn't seem terribly valuable to flood the
eden logs with these messages by default.  These can always be enabled at
runtime if desired when debugging buck performance.

Reviewed By: bolinfest

Differential Revision: D6295566

fbshipit-source-id: dd344c1dea773f4f2a56e2b0dbb18b8738303944
2017-11-10 11:15:27 -08:00
Chad Austin
32ba74fe46 rename FileInode::getSHA1 to FileInode::getSha1
Summary: Make this function match our C++ guidelines.

Reviewed By: wez

Differential Revision: D6288591

fbshipit-source-id: 1a4f52a8c1e0497df938533fe29da10264eb1ccf
2017-11-09 15:52:42 -08:00
Philip Jameson
db24c8b7c9 Fix the include prefix for c++ generated code
Summary: We were previously potentially deleting the include prefix. We also weren't using the cpp2 include prefix if the global one wasn't set. This makes sure we use it, and fixes a bug where 'X_types.h' was included without a full prefix.

Reviewed By: yfeldblum

Differential Revision: D6236108

fbshipit-source-id: 076747fcab2b1414bafa42c9e481ba1e1e5df4b1
2017-11-09 15:35:58 -08:00
Scott Michelson
0431efa7f8 remove all setMaxConnections calls
Summary:
This setting is bad for a few reasons

1) there is no correct value; rejecting connections is typically something you want to do when under pressure, which no fixed number of connections can indicate
2) it's not discoverable when it trips, and pretty much always confuses people
3) the effect is generally not better than the thing it is theoretically supposed to prevent, when it trips, servers crash and exhibit other weird behaviors; I don't think I've ever seen a situation where I thought this was needed, and I've seen many were it created problems where none would have existed.

About a year ago, I removed almost all calls to this; however, I left some behind since they were slightly harder to clean up or had unique flag names. Since then, 2 things have happened

1) more copypasta instances have cropped up
2) More people have run into issues with this flag, notably up2x recently

This removes all calls except one (realtime Pylon has some somewhat more complex config here, I'll need to talk to them). This is somewhat aggressive; some of the calls are totally safe to remove, as they're either copypasted or set it to 0 or some absurdly high number. Others are less obvious. I've decided to adopt a door-in-the-face strategy here and see if anyone tells me I should be more conservative

I couldn't delete all the flags; these ones are in use

zdb_thrift_max_connection
thrift2_max_connections (commerce ranker)
max_connections (a bunch of places)
maxConnections (presence)

I'll need a separate diff/set of diffs to delete those

Reviewed By: eduardo-elizondo, yfeldblum

Differential Revision: D6182563

fbshipit-source-id: e778edd9da582f4ca90a902621cb49f1e04ca26e
2017-11-09 12:18:50 -08:00
Michael Bolin
41f4127e8d Fix INSTRUMENT_THRIFT_CALL macro so it builds in opt mode.
Summary:
Building with gcc/opt mode complains that `strlen()` is not a constant
expression, so I'm dropping the `constexpr` to fix the opt build.

Reviewed By: chadaustin

Differential Revision: D6272580

fbshipit-source-id: a8e3a93ea04e878353c6cab31cb0b1b4705276fe
2017-11-08 10:49:35 -08:00
Michael Bolin
489e9071c4 Give each Thrift endpoint its own logging category.
Summary:
In Eden, some Thrift endpoints get a lot of hits whereas others are used less
frequently. By giving each endpoint its own logging category, we can toggle the
logging for each one independently.

Each of our Thrift endpoint methods should start with the following macro:

    INSTRUMENT_THRIFT_CALL();

then within the method, the macro `TLOG()` should be used everywhere
`XLOG(LEVEL)` would normally used. `LOG()` ensures the logger with the category
specific to the method will be used.

For an endpoint named `XXX`, the logging category will be `eden.thrift.XXX`.

Reviewed By: simpkins

Differential Revision: D6108311

fbshipit-source-id: 23a34179811359b0819434de31a3601d29c3b4f0
2017-11-07 20:56:12 -08:00
Adam Simpkins
c0e048148e log around opening the RocksDB local store
Summary:
Opening the RocksDB store can take a long time if there is a fair amount of
data in the store.  In my current .eden directory I have around 10GB of data in
the local store, and it takes RocksDB nearly 60 seconds to open the database.

These log messages help provide a little more information about what edenfs is
doing during this startup delay.

Reviewed By: bolinfest

Differential Revision: D6263603

fbshipit-source-id: a0945aa1cc020b95944b365b17869756dcc27407
2017-11-07 14:17:37 -08:00