Commit Graph

61 Commits

Author SHA1 Message Date
Katie Mancini
ce0287d81c get oss windows build back to functioning
Summary:
The external OSS build is broken because thrift has a really long path and
cargo/git on Windows do not lick such long paths.

I'm not fixing that. But the external OSS build is the only one we run in CI any
more. In the meantime the internal build has broken because no one has been
watching.

There were 4 different breakages.

Reviewed By: chadaustin

Differential Revision: D44189633

fbshipit-source-id: 2eedbc2b3bbf5d1def075d99f11f2273dbb1f4ab
2023-03-29 12:52:44 -07:00
Xavier Deguillard
00db4f8673 integration: remove FOLLY_INIT_LOGGING_CONFIG from helpers
Summary:
This one slipped through D44263797 and allows for integration tests to be run
with Buck2 on macOS.

Reviewed By: fanzeyi

Differential Revision: D44315942

fbshipit-source-id: d2de0773ba68f13fca9e8d5c067b82653646c757
2023-03-22 17:05:50 -07:00
Katie Mancini
5d22eba56f add and test renamed placeholder detection
Summary:
Renamed files are treated specially in EdenFS. We materialize all renamed
files, but projfs still has these as placeholders. Generally we assume
placeholder == scm data, and full == materialized inode, but
renamed files break this rule. We need to be able to specially handle renamed
placeholders in fsck.

That's comming later in this stack. This diff adds logic to peak into the
reparse buffer and detect renames. It also comes with a full integration test
that confirms it properly detects names.

The original path is stored in the reparse buffer as well, we could use this
to detect renamed, but there seems to be a bit that indicates renames.
so it seems simpler to just use this instead.

Reviewed By: xavierd

Differential Revision: D42268211

fbshipit-source-id: 4bca09601c88297f5b33573ebe2ccad060863dc8
2023-02-06 16:55:36 -08:00
Katie Mancini
f1bef5aa06 assert our assumptions about the reparse buffer and file attrs
Summary:
Our current FSCK implementation does not properly fsck renamed files, and to
do that we are going to need to peek inside the reparse buffer (there doesn't
seem to be anything else that indicates renamed files :( )

The format of the reparse buffer (the projFS bit) is not externally documented.
Microsoft folks have hinted at it's format on github issues:
https://github.com/microsoft/ProjFS-Managed-API/issues/55
And there are some bits with in the buffer that seem to have some clear meanings.
If we are going to rely on introspecting this buffer we should make sure we
become aware if the format changes from what we are expecting.

We also wanted to maybe limit how often we need to peek into the reparse buffer
based on file attributes. However, it does not look like that will be very
possible. To make this clear I am asserting theattributes along with the bit of
the reparse buffer we care about.

Additionally, FSCK already assumes things about what file attributes mean, and
these are kinda implementation details of ProjFS. We are exposed to those
breaking, so we should probably have tests asserting our assumptions anyways.

Reviewed By: xavierd

Differential Revision: D42766825

fbshipit-source-id: d2c022b1379e7bd57af1cd473a794607698115eb
2023-02-06 16:55:36 -08:00
Katie Mancini
633e4cb6b2 expose startup status on thrift server
Summary:
EdenFS is often started in the background. Because of this users can not easily
see the startup status. The only place it is avaiable other than the eden start
command is the logs. Those are hard to find and not really intended to be easy
to read.

Because users don't see EdenFS making progress while it is starting, users
reachout for help in the support group (leading to the oncall having to read out
the logs to them), or they will try to self service and kill EdenFS which land
themselves in a worse state, and lead to start up taking even longer due to
unclean startup or EdenFS corruption (proxy hashes).

In this diff I am exposing a way to subscribe to the status of EdenFS's startup
process. I also introduce the `--verbose` flag on the `status` command that
displays this startup status when EdenFS is running.

Reviewed By: xavierd

Differential Revision: D38962809

fbshipit-source-id: 2a8f03218df33e4fbfd4d0be919d48d9523c85b8
2023-01-27 15:48:56 -08:00
Xavier Deguillard
b7d87d807a utils: deny building an AbsolutePath from a literal
Summary:
We've had a couple of failures after landing D40818724 (f199e93924) due to some places still
building AbsolutePath from a literal. On Windows, this is almost always a bug
due to paths usually not being UNC.

To avoid this issue, we can simply forbid an AbsolutePath from being built in
the first place from a path, instead forcing the use of canonicalPath. This was
suggeted by mshroyer in the orignal diff.

Reviewed By: chadaustin

Differential Revision: D41229194

fbshipit-source-id: 3451bdcba276c87f98326b025e05f6a4eddbd1b7
2022-11-22 16:20:30 -08:00
Chad Austin
a1bc268963 port Path and PathPiece from folly::StringPiece to std::string_view
Summary:
std::string_view has noexcept accessors and folly::Range doesn't, so
this allows us to make Path and PathPiece noexcept.

Reviewed By: kmancini

Differential Revision: D41145426

fbshipit-source-id: 046f6f6a532d8d0da8508ccf7896c914e19a25ec
2022-11-16 18:09:51 -08:00
Katie Mancini
4b80db25dc migrate integration helpers to folly portability
Summary:
lint complained, and it does indeed look like we need to update our includes here.

folly portability avoids strange windows compile issues

Reviewed By: chadaustin

Differential Revision: D38867569

fbshipit-source-id: 91e4530f57151d5fd0cc60f6dae98c8f94577930
2022-08-19 12:38:22 -07:00
Katie Mancini
1821a44656 Allow Delayed opening of RocksDb
Summary:
Opening rocks db can be slow, and prevents the eden thrift server from starting today.

lets split up creating the rocksdb localstore and using it, so that we can get the thrift server started before opening rocksdb.

Reviewed By: xavierd

Differential Revision: D38839390

fbshipit-source-id: e86d0dd5eb456445e3b83ffedfd1283c8caba827
2022-08-19 10:45:59 -07:00
Mark Shroyer
f4b2712cef Fix buck2 run edenfsctl
Summary:
buck2 changes the locations of binary outputs.  We currently hard-code some
assumptions about the relative locations of edenfsctl, edenfs, and
edenfs_privhelper that no longer hold with buck2.

This diff:
- Makes edenfsctl look for edenfs at a location optionally given by the
  EDENFS_SERVER_PATH environment variable.
- Makes edenfs look for edenfs_privhelper at a location optionally given by the
  EDENFS_PRIVHELPER_PATH environment variable.
- Updates edenfsctl run targets to set these variables.

Reviewed By: kmancini

Differential Revision: D38473942

fbshipit-source-id: 4af0f2eca5a88ad0f263968af397a82bd2cf9fd4
2022-08-16 11:38:49 -07:00
Xavier Deguillard
cb0d447098 service: goodbye systemd
Summary:
Our systemd code is unused, and is causing a bunch of tests to fail. In the
case where systemd is needed again in the future, its shape will be
significantly different than what we have today, so there even less need to
keep this version around.

Reviewed By: fanzeyi

Differential Revision: D35829902

fbshipit-source-id: 6013f5fe7c3debdcfbb9929d2748819a96403fe7
2022-04-28 09:18:22 -07:00
Chad Austin
11cf537c45 allow building some integration test scaffolding on macOS
Summary:
Some of EdenFS's integration test functionality can build on macOS, so
implement the required tweaks.

Reviewed By: kmancini

Differential Revision: D34291419

fbshipit-source-id: 296f077f4a3311d7fc8f04ed705fd06019e34212
2022-02-25 13:32:21 -08:00
Xavier Deguillard
a29d465ee8 fs: fix license header
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.

Reviewed By: fanzeyi

Differential Revision: D33407812

fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
2022-01-04 15:00:07 -08:00
Katie Mancini
43fc599083 teach Takeover Server/Client how to pass NFS info
Summary:
Now that we have NFS mounts we need to support takeover for those mounts.
This commit teaches the takeover server and client to serialize and deserialize
NFS information.

NFS information includes:
- the mountd socket. Mountd is used to register mount points, and should keep
the same socket across graceful restarts.
- a connected socket for each of the NFS mount points.
- All the rest of the mount point information that we send for fuse mounts (with
the exception of the fuse negotiated parameters.

Future commits will teach the NFS components to graceful stop and yield this info
as well as restart with this info.

Reviewed By: xavierd

Differential Revision: D31802787

fbshipit-source-id: 667625589b77eed79f7e17390013a818d4be7068
2021-11-17 17:13:11 -08:00
Andrey Chursin
0af2511a3f separate out ObjectId [proxy hash removal 1/n]
Summary:
The goal of this stack is to remove Proxy Hash type, but to achieve that we need first to address some tech debt in Eden codebase.

For the long time EdenFs had single Hash type that was used for many different use cases.

One of major uses for Hash type is identifies internal EdenFs objects such as blobs, trees, and others.

We seem to reach agreement that we need a different type for those identifiers, so we introduce separate ObjectId type in this diff to denote new identifier type and replace _some_ usage of Hash with ObjectId.

We still retain original Hash type for other use cases.

Roughly speaking, this is how this diff separates between Hash and ObjectId:

**ObjectId**:
* Everything that is stored in local store(blobs, trees, commits)

**Hash20**:
* Explicit hashes(Sha1 of the blob)
* Hg identifiers: manifest id and blob hg ig

For now, in this diff ObjectId has exactly same content as Hash, but this will change in the future diffs. Doing this way allows to keep diff size manageable, while migrating to new ObjectId right away would produce insanely large diff that would be both hard to make and review.

There are few more things that needs to be done before we can get to the meat of removing proxy hashes:

1) Replace include Hash.h with ObjectId.h where needed
2) Remove Hash type, explicitly rename rest of Hash usages to Hash20
3) Modify content of ObjectId to support new use cases
4) Modify serialized metadata and possibly other places that assume ObjectId size is fixed and equal to Hash20 size

Reviewed By: chadaustin

Differential Revision: D31316477

fbshipit-source-id: 0d5e4460a461bcaac6b9fd884517e129aeaf4baf
2021-10-01 10:25:46 -07:00
Zeyi (Rice) Fan
3921cd1872 cli_rs: fix health test, enable Rust edenfsctl in tests
Summary: This diff fixes some integration test errors after enabling the new edenfsctl.

Reviewed By: xavierd

Differential Revision: D30789741

fbshipit-source-id: 02d74defc41def4fb6ea0cc4694f944b4c0044e2
2021-09-29 10:02:09 -07:00
Stiopa Koltsov
c247025da9 Log how long it took for eden to start
Summary:
I see in Sandcastle logs it was 10 minutes between `eden start` command and the following command.

So I'm curious, is it eden took 10 minutes to start or Sandcastle did something else but did not log it?

This little message in log will help to understand that.

Reviewed By: kmancini

Differential Revision: D27491709

fbshipit-source-id: 796c8db5abe49b056bd81b03ea57585a761c3cb6
2021-04-01 10:32:18 -07:00
Wez Furlong
b39f678b85 edenfs: remove use of fork from StartupLogger
Summary:
on macOS we cannot safely use `fork`.

This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:

```
edenfs (setuid)
 \-----edenfs (privhelper, as root)
  \------edenfs (daemonized)
```

The forked children take advantage of being able to implicitly pass state to
the child processes from the parent.  That data flow needs to become explicit
when removing the fork which makes some things a little awkward.

With fork removed:

* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
  root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
  process which starts from scratch, but that which needs to
  run as the real server instance
* The original `edenfs` instance needs to linger for a while
  to remain connected to the controlling tty to pass back the
  startup state to the user, before terminating.

This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init.  With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid.  I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.

Reviewed By: chadaustin

Differential Revision: D23696569

fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
2020-09-18 17:22:39 -07:00
Victor Zverovich
e3f4a56f6b Migrate to field_ref Thrift API
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.

This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing

```
thrift_obj.field
```

with

```
*thrift_obj.field_ref()
```

The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.

The output of this codemod has been reviewed in D20039637.

The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.

drop-conflicts

Reviewed By: yfeldblum

Differential Revision: D22631599

fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
2020-07-21 11:23:35 -07:00
Adam Simpkins
c55781c666 move UserInfo to eden/fs/utils/
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.

This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper.  However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.

Reviewed By: wez

Differential Revision: D21296594

fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
2020-04-29 17:21:12 -07:00
Adam Simpkins
cadcd0ab39 enable building the integration helper tools in the CMake build
Summary:
Enable building fake_edenfs.cpp and TakeoverTool.cpp in the CMake build.

This includes a few changes to get `fake_edenfs.cpp` closer to building on
Windows, but at the moment it still does not fully build since the
`StartupLogger` is currently not compiled on Windows.

TakeoverTool.cpp is specific to the graceful restart tests, and doesn't make
sense to ever build on Windows.

Reviewed By: wez

Differential Revision: D21214614

fbshipit-source-id: a7a8306ea3f69579fa02bd3753a66d3095ea004a
2020-04-24 14:48:05 -07:00
Genevieve Helsel
9944a5dff5 add EdenServer recovery step and recover after failed takeover data send handshake
Summary:
* This adds a `EdenServer::recover()` method to start back up on unsuccessful takeover data send.
    * On an unsuccessful ping, filfill the `shutdownPromise` with a `TakeoverSendError` continaing the constructed `TakeoverData`. After this `recover` function is called, `takeoverPromise_` is reset, `takeoverShutdown` is set to `false`, and the `runningState_` is set to `RUNNING`.
With taking over from the returned `TakeoverData`, the user will not encounter `Transport not connected` errors on recovery.

* This adds a `EdenServer::closeStorage()` method to defer closing the `backingStore_` and `localStore_` until after our ready handshake is successful.
* This defers the shutdown of the `PrivHelper` until a successful ready handshake.

I also update the takeover documentation here with the new logic (and fix some formatting issues)

Reviewed By: simpkins

Differential Revision: D20433433

fbshipit-source-id: f59e660922674d281957e80aee5049735b901a2c
2020-04-07 09:52:21 -07:00
Genevieve Helsel
003bdfe7a5 add additional takeover "ready" handshake
Summary:
For graceful restart takeovers, we would like to implement an additional handshake. This handshake will occur right after the takeover data is ready to be sent to the client, but before actually sending it. This is to make sure the old daemon can recover in case of the client not being responsive (the client replies back to the server, and if no response is recieved in 5 seconds, the server will recover).

There are a few cases here:
* **Server sends ping (two cases discussed below)**
I introduced a new ProtocolVersion. Daemons with this change will now have ProtocolVersion4. The Server checks the max version of the client, and if this version is ProtocolVersion4, we know the client can listen for pings. So we will send the ping. Otherwise, we don't send a ping. With this, we will only send pings if we know the client will be listening for one. The case in which a client isn't listening is if we adopt this change and we downgrade past the change.

* **Server does not send ping and Client knows to listen for ping**
This will be a common case immediately after this change. The client will parse the sent data and check if it matches the "ready" ping, and if it doesn't, the client assumes the server simply sent the Takeover Data.

* **Server does not sends ping and Client doesn't know to listen for ping**
This is the case before this change.

Reviewed By: simpkins

Differential Revision: D20290271

fbshipit-source-id: b68e4df6264fb071d770671a80e28c90ddb0d3f2
2020-04-07 09:52:21 -07:00
Chad Austin
d0ce25afba log to scuba upon automatic gc
Summary:
A spike in automatic GCs usually implies something has gone wrong. Log
an event for each one, recording the cache size prior to the GC and
the cache size after.

Reviewed By: simpkins

Differential Revision: D18902580

fbshipit-source-id: 158b2635733a415a9fcc7c412b2c0f44ed04aa01
2019-12-10 16:16:35 -08:00
Chad Austin
4f360eafd2 clang-format
Summary: Formatting had diverged in a few places. Fix that up.

Reviewed By: fanzeyi

Differential Revision: D18123219

fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
2019-10-24 14:56:43 -07:00
Chad Austin
4b47257165 remove EdenConfig accessors
Summary:
Instead of having accessors for every config setting in EdenConfig,
just expose the ConfigSettings directly.

Reviewed By: fanzeyi

Differential Revision: D17847805

fbshipit-source-id: 8c6c1010c010113cf859677449797ea916f2a2a5
2019-10-11 17:55:19 -07:00
Chad Austin
8cac2bfe6a Remove dead includes in eden
Reviewed By: wez

Differential Revision: D17877514

fbshipit-source-id: e7f8ed8364bdb7a77f293cbdf4b48e8f15e64c30
2019-10-11 16:45:01 -07:00
Andres Suarez
fbdb46f5cb Tidy up license headers
Reviewed By: chadaustin

Differential Revision: D17872966

fbshipit-source-id: cd60a364a2146f0dadbeca693b1d4a5d7c97ff63
2019-10-11 05:28:23 -07:00
Wez Furlong
fed51ee453 eden: fixup fake_edenfs tests when compiled in @mode/opt
Summary:
Force a reference to the edenfsctlPath flag, otherwise the
linker will discard it and a large number of tests will fail.

Reviewed By: simpkins

Differential Revision: D17683222

fbshipit-source-id: b7cb29e74af85b544f45a228770ad2613c8e6efc
2019-10-01 08:17:29 -07:00
Chad Austin
a4311562e4 build integration test helpers with open source gflags
Summary:
Integration test helpers relied on an implicit gflags include. Make
that explicit so they compile against open source gflags and glog.

Reviewed By: wez

Differential Revision: D17264335

fbshipit-source-id: e336423b71c0f15e29b0e4ad604328b7624080a8
2019-09-16 13:32:15 -07:00
Chad Austin
fe64ec3874 use fb303 repo in open source build
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.

Reviewed By: simpkins

Differential Revision: D15528156

fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
2019-07-24 21:07:04 -07:00
Chad Austin
ae35e76c9c add a getDaemonInfo() thrift method
Summary:
Open source fb303 will not have getPid() or getCommandLine(), so
introduce a new method for Eden's tests.

Reviewed By: fanzeyi

Differential Revision: D16292993

fbshipit-source-id: 5cdc006ec0ee15f50a3e1cebe9b46a3ea275ff78
2019-07-17 13:47:02 -07:00
Adam Simpkins
aa5e6c7295 update license headers in C++ files
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2

Reviewed By: wez

Differential Revision: D15487078

fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
2019-06-19 17:02:45 -07:00
Adam Simpkins
336d41616f refactor some of the argument parsing and config setup logic
Summary:
Move some of the argument parsing and config setup code out into a new
EdenInit.h header file.  This makes it possible to re-use this logic for other
standalone utilities that want to be able to find the Eden state directory and
config information.

For now I have updated the `fake_edenfs` helper tool used by the integration
tests to use this.  This may also be useful for writing standalone tools that
can perform garbage collection of the LocalStore or checking of the overlay
state.

Reviewed By: chadaustin

Differential Revision: D14889616

fbshipit-source-id: b0b193a42cb2f52177d0c44592426b42e27242aa
2019-04-16 21:17:10 -07:00
Matt Glazar
3b9a0310a1 Fix deadlock when restarting during RocksDbLocalStore::get()
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:

1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.

Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.

Reviewed By: simpkins

Differential Revision: D14337058

fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
2019-03-12 19:29:35 -07:00
Adam Simpkins
b2869048eb update fake_edenfs to return its original arguments via thrift
Summary:
Change some of the integration tests to read back the original command line
arguments from fake_edenfs over thrift rather than by writing it out to a file
on disk.

This shouldn't really change much, it just seemed slightly simpler.

Reviewed By: strager

Differential Revision: D13515855

fbshipit-source-id: 386207c00f28626e2125958895387a870ca87b82
2019-01-11 18:31:48 -08:00
Matt Glazar
7f97ecc7bd Make fake_edenfs use same log path as real edenfs
Summary:
I want to use fake_edenfs to test logging via EdenFS' systemd service. Make fake_edenfs and the real edenfs use similar logic to determine the log file path.

This diff should not change behavior for the real edenfs.

Reviewed By: simpkins

Differential Revision: D13424470

fbshipit-source-id: d0c2e035fdb5884dbd2d9704c7e0244d35e052f2
2018-12-13 19:47:47 -08:00
Matt Glazar
c20d275a6b Make 'eden start' run a bare minimum systemd service (opt-in)
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.

Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.

'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).

Reviewed By: simpkins

Differential Revision: D10849390

fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
2018-12-06 16:01:43 -08:00
Chad Austin
259b070e1d move external benchmarks into a top-level benchmarks/
Summary:
eden/benchmarks/ is a good place to write external benchmarks for FUSE
and Thrift APIs.

(Note: this ignores all push blocking failures!)

Reviewed By: wez, strager

Differential Revision: D12970347

fbshipit-source-id: 81fa35897fc6e626254eea6e1ee44a8d35052261
2018-11-13 15:27:50 -08:00
Chad Austin
60dccd2e38 standardize some of the benchmark infrastructure
Summary:
As we start to build out both FUSE and Thrift benchmarks, we'll want a
standard library. Introduce a benchharness and have both the thrift
sha-1 and parallel_open_close benchmarks use it.

(Note: this ignores all push blocking failures!)

Differential Revision: D12969306

fbshipit-source-id: 89c8bbcc37d53560decffb9281af4aba20345787
2018-11-13 15:27:50 -08:00
Adam Simpkins
4800e57677 fix the eden start integration tests
Summary:
My changes to command line argument parsing in D12927803 conflicted with one
of the new tests added in D10863987.

My changes made `edenfs` and `fake_edenfs` reject any trailing non-option
arguments.  The new test runs `fake_edenfs` with an extra
`--commandArgumentsLogFile` argument asking it to log all non-option arguments
to a file.

This diff updates `fake_edenfs` to allow non-option arguments when
`--commandArgumentsLogFile` is specified.

Reviewed By: strager

Differential Revision: D13014079

fbshipit-source-id: 82ec5fd758716c2b66ac0738b7aacdf884f31233
2018-11-09 19:29:55 -08:00
Adam Simpkins
75a65ff073 error out if users run "edenfs" directly instead of "eden"
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`.  This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.

This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required.  Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.

Reviewed By: wez

Differential Revision: D12927803

fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
2018-11-08 12:09:42 -08:00
Matt Glazar
3700be90ec Fix handling of -- in 'eden start'
Summary:
When giving arguments to the edenfs program, 'eden start' only strips `--` if it is the first positional argument. The position of `--` shouldn't matter as long (as it's before any options).

In other words, given the following command invocation:

  $ eden start --opt-a arg-b -- --opt-c arg-d

Current behavior:

* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b -- --opt-c arg-d` is passed to edenfs as four arguments.

Desired behavior:

* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b --opt-c arg-d` is passed to edenfs as three arguments.
* The `--` argument is not passed to edenfs.

Fix 'eden start' by stripping `--` regardless of its position.

Reviewed By: chadaustin

Differential Revision: D10863987

fbshipit-source-id: 094da3f3674e775fe3e7eb8441ec95c37c34ff05
2018-11-07 12:08:18 -08:00
Matt Glazar
8f73588127 Improve test coverage of 'eden start', 'eden stop', and 'eden restart'
Summary: While creating a systemd service unit for EdenFS, I noticed a few features of the current implementation of 'eden start', 'eden stop', and 'eden restart' which are not covered by tests. Write tests for these features to ensure they don't regress with systemd integration.

Reviewed By: wez

Differential Revision: D10434379

fbshipit-source-id: 6c828a85d59179bbc4beda87e1bf2534543b60b2
2018-10-31 15:26:39 -07:00
Matt Glazar
92848b963c Create test utilities for systemd service management
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.

Unfortunately, we can't create a new instance of systemd if a machine is not managed by systemd. However, we can trick systemd into thinking that the machine *is* managed by systemd. When we do that, the systemd instance seems to work without any issues.

Create a library which, when injected into systemd using LD_PRELOAD, makes systemd thinks the machine is managed by systemd.

The library will be used in a subsequent diff (D10286940). This diff is separate so it's easier to review.

Reviewed By: simpkins

Differential Revision: D10371831

fbshipit-source-id: b54b704dc631ebdc5091320dd6b680ca255f3ba2
2018-10-26 11:27:23 -07:00
Matt Glazar
07620c63c4 Fix stderr fd leak in fake_edenfs
Summary:
When running fake_edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdout and stderr from the parent and never closes them. This has two consequences:

* The daemon process holds a reference to stdout/stderr even after the parent exits. This causes tests to hang if they slurp stdout/stderr after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* fake_edenfs logs to stdout/stderr after 'eden start' has returned. I think this behavior is surprising, but it doesn't really hurt anyone because fake_edenfs is just a developer tool.

Promptly close stdout and stderr after startup finishes by redirecting further output to a log file. This mimics the behavior of the real edenfs.

Note: stdin is still leaked. stdin will be addressed in D10442346.

Reviewed By: chadaustin

Differential Revision: D10439299

fbshipit-source-id: a7a423aa901982f3d35e81940ffc27db3a985d76
2018-10-23 15:57:08 -07:00
Matt Glazar
13549e7109 Remove unused --logPath flag from fake_edenfs
Summary:
No one uses --logPath with fake_edenfs. Remove the dead flag, inlining its default value.

This diff should not change behavior.

Reviewed By: chadaustin

Differential Revision: D10439298

fbshipit-source-id: 068e5677db1bec81a8da174b136ef9f5b54c7e34
2018-10-23 15:57:08 -07:00
Matt Glazar
8bb48d1b2c Share daemonizeIfRequested between edenfs and fake_edenfs
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.

This diff should not change behavior.

Reviewed By: chadaustin

Differential Revision: D10439300

fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
2018-10-23 14:35:38 -07:00
Matt Glazar
78abaa20ad Separate daemon mode from foreground mode in StartupLogger
Summary:
StartupLogger handles two independent scenarios: edenfs in daemon mode, and edenfs in foreground mode. I need to add a third for systemd support: edenfs in foreground mode but with startup logs written to a file.

Refactor the existing StartupLogger into an abstract base class with two concrete derived classes: DaemonStartupLogger and ForegroundStartupLogger. This will make adding the systemd feature easy without confusing the logic for the existing scenarios.

Aside from dropping the setsid() call for ForegroundStartupLogger, this diff should not change behavior.

Reviewed By: simpkins

Differential Revision: D10228156

fbshipit-source-id: fbc2c771c6fbb675779ef05ca3b7395b70b67485
2018-10-22 20:27:26 -07:00
Matt Glazar
34eeb349ae Add opt-in for experimental systemd mode
Summary:
To ease the migration from the current custom daemon setup to a systemd-managed daemon, add a way to enable the to-be-implemented systemd behavior.

* To the edenfs daemon, support the --experimentalSystemd option.
* To the Eden CLI, support the EDEN_EXPERIMENTAL_SYSTEMD environment variable. When set, invoke the edenfs daemon with --experimentalSystemd.

Aside from adding a log message, this diff does not change any behavior. This diff just sets up scaffolding for future diffs.

Reviewed By: simpkins

Differential Revision: D10248055

fbshipit-source-id: 30ca5a4bfde00ff43fd6f2a5d0282ced4f177fed
2018-10-22 20:27:25 -07:00