Commit Graph

257 Commits

Author SHA1 Message Date
Chad Austin
200fefb024 set in-memory ctime properly when creating new FileInodes
Summary:
This corrects a bug where the in-memory timestamp for new files would
be set to the last checkout time instead of when the file was created.

Reviewed By: simpkins

Differential Revision: D6366189

fbshipit-source-id: c5fa8c779726d3a75c2d57b2a161293297eb9271
2017-11-21 09:21:29 -08:00
Chad Austin
7389babe42 update comments regarding O_EXCL behavior
Summary:
Per a discussion with wez, let's make it explicit in the comments that the kernel handles
O_EXCL for us.

Reviewed By: simpkins

Differential Revision: D6365157

fbshipit-source-id: 638f67cd89a9450ff084e0ef77c731ef738bf518
2017-11-21 09:21:29 -08:00
Chad Austin
0c8f577cd8 close FileInode file handle when there are no open file handles
Summary:
In some workloads we're seeing folks run out of file descriptors.
We forgot that we'd taken out the code that closes the underlying fds.
This diff takes a run at adding a simple counter of the open file handle
objects that is incremented when they are constructed and decremented
when they are destroyed.

When the count falls to zero we release the file handle.

Note that we unconditionally open files when we first load the inodes
from the overlay.  I tried to defer that open attempt and it broke
the timestamp overlay test.  I think we can revisit that aspect in
a follow on diff; for now we should be more resilient to transiently
opened files from things like ripgrep or similar.

Reviewed By: simpkins

Differential Revision: D6097090

fbshipit-source-id: 9a48220002e760fb1ffb8d7e2a68fa7036558b78
2017-11-21 09:21:29 -08:00
Michael Bolin
9ab42310c5 Directory was showing up as UNTRACKED_ADDED instead of its contents.
Summary:
This bug is part of a bigger issue in our Mercurial integration where
`UNTRACKED_ADDED` conflicts are being silently swallowed in our Hg extension
whereas stock Mercurial presents these as conflicts and forces the user to deal
with them. The Mercurial issues will be addressed in a follow-up change.

Reviewed By: simpkins

Differential Revision: D6365580

fbshipit-source-id: 831e27ce1da90ea605033b2b9988fe400ba404aa
2017-11-20 15:56:37 -08:00
Chad Austin
daa0f8ad5a Give FileInode's state an explicit tag
Summary:
In preparation for bringing D6097090 across the finish line, we need a
more explicit mechanism for determining which state the inode is in.
The issue is that whether an inode is materialized was checked in a
couple ways: the nonexistence of the hash field or the definedness of
the file field.  This diff introduces an explicit enum indicating the
state.

Reviewed By: simpkins

Differential Revision: D6325955

fbshipit-source-id: 3682a4ebc9330193baadbb33a4dd9845f26e59a6
2017-11-20 12:15:53 -08:00
Adam Simpkins
fbeb35cbdc fix issues tracking the last checkout time
Summary:
Eden attempts to initialize timestamps of newly loaded inodes with the time of
the last checkout operation performed in this mount.  Unfortunately it had a
number of bugs in this logic:

EdenMount had two separate fields attempting to track the last checkout time:
`lastCheckoutTime_` and `parentInfo_.lastCheckoutTime`.

Unfortunately neither field was actually updated on checkout operations.
Additionally, `lastCheckoutTime_` did not have any locking to allow it to be
updated.  `parentInfo_.lastCheckoutTime` did have locking, but it used the
mount point's checkout lock, so it could not be accessed during checkout
operations.

This diff removes `parentInfo_.lastCheckoutTime`, keeping only
`lastCheckoutTime_`.  It also converts `lastCheckoutTime_` to a
`struct timespec` since this is most often needed as a `timespec`.  It also
adds a new mountpoint-wide lock for synchronizing accessing to this variable.

Reviewed By: bolinfest

Differential Revision: D6356698

fbshipit-source-id: db54f9bb297b5febe4642e2b3fcc8055a6afc199
2017-11-19 15:47:13 -08:00
Chad Austin
38360c8563 make materializeForWrite private
Summary: More work towards encapsulating a FileInode's internal state machine.

Reviewed By: wez

Differential Revision: D6316013

fbshipit-source-id: 9c8303b35a0de1ba69207c7f59be88c5fb037ad8
2017-11-16 09:58:15 -08:00
Chad Austin
9217c5b56b refactor inode creation a bit, remove finishCreate()
Summary:
This is preparation for a change I'm preparing that makes the state
transitions in FileInode clearer and safer.

Reviewed By: wez

Differential Revision: D6304272

fbshipit-source-id: 493ee80517443432f790abf9806000eecb03651c
2017-11-16 00:29:24 -08:00
Adam Simpkins
c8c1ba5eab remove eden/fs/utils/test/TestChecks.h
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.

Reviewed By: chadaustin

Differential Revision: D6301759

fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
2017-11-15 12:53:55 -08:00
Michael Bolin
e8d25ee211 Move the post-clone step from C++ to Python.
Summary:
Moving the post-clone step out of C++ makes it so that
`ClientConfig` in C++ no longer knows about `hooks` and requires
`RepoConfig` to know about `hooks`. This helps us reduce
`ClientConfig`'s dependency on `ConfigData`, as my next step
is to move the remaining information that `ClientConfig` gets from
`ConfigData` directly into the client's `edenrc` file under
`~/local/.eden`.

Reviewed By: chadaustin

Differential Revision: D6310544

fbshipit-source-id: dec7a21281ab49e0416b8872757970a4eff2d943
2017-11-14 14:06:25 -08:00
Michael Bolin
2d1eade7a7 Run the autodeps script on our build files.
Summary:
I created this change by running:

```
find eden -name TARGETS | grep -v eden/fs/fuse/TARGETS | grep -v eden/fs/service/TARGETS | xargs autodeps
```

apparently `eden/fs/fuse/TARGETS` and `eden/fs/service/TARGETS` have some
constructions that `autodeps` does not understand, so I filtered those out.

Reviewed By: StanislavGlebik

Differential Revision: D6319982

fbshipit-source-id: 7b3683d1507409dde6d6570e9b13811168aa6859
2017-11-14 11:18:19 -08:00
Chad Austin
587d0ad1ee Move Journal locking inside of Journal itself
Summary:
This is follow-up to the lock ordering issues in
StreamingSubscriber.  The Journal locks are now finer-grained and no
locks are held while the subscribers are invoked.  This change
prevents future deadlocks.

Reviewed By: wez

Differential Revision: D6281410

fbshipit-source-id: 797c164395831752f61cc15928b6d4ce4dab1b68
2017-11-10 19:57:49 -08:00
Chad Austin
75bd0c08ea remove future.get() call from FileInode::ensureDataLoaded
Summary:
With the subsequent diff that enables multiple concurrent
hg importers, I was seeing this deadlock during rebase; each of
the worker threads was being blocked until it saturated the various
thread pools and locked up the server.

This removes the blocking call and replaces it with a SharedPromise
to allow multiple callers to wait for the load.

Note that this changes the semantics of ensureDataLoaded slightly:
previously the blob load was synchronous while the write lock was
held.  Now we release the write lock until the load completes.

I've added some post-condition checks to validate that we don't
break any state.  I believe that our usage is such that we don't
do anything that might mess things up.  Am I missing anything?

Reviewed By: simpkins

Differential Revision: D6264900

fbshipit-source-id: 4aa2870d95f0f0ec48d87299978cb87af99e3969
2017-11-10 13:24:33 -08:00
Chad Austin
9981d7526c Avoid blocking to fetch the metadata when prefetching sha-1 is purely an optimization.
Summary:
Address some feedback on D6264900 by only storing the sha-1 in the
overlay when it's immediately available.  This avoids the possibility
of the FUSE thread getting blocked.

Reviewed By: simpkins

Differential Revision: D6292175

fbshipit-source-id: 06c2372724e58c485d9a302dde36c34885109acf
2017-11-10 09:07:59 -08:00
Michael Bolin
c6f59d25b8 Fix a crash that could occur when doing hg update .^ --merge.
Summary:
The underlying issue is that we were reporting a `MODIFIED_REMOVED`
conflict as a `MODIFIED_MODIFIED` conflict. This put us in a state where
Mercurial expected to find a file in the new manifest, but failed because the
file was not present in that revision, so no such file could be found.

Somewhat surprisingly, the appropriate handler for a `MODIFIED_REMOVED`
conflict already existed in our Mercurial extension, but there was no logic on
the server that would generate a `MODIFIED_REMOVED` conflict previous to
this change.

Like D6204916, this was an issue I ran into when trying to create a repro case
for the issue that was fixed in D6199215.

Reviewed By: wez

Differential Revision: D6270272

fbshipit-source-id: 6604eea00b0794cd44b01d2ba6b9ea10db32d556
2017-11-09 16:29:55 -08:00
Chad Austin
32ba74fe46 rename FileInode::getSHA1 to FileInode::getSha1
Summary: Make this function match our C++ guidelines.

Reviewed By: wez

Differential Revision: D6288591

fbshipit-source-id: 1a4f52a8c1e0497df938533fe29da10264eb1ccf
2017-11-09 15:52:42 -08:00
Adam Simpkins
107c2e1753 add debug logging for TreeInode::diff()
Summary:
Add log messages that will help debug the behavior of `TreeInode::diff()`

I added this since I saw a case where `getScmStatus()` was returning a file as
modified when it did not appear to have mode or contents changes.
Unfortunately we did not have any existing debug log messages for this, and the
problem went away after restarting edenfs, so I wasn't able to track it down.
Having these log messages will help debug this in the future if we run into it
again.

Reviewed By: bolinfest

Differential Revision: D6281905

fbshipit-source-id: 736028d6efe46a387bed6bf98d971685f83da3ec
2017-11-08 21:39:07 -08:00
Chad Austin
8b605a38c4 Document and enforce the invariants in FileInode's State variant
Summary: The comments and the code did not agree on what fields were valid when, so I corrected them and added runtime enforcement.

Reviewed By: wez

Differential Revision: D6260555

fbshipit-source-id: 13e35e22c44b9a5b4cb30cdfd48e2bf7b0e116d3
2017-11-07 16:27:59 -08:00
Michael Bolin
5d738193e5 Store Hg dirstate data in Hg instead of Eden.
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.

Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.

This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.

Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.

In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.

Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.

In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.

Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.

Reviewed By: simpkins

Differential Revision: D6179950

fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
2017-11-06 19:56:49 -08:00
Chad Austin
0003adc4fb make FileInode::stat() asynchronous
Summary: Continue moving the asynchrony of loading closer to the state access so, in a future world, we still behave if unloading occurs during access.

Reviewed By: simpkins

Differential Revision: D6238523

fbshipit-source-id: 722fe5bba90b55204d50393314d225f42680409b
2017-11-03 19:08:25 -07:00
Chad Austin
536bd6e298 Change FileInode::readAll's public API to be async
Summary: To fix up the invariants in FileInode's API, we intend to remove ensureDataLoaded() and materializeForWrite().  But first I'm going to see if there is any pain caused by removing their calls.

Reviewed By: simpkins

Differential Revision: D6234049

fbshipit-source-id: c39c6d018d164b32903414d3750b875a897af210
2017-11-03 17:51:42 -07:00
Chad Austin
8b9261f2a1 run clang-format across all C++ files
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.

This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`

Reviewed By: bolinfest

Differential Revision: D6232695

fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
2017-11-03 16:02:03 -07:00
Chad Austin
f79672e8ec clean up partially-written symlinks if symlink creation fails
Summary: If writing out the symlink file in the overlay fails, clean up the file before releasing the lock.

Reviewed By: wez

Differential Revision: D6234214

fbshipit-source-id: 91ccf917a26cc1a73c3963dea7dd87364857fa03
2017-11-03 15:25:34 -07:00
Adam Simpkins
07b61591ca avoid holding TreeInode locks while locking the Journal
Summary:
Most functions in TreeInode that call Journal::addDelta() release the
TreeInode::contents_ lock before locking the journal.  However, `doRename()`
locked the journal while still holding the contents_ locks for the directories
affected by the rename.

This updates `doRename()` to make sure we release the TreeInode locks properly
before attempting to update the journal.  This should help ensure that we don't
have any lock ordering issues here.

Reviewed By: bolinfest

Differential Revision: D6212075

fbshipit-source-id: 84503acf47e9d97b81075932326825832cef9514
2017-11-01 20:43:43 -07:00
Ryan Menezes
22b2ccf616 thrift_library rename java{,deprecated}
Summary:
For a while, the recommended approach (e.g. from folks on the thrift team, from folks on the javafoundations team) has been to use `languages = ["java-swift"]`. This renaming brings the code in line with the recommendation. There are no actual changes to any builds, this is just a rename.

This is the second diff in a small stack. It's a fully automated change that uses buildozer to rename `*-java` rules to `*-javadeprecated`.

Differential Revision: D6189950

fbshipit-source-id: a08ba86cb92293a07eceb4b6b29385c7bf647b72
2017-11-01 14:34:07 -07:00
Yedidya Feldblum
b2f5376cea Move folly/Array.h
Summary: [Folly] Move `folly/Array.h` to `folly/container/`.

Reviewed By: luciang

Differential Revision: D6182858

fbshipit-source-id: 59340b96058cc6d0c7a0289e316bbde98c15d724
2017-10-29 03:42:55 -07:00
Michael Bolin
0f4ba60602 Update the hg/post-clone hook to write the .hg/dirstate file.
Summary:
Now that we make use of the `.hg/dirstate` file, we should ensure that it is
initialized correctly. This takes the commit hash from the `SNAPSHOT` file and
passes it as an argument to the `post-clone` hook. In the Hg `post-clone` hook,
it writes the `.hg/dirstate` file such that the first 20 bytes are the binary
version of the commit hash and the next 20 bytes are nul bytes.

Reviewed By: simpkins

Differential Revision: D6174440

fbshipit-source-id: 29d17f88ca3f63fa6490e1db88bc2121aced74d6
2017-10-27 13:35:26 -07:00
Wez Furlong
d3202383c1 tidy up journal subscribers on unmount
Summary:
We have a couple of issues with the watchman/eden integration:

1. If you "win" the race, you can cause a segfault during shutdown
by changing files during unmount.  This causes the journal updating
code to trigger a send to the client, but the associated eventBase
has already been destroyed.

2. We don't proactively send any signal to the subscriber (in practice: watchman)
when we unmount.  Watchman detects the eden shutdown by noticing that its
socket has closed but has no way to detect an unmount.

This diff tries to connect the unmount with the set of subscribers and tries
to cause the thrift socket to close out.

Reviewed By: bolinfest

Differential Revision: D6162717

fbshipit-source-id: 42d4a005089cd9cddf204997b1643570488f04c3
2017-10-27 09:01:57 -07:00
Michael Bolin
ac5b213e92 Include the dirstate tuples and copymap when backing up the dirstate.
Summary:
Previously, the `savebackup()` and `restorebackup()` methods in `eden_dirstate`
only retained the parent commit hashes. With this change, now the dirstate tuples
and entries in the copymap for the dirstate are also included as part of the saved
state.

Failing to restore all of the state caused issues when doing things like aborting
an `hg split`, as observed by one of our users. Although this fix works, we ultimately
plan to move the responsibility for persisting dirstate data out of Eden and into the
Hg extension. Then the data will live in `.hg/dirstate` like it would for the default
dirstate implementation.

Reviewed By: simpkins

Differential Revision: D6145420

fbshipit-source-id: baa077dee73847a47cc171cd980cdd272b3a3a99
2017-10-25 22:36:06 -07:00
Adam Simpkins
2e6ed25612 flush kernel caches properly for empty directories removed by checkout
Summary:
When performing an source control checkout operation, we attempt to remove
directories that are empty after the checkout.  However, this code path was
missing a call to flush the kernel cache for these directories.

As a result, even though eden thought the directory not longer existed, and
would not report it in `readdir()` results, the kernel would return stale
information from its cache when explicitly accessing this path.

Reviewed By: bolinfest

Differential Revision: D6151543

fbshipit-source-id: 6031feb268ff6f980c885efb26c3d43243dec3f4
2017-10-25 16:51:56 -07:00
Adam Simpkins
35c50ea4e3 add more debug logging of checkout operations
Summary:
Add a few extra debug logs to record `processCheckoutEntry()` and
`saveOverlayPostCheckout()` calls.

Reviewed By: bolinfest

Differential Revision: D6151544

fbshipit-source-id: ca6faa8fd1fe53df1e70305f5527360c918841d1
2017-10-25 16:51:56 -07:00
Yedidya Feldblum
281d1ec904 CodeMod: Replace wangle/concurrency with folly/executors
Summary:
CodeMod: Replace `wangle/concurrency` with `folly/executors`.

The headers in `wangle/concurrency/` are now but shims to equivalent headers in `folly/executors/`.

Reviewed By: jsedgwick

Differential Revision: D6120852

fbshipit-source-id: 358ceabea7ad79f84b803ed8e3aecb2a57fdd077
2017-10-22 14:42:12 -07:00
James Sedgwick
6b68c23aa1 move MemoryMapping, Shell, ThreadId, ThreadName, and VersionCheck to system/
Summary:
Everything that's going in system/ besides CpuId and Subprocess,
which are included in hphp

Reviewed By: mzlee

Differential Revision: D6102263

fbshipit-source-id: 564ef584c341a4ac79db14a9d58fe23ce51e78b3
2017-10-21 13:06:02 -07:00
Christopher Dykes
ea9b3416c5 Ensure explicit dependency on boost_filesystem
Summary:
This is needed to unblock some other work on folly.

```
foundation/dependency_management/ensure-explicit-dependencies.sh -e '("boost", None, "boost_filesystem")' boost/filesystem.hpp"
```

Reviewed By: yfeldblum

Differential Revision: D6094896

fbshipit-source-id: 3d4f7b42fa54514cf8c055f5fc477c3a2463bb0a
2017-10-18 20:34:41 -07:00
Wez Furlong
66a3a0fcea urgh, fixup move ordering issue
Summary:
I changed this around at the last minute before landing D5896494 and
it seemed to work, but the optimized build is pretty consistent at falling over
at runtime with a zero page non-null address.

Reviewed By: bolinfest

Differential Revision: D6092224

fbshipit-source-id: 93448615ac3bbea30bfe7729085808d9926f7dab
2017-10-18 15:25:01 -07:00
Chad Austin
7e93931142 remove the blocking getSha1ForBlob method
Reviewed By: wez

Differential Revision: D6083124

fbshipit-source-id: b68db91f6ebeebca9888b27eb227312893d451de
2017-10-18 10:36:29 -07:00
Wez Furlong
aa7a07f209 remove warning about already loaded inodes
Summary:
Since adding prefetch support, we're triggering this code
each time we perform a lookup operation on a dir which means that
the eden logs are made a bit more noisy with this output than
previously.

So, let's remove the warning.

Reviewed By: simpkins

Differential Revision: D5905454

fbshipit-source-id: 0abc9b42284a5224cef908293ab377d8858977ec
2017-10-17 13:07:50 -07:00
Wez Furlong
25a9786ca5 augment JournalDelta with unclean paths on snapshot hash change
Summary:
We were previously generating a simple JournalDelta consisting of
just the from/to snapshot hashes.  This is great from a `!O(repo)` perspective
when recording what changed but makes it difficult for clients downstream
to reason about changes that are not tracked in source control.

This diff adds a concept of `uncleanPaths` to the journal; these are paths
that we think are/were different from the hashes in the journal entry.

Since JournalDelta needs to be able to be merged I've opted for a simple
list of the paths that have a differing status; I'm not including all of
the various dirstate states for this because it is not obvious how to
reconcile the state across successive snapshot change events.

The `uncleanPaths` set is populated with an initial set of different paths as
the first part of the checkout call (prior to changing the hash), and then is
updated after the hash has changed to capture any additional differences.

Care needs to be taken to avoid recursively attempting to grab the parents lock
so I'm replicating just a little bit of the state management glue in the
`performDiff` method.

The Journal was not setting the from/to snapshot hashes when merging deltas.
This manifested in the watchman integration tests; we'd see the null revision
as the `from` and the `to` revision held the `from` revision(!).

On the watchman side we need to ask source control to expand the list of
files that changed when the from/to hashes are different; I've added code
to handle this.  This doesn't do anything smart in the case that the
source control aware queries are in use.  We'll look at that in a following
diff as it isn't strictly eden specific.

`watchman clock` was returning a basically empty clock unconditionally,
which meant that most since queries would report everything since the start
of time.  This is most likely contributing to poor Buck performance, although
I have not investigated the performance aspect of this.  It manifested itself
in the watchman integration tests.

Reviewed By: simpkins

Differential Revision: D5896494

fbshipit-source-id: a88be6448862781a1d8f5e15285ca07b4240593a
2017-10-16 22:46:54 -07:00
Michael Bolin
9e8e24d7df Create fix and test for hg merge.
Summary:
Running Mercurial's own integration tests revealed that we had a bug here:
https://www.mercurial-scm.org/repo/hg/file/tip/tests/test-histedit-arguments.t

Somewhat unsurprisingly, it was time to finally address a longstanding `TODO`
in `Dirstate.cpp`. The issue was that, after running `hg merge --tool :local`,
`hg status` was not including a merged file in the list of modified files. Because
the information from `hg status` is used to create a commit context, that meant
that when a commit was made after running `hg merge`, the commit did not
include the merged file in the list of files for the commit, which differs from
Mercurial's behavior.

Most of the implementation of `hg status` on the Eden side is done by
`EdenMount.diff()`. However, in this case, `diff()` does not categorize the
merged file by invoking one of the methods of `InodeDiffCallback` because
as far as `EdenMount` is concerned, the file has not changed because `EdenMount`
is unaware of the `Dirstate`. We already have some existing cases where we have
to do some post-processing on the result of `EdenMount.diff()` using information
in the `Dirstate` (e.g., files that are marked for addition or removal), so the fix was
to add a check for the case when the file is flagged as "needs merging" and
then including it as modified in the `hg status` output, as appropriate.

Reviewed By: wez

Differential Revision: D6005603

fbshipit-source-id: 7d4dd80e1a2e9f4b98243da80989e0e9119a566d
2017-10-09 11:55:34 -07:00
Yedidya Feldblum
eb45944ea0 CodeMod: Change #include's of wangle/concurrent/GlobalExecutor.h to use folly
Summary:
CodeMod: Change `#include`'s of `wangle/concurrent/GlobalExecutor.h` to use `folly`.

The file in `wangle/` is just a shim around the same file in `folly/executors/GlobalExecutor.h`. Just codemod all the `#include` sites.

Reviewed By: Orvid

Differential Revision: D5981467

fbshipit-source-id: ad7f0dce959e2760d3977b04925945e0447abc1d
2017-10-05 13:07:45 -07:00
Jeremy Fitzhardinge
fe4a7b6765 eden: enable rust Thrift generation
Reviewed By: wez

Differential Revision: D5909038

fbshipit-source-id: a4bdeac30a17682b2a92e182ab53d0413ad54256
2017-10-03 00:54:17 -07:00
Wez Furlong
364398cf94 ObjectStore now returns shared_ptr<const>
Summary:
Originally I thought this would help move towards removing a
`future.get()` call from FileInode, but it turned out to not make a difference
to that code.

It does make it a bit less of a chore to deal with the Journal related diff
callbacks added in D5896494 though, and is a move towards a future where we
could potentially return cached and shared instances of these objects.

This diff is a mechanical change to alter the return type so that we can share
instances returned from the object store interface.  It doesn't change any
functionality.

Reviewed By: simpkins

Differential Revision: D5919268

fbshipit-source-id: efe4b3af74e80cf1df20e81b4386450b72fa2c94
2017-09-29 16:54:05 -07:00
Chad Austin
3fb5680152 Rename ConflictType::MODIFIED to ConflictType::MODIFIED_MODIFIED
Summary: Per wez, this makes the MODIFIED case consistent with the other conflict types (e.g. local_remote).  Side benefit of avoiding some naming conflicts in the Haskell/Rust thrift tooling.

Reviewed By: wez

Differential Revision: D5882327

fbshipit-source-id: 3ec68c44d8c8a5c5675f1ced3842d29376d46fe2
2017-09-21 16:54:37 -07:00
Chad Austin
24c0fe9e8f prefetch inode contents when they're looked up
Reviewed By: wez

Differential Revision: D5817915

fbshipit-source-id: 3820f635cc6251ae5e13a4e214ba66df25ab6687
2017-09-19 11:10:11 -07:00
Wez Furlong
fab0c4071c centralize the post-mount functionality
Summary:
This moves the bind mounting and post-clone script running
functionality to methods of the EdenMount class and makes the whole
mount flow return a `Future<>`.  The higher level goal is to make it
easier to see where and how we want to tweak this flow to support
graceful restart.

This is mostly straight forward but care is required to avoid deadlocks; there
are two scenarios:

  # We fulfill the fuse start promise in the context of the fuse thread that is
  handling the fuse initialization packet before it has signalled to the kernel
  that it has come up.  This can be solved by using `via(mainEventBase_)`, but...
  # When remounting all the mounts on startup, we're running in the
  `mainEventBase_` thread so the simplistic solution to 1. would cause us to
  deadlock on startup (visible in the remount integration tests).

So to avoid this, we shunt the completion of the future via a CPU pool.

Also worth noting: the way we were setting up the global CPU pool with
wangle wasn't correct; it takes a weak reference to the pool which was
then getting destroyed when our prepare method returned.  It just happened
to work for us in the facebook specific build because something else was
setting up a different CPU executor.

I've reconciled this by just setting up a thread pool of our own and
using that explicitly.

Reviewed By: bolinfest

Differential Revision: D5798659

fbshipit-source-id: f1c48730f283f6962f6cd706c02d82ea2952e369
2017-09-13 08:42:21 -07:00
Michael Bolin
e837848da5 Introduce a special NoValueForKeyError for hgGetDirstateTuple() and hgCopyMapGet().
Summary:
Previously, we were generating a bit of disconcerting noise in our logs when
requesting a non-existent key in the dirstate or its copy map. We were also
susceptible to a logical error in the Eden side being silently translated to
a `KeyError` on the Python side.

Now we make things more explicit by converting a `std::out_of_range` on the C++
side to an explicit `NoValueForKeyError` that is defined in `eden.thrift`.
Now the Python side catches a `NoValueForKeyError` explicitly and converts it
into a `KeyError`. Other types of exceptions should pass through rather than be
swallowed.

This also updates the log messages to communicate when a there is no value for a
key. The messaging is improved so that it no longer appears to be a logical
error.

Reviewed By: wez

Differential Revision: D5800833

fbshipit-source-id: c44f2caf04622475d218593037cc6616bbb1c701
2017-09-11 10:52:09 -07:00
Michael Bolin
1c695e6dc6 Fixed a bug in how we update the dirstate upon a snapshot change.
Summary:
Previously, we were clearing entries in `hgDirstateTuples` for which:

```
mergeState == NotApplicable
```

but we should have been checking for:

```
mergeState == NotApplicable AND status == Normal
```

The previous logic was causing us to erroneously clear entries in a state like:

```
mergeState == NotApplicable AND status == MarkedForRemoval
```

This bug manifested itself when grafting a change that removed a file.
The file was removed from disk, but Eden did not know that it had been
`MarkedForRemoval`, so it would report the removed file as "missing" in
`hg status`.

Reviewed By: wez

Differential Revision: D5797270

fbshipit-source-id: 29740dfaa8102db868b95e932716773787f317ac
2017-09-08 19:25:34 -07:00
Wez Furlong
d910ab6594 unify state and fuseStatus in EdenMount
Summary:
After performing the dumb merge of EdenMount and MountPoint in
the prior commit, this one tidies up the state tracking and the interface
by which clients of the object can be notified of state changes.

I've introduced two Promises; the first of these can be used to wait
for the fuse mount to come up or error out.  It logically replaces
the cond wait in the `start` method and is exposed to the caller
as a Future, allowing them to wait and react to the outcome.

The second of the promises is associated with the fuse thread pool
winding down.  The attached future can be extracted and used by the
client of the EdenMount class.  This future yields the fuse device
descriptor which we can then choose to pass on during graceful
restart or simply close.  In the current integration, since we ignore
the result of that future, the handle is implicitly closed.

These promises allow us to remove the reference cycle that we had with the
`onStop` function and to potentially make the mount/unmount sequence more
async.

Reviewed By: bolinfest

Differential Revision: D5778214

fbshipit-source-id: 00b293009b7251ddd8bfb10795a115188e97aa3a
2017-09-08 19:25:34 -07:00
Wez Furlong
3da68e5adc dumb merge of MountPoint into EdenMount
Summary:
This is a mechanical and dumb move of the code from MountPoint
and into the EdenMount class.

Of note, it doesn't merge together the two different state/status fields
into a unified thing; that will be tackled in a follow on diff.

Reviewed By: bolinfest

Differential Revision: D5778212

fbshipit-source-id: 6e91a90a5cc760429d87a475ec12f81b93f87be0
2017-09-08 19:25:34 -07:00
Wez Furlong
e72a4cc187 Dispatcher no longer needs to know about MountPoint
Summary:
This is leading up to folding the MountPoint code into
the EdenMount class.

There's still a mention of the MountPoint in Dispatcher.h; that is
being dealt with in the following diff.

Reviewed By: bolinfest

Differential Revision: D5778215

fbshipit-source-id: 996640b3773988a4738ad55bb13de45e1ffe1880
2017-09-08 19:25:34 -07:00