Commit Graph

86 Commits

Author SHA1 Message Date
Chad Austin
d0823ab865 unbox Dir Entries
Summary:
Dir's contents were represented as a vector of 64-bit
pointers to 48-byte structs.  This change removes that layer of
indirection, reducing memory usage and slightly pessimizing insertion.

The diff is mostly mechanical outside of the TreeInode.h changes and
calls to emplace..

I'll run memory tests tomorrow, though it's a gamble as to whether
private bytes will show a difference.  I may need to shrink the Entry
struct too.

Reviewed By: wez

Differential Revision: D6804957

fbshipit-source-id: b126656dbc7951565e74b6401adde6353e809056
2018-01-30 15:11:55 -08:00
Chad Austin
697eb8a6fd run clang-format across eden
Summary:
```
find . \( -iname '*.cpp' -o -iname '*.h' \) -exec arc lint --apply-patches {} +
```

Differential Revision: D6820436

fbshipit-source-id: 173c0e3b5c023c1c9276f34e17d732f1dd161892
2018-01-26 11:20:31 -08:00
Sergey Zhupanov
7c1dac9a29 Replaced unnecessary memset calls with C++11 struct = {} init syntax.
Summary: Replaced memset used to 0-init structs with POD fields with = {} init.

Reviewed By: simpkins

Differential Revision: D6770868

fbshipit-source-id: 190205a277e3c73711813bc515b860b107d41f83
2018-01-22 17:55:24 -08:00
Wez Furlong
530346887d split EdenMount::{create,initialize} -> safer takeover mounting
Summary:
Previously, `EdenMount::create` would implicitly call
`EdenMount::initialize` which would load the root inode and the `.eden` magical
directory.  That's fine for the fresh mount case, but in the case of the
graceful restart we need to take the opportunity to apply the `InodeMap`
from the old process before we start muddying its state.

This diff breaks out the `initialize` method from the `create` method and
makes the mount code call it after potentially loading the `InodeMap` from
the takeover data.

In addition, this diff removes the the `root->loadMaterializedChildren()`
call from the mount initialization code.  It is no longer required to do
this eagerly and it makes things simpler and our memory profile a little
smaller to defer this (I haven't measured how much that impacts things).

Reviewed By: simpkins

Differential Revision: D6691182

fbshipit-source-id: 52033a6d64105b658314a919f69dbfcd4eea242b
2018-01-10 13:13:50 -08:00
Sergey Zhupanov
4f40f9db60 Added TestMount::addSymlink().
Summary: Added TestMount::addSymlink() method which creates a symlink.

Reviewed By: simpkins

Differential Revision: D6659638

fbshipit-source-id: 49c26ca8c5c6783343c1e787b594ccc87f4c03a8
2018-01-04 10:11:38 -08:00
Chad Austin
36a51e8ac6 make readIntoBuffer private
Summary:
This came up when I was auditing the rules about when it's
safe to read from a FileInode. read() must only be called while openCount > 0.

Reviewed By: wez

Differential Revision: D6604898

fbshipit-source-id: 829ddc335bd58201c2b456ee544cdc6253ebf66c
2018-01-03 17:38:48 -08:00
Sergey Zhupanov
29c2999a65 Removed all folly::fbstring uses from eden.
Summary: Removed all folly::fbstring uses from eden.

Reviewed By: wez

Differential Revision: D6597696

fbshipit-source-id: 816ea91fad063eccf9e6a46bf5c5e32bc69c6bf7
2018-01-03 16:41:34 -08:00
Wez Furlong
6ff492d11c remove dep on libfuse
Summary:
This serves a few purposes:

1. We can avoid some conditional code inside eden if we know that
   we have a specific fuse_kernel.h header implementation.
2. We don't have to figure out a way to propagate the kernel
   capabilities through the graceful restart process.
3. libfuse3 removed the channel/session hooks that we've been
   using thus far to interject ourselves for mounting and
   graceful restarting, so we were already effectively the
   walking dead here.
4. We're now able to take advtange of the latest aspects of
   the fuse kernel interface without being tied to the implementation
   of libfuse2 or libfuse3.  We're interested in the readdirplus
   functionality and will look at enabling that in a future diff.

This may make some things slightly harder for the more immediate
macOS port but I belive that we're in a much better place overall.

This diff is relatively mechanical and sadly is (unavoidably) large.

The main aspects of this diff are:

1. The `fuse_ino_t` type was provided by libfuse so we needed to
   replace it with our own definition.  This has decent penetration
   throughout the codebase.
2. The confusing `fuse_file_info` type that was multi-purpose and
   had fields that were sometimes *in* parameters and sometimes *out*
   parameters has been removed and replaced with a simpler *flags*
   parameter that corresponds to the `open(2)` flags parameter.
   The *out* portions are subsumed by existing file handle metadata
   methods.
3. The fuse parameters returned from variations of the `LOOKUP` opcode
   now return the fuse kernel type for this directly.  I suspect
   that we may need to introduce a compatibility type when we revisit
   the macOS port, but this at least makes this diff slightly simpler.
   You'll notice that some field and symbol name prefixes vary as
   a result of this.
4. Similarly for `setattr`, libfuse separated the kernel data into
   two parameters that were a little awkward to use; we're now just
   passing the kernel data through and this, IMO, makes the interface
   slightly more understandable.
5. The bulk of the code from `Dispatcher.cpp` that shimmed the
   libfuse callbacks into the C++ virtual methods has been removed
   and replaced by a `switch` statement based dispatcher in
   `FuseChannel`.   I'm not married to this being `switch` based
   and may revise this to be driven by an `unordered_map` of
   opcode -> dispatcher method defined in `FuseChannel`.  Regardless,
   `Dispatcher.cpp` is now much slimmer and should be easier to
   replace by rolling it together into `EdenDispatcher.cpp` in
   the future should we desire to do so.
6. This diff disables dispatching `poll` and `ioctl` calls.  We
   didn't make use of them and their interfaces are a bit fiddly.
7. `INTERRUPT` is also disabled here.  I will re-enable it in
   a follow-up diff where I can also revise how we track outstanding
   requests for graceful shutdown.
8. I've imported `fuse_kernel.h` from libfuse.  This is included
   under the permissive 2-clause BSD license that it allows for
   exactly this integration purpose.

Reviewed By: simpkins

Differential Revision: D6576472

fbshipit-source-id: 7cb088af5e06fe27bf22a1bed295c18c17d8006c
2018-01-02 16:36:16 -08:00
Philip Jameson
8604b8f5b0 Migrate TARGETS files from @/ to //
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros

The only thing left of the old format should be @/third-party:foo:bar

drop-conflicts

Reviewed By: ttsugriy

Differential Revision: D6605465

fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
2017-12-20 16:57:41 -08:00
Chad Austin
d112d63870 remove all direct calls to clock_gettime and system_clock::now
Summary:
Follow-up to comments in D6466209.  All access to the clock goes
through the Clock interface, making time deterministic in unit tests.

Reviewed By: simpkins

Differential Revision: D6477973

fbshipit-source-id: 24e51bdb52d0d079b34d91598d2e787d361f2525
2017-12-05 10:06:50 -08:00
Chad Austin
70f1bba3d6 test that asserts a new file inode's timestamps match the creation time, not the last checkout time
Summary: Follow-up from D6366189. First use of the new FakeClock!

Reviewed By: simpkins

Differential Revision: D6466209

fbshipit-source-id: 4d4d8a9a83df2bee11149e7a0cbddaaf734d0e04
2017-12-05 10:06:50 -08:00
Chad Austin
f91ae520b2 introduce a Clock seam for testability
Summary:
Introduce a Clock seam.  This will allow us to write tests around
ctime, mtime, and atime logic.

Reviewed By: wez

Differential Revision: D6392543

fbshipit-source-id: 1721d76d2364b135b4ef5c078ef60f7f8526259e
2017-12-05 10:06:48 -08:00
Adam Simpkins
fbeb35cbdc fix issues tracking the last checkout time
Summary:
Eden attempts to initialize timestamps of newly loaded inodes with the time of
the last checkout operation performed in this mount.  Unfortunately it had a
number of bugs in this logic:

EdenMount had two separate fields attempting to track the last checkout time:
`lastCheckoutTime_` and `parentInfo_.lastCheckoutTime`.

Unfortunately neither field was actually updated on checkout operations.
Additionally, `lastCheckoutTime_` did not have any locking to allow it to be
updated.  `parentInfo_.lastCheckoutTime` did have locking, but it used the
mount point's checkout lock, so it could not be accessed during checkout
operations.

This diff removes `parentInfo_.lastCheckoutTime`, keeping only
`lastCheckoutTime_`.  It also converts `lastCheckoutTime_` to a
`struct timespec` since this is most often needed as a `timespec`.  It also
adds a new mountpoint-wide lock for synchronizing accessing to this variable.

Reviewed By: bolinfest

Differential Revision: D6356698

fbshipit-source-id: db54f9bb297b5febe4642e2b3fcc8055a6afc199
2017-11-19 15:47:13 -08:00
Adam Simpkins
c8c1ba5eab remove eden/fs/utils/test/TestChecks.h
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.

Reviewed By: chadaustin

Differential Revision: D6301759

fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
2017-11-15 12:53:55 -08:00
Michael Bolin
2d1eade7a7 Run the autodeps script on our build files.
Summary:
I created this change by running:

```
find eden -name TARGETS | grep -v eden/fs/fuse/TARGETS | grep -v eden/fs/service/TARGETS | xargs autodeps
```

apparently `eden/fs/fuse/TARGETS` and `eden/fs/service/TARGETS` have some
constructions that `autodeps` does not understand, so I filtered those out.

Reviewed By: StanislavGlebik

Differential Revision: D6319982

fbshipit-source-id: 7b3683d1507409dde6d6570e9b13811168aa6859
2017-11-14 11:18:19 -08:00
Michael Bolin
5d738193e5 Store Hg dirstate data in Hg instead of Eden.
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.

Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.

This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.

Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.

In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.

Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.

In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.

Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.

Reviewed By: simpkins

Differential Revision: D6179950

fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
2017-11-06 19:56:49 -08:00
Chad Austin
536bd6e298 Change FileInode::readAll's public API to be async
Summary: To fix up the invariants in FileInode's API, we intend to remove ensureDataLoaded() and materializeForWrite().  But first I'm going to see if there is any pain caused by removing their calls.

Reviewed By: simpkins

Differential Revision: D6234049

fbshipit-source-id: c39c6d018d164b32903414d3750b875a897af210
2017-11-03 17:51:42 -07:00
Chad Austin
8b9261f2a1 run clang-format across all C++ files
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.

This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`

Reviewed By: bolinfest

Differential Revision: D6232695

fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
2017-11-03 16:02:03 -07:00
Chad Austin
7e93931142 remove the blocking getSha1ForBlob method
Reviewed By: wez

Differential Revision: D6083124

fbshipit-source-id: b68db91f6ebeebca9888b27eb227312893d451de
2017-10-18 10:36:29 -07:00
Wez Furlong
364398cf94 ObjectStore now returns shared_ptr<const>
Summary:
Originally I thought this would help move towards removing a
`future.get()` call from FileInode, but it turned out to not make a difference
to that code.

It does make it a bit less of a chore to deal with the Journal related diff
callbacks added in D5896494 though, and is a move towards a future where we
could potentially return cached and shared instances of these objects.

This diff is a mechanical change to alter the return type so that we can share
instances returned from the object store interface.  It doesn't change any
functionality.

Reviewed By: simpkins

Differential Revision: D5919268

fbshipit-source-id: efe4b3af74e80cf1df20e81b4386450b72fa2c94
2017-09-29 16:54:05 -07:00
Andrew Gallagher
baea19f70d Re-run autodeps on all TARGETS
Reviewed By: yfeldblum

Differential Revision: D5801431

fbshipit-source-id: f96c482a43510372324d9de6a4bb86edc5462475
2017-09-09 18:35:02 -07:00
Wez Furlong
3da68e5adc dumb merge of MountPoint into EdenMount
Summary:
This is a mechanical and dumb move of the code from MountPoint
and into the EdenMount class.

Of note, it doesn't merge together the two different state/status fields
into a unified thing; that will be tackled in a follow on diff.

Reviewed By: bolinfest

Differential Revision: D5778212

fbshipit-source-id: 6e91a90a5cc760429d87a475ec12f81b93f87be0
2017-09-08 19:25:34 -07:00
Adam Simpkins
4917682fc6 add a tag parameter to ThreadLocal<EdenStats>
Summary:
Update all of the code using ThreadLocal<EdenStats> to pass in a non-default
Tag parameter to the ThreadLocal template.

A non-default tag parameter is required to use the accessAllThreads() method on
the ThreadLocal object.  We need to use accessAllThreads() to perform stats
aggregation correctly.  Currently the EdenServer code is only performing
aggregation for stats in the FunctionScheduler.

This diff only updates the ThreadLocal<EdenStats> type, but does not contain
any behavior changes.  I will fix the stats aggregation in a subsequent diff.

Reviewed By: bolinfest

Differential Revision: D5657268

fbshipit-source-id: bc4b6f56324eb8d3052c023fd3f6f64f99b1d4e0
2017-08-18 11:50:56 -07:00
Jyothsna Konisa
19df19d994 Adding lastCheckoutTime to EdenMount and initializing timestamps of FileInode with lastCheckoutTime
Summary:
Added a new data member lastCheckoutTime to EdenMount class to store the time when checkout operation is performed. Also added a method to get the last checkout time which internally returns the lastCheckoutTime in EdenMount class.

Added new fields atime,mtime,ctime in FileInode::State structure to keep track of timestamps in memory. Initialzed timestamps in FileInode::State constructor by calling getLastCheckOutTime from EdenMount class.

Still have to add timestamp tracking for directories and have to initialize them with lastCheckout time.This probably will be done in a seperate diff.

Reviewed By: bolinfest

Differential Revision: D5437682

fbshipit-source-id: e3d6b1bc0c2192538dd3b0d9a6017ceb3ca0843d
2017-07-27 11:52:31 -07:00
Jyothsna Konisa
20cd12b31a Moving FileData methods to FileInode
Summary:
Moved all the member functions from `FileData` class to `FileInode` class
and made `FileInode` methods independent of shared `FileData` object.
Removed `FileData.h` and `FileData.cpp` files as they are not needed anymore.

Modified functions `FileInode::getSHA1()` and `FileInode::isSameAsFast` and
modified few testcases which are currently using `FileData` class and made
sure that all the test cases are passing.

Reviewed By: bolinfest

Differential Revision: D5430128

fbshipit-source-id: 3e8e6c490e92e4e602355e4ce39b67c450ec53f8
2017-07-26 23:39:35 -07:00
Eamonn Kent
27c7865526 Update eden mount point to be created asyncronously
Summary:
- Updated makeShared() to return a folly::Future<std::shared_ptr<EdenMount>> instead of just a std::shared_ptr<EdenMount>.
- Updated callers to use the returned future (get() for now)
- Refactor makeShared() and the EdenMount constructor to avoid blocking.

Reviewed By: simpkins

Differential Revision: D5424088

fbshipit-source-id: f026a3a3e4abb3593bafda76673e12c55da26322
2017-07-17 13:06:52 -07:00
Adam Simpkins
38100d0f08 fix some bugs in the treemanifest import code
Summary:
This adds unit tests for the treemanifest import code, and fixes a couple of
bugs triggered by the tests: directory permissions were being set incorrectly,
and trees were being stored in the LocalStore with the wrong hash, causing
lookup errors later.

The HgImportTest code previously used HgImporter::importManifest(), which ended
up exercising only the flatmanifest code.  This updates HgImporter to expose
separate importFlatManifest() and importTreeManifest() APIs, and updates the
tests to check both import mechanisms.

Reviewed By: wez

Differential Revision: D5365959

fbshipit-source-id: 3799ee3b937296e8025b666dd7176dbe7e268a31
2017-07-05 11:21:30 -07:00
Christopher Dykes
658a89174a Auto-dep the auto-dep'd TARGETS
Summary:
This is mostly needed because of the `folly:file` target split, but there are a few other that had changes that weren't reflected.

This also re-enables auto-deps for `multifeed/aggregator/processor/TARGETS` as it works again.

Reviewed By: yfeldblum

Differential Revision: D5362875

fbshipit-source-id: f9d2793249c0bfe644d0504f19839c108648ea3b
2017-07-02 15:50:51 -07:00
Adam Simpkins
6e7e8f9200 add basic unit test for remounting
Summary:
Add a basic unit test that adds a new file, remounts the mount point, and
confirms the materialized contents are loaded from the overlay correctly on
remount.

Differential Revision: D5326024

fbshipit-source-id: d2446030802cc4afe5af09460d590ccf8c43e525
2017-06-30 19:10:53 -07:00
Christopher Dykes
0970c1e12a Merge StringBase.cpp into String.cpp
Summary: It doesn't need to exist anymore

Reviewed By: yfeldblum

Differential Revision: D5318746

fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
2017-06-29 20:20:11 -07:00
Adam Simpkins
429f737816 format eden/fs TARGETS files with autodeps
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.

A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from.  The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.

Reviewed By: wez

Differential Revision: D5316000

fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
2017-06-27 21:20:15 -07:00
Adam Simpkins
f7756af4c7 add unit tests for HgImporter
Summary: Add unit tests for HgImporter to better test this import logic.

Reviewed By: bolinfest

Differential Revision: D5309171

fbshipit-source-id: b5619d8271fef1cc31a0f642ec2fcbf9eccced54
2017-06-23 18:25:32 -07:00
Adam Simpkins
6b62f186c7 drop Future from the name of ObjectStore APIs
Summary:
Now that the non-future versions of these APIs have been removed, rename
getBlobFuture() to getBlob(), and getTreeFuture() to getTree()

Reviewed By: wez

Differential Revision: D5295690

fbshipit-source-id: 30dcb88854b23160692b9cd83a632f863e07b491
2017-06-21 17:20:50 -07:00
Adam Simpkins
33cb16d97e remove the deprecated ObjectStore::getBlob() API
Summary:
Remove the blocking getBlob() API.

There were a few call sites in FileData still using this blocking API.  For now
I simply updated them to use getBlobFuture() and make a blocking get() call on
the returned future.  These call sites already had TODO comments documenting
the blocking behavior.

I plan to rename getBlobFuture() to getBlob() in a subsequent diff.

Reviewed By: wez

Differential Revision: D5295726

fbshipit-source-id: 748fd7a140b9b59da339a330071f732bba38cb35
2017-06-21 17:20:50 -07:00
Adam Simpkins
9cd3d175e0 remove the deprecated ObjectStore::getTree() API
Summary:
Remove the blocking getTree() API.  All call sites are using getTreeFuture()
instead now.

I plan to rename getTreeFuture() to getTree() in a subsequent diff.

Reviewed By: wez

Differential Revision: D5295725

fbshipit-source-id: 6b40b4c808da94a9c68decae3ce38c7d13fbe9f5
2017-06-21 17:20:49 -07:00
Andrew Gallagher
03bdaff954 codemod: format TARGETS with buildifier [4/5] (D5092623)
Reviewed By: igorsugak

fbshipit-source-id: 277a9d2bdc1d7e3ff3075bfe2d7307502fd0a507
2017-06-01 17:52:40 -07:00
Michael Bolin
57f5d72a27 Reimplement dirstate used by Eden's Hg extension as a subclass of Hg's dirstate.
Summary:
This is a major change to Eden's Hg extension.

Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.

In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.

However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.

Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).

In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)

That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.

In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.

One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).

In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!

I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.

For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.

Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)

The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.

Reviewed By: simpkins

Differential Revision: D5071778

fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
2017-05-26 12:05:29 -07:00
Eric Niebler
801fce3a91 Replace FOLLY_WARN_UNUSED_RESULT with FOLLY_NODISCARD everywhere
Summary: `[[nodiscard]]` is the future. Let's start now.

Reviewed By: yfeldblum

Differential Revision: D5108297

fbshipit-source-id: c98f44af9e282616af92f9171516b6ea18e68c6d
2017-05-24 20:40:38 -07:00
Victor Gao
f5c853a552 apply clang-tidy modernize-use-override
Summary:
This is generated by applying clang-tidy `-checks=modernize-use-override` to all the c++ code in project eden.
It enforces the use of the keywords `virtual`, `verride` and `final` in a way compliant to the style guide.

Reviewed By: igorsugak

Differential Revision: D5108807

fbshipit-source-id: 596f2d73f1137de350114416edb1c37da5423ed5
2017-05-23 16:10:26 -07:00
Eric Niebler
27c4156b1f Add FOLLY_NODISCARD for [[nodiscard]] attribute when it exists, FOLLY_WARN_UNUSED_RESULT uses FOLLY_NODISCARD.
Summary: The `nodiscard` attribute was added to standard C++ in C++17. Prefer the standard formulation when it is available; otherwise, use `__attribute__((__warn_unused_result__))` on compilers that support the GNU extensions, and `_Check_return_` on MSVC. The old `FOLLY_WARN_UNUSED_RESULT` is now expands to `FOLLY_NODISCARD`.

Reviewed By: yfeldblum

Differential Revision: D5105137

fbshipit-source-id: 9aa22e81cd9f0b89f9343433aeae3ba365227ccb
2017-05-23 09:05:33 -07:00
Adam Simpkins
a9b2c41328 update the hg extension to support multiple parents
Summary:
Update the eden thrift APIs to support multiple thrift parents, and update the
hg extension to use the new thrift APIs.

Reviewed By: bolinfest

Differential Revision: D4949139

fbshipit-source-id: 98a7bd17ccad7be6a429df34ecfaf3fe7ae0b39a
2017-04-27 17:37:03 -07:00
Yedidya Feldblum
d5837dca4b Casing consistency for exception_wrapper::throw_exception
Summary: [Folly] Casing consistency for `exception_wrapper::throw_exception`.

Reviewed By: Orvid

Differential Revision: D4944818

fbshipit-source-id: 72056fb24ab6362e9a0319f73b5bbf8c92d658ca
2017-04-27 01:21:04 -07:00
Adam Simpkins
87cbfe142b update the in-memory snapshot correctly after a commit
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout().  The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.

This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().

Reviewed By: bolinfest

Differential Revision: D4935943

fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
2017-04-24 18:06:59 -07:00
Adam Simpkins
a6ae3edab9 move eden/utils and eden/fuse into eden/fs
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.

Reviewed By: bolinfest, wez

Differential Revision: D4889053

fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
2017-04-14 11:39:02 -07:00
Igor Sugak
ad6811a301 use googletest instead of gtest with gmock
Summary:
This updates all of the references to gtest and gmock with googletest.

The change is mechanilcal, generated with the following one-liner:
```lang=bash
hg grep -lwE '(gtest|gmock)' 'glob:**/TARGETS' | grep -v '^third-party-buck' | xargs perl -pi -e '
$gt=qr!(["'"'"'])gtest\g1!;
(
  s!$gt(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gtest\1\),!g or
  s!$gt((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
  s!\(\s*$gt,?\s*\),?!\(\1googletest\1, None, \1gtest\1\),!g or
  s!$gt,?!\(\1googletest\1, None, \1gtest\1\),!g
) unless /(name|type) *=/;

$gm=qr!(["'"'"'])gmock\g1!;
(
  s!$gm(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gmock\1\),!g or
  s!$gm((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
  s!\(\s*$gm,?\s*\),?!\(\1googletest\1, None, \1gmock\1\),!g or
  s!$gm,?!\(\1googletest\1, None, \1gmock\1\),!g
) unless /(name|type) *=/;
'
```

Reviewed By: meyering

Differential Revision: D4643237

fbshipit-source-id: fda7f41760c7e44254231df87634631c343e6355
2017-04-01 09:21:54 -07:00
Wez Furlong
66ec00f294 connect the fuse stats to fb303
Summary:
Tweaks the stats stuff so that we can name the histograms and export them with p50, p90, p99 percentiles.

This is made a bit ugly because the fb303 stats code isn't yet open source, so
there's a moderately ugly preprocessor directive that we assume to be true for
our internal build.   We'll need to force that to be 0 for the open source build
until we open the stats code and make it available.

Reviewed By: bolinfest

Differential Revision: D4801868

fbshipit-source-id: 643909e63bd4a74b2cfa580be131f65c5673bc94
2017-03-31 11:39:48 -07:00
Wez Furlong
4235784907 add .eden "magic" dir
Summary:
It's not really magic because we don't have a virtual directory
inode base any more.  Instead, we mkdir and populate it at mount time.

What is slightly magical about it is that we give it some special powers:

* We know the inode number of the eden dir and prevent unlink operations
  on it or inside it.
* The .eden dir is present in the contents of the root inode and will
  show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
  magic `.eden` inode number.  This means that it is possible to `stat`
  and consume the `.eden` directory from any directory inside the eden
  mount, even though it won't show up in `readdir` for those child dirs.

The contents of the `.eden` dir are:

* `socket` - a symlink back to the unix domain socket that our thrift
  server is listening on.  This means that it is a simple
  `readlink(".eden/socket")` operation to discover both whether a directory
  is part of an eden mount and how to talk to the server.

* `root` - a symlink back to the root of this eden mount.  This allows
  using `readlink(".eden/root")` as a simple 1-step operation to find
  the root of an eden mount, and avoids needing to walk up directory
  by directory as is the common pattern for locating `.hg` or `.git`
  dirs.

Reviewed By: simpkins

Differential Revision: D4637285

fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
2017-03-24 23:07:42 -07:00
Adam Simpkins
d101f404dc add an EdenMount::diff() method
Summary:
This begins adding an EdenMount::diff() method, which walks through the inode
tree and reports differences from the current source control tree state.

My intent is to eventually update the Dirstate code to use this diff()
function, and to remove the existing getModifiedDirectories() logic.  The
getModifiedDirectories() code is currently incorrect, as it reports
materialized directories, which is not the same as directories that are
modified from the current source control state.

The ignore processing logic is not currently implemented in EdenMount::diff(),
but it should be relatively straightforward to add in a subsequent diff.

Reviewed By: bolinfest

Differential Revision: D4726048

fbshipit-source-id: ad0bb3b5d72bb3830f60fc2b2e56a81217c35353
2017-03-21 12:06:44 -07:00
Adam Simpkins
d1e1e4c621 refactor FileInode to avoid using its parent's TreeInode::Entry
Summary:
Refactor FileInode and FileData so that they no longer store the
TreeInode::Entry that refers to this file.  The TreeInode::Entry is owned by
the parent TreeInode and is not safe to access without holding the TreeInode
contents_ lock, which the FileInode code never did.  Additionally, once a
FileInode is unlocked the TreeInode::Entry is destroyed.  Previously the
FileInode code would retain its pointer to this now invalid data, and could
still dereference it.

In the future I think it might be worth refactoring the FileInode/FileData
split a bit more, but this at least addresses the issues around invalid
accesses to entry_.

Reviewed By: bolinfest

Differential Revision: D4688058

fbshipit-source-id: 17d5d0c4e756afc6e3c4bb279cb1bb5231f3134f
2017-03-10 18:29:29 -08:00
Adam Simpkins
aec52ad41d update unlink() to load the child inode before removing it
Summary:
Loading the child inode is necessary to avoid race conditions with other
threads that may be currently trying to load the child inode in question.

This updates the code so that rmdir() and unlink() now share most of their
implementation.

Reviewed By: bolinfest

Differential Revision: D4664110

fbshipit-source-id: 871ca7b3c22819b24bc2639d9550bb45f6541f50
2017-03-08 17:09:36 -08:00