Summary: This will make it easier to compare a `TreeEntry` with a `TreeInode::Entry`.
Reviewed By: simpkins
Differential Revision: D4034298
fbshipit-source-id: 29674e2902661bf46394ea71b81537b35bd4b107
Summary:
I need this for the upcoming test harness so I can avoid creating a
`ClientConfig`, which is currently a huge pain to do from a unit test.
Reviewed By: simpkins
Differential Revision: D4010842
fbshipit-source-id: 03d1e1de9c3047340a6f26202d4b432f4a8620b4
Summary:
We were hitting an assertion in the case where we did a `mkdir`
followed by a `rename` followed by `getMaterializedEntries`.
The issue is that our in-memory representation has a boolean to indicate
whether a dir inode is materialized, but our serialization format does
not have this bit. When we loaded the data we were not setting the
field to true and this was caught by the DCHECK.
If we have serialized data for a dir then it is, by definition, materialized
and we should just set that field to true.
Reviewed By: bolinfest
Differential Revision: D3900795
fbshipit-source-id: 62d8281e7a1009056d274888c9aff87664d2e09f
Summary:
simpkins spotted this; we were passing the wrong path down to the overlay saving dir.
This adds a test to prove that the source and destination directory contents
are correct both immediately after performing the rename and after remounting,
where we just read the serialized data.
Reviewed By: simpkins
Differential Revision: D3888694
fbshipit-source-id: 7f5fb5be417db5c693ac8a07b85abbffdbfe0fff
Summary:
populate the position from the latest journal delta.
To facilitate this, we also define the mountGeneration value to be a
combination of the pid and the time at which we created the EdenMount object,
as well as a global counter that we bump for each mount.
The precise value and meaning of this bits really doesn't matter, just that we
are unlikely to pick the same value for this same mountPoint path again if we
were to remount in the future.
Since we are now in a position to report JournalPosition values to clients, now
is also a good time to fill out the `currentPosition` field for the
`getMaterializedEntries` thrift call, and to check that this value is
consistent with the value we return via `getCurrentJournalPosition`.
Reviewed By: simpkins
Differential Revision: D3872952
fbshipit-source-id: 2fbc25d2e9711035b66ab1bf5d746507b72de265
Summary:
This is pretty simplistic: we just wlock and add a delta for the set
of file(s) that were changed in a given fuse operation (this is typically 1
file, but rename affects 2).
To reduce boilerplate very slightly, I've added an initializer_list constructor
for JournalDelta that makes it less cumbersome to create a JournalDelta for a
list of files.
Reviewed By: simpkins
Differential Revision: D3866053
fbshipit-source-id: cd918e2c98c022d5ef79430cd8ab4aef88875239
Summary:
Adds a thrift call that returns the list of materialized entries from the whole tree.
This is intended to be plugged into the mercurial dirstate extension.
Reviewed By: simpkins
Differential Revision: D3851805
fbshipit-source-id: 8429fdb4eeccc32928e8abc154d4e6fd49343556
Summary:
Annoying that gcc and clang behave differently here. The compilation
error is due to gcc not seeing the implicit this pointer for some of these
method calls, so we need to explicitly use it.
Reviewed By: simpkins
Differential Revision: D3846973
fbshipit-source-id: 3d5b8b8b8c9bbab1e7935cff0e65677f76d116fb
Summary:
Buck needs this API so that it knows which paths under a project
root it should exclude when deciding whether it can ask Eden for its
SHA-1 or if it must compute it on its own.
Reviewed By: simpkins
Differential Revision: D3840658
fbshipit-source-id: 5eddc0bef423d3b3ee165d2a4b0bbf193f94f61a
Summary:
we now serialize the overlay data for each directory independently.
When we mount, we try to load the root overlay data. The children are lazy
loaded as the inodes are instantiated.
Structural changes cause the overlay data for the impacted dirs to get saved out.
I need to make a pass over this to fixup comments and so on, I just wanted to get this diff out first.
I moved the overlay stuff from `eden/fs/overlay` -> `eden/fs/inodes` since most
of the overlay-ness is handled in `TreeInode` now; the `Overlay` class is
really just for carrying around the paths and providing the serialization
helpers.
Reviewed By: simpkins
Differential Revision: D3787108
fbshipit-source-id: f0e089a829defd953535b9d0a96b102ac729261b
Summary:
It was starting to get pretty complex to manage locking across the
inodes, filedata, overlay and soon the journal, so as a simplifying step, this
folds data that was tracked by the overlay into the TreeInode itself.
This is the first diff in a short series for this. This one:
1. Breaks the persistent overlay information, so shutting down eden and
bringing it back up will lose your changes (to be restored in the
following diff)
2. Allows deferring materialization of file data in more cases
3. Allows renaming dirs.
The approach here is now to keep just one source of information about the
directory contents; when we construct a TreeInode we import this data from the
Tree and then apply mutations to it locally.
Each inode can be mutated indepdently from others; we only need to lock the 1,
2 or 3 participating inodes in the various mutation operations.
I'll tackle persistence of the mutations in the following diff, but the high
level plan for that (to help understand this diff) is to always keep the
directory inodes for mutations alive as inode objects. We make use of the
canForget functionality introduced by D3774269 to ensure that these don't
get evicted early. On startup we'll load this information from the overlay
area.
This model simplifies some of the processing around reading dirs and looking up
children.
Since the overlay data now tracks the appropriate tree or content hash
we can be much more lazy at materializing data, especially in the rename
case. For example, renaming "fbcode" to "fbcod" doesn't require us to
recursively materialize the "fbcode" tree.
Depends on D3653706
Reviewed By: simpkins
Differential Revision: D3657894
fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
Summary:
Previously we would simply report the raw pointer address to the kernel and
rely on it to return that same number to us as the file handle, and make sure
that it told us to shut down the handle when it was closed.
This meant that we had no real idea about which files were still open.
For our future live upgrade plans we need to be able to know this so that we
can transfer the appropriate information to our replacement process.
To facilitate this this diff implements a FileHandleMap class that will assign
file handle numbers and keep track of the instances. The number assignment
strategy is the same as it used to be in the common case: we take the address
of the newly created instance and use that 64-bit number as the file handle
number. However, in the future when we transfer the mapping to a new process,
we may experience a collision when subsequently opening a file handle. To deal
with that, we have a bounded number of attempts to assign a random file handle
number.
We don't yet offer a means to iterate the map, but it would be trivial to
expose such an accessor when we're ready to use it.
Since we now keep track of these things via shared_ptr this changes the
appropriate portions of the fuse interface from unique_ptr to shared_ptr.
Reviewed By: simpkins
Differential Revision: D3602364
fbshipit-source-id: dd996339c2838225a2caeee9da16ef99a06c1e2b
Summary:
Implements basic rename(2) support. We only do this for files at this
time as directory renames require that we recursively materialize a tree and
are a bit more complex. I'll look at that in a follow-on diff; there's
potential for optimizing it to avoid eager materialization, but that depends on
the trie work that I'm tackling concurrently with this.
rename for files is the last piece needed to allow `sed -i` to operate correctly.
Reviewed By: bolinfest
Differential Revision: D3515361
fbshipit-source-id: 9c8cc5f9b8db6b5a9372ca9286336647f50490f8
Summary:
This enables O_EXCL to function by allowing the create routine to
move its folly::File instance down in to the underlying FileData instance.
Previously we would close and then re-open the file; this would discard
any of the natural gating for the open call that is performed by the kernel
for the underlying filesystem in the overlay.
Reviewed By: bolinfest
Differential Revision: D3513758
fbshipit-source-id: 85967a3b7affa1b1df46842be8ba21c8fbb843a6
Summary:
Despite being handy things for filesystems in general, these are
needed to support the `sed -i` integration test.
Reviewed By: bolinfest
Differential Revision: D3513754
fbshipit-source-id: 505b4bd58b254141e2ef996f01e3347fc1a77584
Summary:
This eliminates a linear scan from TreeInode and replaces it with a
binary search, exploiting the sorted order of the entries vector.
Two new methods are introduced: getEntryPtr which returns a pointer to the
entry with the matching name, and getEntryAt() which returns a reference
(throwing a range error if there is no such entry).
I wanted to use the PathMap class here, but that would cause us to duplicate
the name string as both the key and value in the map.
Reviewed By: bolinfest
Differential Revision: D3515723
fbshipit-source-id: 4ee0371f3ec08cbcf110cf28f5c1e1529b120fb6
Summary:
setattr is a bit of a multi-purpose interface; depending on the flag
values, this is responsible for:
* ftruncate(2)
* fchmod(2)
* fchown(2)
* futimens(2)
In order to apply any of these things, we have to materialize the file. In
the future we may want to allow setting the utimes without materializing the file.
We don't allow chown to actually chown anything. We may want to relax that in
the future, but at the moment we will return an error if an attempt is made to
change the ownership of a file.
Reviewed By: bolinfest
Differential Revision: D3511011
fbshipit-source-id: 858d2c07686fcbe2dcdb60a07527f739a9726be3
Summary:
When running the integration tests when built with gcc, the tests would crash
in TreeInode::create(). It appears that the unique_ptr<FileHandle> object was
getting passed to the lambda before dereferencing it to call getattr().
Reviewed By: wez
Differential Revision: D3459605
fbshipit-source-id: 5e2ce98d268a85731acaf7d7f37f22c77fb571cf
Summary:
This fixes inode number handling problems in TreeInode and TreeEntryFileInode.
Previosly these classes each had an ino_ member variable, despite deriving from
InodeBase which has its own (private) ino_ member.
TreeEntryFileInode never actually initialized its own local ino_ variable.
This was causing problems for many applications which use inode numbers to
cache file data. TreeEntryFileInode returned garbage data in the inode field,
resulting in incorrect cache collisions.
This fixes the inode handling, and also fixes the stat data returned by
FileData. It now sets the uid, gid, and inode fields correctly. It 0s out
several other fields rather than leaving them uninitialized.
Reviewed By: bolinfest
Differential Revision: D3455126
fbshipit-source-id: 631276b01676733f96035bc153219ef84406dcc9
Summary:
If a directory is present in the overlay, we still need to check if a TreeEntry
exists from the source control data structures. Previously this was causing us
to incorrectly report directories as empty if they exist in the local overlay,
even when they had contents from the main Tree.
Reviewed By: wez
Differential Revision: D3434219
fbshipit-source-id: f872f90075602dfdc7b217f50eefcd7c248512e7
Summary:
Add a new ObjectStore class, which will eventually contain both a LocalStore
and a BackingStore. The LocalStore will be a cache of data loaded from the
authoritative BackingStore. The ObjectStore API will hide the work of querying
the BackingStore and updating the LocalStore when data is not already available
in the LocalStore.
For now ObjectStore only contains the LocalStore, but I will add BackingStore
functionality in subsequent diffs. This diff simply updates all call sites to
use the ObjectStore instead of directly accessing the LocalStore.
Reviewed By: bolinfest
Differential Revision: D3403898
fbshipit-source-id: 47b8c51a7717a4c7c29911a7085b382521a8c0db
Summary:
This avoids translation from string->Hash in the common case
where the file is unmodified and its hash is read directly from
the store rather than computed from the overlay.
I'm guessing I should use `unique_ptr` as the return value throughout?
Reviewed By: simpkins
Differential Revision: D3355773
fbshipit-source-id: 50dff879a78b3d6ff49f86b856866ca28808c4f7
Summary:
Add a function to compute the sha1 content hash for an overlay file.
We persist the computed hash in an extended attribute in the underlying overlay
file so that a subsequent read of the attribute doesn't require opening the
file to recompute it.
Each time the file is mutated, we blow the cached status of the file.
Each time the sha1 attribute is read, if the cache is blown, the content
hash will be recomputed and set in the overlay file.
Each time the file is flushed or sync'd, if the cache is blown, the content
hash will be computed and set in the overlay file.
Reviewed By: bolinfest
Differential Revision: D3302412
fbshipit-source-id: bd45c7a24b732bd0b7474b7f96e82936870b2117
Summary:
This is part 1 of 2 diffs. This one adds some plumbing to make it possible
to read the xattr attribute from an overlay.
It doesn't do anything to ensure that it is set; the next diff in this series will take care of that.
Reviewed By: bolinfest
Differential Revision: D3302410
fbshipit-source-id: 47406a9c75f29743691d396676c691bcb99c4760
Summary:
We look this up via the mount point or eden mount object instead.
I've also removed the mercurial library stuff that was added to support the now defunct lamehg fuse we had in the earlier days.
simpkins' new importer doesn't use these and it resolves our CI mismatch issue.
Reviewed By: bolinfest
Differential Revision: D3349698
fbshipit-source-id: 5f4ec16b76042959cd1e3184f46bb3526fbaf74c
Summary:
Add a new class to serve as a single location where we can store all
information about a single eden mount point. Currently this contains the
MountPoint, LocalStore, and Overlay objects. This allows the TreeInode class
to just store a single pointer to the EdenMount, rather than having to track
these three objects separately.
In the future we could consider also keeping a copy of the ClientConfig in the
EdenMount object, but I haven't done that for now.
Reviewed By: bolinfest
Differential Revision: D3321355
fbshipit-source-id: 8a39bb49822ca8e90c88b2a834b59230d2f91435
Summary:
Enables mkdir in the overlay area.
I had to add some `lstat` calls in to the overlay dir reader because we depend
on knowing at least whether a node is a dir or not at the next level up.
When I run the test suite, the mounts are on my `/tmp` filesystem. When I run
eden manually, they are on my `/data` filesystem. The latter (xfs) does not
populate the type bits. This meant that the test suite passed but manual
testing did not.
Adding the `lstat` calls is a little unfortunate. On OS X there is a bulk
operation that combines `readdir` and `lstat` so that there are fewer syscalls.
We don't have an equivalent for Linux.
Reviewed By: bolinfest
Differential Revision: D3301532
fbshipit-source-id: e228f4a392f90aa491fec62e8b98471a8acecff2
Summary:
We still have naked pointers at the handoff to the kernel, but now
have a cleaner implementation at the level that we're going to be working at
day to day.
I also renamed `FileHandle::release` to `FileHandle::releasefile` so that it
isn't visually ambiguous with `std::unique_ptr::release` in the
`Dispatcher.cpp` code: `fh.release()` vs `fh->release()` look similar but are
dramatically different in behavior.
Reviewed By: bolinfest
Differential Revision: D3309455
fbshipit-source-id: f8cf055bcd51121048a20f0202988cf0aef1f085
Summary:
`creat(2)` and `open(2)` could decide to create a PassThru file handle.
This diff removes that usage. There is a TODO here around handling `O_EXCL` properly.
I'm punting this to a follow-up diff.
Reviewed By: bolinfest
Differential Revision: D3301387
fbshipit-source-id: d35104c536396e7fd064d786f3d5592ecfcbfecf
Summary:
Centralize and delegate most (all?) of the content sensitive portions of file accesses into the FileData class.
Add tests to show that we can write to the overlay file and that the stat data is consistent with the result.
Reviewed By: bolinfest
Differential Revision: D3301251
fbshipit-source-id: a09316ad61c6ef4c656bc5d6dbd43f906abb7932
Summary:
This is the workhorse for adjusting the state that we track for the file data.
It handles both overlay and Tree backed data cases and moving from the latter
to the former depending on the open flags provided when a file handle is
opened.
This diff handles more cases than we have tests for. Those will be covered later in this stack of diffs.
Reviewed By: bolinfest
Differential Revision: D3301213
fbshipit-source-id: c1dab40c0ad205ce6cee820043b70dd886e78431
Summary:
Another step towards making TreeEntryFileInode overlay aware, this diff implements:
- stat
- readlink (although it is not possible to create symlinks yet)
- open
Reviewed By: bolinfest
Differential Revision: D3255158
fbshipit-source-id: 3f90b624e629ef279d6cc32e1d82787ee24796eb
Summary:
This allows us to share the same state between multiple open files and
also helps to reduce the size of inode instances that are otherwise inactive;
when there are no outstanding references to the data, we can drop it and forget
it.
Reviewed By: bolinfest
Differential Revision: D3301198
fbshipit-source-id: f0b3fc73a666ec2033c7a22e9eb587d3212cf966
