sapling/eden/fs/inodes/TreeInode.h

757 lines
27 KiB
C
Raw Normal View History

/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
*/
#pragma once
#include <folly/File.h>
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
#include <folly/Optional.h>
#include <folly/Portability.h>
#include <folly/Synchronized.h>
#include "eden/fs/inodes/InodeBase.h"
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
#include "eden/fs/model/Hash.h"
#include "eden/fs/utils/DirType.h"
#include "eden/fs/utils/PathMap.h"
namespace facebook {
namespace eden {
class CheckoutAction;
class CheckoutContext;
class DiffContext;
class EdenMount;
class FileHandle;
class GitIgnoreStack;
class InodeDiffCallback;
class InodeMap;
class ObjectStore;
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
class Overlay;
class RenameLock;
class Tree;
class TreeEntry;
class TreeInodeDebugInfo;
constexpr folly::StringPiece kDotEdenName{".eden"};
/**
* Represents a directory in the file system.
*/
class TreeInode : public InodeBase {
public:
enum : int { WRONG_TYPE_ERRNO = ENOTDIR };
enum class Recurse {
SHALLOW,
DEEP,
};
/**
* Represents a directory entry.
*
* A directory entry can be in one of several states:
*
* - An InodeBase object for the entry may or may not exist. If it does
* exist, it is the authoritative source of data for the entry.
*
* - If the child InodeBase object does not exist, we may or may not have an
* inode number already allocated for the child. An inode number can be
* allocated on-demand if necessary, without fully creating a child
* InodeBase object.
*
* - The child may or may not be materialized in the overlay.
*
* If the child contents are identical to an existing source control Tree
* or Blob then it does not need to be materialized, and the Entry may only
* contain the hash identifying the Tree/Blob.
*
* If the child is materialized in the overlay, then it must have an inode
* number allocated to it.
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
*/
class Entry {
public:
/**
* Create a hash for a non-materialized entry.
*/
Entry(mode_t m, Hash hash) : mode_(m), hash_{hash} {}
/**
* Create a hash for a materialized entry.
*/
Entry(mode_t m, fuse_ino_t number, dev_t rdev = 0)
: mode_(m), inodeNumber_{number}, rdev_(rdev) {}
Entry(Entry&& e) = default;
Entry& operator=(Entry&& e) = default;
bool isMaterialized() const {
// TODO: In the future we should probably only allow callers to invoke
// this method when inode is not set. If inode is set it should be the
// authoritative source of data.
return !hash_.hasValue();
}
Hash getHash() const {
// TODO: In the future we should probably only allow callers to invoke
// this method when inode is not set. If inode is set it should be the
// authoritative source of data.
return hash_.value();
}
const folly::Optional<Hash>& getOptionalHash() const {
return hash_;
}
bool hasInodeNumber() const {
return inodeNumber_ != 0;
}
fuse_ino_t getInodeNumber() const {
DCHECK_NE(inodeNumber_, 0);
return inodeNumber_;
}
void setInodeNumber(fuse_ino_t number) {
DCHECK_EQ(inodeNumber_, 0);
DCHECK(!inode_);
inodeNumber_ = number;
}
void setMaterialized(fuse_ino_t inode) {
DCHECK(inodeNumber_ == 0 || inode == inodeNumber_);
inodeNumber_ = inode;
hash_.clear();
}
void setDematerialized(Hash hash) {
hash_ = hash;
}
mode_t getMode() const {
// Callers should not check getMode() if an inode is loaded.
// If the child inode is loaded it is the authoritative source for
// the mode bits.
DCHECK(!inode_);
return mode_;
}
mode_t getModeUnsafe() const {
// TODO: T20354866 Remove this method once all callers are refactored.
//
// Callers should always call getMode() instead. This method only exists
// for supporting legacy code which will be refactored eventually.
return mode_;
}
/**
* Get the file type, as a dtype_t value as used by readdir()
*
* It is okay for callers to call getDtype() even if the inode is
* loaded. The file type for an existing entry never changes.
*/
dtype_t getDtype() const {
return mode_to_dtype(mode_);
}
/**
* Check if the entry is a directory or not.
*
* It is okay for callers to call isDirectory() even if the inode is
* loaded. The file type for an existing entry never changes.
*/
bool isDirectory() const {
return getDtype() == dtype_t::Dir;
}
dev_t getRdev() const {
// Callers should not check getRdev() if an inode is loaded.
// If the child inode is loaded it is the authoritative source for
// the mode bits.
DCHECK(!inode_);
return rdev_;
}
InodeBase* getInode() const {
return inode_;
}
void setInode(InodeBase* inode) {
DCHECK(!inode_);
DCHECK(inode);
inode_ = inode;
}
void clearInode() {
inode_ = nullptr;
}
private:
/** The complete st_mode value for this entry */
mode_t mode_{0};
/**
* If the entry is not materialized, this contains the hash
* identifying the source control Tree (if this is a directory) or Blob
* (if this is a file) that contains the entry contents.
*
* If the entry is materialized, this field is not set.
*
* TODO: If inode is set, this field generally should not be used, and the
* child InodeBase should be consulted instead.
*/
folly::Optional<Hash> hash_;
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
/**
* The inode number, if one is allocated for this entry, or 0 if one is not
* allocated.
*
* An inode number is required for materialized entries, so this is always
* non-zero if hash_ is not set. (It may also be non-zero even when hash_
* is set.)
*/
fuse_ino_t inodeNumber_{0};
/**
* The value of the rdev field that we report in stat.
* This is used for mknod and thus for unix domain sockets.
**/
dev_t rdev_{0};
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
/**
* A pointer to the child inode, if it is loaded, or null if it is not
* loaded.
*
* Note that we store this as a raw pointer. Children inodes hold a
* reference to their parent TreeInode, not the other way around.
* Children inodes can be destroyed only in one of two ways:
* - Being unlinked, then having their last reference go away.
* In this case they will be removed from our entries list when they are
* unlinked.
* - Being unloaded (after their reference count is already 0). In this
* case the parent TreeInodes responsible for triggering unloading of its
* children, so it resets this pointer to null when it unloads the child.
*/
InodeBase* inode_{nullptr};
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
};
/** Represents a directory in the overlay */
struct Dir {
/** The direct children of this directory */
PathMap<std::unique_ptr<Entry>> entries;
InodeTimestamps timeStamps;
/**
* If this TreeInode is unmaterialized (identical to an existing source
* control Tree), treeHash contains the ID of the source control Tree
* that this TreeInode is identical to.
*
* If this TreeInode is materialized (possibly modified from source
* control, and backed by the Overlay instead of a source control Tree),
* treeHash will be none.
*/
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
folly::Optional<Hash> treeHash;
bool isMaterialized() const {
return !treeHash.hasValue();
}
void setMaterialized() {
treeHash = folly::none;
}
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
};
/** Holds the results of a create operation.
*
* It is important that the file handle creation respect O_EXCL if
* it set in the flags parameter to TreeInode::create.
*/
struct CreateResult {
/// file attributes and cache ttls.
fusell::Dispatcher::Attr attr;
/// The newly created inode instance.
InodePtr inode;
/// The newly opened file handle.
std::shared_ptr<FileHandle> file;
explicit CreateResult(const EdenMount* mount);
};
TreeInode(
fuse_ino_t ino,
TreeInodePtr parent,
PathComponentPiece name,
std::unique_ptr<Tree>&& tree);
/// Construct an inode that only has backing in the Overlay area
TreeInode(
fuse_ino_t ino,
TreeInodePtr parent,
PathComponentPiece name,
Dir&& dir);
/// Constructors for the root TreeInode
TreeInode(EdenMount* mount, std::unique_ptr<Tree>&& tree);
TreeInode(EdenMount* mount, Dir&& tree);
~TreeInode() override;
folly::Future<fusell::Dispatcher::Attr> getattr() override;
folly::Future<folly::Unit> prefetch() override;
void updateOverlayHeader() const override;
fusell::Dispatcher::Attr getAttrLocked(const Dir* contents);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
/** Implements the InodeBase method used by the Dispatcher
* to create the Inode instance for a given name */
folly::Future<InodePtr> getChildByName(PathComponentPiece namepiece);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
/**
* Get the inode object for a child of this directory.
*
* The Inode object will be loaded if it is not already loaded.
*/
folly::Future<InodePtr> getOrLoadChild(PathComponentPiece name);
folly::Future<TreeInodePtr> getOrLoadChildTree(PathComponentPiece name);
/**
* Recursively look up a child inode.
*
* The Inode object in question, and all intervening TreeInode objects,
* will be loaded if they are not already loaded.
*/
folly::Future<InodePtr> getChildRecursive(RelativePathPiece name);
fuse_ino_t getChildInodeNumber(PathComponentPiece name);
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
folly::Future<std::shared_ptr<fusell::DirHandle>> opendir(
const struct fuse_file_info& fi);
folly::Future<folly::Unit> rename(
PathComponentPiece name,
TreeInodePtr newParent,
PathComponentPiece newName);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
const folly::Synchronized<Dir>& getContents() const {
return contents_;
}
folly::Synchronized<Dir>& getContents() {
return contents_;
}
/**
* Get the InodeMap for this tree's EdenMount.
*
* The InodeMap is guaranteed to remain valid for at least the lifetime of
* the TreeInode object.
*/
InodeMap* getInodeMap() const;
/**
* Get the ObjectStore for this mount point.
*
* The ObjectStore is guaranteed to remain valid for at least the lifetime of
* the TreeInode object. (The ObjectStore is owned by the EdenMount.)
*/
ObjectStore* getStore() const;
const std::shared_ptr<Overlay>& getOverlay() const;
folly::Future<CreateResult>
create(PathComponentPiece name, mode_t mode, int flags);
FileInodePtr symlink(PathComponentPiece name, folly::StringPiece contents);
TreeInodePtr mkdir(PathComponentPiece name, mode_t mode);
folly::Future<folly::Unit> unlink(PathComponentPiece name);
folly::Future<folly::Unit> rmdir(PathComponentPiece name);
/**
* Create a special filesystem node.
* Only unix domain sockets are supported; attempting to create any
* other kind of node will fail.
*/
FileInodePtr mknod(PathComponentPiece name, mode_t mode, dev_t rdev);
/**
* Compute differences between a source control Tree and the current inode
* state.
*
* @param context A pointer to the DiffContext containing parameters for the
* current diff operation. The caller is responsible for ensuring that
* the DiffContext object remains valid until this diff completes.
* @param currentPath The path to this Tree, as used for the purpose of diff
* computation. Note that we do not block renames and other filesystem
* layout changes during diff operations, so this might not actually
* correspond to the current TreeInode's path. However, it was the path
* that we used for computing ignored status, so we want to report diff
* results using this path. Even if it may not currently be the
* TreeInode's path it reflects the path that used to be correct at some
* point since the diff started.
* @param tree The source control Tree to compare the current state against.
* This may be null when comparing a portion of the file system tree that
* does not exist in source control.
* @param parentIgnore A GitIgnoreStack containing the gitignore data for all
* parent directories of this one. This parameter may be null if
* isIgnored is true. The caller must ensure that this GitIgnoreStack
* object remains valid until the returned Future object completes.
* @param isIgnored Whether or not the current directory is ignored
* according to source control ignore rules.
*
* @return Returns a Future that will be fulfilled when the diff operation
* completes. The caller must ensure that the InodeDiffCallback parameter
* remains valid until this Future completes.
*/
folly::Future<folly::Unit> diff(
const DiffContext* context,
RelativePathPiece currentPath,
std::unique_ptr<Tree> tree,
const GitIgnoreStack* parentIgnore,
bool isIgnored);
/**
* Update this directory so that it matches the specified source control Tree
* object.
*
* @param ctx The CheckoutContext for the current checkout operation.
* The caller guarantees that the CheckoutContext argument will remain
* valid until the returned Future completes.
* @param fromTree The Tree object that the checkout operation is moving
* from. This argument is necessary to detect conflicts between the
* current inode state and the expected previous source control state.
* This argument may be null when updating a TreeInode that did not exist
* in source control in the previous commit state.
* @param toTree The Tree object that the checkout operation is moving to.
* This argument may be null if this path no longer exists in the
* destination commit. If the destination location is empty, and this
* TreeInode is empty after the checkout operation (no untracked entries
* remain inside it), then this TreeInode itself will be unlinked as
* well.
*
* @return Returns a future that will be fulfilled once this tree and all of
* its children have been updated.
*/
folly::Future<folly::Unit> checkout(
CheckoutContext* ctx,
std::unique_ptr<Tree> fromTree,
std::unique_ptr<Tree> toTree);
/**
* Update this directory when a child entry is materialized.
*
* This will materialize this directory if it is not already materialized,
* and will record that the child in question is materialized.
*
* This method should only be called by the child inode in question.
*
* With regards to specific implementation details of this API:
* - The child inode must not be holding locks on itself when calling this
* method. Typically the child updates its own in-memory state first, then
* releases its lock before calling childMaterialized() on its parent.
* - The child should have written out its overlay data on disk before
* calling this method. This ensures that the child always has overlay
* data on disk whenever its parent directory's overlay data indicates that
* the child is materialized.
*/
void childMaterialized(
const RenameLock& renameLock,
PathComponentPiece childName,
fuse_ino_t childNodeId);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
/**
* Update this directory when a child entry is dematerialized.
*
* This method should only be called by the child inode in question.
*
* With regards to specific implementation details of this API:
* - The child inode must not be holding locks on itself when calling this
* method. Typically the child updates its own in-memory state first, then
* releases its lock before calling childMaterialized() on its parent.
* - The child should delay removing its on-disk overlay state until after
* this method returns. This ensures that the child always has overlay
* data on disk whenever its parent directory's overlay data indicates that
* the child is materialized.
*/
void childDematerialized(
const RenameLock& renameLock,
PathComponentPiece childName,
Hash childScmHash);
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
/**
* Internal API only for use by InodeMap.
*
* InodeMap will call this API when a child inode needs to be loaded.
* The TreeInode will call InodeMap::inodeLoadComplete() or
* InodeMap::inodeLoadFailed() when the load finishes.
*/
void loadChildInode(PathComponentPiece name, fuse_ino_t number);
/**
* Unload all unreferenced children under this tree (recursively).
*
* This walks the children underneath this tree, unloading any inodes that
* are unreferenced.
*/
void unloadChildrenNow();
/**
* unloadChildrenNow function is used to unload inodes from a directory based
* on the age of the inode.
* Age of the inode is determined by the last access time of the inode, if
* time since last access time is greater than the 'age' parameter, we will
* unload the inode.
*/
uint64_t unloadChildrenNow(std::chrono::nanoseconds age);
/**
* Load all materialized children underneath this TreeInode.
*
* This recursively descends into children directories.
*
* This method is intended to be called during the mount point initialization
* to trigger loading of materialized inodes. This allows other parts of the
* code to assume that materialized inodes are always loaded once the mount
* point has been initialized.
*
* Returns a Future that completes once all materialized inodes have been
* loaded.
*/
folly::Future<folly::Unit> loadMaterializedChildren(
Recurse recurse = Recurse::DEEP);
/*
* Update a tree entry as part of a checkout operation.
*
* This helper function is only to be used by CheckoutAction.
*
* @param ctx The CheckoutContext for the current checkout operation.
* The caller guarantees that the CheckoutContext argument will remain
* valid until the returned Future completes.
* @param name The name of the child entry being replaced.
* @param inode A pointer to the child InodeBase that is being updated.
* The path to this inode is guaranteed to match the name parameter.
* @param oldTree If this entry referred to Tree in the source commit,
* then oldTree will be a pointer to its source control state. oldTree
* will be null if this entry did not exist or if it referred to a Blob
* in the source commit.
* @param newTree If this entry refers to Tree in the destination commit,
* then newTree will be a pointer to its source control state. newTree
* will be null if this entry does not exist or if it refers to a Blob in
* the source commit.
* @param newScmEntry The desired source control state for the new entry,
* or folly::none if the entry does not exist in the destination commit.
* This entry will refer to a tree if and only if the newTree parameter
* is non-null.
*/
folly::Future<folly::Unit> checkoutUpdateEntry(
CheckoutContext* ctx,
PathComponentPiece name,
InodePtr inode,
std::unique_ptr<Tree> oldTree,
std::unique_ptr<Tree> newTree,
const folly::Optional<TreeEntry>& newScmEntry);
/**
* Get debug data about this TreeInode and all of its children (recursively).
*
* This populates the results argument with TreeInodeDebugInfo objects for
* this TreeInode and all subdirectories inside of it.
*/
void getDebugStatus(std::vector<TreeInodeDebugInfo>& results) const;
/**
* Get the timestamps of the inode.
*/
InodeTimestamps getTimestamps() const;
/**
* Helper function to set the atime of a TreeInode. In order to set atime of a
* file in TreeInodeDirHandle::readdir which doesnot have access to.
* TreeInode::contents_ we have this function. This has to be public since we
* are using it TreeInodeDirHandle class.
*/
void setAtime(struct timespec& atime);
private:
class TreeRenameLocks;
class IncompleteInodeLoad;
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
void registerInodeLoadComplete(
folly::Future<std::unique_ptr<InodeBase>>& future,
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
PathComponentPiece name,
fuse_ino_t number);
void inodeLoadComplete(
PathComponentPiece childName,
std::unique_ptr<InodeBase> childInode);
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
folly::Future<std::unique_ptr<InodeBase>> startLoadingInodeNoThrow(
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
Entry* entry,
PathComponentPiece name,
fuse_ino_t number) noexcept;
folly::Future<std::unique_ptr<InodeBase>>
introduce a new InodeMap class Summary: This diff starts adding a new InodeMap class. This class will eventually consolidate the functionality of InodeNameMap plus the inode map stored in EdenDispatcher. This new class will bring several new benefits: - All inode mapping logic consolidated into a single class, with a single lock protecting the maps. - A well-defined model for loaded vs unloaded inodes. InodeMap explicitly tracks inodes that have InodeBase objects created for them vs inodes that have an inode number allocated (for FUSE) but do not have an InodeBase object in memory. This will make it possible to unload Inode objects on demand to reduce memory usage. - Tracking of pending loads, and de-duplicating load requests. This ensures that only one Inode object ever exists for a given inode number / path. If a second request to load an Inode arrives when a previous load request is still in progress, InodeMap deals with this situation properly. - Better support for using Inode objects without FUSE. With the old code, attempts to interact with Inode objects without going through the FUSE dispatch (say, when processing a thrift call) could result in inconsistent state. New inodes created would not be put into the EdenDispatcher map, which could result in problems. - More convenient child inode access from TreeInode. With this change, the TreeInode class can easily tell which of its children are loaded. This makes it easier to do tasks which only need to operate on existing loaded inode state (for instance, dirstate computation). - Support for saving and loading state, to implement graceful restart.. InodeMap provides a central place to write out inode state on shutdown and restoring it on startup. Saved inodes can easily be restored to an "unloaded" state on startup. This code is not implemented yet as part of this diff, but it should be straightforward to add in future diffs. Reviewed By: bolinfest Differential Revision: D4318060 fbshipit-source-id: d9b16430fc8367e3516e788d1e991e5163aa6997
2016-12-23 02:34:54 +03:00
startLoadingInode(Entry* entry, PathComponentPiece name, fuse_ino_t number);
/**
* Materialize this directory in the overlay.
*
* This is required whenever we are about to make a structural change
* in the tree; renames, creation, deletion.
*/
void materialize(const RenameLock* renameLock = nullptr);
folly::Future<folly::Unit> doRename(
TreeRenameLocks&& locks,
PathComponentPiece srcName,
PathMap<std::unique_ptr<Entry>>::iterator srcIter,
TreeInodePtr destParent,
PathComponentPiece destName);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
/** Translates a Tree object from our store into a Dir object
* used to track the directory in the inode */
static Dir buildDirFromTree(
const Tree* tree,
const struct timespec& lastCheckoutTime);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
/**
* Get a TreeInodePtr to ourself.
*
* This uses TreeInodePtr::newPtrFromExisting() internally.
*
* This should only be called in contexts where we know an external caller
* already has an existing reference to us. (Which is most places--a caller
* has to have a reference to us in order to call any of our APIs.)
*/
TreeInodePtr inodePtrFromThis() {
return TreeInodePtr::newPtrFromExisting(this);
}
/**
* removeImpl() is the actual implementation used for unlink() and rmdir().
*
* The child inode in question must already be loaded. removeImpl() will
* confirm that this is still the correct inode for the given name, and
* remove it if so. If not it will attempt to load the child again, and will
* retry the remove again (hence the attemptNum parameter).
*/
template <typename InodePtrType>
folly::Future<folly::Unit>
removeImpl(PathComponent name, InodePtr child, unsigned int attemptNum);
/**
* tryRemoveChild() actually unlinks a child from our entry list.
*
* The caller must already be holding the mountpoint-wide RenameLock.
*
* This method also updates the overlay state if the child was removed
* successfully.
*
* @param renameLock A reference to the rename lock (this parameter is
* required mostly to ensure that the caller is holding it).
* @param name The entry name to remove.
* @param child If this parameter is non-null, then only remove the entry if
* it refers to the specified inode. If the entry does not refer to the
* inode in question, EBADF will be returned.
*
* @return Returns an errno value on error, or 0 on success. Notable errors
* include:
* - ENOENT: no entry exists the specified name
* - EBADF: An entry exists with the specified name, but the InodeBase object
* for it is not loaded, or it does not refer to the same inode as the
* child parameter (if child was non-null).
* - EISDIR: the entry with the specified name is a directory (only returned
* if InodePtrType is FileInodePtr).
* - ENOTDIR: the entry with the specified name is not a directory (only
* returned if InodePtrType is TreeInodePtr).
* - ENOTEMPTY: the directory being removed is not empty.
*
* Callers should assume that tryRemoveChild() may still throw an exception
* on other unexpected error cases.
*/
template <typename InodePtrType>
FOLLY_NODISCARD int tryRemoveChild(
const RenameLock& renameLock,
PathComponentPiece name,
InodePtrType child);
/**
* checkPreRemove() is called by tryRemoveChild() for file or directory
* specific checks before unlinking an entry. Returns an errno value or 0.
*/
FOLLY_NODISCARD static int checkPreRemove(const TreeInodePtr& child);
FOLLY_NODISCARD static int checkPreRemove(const FileInodePtr& child);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
/**
* This helper function starts loading a currently unloaded child inode.
* It must be held with the contents_ lock held. (The Dir argument is only
* required as a parameter to ensure that the caller is actually holding the
* lock.)
*/
folly::Future<InodePtr> loadChildLocked(
Dir& dir,
PathComponentPiece name,
Entry* entry,
std::vector<IncompleteInodeLoad>* pendingLoads);
/**
* Load the .gitignore file for this directory, then call computeDiff() once
* it is loaded.
*/
folly::Future<folly::Unit> loadGitIgnoreThenDiff(
InodePtr gitignoreInode,
const DiffContext* context,
RelativePathPiece currentPath,
std::unique_ptr<Tree> tree,
const GitIgnoreStack* parentIgnore,
bool isIgnored);
/**
* The bulk of the actual implementation of diff()
*
* The main diff() function's GitIgnoreStack parameter contains the ignore
* data for the ancestors of this directory. diff() loads .gitignore data
* for the current directory and then invokes computeDiff() to perform the
* diff once all .gitignore data is loaded.
*/
folly::Future<folly::Unit> computeDiff(
folly::Synchronized<Dir>::LockedPtr contentsLock,
const DiffContext* context,
RelativePathPiece currentPath,
std::unique_ptr<Tree> tree,
std::unique_ptr<GitIgnoreStack> ignore,
bool isIgnored);
void computeCheckoutActions(
CheckoutContext* ctx,
const Tree* fromTree,
const Tree* toTree,
std::vector<std::unique_ptr<CheckoutAction>>* actions,
std::vector<IncompleteInodeLoad>* pendingLoads);
std::unique_ptr<CheckoutAction> processCheckoutEntry(
CheckoutContext* ctx,
Dir& contents,
const TreeEntry* oldScmEntry,
const TreeEntry* newScmEntry,
std::vector<IncompleteInodeLoad>* pendingLoads);
void saveOverlayPostCheckout(CheckoutContext* ctx, const Tree* tree);
/**
* Attempt to remove an empty directory during a checkout operation.
*
* Returns true on success, or false if the directory could not be removed.
* The most likely cause of a failure is an ENOTEMPTY error if someone else
* has already created a new file in a directory made empty by a checkout.
*/
FOLLY_NODISCARD bool checkoutTryRemoveEmptyDir(CheckoutContext* ctx);
/**
* Helper function called inside InodeBase::setattr to perform TreeInode
* specific operation during setattr.
*/
folly::Future<fusell::Dispatcher::Attr> setInodeAttr(
const struct stat& attr,
int to_set) override;
/**
* Internal function used by unloadChildrenNow(std::chrono::nanoseconds age)
* which takes struct timespec as parameter. We will unload inodes under a
* directory whose atime lies before 'timePointAge'.
*/
uint64_t unloadChildrenNow(const timespec& timePointAge);
eden: merge overlay into the inode objects Summary: It was starting to get pretty complex to manage locking across the inodes, filedata, overlay and soon the journal, so as a simplifying step, this folds data that was tracked by the overlay into the TreeInode itself. This is the first diff in a short series for this. This one: 1. Breaks the persistent overlay information, so shutting down eden and bringing it back up will lose your changes (to be restored in the following diff) 2. Allows deferring materialization of file data in more cases 3. Allows renaming dirs. The approach here is now to keep just one source of information about the directory contents; when we construct a TreeInode we import this data from the Tree and then apply mutations to it locally. Each inode can be mutated indepdently from others; we only need to lock the 1, 2 or 3 participating inodes in the various mutation operations. I'll tackle persistence of the mutations in the following diff, but the high level plan for that (to help understand this diff) is to always keep the directory inodes for mutations alive as inode objects. We make use of the canForget functionality introduced by D3774269 to ensure that these don't get evicted early. On startup we'll load this information from the overlay area. This model simplifies some of the processing around reading dirs and looking up children. Since the overlay data now tracks the appropriate tree or content hash we can be much more lazy at materializing data, especially in the rename case. For example, renaming "fbcode" to "fbcod" doesn't require us to recursively materialize the "fbcode" tree. Depends on D3653706 Reviewed By: simpkins Differential Revision: D3657894 fbshipit-source-id: d4561639845ca93b93487dc84bf11ad795927b1f
2016-09-10 02:56:00 +03:00
folly::Synchronized<Dir> contents_;
};
}
}