Summary:
Add a call to tell the privhelper process to redirect its log messages to a
different file handle.
This call will make it possible for the privhelper processed to be forked
before the main edenfs process has processed the --logPath argument. The main
edenfs process can now drop privileges before it processes arguments and opens
the log file, and it can then pass the file handle to the privhelper process.
Reviewed By: wez
Differential Revision: D8212776
fbshipit-source-id: 3ec6bfc82ee090216d66c5bfd1b8c2b8819c1f45
Summary:
Add a PrivHelper::detachEventBase() method, and rename PrivHelper::start() to
attachEventBase().
This makes it possible to detach a running PrivHelper from its EventBase and
re-attach it to an EventBase later to restart it. This will be useful in
upcoming diffs to allow performing calls to the PrivHelper before the main
thrift server EventBase has started.
Reviewed By: wez
Differential Revision: D8212777
fbshipit-source-id: d5a9bf672afa8b16e53201ac747d77337e1cc307
Summary:
The `HgImporter::prefetchFiles()` method was previously sending the file
information to the `hg_import_helper.py` script as JSON. This caused
problems since the python JSON code decodes the path names as Unicode strings.
These then get passed to the mercurial code that expects binary data. It
tries encoding the unicode strings to ASCII, which can throw
UnicodeEncodeErrors.
This updates the code to use a simple binary encoding scheme, since we don't
really want to convert these pathnames from binary to unicode and back again.
The file names are not necessarily even guaranteed to be valid unicode data.
Reviewed By: chadaustin
Differential Revision: D8393757
fbshipit-source-id: 3ec4dcf2bea57e5400af94e9139d0636c446c1f0
Summary:
Update the HgRepo class to set the `HGRCPATH` environment variable to the
empty string to ensure that we always use consistent settings, and are not
affected by the current system's hgrc files.
Reviewed By: chadaustin
Differential Revision: D8410265
fbshipit-source-id: 477a3721860c067f39fbef4bb7d3d14cae3a14b6
Summary:
The two FileInode::isSameAs() methods called FileInode::getSha1(), which
returns a folly::Future object, and immediately called `value()` on the
resulting future without waiting for it to complete first.
This bug dates back to D5430128. However, I suspect it was D7888344 that
actually exposed this issue recently. D7888344 updates the
`RocksDbLocalStore` code to perform work in an I/O thread pool. Before this
change the SHA1 futures were likely always immediately ready, so this bug was
not causing problems.
Reviewed By: wez
Differential Revision: D8393671
fbshipit-source-id: ec2116751ddda31a119bfe85eab5612b622f83cf
Summary:
This updates the privhelper code to use the UnixSocket class for performing
I/O. This reduces the number of separate implementations of code we have for
sending file descriptors across Unix domain sockets, and also makes the
privhelper APIs non-blocking.
This will make it easier to clean up some of the initialization ordering in
the future. It will also make it easier to send file descriptors to the
privhelper server, instead of just receiving them. This may be helpful for
passing a file descriptor to use for logging to the privhelper process, which
will make it easier to fork the privhelper before logging redirection has
occurred.
Reviewed By: bolinfest
Differential Revision: D8053422
fbshipit-source-id: 1f8fdf22afc797eead0213be1352ea530762140d
Summary:
Up until now all of the privhelper APIs have been blocking calls. This
changes the privhelper functions to return Futures, and updates all users of
these APIs to be able to handle the results using Futures.
One benefit of this change is that all existing mount points are remounted in
parallel now during startup, rather than being mounted serially. The old code
performed a blocking `get()` call on the future returned by
`EdenServer::mount()`.
The privhelper calls themselves are still blocking for now--they block until
complete and always return completed Future objects. I will update the
privhelper code in a subsequent diff to actually make it asynchronous.
Reviewed By: bolinfest
Differential Revision: D8053421
fbshipit-source-id: 342d38697f67518f6ca96a37c12dd9812ddb151d
Summary:
Update the HgRepo class to pass through the original $PATH environment
variable. Previously it was incorrectly setting $PATH to the repository path.
Reviewed By: chadaustin
Differential Revision: D8359187
fbshipit-source-id: ce9ce374d70c325aa45b5fd643c3d55d30d34cdc
Summary:
1. Enabled a number of additional C++ compiler warnings in Eden.
2. Fixed warnings-turned-errors that resulted from this change.
Reviewed By: simpkins
Differential Revision: D8132543
fbshipit-source-id: 2290ffaaab55024d582e29201a1bcaa1152e6b3e
Summary:
Like D7867399, split TreeInode's synchronized state into a top-level
class. This is a step towards using the type system to perform
lock-safe metadata updates.
Reviewed By: simpkins
Differential Revision: D7882648
fbshipit-source-id: 27262df8ed9137c8478c68ebf4c4f13878655754
Summary:
Remove getTreeFuture and have getTree always return a Future. It's a
little unfortunate that this will always allocate, but it sounds like
we might decide to put all RocksDB access on a background thread to
increase CPU parallelism.
Reviewed By: bolinfest
Differential Revision: D8101430
fbshipit-source-id: e12b7ab07b3468114a58753768655c107265b8af
Summary:
Most filesystems limit path components to 255. To remain consistent,
let's have Eden do the same, at least for write operations.
Reviewed By: simpkins
Differential Revision: D8151020
fbshipit-source-id: 251da94a076f5765111c8e3d9d8a25c37682e2e3
Summary:
This removes the main point of contention for eden prefetch
in two ways:
1. We batch up the complete list of blobs so that they can be processed
in bulk rather than stalling the tree walk
2. We can ask remotefilelog to check and fetch that list to the local
hgcache, again as a batch, rather than by forcing the data to be
loaded through into the local store
The goal of this prefetch is to bulk load data from the mercurial server
so that a subsequent file access doesn't have to make a one-off ssh session
for each one, rather than making sure that all the data is loaded into
the local store.
Reviewed By: chadaustin
Differential Revision: D7965818
fbshipit-source-id: 753400460d633b5467c5110e3f5608ce06106e00
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary:
This is not at all clear from cppreference.com, but per
https://www.youtube.com/watch?v=dTeKf5Oek2c, it sounds to me like
recommended practice is to either:
`using namespace std::chrono_literals` (or string_literals or
whatever) to pull in a focused set of literals.
or
`using namespace std::literals` to pull in all standard literals.
or
`using namespace std` to pull in everything.
`using namespace std::literals::chrono_literals` is unnecessarily
verbose.
Adopt those standards in Eden.
Reviewed By: simpkins
Differential Revision: D8060944
fbshipit-source-id: 4d9dd4329698b7ff5e5c81b5b28780ca4d81a2a1
Summary:
igorsugak reported an LSAN failure in Eden. Relatively straightforward to
track down - pending requests could sustain an Inode which ultimately
sustains EdenMount which owns the BackingStore, resulting in a
reference cycle for the duration of a fetch request. During EdenMount
shutdown in general we may want to consider aborting or discarding any
pending fetch requests.
Reviewed By: bolinfest, igorsugak
Differential Revision: D8047650
fbshipit-source-id: 32dafaff60570cf54a74ca4f57da61b6657d8ccb
Summary:
TestMount used to call through the EdenDispatcher when creating nodes
in the filesystem which had the unfortunate side effect of
incrementing the FUSE refcount. This diff asserts the FUSE refcounts
remain zero after inodes are created.
Reviewed By: simpkins
Differential Revision: D7957177
fbshipit-source-id: 7b0865a37ebbf39fdb34db409edd70d606295a0f
Summary:
I got tired of typing PathComponentPiece{"..."} in tests so here are
some operator literals.
Reviewed By: simpkins
Differential Revision: D7956732
fbshipit-source-id: 85d9f3fd725853a54da9e70fc659bd7eb9e0862c
Summary:
Historically, we have seen a number of messages like the following in the Eden
logs:
```
Journal for .hg/blackbox.log holds invalid Created, Created sequence
```
Apparently we were getting these invalid sequences because we were not always
recording a "rename" correctly. The "rename" constructor for a `JournalDelta`
assumed that the source path should be included in the list of "removed" files
while the destination path should be included in the list of "created" files.
However, that is not accurate if the destination path already existed before
the user ran `mv`.
Fortunately, we already check whether the destination file exists in
`TreeInode::doRename()`, so it is straightforward to determine whether the
action is a "rename" (destination does not exist) or an "replace" (destination
already exists) and then classify the destination path accordingly.
As demonstrated by the new test introduced in this commit
(`JournalUpdateTest::moveFileReplace`), in the old implementation,
a file that was removed after it was overwritten would not show up as
removed in the merged `JournalDelta`. Because Watchman relies on
`JournalDelta::merge()` via the Thrift method `getFilesChangedSince()`,
this would cause Watchman to report such a file as still existing even
though it was removed.
This definitely caused bugs in Nuclide. It is likely that other tools that rely
on Watchman in Eden (such as Buck) may have also done incorrect things
because of this bug, so this could explain past reported issues.
Reviewed By: simpkins
Differential Revision: D7888249
fbshipit-source-id: 3e57963f27c5421a6175d1a759db8d9597ed76f3
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
To avoid bugs similar to the ones fixed in D7781691, mark a bunch of
folly::Future<folly::Unit> functions with FOLLY_NODISCARD.
Reviewed By: simpkins
Differential Revision: D7782224
fbshipit-source-id: 23ba42aa63011cc33e5a6e18d5bc6d00403a78d3
Summary: Also print the inode path if the assertions in EXPECT_FILE_INODE fail.
Reviewed By: simpkins
Differential Revision: D7035517
fbshipit-source-id: 6c50acb588d1c985c7e7a1586c06f04a657698f9
Summary:
This updates the code to store ServerState using a shared_ptr rather than
having it be an inlined member variable of EdenServer.
Previously EdenMount objects contained a raw pointer to the ServerState, with
the reasoning that the EdenServer object should outlive the EdenMount objects.
It turns out that this is not quite true in practice--EdenMount::destroy() will
normally be called before the EdenServer is destroyed, but this may not
actually destroy the EdenMount object immediately.
This fixes a race condition in the FuseTest.initMount() test that could cause
this test to occasionally fail when run on a heavily loaded system.
Reviewed By: chadaustin
Differential Revision: D7720509
fbshipit-source-id: 056ff5985587c8d8c32c11d17ba637ebd7598677
Summary:
Now that D7451330 writes saved inode numbers into the overlay, we
need to make sure they get deleted when checking out between trees.
Reviewed By: simpkins
Differential Revision: D7327942
fbshipit-source-id: 9593c7abe9d2e424b9ca1d9c5a5ab8b285867e6e
Summary:
Decouple inode number assignment from materialization status.
The idea is that we will always assign entries an inode number and
track whether an entry is materialized otherwise. This is necessary
to give consistent inode values across remounts.
Reviewed By: simpkins
Differential Revision: D7052470
fbshipit-source-id: 80d3f2a2938463198a3132182537e6223c79d509
Summary: Now we can write unit tests for takeover. :)
Reviewed By: simpkins
Differential Revision: D7345419
fbshipit-source-id: 05741dab65016c59109bf190c83be2f676a6141d
Summary:
Add a function for diffing two source control commits without needing to
instantiate TreeInode objects.
Reviewed By: wez
Differential Revision: D7341604
fbshipit-source-id: 557eef87faa2785ab96d51b09569a46f892a71f6
Summary:
This helps distinguish `facebook::eden::FileHandle` from
`facebook::eden::fusell::FileHandle`, and will make it possible to eliminate
the separate `facebook::eden::fusell` namespace in the future.
Reviewed By: chadaustin
Differential Revision: D7314456
fbshipit-source-id: 0869427766fe666d119a59c7df1c97825a8ad36e
Summary:
The FuseChannel destructor previously could be invoked while there are still
outstanding requests, which would cause problems when responses were generated
for these requests.
This makes the FuseChannel destructor private, requiring users to always
allocate FuseChannel objects on the heap and destroy them using a destroy()
method. The destroy() method still blocks waiting for the FUSE worker threads
to exit, but if there are still outstanding FUSE requests it defers the
deletion of the FuseChannel until they are complete.
The EdenMount class currently couldn't ever destroy FuseChannel objects with
outstanding requests, since it waits for the completion future to fire before
invoking the destructor. Fixing this behavior is mostly for correctness of the
FuseChannel API and for the benefit of our unit tests.
Reviewed By: chadaustin
Differential Revision: D7297829
fbshipit-source-id: 643d1332be84a1f25ee30ba2a2ea3515806f00ef
Summary:
Make ServerState also store the clock as a member variable. This moves more
server-wide state into the ServerState class, and allows us to eliminate
another argument to the EdenMount constructor.
Reviewed By: chadaustin
Differential Revision: D7297832
fbshipit-source-id: 2a063d67752f46686987390b3faefb304455568a
Summary:
Move the EdenCPUThreadPool from EdenServer to ServerState.
This ensures that all EdenMount objects always have a thread pool. Previously
most unit tests were creating an EdenMount without a thread pool set, but the
FileInode code and TreeInode code could end up wanting to use the thread pool
even if no FuseChannel is running.
Reviewed By: chadaustin
Differential Revision: D7297828
fbshipit-source-id: 9861b3439cd091afe31fa13f449aa14656983318
Summary: Cleaning up the takeover initialization code for EdenMount and InodeMap.
Reviewed By: simpkins
Differential Revision: D7294419
fbshipit-source-id: 58506f04259bb1017e6cd2e80e40e5820de6200f
Summary:
- Update FuseChannel::initializate() to not require an Executor. Rather than
performing initialization using the supplied Executor, it simply starts one
worker thread first and performs initialization there.
- Change the API semantics slightly so that the session complete future is
invoked only if initialization succeeds. Previously the session complete
future could be invoked if initialization failed as well.
- Replace the activeThreads counter with a stoppedThreads counter to determine
when the session complete future should be invoked. The previous
activeThreads behavior was somewhat racy, since the activeThreads counter was
incremented inside each worker thread, but most places that checked it did
not wait to ensure that all worker threads had successfully incremented it
yet.
Reviewed By: chadaustin
Differential Revision: D7243910
fbshipit-source-id: 93b3465509bd9bf6fa90ea097e70dac3193172f9
Summary:
If the root TreeInode wants to allocate inode numbers, the inode
allocator must be initialized first. But complete InodeMap
initialization requires the root TreeInode. So split this into two
parts.
Also, I changed the inode allocator to a single atomic increment instead
of a lock acquisiton.
Finally, the extra assertions in this diff uncovered what looks like a
bug in the takeover logic where nextInodeNumber_ could end up being
smaller than the value in the takeover data, since the max inode
number from the overlay was assigned after loading from takeover data.
Reviewed By: simpkins
Differential Revision: D7107706
fbshipit-source-id: ec43cc81c11d709261598739c622609b372433a2
Summary:
This adds a FakeFuse class to simulate a FUSE connection to the kernel, but
that is actually communicating with our userspace test harness code. This also
adds a FakePrivHelper class which implements the PrivHelper interface but
returns FakeFuse connections rather than real FUSE connections to the kernel.
This will enable us to write unit tests that exercise more of the FUSE and
EdenMount logic, and will also enable us to test error conditions and ordering
behaviors that are difficult to reliably reproduce with a real FUSE mount.
This also includes some very basic tests using this new code. The code in
fuse/test/FuseChannelTest.cpp creates a FuseChannel using a FakeFuse object,
and the code in inodes/test/FuseTest.cpp creates a full EdenMount object using
FakePrivHelper and FakeFuse. The tests are pretty similar for now, and only
exercise the FUSE initialization code. I will expand these tests in subsequent
diffs.
Reviewed By: wez
Differential Revision: D7050826
fbshipit-source-id: 4f82375d65664ca3a5a228a577caa4d1d47454fe
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
mode_t isn't really part of a TreeEntry and I also wanted to see all
the places where we convert an entry type from source control into
mode bits.
Reviewed By: simpkins
Differential Revision: D6983198
fbshipit-source-id: ce1d0976f5fc5130c34a8c93c07a4e26a7cdaf71
Summary:
This is the type of a tree entry, which may be another tree, so
FileType is not an accurate name.
Reviewed By: simpkins
Differential Revision: D6981168
fbshipit-source-id: 997eb8a27f599310ed678ce221c8083722db8bff
Summary:
I want to rename FileType to TreeEntryType so I removed this one first
and replaced all of its uses with an isTree() method.
Reviewed By: simpkins
Differential Revision: D6980501
fbshipit-source-id: 105b8c599585e63efd44043e761db40e2824e77e
Summary:
Our Model TreeEntry code was a bit too general - in reality, both git
and hg only support a handful of specific tree entries: regular files,
executable files, symlinks, and trees. (git also supports
submodules.) This diff delays the expansion of a TreeEntry's type
into a full mode_t.
Reviewed By: simpkins
Differential Revision: D6980003
fbshipit-source-id: 73729208000668078a180b728d7e0bb9169c6f3c
Summary:
Remove the declaration of HgRepo::initHg(), which is never used or defined
anywhere. (There is a public HgRepo::hgInit() method.)
Reviewed By: chadaustin
Differential Revision: D6996812
fbshipit-source-id: 1ad15f624b3839ccb8a2005f3bcf0897563f82f5
Summary:
Add a new ServerState class to store process-wide state that is shared across
multiple mounts. Up until now we have passed around the shared state data as
separate variables.
This is intentionally separate from the existing EdenServer class to allow unit
tests to create EdenMount objects without an EdenServer object.
Reviewed By: wez
Differential Revision: D6929861
fbshipit-source-id: 5f22efb6d79dfd70031be1dc37f494c2ad8af902
Summary:
This enables dropping in alternative implementations
of LocalStore and adds a MemoryLocalStore implementation for
use in our tests.
This diff doesn't change the default storage option for the
eden server. I'll look at adding such an option in a follow up diff.
Reviewed By: chadaustin
Differential Revision: D6910413
fbshipit-source-id: 018bf04e0bff101e1f0ab35e8580ca2a2622e5ef
Summary: This API seems like it should be const, as it does not modify the clock.
Reviewed By: chadaustin, zhupanov
Differential Revision: D6869719
fbshipit-source-id: c8bf4ccab34538b59e6baeedd0b0ff88b328236e
Summary:
Dir's contents were represented as a vector of 64-bit
pointers to 48-byte structs. This change removes that layer of
indirection, reducing memory usage and slightly pessimizing insertion.
The diff is mostly mechanical outside of the TreeInode.h changes and
calls to emplace..
I'll run memory tests tomorrow, though it's a gamble as to whether
private bytes will show a difference. I may need to shrink the Entry
struct too.
Reviewed By: wez
Differential Revision: D6804957
fbshipit-source-id: b126656dbc7951565e74b6401adde6353e809056
Summary: Replaced memset used to 0-init structs with POD fields with = {} init.
Reviewed By: simpkins
Differential Revision: D6770868
fbshipit-source-id: 190205a277e3c73711813bc515b860b107d41f83
Summary:
Previously, `EdenMount::create` would implicitly call
`EdenMount::initialize` which would load the root inode and the `.eden` magical
directory. That's fine for the fresh mount case, but in the case of the
graceful restart we need to take the opportunity to apply the `InodeMap`
from the old process before we start muddying its state.
This diff breaks out the `initialize` method from the `create` method and
makes the mount code call it after potentially loading the `InodeMap` from
the takeover data.
In addition, this diff removes the the `root->loadMaterializedChildren()`
call from the mount initialization code. It is no longer required to do
this eagerly and it makes things simpler and our memory profile a little
smaller to defer this (I haven't measured how much that impacts things).
Reviewed By: simpkins
Differential Revision: D6691182
fbshipit-source-id: 52033a6d64105b658314a919f69dbfcd4eea242b
Summary:
This came up when I was auditing the rules about when it's
safe to read from a FileInode. read() must only be called while openCount > 0.
Reviewed By: wez
Differential Revision: D6604898
fbshipit-source-id: 829ddc335bd58201c2b456ee544cdc6253ebf66c
Summary:
This serves a few purposes:
1. We can avoid some conditional code inside eden if we know that
we have a specific fuse_kernel.h header implementation.
2. We don't have to figure out a way to propagate the kernel
capabilities through the graceful restart process.
3. libfuse3 removed the channel/session hooks that we've been
using thus far to interject ourselves for mounting and
graceful restarting, so we were already effectively the
walking dead here.
4. We're now able to take advtange of the latest aspects of
the fuse kernel interface without being tied to the implementation
of libfuse2 or libfuse3. We're interested in the readdirplus
functionality and will look at enabling that in a future diff.
This may make some things slightly harder for the more immediate
macOS port but I belive that we're in a much better place overall.
This diff is relatively mechanical and sadly is (unavoidably) large.
The main aspects of this diff are:
1. The `fuse_ino_t` type was provided by libfuse so we needed to
replace it with our own definition. This has decent penetration
throughout the codebase.
2. The confusing `fuse_file_info` type that was multi-purpose and
had fields that were sometimes *in* parameters and sometimes *out*
parameters has been removed and replaced with a simpler *flags*
parameter that corresponds to the `open(2)` flags parameter.
The *out* portions are subsumed by existing file handle metadata
methods.
3. The fuse parameters returned from variations of the `LOOKUP` opcode
now return the fuse kernel type for this directly. I suspect
that we may need to introduce a compatibility type when we revisit
the macOS port, but this at least makes this diff slightly simpler.
You'll notice that some field and symbol name prefixes vary as
a result of this.
4. Similarly for `setattr`, libfuse separated the kernel data into
two parameters that were a little awkward to use; we're now just
passing the kernel data through and this, IMO, makes the interface
slightly more understandable.
5. The bulk of the code from `Dispatcher.cpp` that shimmed the
libfuse callbacks into the C++ virtual methods has been removed
and replaced by a `switch` statement based dispatcher in
`FuseChannel`. I'm not married to this being `switch` based
and may revise this to be driven by an `unordered_map` of
opcode -> dispatcher method defined in `FuseChannel`. Regardless,
`Dispatcher.cpp` is now much slimmer and should be easier to
replace by rolling it together into `EdenDispatcher.cpp` in
the future should we desire to do so.
6. This diff disables dispatching `poll` and `ioctl` calls. We
didn't make use of them and their interfaces are a bit fiddly.
7. `INTERRUPT` is also disabled here. I will re-enable it in
a follow-up diff where I can also revise how we track outstanding
requests for graceful shutdown.
8. I've imported `fuse_kernel.h` from libfuse. This is included
under the permissive 2-clause BSD license that it allows for
exactly this integration purpose.
Reviewed By: simpkins
Differential Revision: D6576472
fbshipit-source-id: 7cb088af5e06fe27bf22a1bed295c18c17d8006c
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
Follow-up to comments in D6466209. All access to the clock goes
through the Clock interface, making time deterministic in unit tests.
Reviewed By: simpkins
Differential Revision: D6477973
fbshipit-source-id: 24e51bdb52d0d079b34d91598d2e787d361f2525
Summary: Follow-up from D6366189. First use of the new FakeClock!
Reviewed By: simpkins
Differential Revision: D6466209
fbshipit-source-id: 4d4d8a9a83df2bee11149e7a0cbddaaf734d0e04
Summary:
Introduce a Clock seam. This will allow us to write tests around
ctime, mtime, and atime logic.
Reviewed By: wez
Differential Revision: D6392543
fbshipit-source-id: 1721d76d2364b135b4ef5c078ef60f7f8526259e
Summary:
Eden attempts to initialize timestamps of newly loaded inodes with the time of
the last checkout operation performed in this mount. Unfortunately it had a
number of bugs in this logic:
EdenMount had two separate fields attempting to track the last checkout time:
`lastCheckoutTime_` and `parentInfo_.lastCheckoutTime`.
Unfortunately neither field was actually updated on checkout operations.
Additionally, `lastCheckoutTime_` did not have any locking to allow it to be
updated. `parentInfo_.lastCheckoutTime` did have locking, but it used the
mount point's checkout lock, so it could not be accessed during checkout
operations.
This diff removes `parentInfo_.lastCheckoutTime`, keeping only
`lastCheckoutTime_`. It also converts `lastCheckoutTime_` to a
`struct timespec` since this is most often needed as a `timespec`. It also
adds a new mountpoint-wide lock for synchronizing accessing to this variable.
Reviewed By: bolinfest
Differential Revision: D6356698
fbshipit-source-id: db54f9bb297b5febe4642e2b3fcc8055a6afc199
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.
Reviewed By: chadaustin
Differential Revision: D6301759
fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
Summary:
I created this change by running:
```
find eden -name TARGETS | grep -v eden/fs/fuse/TARGETS | grep -v eden/fs/service/TARGETS | xargs autodeps
```
apparently `eden/fs/fuse/TARGETS` and `eden/fs/service/TARGETS` have some
constructions that `autodeps` does not understand, so I filtered those out.
Reviewed By: StanislavGlebik
Differential Revision: D6319982
fbshipit-source-id: 7b3683d1507409dde6d6570e9b13811168aa6859
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.
Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.
This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.
Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.
In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.
Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.
In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.
Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.
Reviewed By: simpkins
Differential Revision: D6179950
fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
Summary: To fix up the invariants in FileInode's API, we intend to remove ensureDataLoaded() and materializeForWrite(). But first I'm going to see if there is any pain caused by removing their calls.
Reviewed By: simpkins
Differential Revision: D6234049
fbshipit-source-id: c39c6d018d164b32903414d3750b875a897af210
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
Originally I thought this would help move towards removing a
`future.get()` call from FileInode, but it turned out to not make a difference
to that code.
It does make it a bit less of a chore to deal with the Journal related diff
callbacks added in D5896494 though, and is a move towards a future where we
could potentially return cached and shared instances of these objects.
This diff is a mechanical change to alter the return type so that we can share
instances returned from the object store interface. It doesn't change any
functionality.
Reviewed By: simpkins
Differential Revision: D5919268
fbshipit-source-id: efe4b3af74e80cf1df20e81b4386450b72fa2c94
Summary:
This is a mechanical and dumb move of the code from MountPoint
and into the EdenMount class.
Of note, it doesn't merge together the two different state/status fields
into a unified thing; that will be tackled in a follow on diff.
Reviewed By: bolinfest
Differential Revision: D5778212
fbshipit-source-id: 6e91a90a5cc760429d87a475ec12f81b93f87be0
Summary:
Update all of the code using ThreadLocal<EdenStats> to pass in a non-default
Tag parameter to the ThreadLocal template.
A non-default tag parameter is required to use the accessAllThreads() method on
the ThreadLocal object. We need to use accessAllThreads() to perform stats
aggregation correctly. Currently the EdenServer code is only performing
aggregation for stats in the FunctionScheduler.
This diff only updates the ThreadLocal<EdenStats> type, but does not contain
any behavior changes. I will fix the stats aggregation in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D5657268
fbshipit-source-id: bc4b6f56324eb8d3052c023fd3f6f64f99b1d4e0
Summary:
Added a new data member lastCheckoutTime to EdenMount class to store the time when checkout operation is performed. Also added a method to get the last checkout time which internally returns the lastCheckoutTime in EdenMount class.
Added new fields atime,mtime,ctime in FileInode::State structure to keep track of timestamps in memory. Initialzed timestamps in FileInode::State constructor by calling getLastCheckOutTime from EdenMount class.
Still have to add timestamp tracking for directories and have to initialize them with lastCheckout time.This probably will be done in a seperate diff.
Reviewed By: bolinfest
Differential Revision: D5437682
fbshipit-source-id: e3d6b1bc0c2192538dd3b0d9a6017ceb3ca0843d
Summary:
Moved all the member functions from `FileData` class to `FileInode` class
and made `FileInode` methods independent of shared `FileData` object.
Removed `FileData.h` and `FileData.cpp` files as they are not needed anymore.
Modified functions `FileInode::getSHA1()` and `FileInode::isSameAsFast` and
modified few testcases which are currently using `FileData` class and made
sure that all the test cases are passing.
Reviewed By: bolinfest
Differential Revision: D5430128
fbshipit-source-id: 3e8e6c490e92e4e602355e4ce39b67c450ec53f8
Summary:
- Updated makeShared() to return a folly::Future<std::shared_ptr<EdenMount>> instead of just a std::shared_ptr<EdenMount>.
- Updated callers to use the returned future (get() for now)
- Refactor makeShared() and the EdenMount constructor to avoid blocking.
Reviewed By: simpkins
Differential Revision: D5424088
fbshipit-source-id: f026a3a3e4abb3593bafda76673e12c55da26322
Summary:
This adds unit tests for the treemanifest import code, and fixes a couple of
bugs triggered by the tests: directory permissions were being set incorrectly,
and trees were being stored in the LocalStore with the wrong hash, causing
lookup errors later.
The HgImportTest code previously used HgImporter::importManifest(), which ended
up exercising only the flatmanifest code. This updates HgImporter to expose
separate importFlatManifest() and importTreeManifest() APIs, and updates the
tests to check both import mechanisms.
Reviewed By: wez
Differential Revision: D5365959
fbshipit-source-id: 3799ee3b937296e8025b666dd7176dbe7e268a31
Summary:
This is mostly needed because of the `folly:file` target split, but there are a few other that had changes that weren't reflected.
This also re-enables auto-deps for `multifeed/aggregator/processor/TARGETS` as it works again.
Reviewed By: yfeldblum
Differential Revision: D5362875
fbshipit-source-id: f9d2793249c0bfe644d0504f19839c108648ea3b
Summary:
Add a basic unit test that adds a new file, remounts the mount point, and
confirms the materialized contents are loaded from the overlay correctly on
remount.
Differential Revision: D5326024
fbshipit-source-id: d2446030802cc4afe5af09460d590ccf8c43e525
Summary: It doesn't need to exist anymore
Reviewed By: yfeldblum
Differential Revision: D5318746
fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.
A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from. The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.
Reviewed By: wez
Differential Revision: D5316000
fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
Summary: Add unit tests for HgImporter to better test this import logic.
Reviewed By: bolinfest
Differential Revision: D5309171
fbshipit-source-id: b5619d8271fef1cc31a0f642ec2fcbf9eccced54
Summary:
Now that the non-future versions of these APIs have been removed, rename
getBlobFuture() to getBlob(), and getTreeFuture() to getTree()
Reviewed By: wez
Differential Revision: D5295690
fbshipit-source-id: 30dcb88854b23160692b9cd83a632f863e07b491
Summary:
Remove the blocking getBlob() API.
There were a few call sites in FileData still using this blocking API. For now
I simply updated them to use getBlobFuture() and make a blocking get() call on
the returned future. These call sites already had TODO comments documenting
the blocking behavior.
I plan to rename getBlobFuture() to getBlob() in a subsequent diff.
Reviewed By: wez
Differential Revision: D5295726
fbshipit-source-id: 748fd7a140b9b59da339a330071f732bba38cb35
Summary:
Remove the blocking getTree() API. All call sites are using getTreeFuture()
instead now.
I plan to rename getTreeFuture() to getTree() in a subsequent diff.
Reviewed By: wez
Differential Revision: D5295725
fbshipit-source-id: 6b40b4c808da94a9c68decae3ce38c7d13fbe9f5
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
Summary:
This is generated by applying clang-tidy `-checks=modernize-use-override` to all the c++ code in project eden.
It enforces the use of the keywords `virtual`, `verride` and `final` in a way compliant to the style guide.
Reviewed By: igorsugak
Differential Revision: D5108807
fbshipit-source-id: 596f2d73f1137de350114416edb1c37da5423ed5
Summary: The `nodiscard` attribute was added to standard C++ in C++17. Prefer the standard formulation when it is available; otherwise, use `__attribute__((__warn_unused_result__))` on compilers that support the GNU extensions, and `_Check_return_` on MSVC. The old `FOLLY_WARN_UNUSED_RESULT` is now expands to `FOLLY_NODISCARD`.
Reviewed By: yfeldblum
Differential Revision: D5105137
fbshipit-source-id: 9aa22e81cd9f0b89f9343433aeae3ba365227ccb
Summary:
Update the eden thrift APIs to support multiple thrift parents, and update the
hg extension to use the new thrift APIs.
Reviewed By: bolinfest
Differential Revision: D4949139
fbshipit-source-id: 98a7bd17ccad7be6a429df34ecfaf3fe7ae0b39a
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout(). The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.
This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().
Reviewed By: bolinfest
Differential Revision: D4935943
fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.
Reviewed By: bolinfest, wez
Differential Revision: D4889053
fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
Summary:
This updates all of the references to gtest and gmock with googletest.
The change is mechanilcal, generated with the following one-liner:
```lang=bash
hg grep -lwE '(gtest|gmock)' 'glob:**/TARGETS' | grep -v '^third-party-buck' | xargs perl -pi -e '
$gt=qr!(["'"'"'])gtest\g1!;
(
s!$gt(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gtest\1\),!g or
s!$gt((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gt,?\s*\),?!\(\1googletest\1, None, \1gtest\1\),!g or
s!$gt,?!\(\1googletest\1, None, \1gtest\1\),!g
) unless /(name|type) *=/;
$gm=qr!(["'"'"'])gmock\g1!;
(
s!$gm(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gmock\1\),!g or
s!$gm((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gm,?\s*\),?!\(\1googletest\1, None, \1gmock\1\),!g or
s!$gm,?!\(\1googletest\1, None, \1gmock\1\),!g
) unless /(name|type) *=/;
'
```
Reviewed By: meyering
Differential Revision: D4643237
fbshipit-source-id: fda7f41760c7e44254231df87634631c343e6355
Summary:
Tweaks the stats stuff so that we can name the histograms and export them with p50, p90, p99 percentiles.
This is made a bit ugly because the fb303 stats code isn't yet open source, so
there's a moderately ugly preprocessor directive that we assume to be true for
our internal build. We'll need to force that to be 0 for the open source build
until we open the stats code and make it available.
Reviewed By: bolinfest
Differential Revision: D4801868
fbshipit-source-id: 643909e63bd4a74b2cfa580be131f65c5673bc94
Summary:
It's not really magic because we don't have a virtual directory
inode base any more. Instead, we mkdir and populate it at mount time.
What is slightly magical about it is that we give it some special powers:
* We know the inode number of the eden dir and prevent unlink operations
on it or inside it.
* The .eden dir is present in the contents of the root inode and will
show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
magic `.eden` inode number. This means that it is possible to `stat`
and consume the `.eden` directory from any directory inside the eden
mount, even though it won't show up in `readdir` for those child dirs.
The contents of the `.eden` dir are:
* `socket` - a symlink back to the unix domain socket that our thrift
server is listening on. This means that it is a simple
`readlink(".eden/socket")` operation to discover both whether a directory
is part of an eden mount and how to talk to the server.
* `root` - a symlink back to the root of this eden mount. This allows
using `readlink(".eden/root")` as a simple 1-step operation to find
the root of an eden mount, and avoids needing to walk up directory
by directory as is the common pattern for locating `.hg` or `.git`
dirs.
Reviewed By: simpkins
Differential Revision: D4637285
fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
Summary:
This begins adding an EdenMount::diff() method, which walks through the inode
tree and reports differences from the current source control tree state.
My intent is to eventually update the Dirstate code to use this diff()
function, and to remove the existing getModifiedDirectories() logic. The
getModifiedDirectories() code is currently incorrect, as it reports
materialized directories, which is not the same as directories that are
modified from the current source control state.
The ignore processing logic is not currently implemented in EdenMount::diff(),
but it should be relatively straightforward to add in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D4726048
fbshipit-source-id: ad0bb3b5d72bb3830f60fc2b2e56a81217c35353
Summary:
Refactor FileInode and FileData so that they no longer store the
TreeInode::Entry that refers to this file. The TreeInode::Entry is owned by
the parent TreeInode and is not safe to access without holding the TreeInode
contents_ lock, which the FileInode code never did. Additionally, once a
FileInode is unlocked the TreeInode::Entry is destroyed. Previously the
FileInode code would retain its pointer to this now invalid data, and could
still dereference it.
In the future I think it might be worth refactoring the FileInode/FileData
split a bit more, but this at least addresses the issues around invalid
accesses to entry_.
Reviewed By: bolinfest
Differential Revision: D4688058
fbshipit-source-id: 17d5d0c4e756afc6e3c4bb279cb1bb5231f3134f
Summary:
Loading the child inode is necessary to avoid race conditions with other
threads that may be currently trying to load the child inode in question.
This updates the code so that rmdir() and unlink() now share most of their
implementation.
Reviewed By: bolinfest
Differential Revision: D4664110
fbshipit-source-id: 871ca7b3c22819b24bc2639d9550bb45f6541f50
Summary:
Replace the one very basic checkout unit test with several tests that
more thoroughly test file modification changes. More tests for
directory changes and file type changes still need to be added, but this is a
first step for now. I will continue adding more tests in upcoming diffs.
Reviewed By: wez
Differential Revision: D4641595
fbshipit-source-id: 443a68e4e3c673fe687446f3469f035b4f48f1b3
Summary:
Add a method to reset the current commit without changing the working
directory state, similar to "hg reset" and "git reset --soft".
This also adds a new EXPECT_FILE_INODE() check for use in the unit tests,
and FileInode::getPermissions() and FileData::readFull() methods to support
this check.
Reviewed By: wez
Differential Revision: D4641476
fbshipit-source-id: 1e516774fe8e292a8d82cc2c354619374a3abe37
Summary:
The FakeTreeBuilder class in D4609587 provides a lot of the same functionality
as the TestMountBuilder class, but is not restricted to being used only at
mount initialization time. (It also provides more powerful functionality for
controlling the ordering of events when loading data from the ObjectStore.)
This switches all of the existing tests to use FakeTreeBuilder rather than
TestMountBuilder, and removes the TestMountBuilder class.
One difference is that FakeTreeBuilder adds data to the FakeBackingStore, while
TestMountBuilder previously put data directly into the LocalStore. This
shouldn't really make much difference for the behavior of the code. Putting
data in the FakeBackingStore gives us more control over when data is available
during load operations, and exercises slightly more of the normal ObjectStore
code path.
Reviewed By: wez
Differential Revision: D4641288
fbshipit-source-id: ca2b45bf69bd373848c12a7b739b286aabe6aa9b
Summary:
Add a FakeTreeBuilder class to make it easier to create Tree structures in the
FakeBackingStore using path names. Previously the test code had to manually
chain together Trees and Blobs. FakeTreeBuilder lets callers call setFile()
with a path name to define the file layout. The files that were defined then
get converted into Tree and Blob data when finalize() is called.
This is similar to the existing TestMountBuilder::addFile() API. However,
TestMountBuilder only populates the LocalStore. FakeTreeBuilder populates
the FakeBackingStore instead. The FakeBackingStore class allows tests to have
much more control about when objects appear ready from the backing store,
allowing control over the order in which Futures are fulfilled during the test.
Reviewed By: bolinfest
Differential Revision: D4609587
fbshipit-source-id: 9642d17daf0dc10f08901e51552bbb3c3e150b53
Summary:
Remove AutoHeaders.RECURSIVE_GLOB from all of our TARGETS files. This is a
Facebook-internal directive that was added as part of a codemod. The vanilla
open source buck does not know about this directive, and can't parse TARGETS
files that contain it.
All of the affected rules are cpp_unittest() and cpp_binary() rules that don't
actually have any header files, so these parameters didn't actually have any
effect.
Reviewed By: andrewjcg
Differential Revision: D4633748
fbshipit-source-id: 6227fe9b7f2ea838a0c72408ca195685bcae9f5a
Summary:
This is the initial code for implementing checkout.
This isn't quite 100% implemented yet, but I think it's worth checking in this
code as-is, and getting the remaining functionality done in separate diffs.
In particular, a few operations aren't implemented:
- Removing a directory that was deleted in the new revision
- Replacing a directory that was replaced with a file or symlink in the new
revision
- When doing a forced update, replacing a file or directory that did not exist
in the old revision, but that was created locally in the working directory,
and also exists in the new revision.
Reviewed By: wez
Differential Revision: D4538516
fbshipit-source-id: 5bb4889b02f23ab2048fcae2c8b7614340181aa6
Summary:
Refactor the Overlay code to store data using inode numbers rather than the
affected file's path in the repository. This simplifies the TreeInode code a
bit, as we no longer have to rename overlay files to stay in sync with the file
paths. This also eliminates some crashes when trying to update overlay files
for inodes that have been unlinked (and hence no longer have a path). This
also includes a few fixes to avoid writing journal entries for unlinked files
too. Additionally this contains a few fixes to how mode bits are stored in the
overlay, and fixes a bug where create() was ignoring the mode argument.
Reviewed By: wez
Differential Revision: D4517578
fbshipit-source-id: c1e31497dcf62c322b0deff72b0a02675b0509ab
Summary:
InodeBase::getNodeId() already returns the inode number, so there is no need
for the TreeInode subclass to provide an alternatively-named method that does
the same thing.
Reviewed By: bolinfest
Differential Revision: D4500215
fbshipit-source-id: fbc595cf9e8f43efe44ed352a97e3a4974b5aed0
Summary:
This updates the TreeInode::rename() code to handle concurrency better.
In particular:
- The code now ensures that both the source inode being renamed and destination
inode (if it exists) are loaded. This simplifies issues when an inode is
being loaded at the same time a rename is in progress. This ensures that any
pending load is processed before the rename takes place. (All promises for
the load might not be fulfilled before the rename completes, but the relevant
TreeInode and InodeMap data structures are updated before the rename occurs.)
This does mean that the rename code potentially might have to retry several
times if the inode it began loading is no longer the affected source or
destination or child once the load completes. However, this seems like a
reasonable trade-off, compared to dealing with the complications that would
arise with the load code having to handle renames occuring before load
completion.
- The code now implements proper lock ordering, to avoid acquiring locks in
conflicting orders that might cause deadlock with other threads also trying
to acquire the same locks. The InodeLocks.md document has been updated to
clarify the TreeInode lock ordering requirements.
Reviewed By: wez
Differential Revision: D4493526
fbshipit-source-id: 627393fafad90eb551aea62be7762d59ed043abe
Summary:
Rename the current EdenMount::getInodeBase() function to
EdenMount::getInodeBaseBlocking() and add a new getInodeBase() function that
performs the lookup asynchronously and returns a Future<InodePtr>.
This is implemented with a TreeInode::getChildRecursive() function that allows
asynchronous recursive lookups at any point in the tree.
Reviewed By: bolinfest
Differential Revision: D4427855
fbshipit-source-id: aca55e681a48c848b085b7fc6a13efe6cf0a4e3a
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function. This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.
This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.
Reviewed By: wez
Differential Revision: D4360558
fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
Summary: I fixed an unrelated bug in HTE that made some more paths lintable - and some more move-abuses visible :-(
Reviewed By: meyering
Differential Revision: D4397732
fbshipit-source-id: 560db05f64ecf160e8198caf80dbd988e0dad75d
Summary:
Rename the existing TestBackingStore class to FakeBackingStore, and fill it out
with an implementation that allows test code to control the store.
The test code can populate the store with Trees and Blobs to return, and can
control when the Futures returned by the store are fulfilled.
Reviewed By: bolinfest
Differential Revision: D4338577
fbshipit-source-id: 79221b04d844bd6011078b799e55182de4ccdfdc
Summary:
Add a helper function to create human-readable hashes in test code without
having to always specify a full 40-byte long string.
Reviewed By: wez
Differential Revision: D4336161
fbshipit-source-id: cf6af58dd788b5553a2a6daef56db43cddbbc04a
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.
This updates the main program to process path arguments using canonicalPath().
Reviewed By: bolinfest
Differential Revision: D4332273
fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.
For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready. I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.
Reviewed By: bolinfest
Differential Revision: D4318055
fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
Summary:
Move the FakeObjectStoreTest class into fs/testharness, along with the
TestMount and TestBackingStore classes. This simply consolidates the test
utility code into a single location.
Reviewed By: bolinfest
Differential Revision: D4317517
fbshipit-source-id: 4e19590c5ffde88b66f2c8d4a964352ec349031c
Summary:
Define InodePtr, TreeInodePtr, and FileInodePtr as aliases for std::shared_ptr
of the underlying inode type. This also updates all of the code to use these
new type names.
This will make it easier swap out std::shared_ptr with a custom pointer type in
the future. (I believe we will need a custom type in the future so that we
can have more precise control of the reference counting so we can load and
unload Inode objects on demand. std::shared_ptr::unique() doesn't quite
provide the flexibility we need, and is also being deprecated in C++17.)
Reviewed By: bolinfest
Differential Revision: D4297791
fbshipit-source-id: 1080945649290e676f62689592159f1166159b20
Summary:
Update the TestMount APIs to use the term "TreeInode" instead of "DirInode" now
that the DirInode class is no more.
Reviewed By: bolinfest
Differential Revision: D4257171
fbshipit-source-id: 2407dfc25405f7161987260c0299f1723831e264
Summary:
Now that the EdenDispatcher class has been moved into eden/fs, we no longer
need the distinction between TreeInode and fusell::DirInode, and FileInode and
fusell::FileInode.
This diff deletes the fusell versions of these classes, and updates all of the
code to always directly use TreeInode and FileInode. This allows us to get rid
of the remaining dynamic_casts between these pairs of classes.
Reviewed By: bolinfest
Differential Revision: D4257165
fbshipit-source-id: e2b6f328b9605ca0e2882f5cf7a3983fb4470cdf
Summary:
Move the InodeDispatcher class out of the lower-level fusell namespace in
eden/fuse and into the higher-level eden code in eden/fs/inodes. I also
renamed it from InodeDispatcher to EdenDispatcher, in anticipation of it
getting more eden-specific functionality in the future.
The fusell::MountPoint class is now independent of the Dispatcher type, and can
work with any Dispatcher subclass. Previously the MountPoint class was
responsible for owning the InodeDispatcher object. Now its caller (EdenMount
in our case) is responsible for supplying a Dispatcher object that is owned
externally.
Several parts of EdenDispatcher had to be updated as a result of the namespace
move, but I tried to keep this change somewhat minimal. I did update it from
using fusell::DirInode and fusell::FileInode to eden's TreeInode and FileInode
classes directly. However, there still remains more clean-up work to do. I
will split remaining changes out into upcoming diffs.
Reviewed By: bolinfest
Differential Revision: D4257163
fbshipit-source-id: dc9c2526640798f9f924ae2531218ba2c45d1d0a
Summary:
Move getInodeBaseForPath(), getDirInodeForPath(), and getFileInodeForPath()
entirely into the EdenMount class, and make sure all call sites are using the
EdenMount methods rather than the MountPoint methods.
Reviewed By: bolinfest
Differential Revision: D4257153
fbshipit-source-id: d528cfad174757c3c9f23e62a0616f8bf1976da7
Summary:
Update all code in eden/fs to call EdenMount::getDispatcher() instead of
getting the underlying MountPoint from the EdenMount and then calling
getDispatcher() on it. This will allow me to move the InodeDispatcher from
MountPoint to EdenMount in a subsequent diff. This also simplifies many of the
callers of this method.
Additionally, add an EdenMount::getRootInode() method, and update call sites to
use this rather than having to look up the InodeDispatcher and call
getRootInode() or getDirInode(FUSE_ROOT_ID) on it.
Reviewed By: bolinfest
Differential Revision: D4257152
fbshipit-source-id: 33e6f6b8853db2a88f4f2c221122eea50e796390
Summary:
This cleans up construction of the EdenMount and Dirstate objects:
- The EdenMount constructor is now responsible for creating the Overlay and
Dirstate objects.
- The Dirstate constructor is now responsible for loading the
DirstatePersistence file.
- The EdenMount now takes ownership of the ClientConfig object, and stores it
for later use.
- The ClientConfig object now has a method to get the path to the
DirstatePersistence file.
- I added a ClientConfig::createTestConfig() method, so that the TestMount code
can now use the same EdenMount constructor as the normal code.
This simplifies the logic in EdenServiceHandler and TestMount, and makes some
of the initialization dependencies a little bit simpler.
This change is necessary in order for me to move some logic from
fusell::MountPoint into EdenMount. The Dirstate object will need a pointer
back to its EdenMount object, and this diff enables that.
Reviewed By: bolinfest
Differential Revision: D4249393
fbshipit-source-id: 439786accbf48c8696dbc6ca4fe77a4c6bdeab65
Summary:
Rename TreeEntryFileInode to FileInode, and TreeEntryFileHandle to FileHandle.
These class names were long and awkward.
It's slightly unfortunate that we now have classes named both
eden::fuse::FileInode and eden::fuse::fusell::FileInode, but I don't believe
this should cause any major problems. If we want to eliminate these name
collisions in the future I would advocate for renaming the fusell versions to
something like "FileInodeIface".
Reviewed By: bolinfest
Differential Revision: D4217909
fbshipit-source-id: 899672a318d7ae39595f2c18e171f8fd6cebedc6
Summary:
Previous to this commit, the `Dirstate` logic only worked correctly when the
changes occurred in the root directory. Obviously that is very limiting, so this
adds support for changes in arbitrary directories at arbitrary depths.
This also introduces support for things like a file being replaced by a
directory of same name or vice versa. The tests have been updated to verify
these cases.
One interesting design change that fell out of this was the addition of the
`removedDirectories` field to the `DirectoryDelta` struct. As you can see,
all entries in a removed directory need to be processed by the new
`addDeletedEntries()` function. These require special handling because deleted
directories do not show up in the traversal of modified directories.
In contrast, new directories do show up in the traversal, so they require a
different type of special handling. Specifically, this call will return `NULL`:
```
auto tree = getTreeForDirectory(directory, rootTree.get(), objectStore);
```
When this happens, we must pass an empty vector of tree entries to
`computeDelta()` rather than `&tree->getTreeEntries()`. Admittedly, the special
case of new directories is much simpler than the special case of deleted ones.
Reviewed By: simpkins
Differential Revision: D4219478
fbshipit-source-id: 4c805ba3d7688c4d12ab2ced003a7f5c19ca07eb
Summary:
This also introduces the change where the `EdenMount` creates
and takes ownership of the `Dirstate`.
To clean some of this up, I had to expose a `getEdenDir()` method on `EdenServer`
that returns an `AbsolutePathPiece`. This was previously stored internally as a
`std::string`, so I had to clean up a bunch of path construction that was using `edenDir_`.
Reviewed By: simpkins
Differential Revision: D4123763
fbshipit-source-id: 270b182521c1a84bb054832f4b5f92af849d67e4
Summary:
Previously, `Dirstate` took a `std::shared_ptr<EdenMount>`, but now it takes
pointers to a `MountPoint` and an `ObjectStore` because it does not need the
entire `EdenMount`. Ultimately, this will enable us to have `EdenMount` create
the `Dirstate` itself, but that will be done in a follow-up commit.
Fortunately, it was pretty easy to remove the references to `edenMount_` in
`Dirstate.cpp` and rewrite them in terms of `mountPoint_` or `objectStore_`.
The one thing that I also decided to move was `getModifiedDirectoriesForMount()`
because I already needed to create an `EdenMounts` file (admittedly not a
great name) to collect some utility functions that use members of an `EdenMount`
while not having access to the `EdenMount` itself.
As part of this change, all of the code in `eden/fs/model/hg` has been moved to
`eden/fs/inodes` so that it is alongside `EdenMount`. We are going to change
the `Dirstate` from an Hg-specific concept to a more general concept.
`LocalDirstatePersistence` is no longer one of two implementations of
`DirstatePersistence`. (The other was `FakeDirstatePersistence`.) Now there is
just one concrete implementation called `DirstatePersistence` that takes its
implementation from `LocalDirstatePersistence`. Because there is no longer a
`FakeDirstatePersistence`, `TestMount` must create a `DirstatePersistence` that
uses a `TemporaryFile`.
Because `TestMount` now takes responsibility for creating the `Dirstate`, it
must also give callers the ability to specify the user directives. To that end,
`TestMountBuilder` got an `addUserDirectives()` method while `TestMount` got a
`getDirstate()` method. Surprisingly, `TestMountTest` did not need to be updated
as part of this revision, but `DirstateTest` needed quite a few updates
(which were generally mechanical).
Reviewed By: simpkins
Differential Revision: D4230154
fbshipit-source-id: 9b8cb52b45ef5d75bc8f5e62a58fcd1cddc32bfa
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
Reviewed By: simpkins
Differential Revision: D4073653
fbshipit-source-id: dda1d8ea8d29aa071a31f8e2afab324f9109e9b2
Summary:
This is a utility that should be generally useful in creating test,
including the test of `TestMount` itself.
As you can see, this helped uncover a bug in the way we were
inserting blobs into `LocalStore`.
Reviewed By: simpkins
Differential Revision: D4073039
fbshipit-source-id: 42683fd0bfdb0a1e77df9324fcaa79091f45e83d
Summary:
This is not a one-liner and this is needed for the upcoming `Dirstate` class,
so moving this code to a place where it is more easily reusable.
Reviewed By: simpkins
Differential Revision: D4032001
fbshipit-source-id: 7d8d87802665ac2993ec0a3ac73c5f645fe4a1aa
Summary:
Performs a depth-first traversal of the overlay to find modified
directories and returns them in that order.
Reviewed By: simpkins
Differential Revision: D4025309
fbshipit-source-id: 09d8ed41b250dddbfb3fe545643ec3fd755a430e
Summary:
These are new helper methods we need to create test scenarios.
They will be used in upcoming revisions.
Reviewed By: wez
Differential Revision: D4046981
fbshipit-source-id: 9c66c456be57006173e4a65eed603de4a426a438
Summary: This should be useful for my upcoming unit tests for the Hg dirstate.
Reviewed By: simpkins
Differential Revision: D4013464
fbshipit-source-id: 46460186abfa104aa026894068cd160e52c94729
