Summary:
The higher level goal is to make it easier to deal
with the graceful restart scenario.
This diff removes the SessionDeleter class and effectively renames
the Channel class to FuseChannel. The FuseChannel represents
the channel to the kernel; it can be constructed from a fuse
device descriptor that has been obtained either from the privhelper
at mount time, or from the graceful restart procedure. Importantly
for graceful restart, it is possible to move the fuse device
descriptor out of the FuseChannel so that it can be transferred
to a new eden process.
The graceful restart procedure requires a bit more control over
the lifecycle of the fuse event loop so this diff also takes over
managing the thread pool for the worker threads. The threads
are owned by the MountPoint class which continues to be responsible
for starting and stopping the fuse session and notifying EdenServer
when it has finished. A nice side effect of this change is that
we can remove a couple of inelegant aspects of the integration;
the stack size env var stuff and the redundant extra thread
to wait for the loop to finish.
I opted to expose the dispatcher ops struct via an `extern` to
simplify the code in the MountPoint class and avoid adding special
interfaces for passing the ops around; they're constant anyway
so this doesn't feel especially egregious.
Reviewed By: bolinfest
Differential Revision: D5751521
fbshipit-source-id: 5ba4fff48f3efb31a809adfc7787555711f649c9
Summary:
This partially fixes up a perf problem when performing status when a large
number of inodes have been loaded but not materialized (eg: by `find /edenfs
-ls`).
For the FileInode case we'd end up requesting the SHA1 from the store
twice in parallel only to compare it and decide that the file has not
been changed(!)
The remediation is to cut this code over to calling `FileInode::isSameAs` so that
we can short-circuit some of this work. In addition, we can avoid loading
subtrees if we haven't materialized them and the hash matches up.
Reviewed By: simpkins
Differential Revision: D5783044
fbshipit-source-id: f40da3fadfcf8d9e19221d41e3a5a980454717db
Summary:
Before this change, `hg split` crashed complaining that `node` was a
`changectxwrapper` instead of a 20-byte hash when it was sent as `parent1`
of `WorkingDirectoryParents` in `resetParentCommits()`. Now we use `node()` to
get the hash from the `destctx` that we have already extracted via this line
earlier in `merge_update()`:
destctx = repo[node]
The change to `eden/hg/eden/__init__.py` eliminated the crash, but was
not sufficient on its own to make `hg split` work correctly. There was also a fix
required in `Dirstate.cpp` where the `onSnapshotChanged()` callback was clearing out
entries of both `NotApplicable` and `BothParents` from `hgDirstateTuples`.
It appears that only `NotApplicable` entries should be cleared. (I tried leaving
`NotApplicable` entries in there, but that broke `eden/integration/hg/graft_test.py`.)
I suspected that the logic to clear out `hgDestToSourceCopyMap` in
`Dirstate::onSnapshotChanged` was also wrong, so I deleted it and all of the
integration tests still pass. Admittedly, we are pretty weak in our test coverage
for use cases that write to the `hgDestToSourceCopyMap`. In general, we should
rely on Mercurial to explicitly remove entries from `hgDestToSourceCopyMap`.
We have a Thrift API, `hgClearDirstate()`, that `eden_dirstate` can use to categorically
clear out `hgDirstateTuples` and `hgDestToSourceCopyMap`, if necessary.
Finally, creating a proper integration test for `hg split` required creating a value for
`HGEDITOR` that could write different commit messages for different commits.
To that end, I added a `create_editor_that_writes_commit_messages()` utility as a
method of `HgExtensionTestBase` and updated its `hg()` method to take `hgeditor`
as an optional parameter.
Reviewed By: wez
Differential Revision: D5758236
fbshipit-source-id: 5cb8bf4207d4e802726cd93108fae4a6d48f45ec
Summary:
The serialized data for each file handle needs to be enough
to re-construct the handle when we load it into a new process later
on. We need the inode number, the file handle number that we communicated
to the kernel and a flag to let us know whether it is a file or a dir.
Note that the file handle allocation strategy already accomodates the
idea of migrating to a new process; we don't need to serialize anything
like a next file handle id number.
This doesn't implement instantiating the handles from the loaded state,
it is just the plumbing for saving and loading that state information.
Reviewed By: bolinfest
Differential Revision: D5733079
fbshipit-source-id: 8fb8afb8ae9694d013ce7a4a82c31bc876ed33c9
Summary:
We're not doing anything with this today. It's not
clear whether we should be doing sanity checks (eg: block attempts
to write to a handle that was opened only for reading) or whether
the kernel is going to do that for us, so I've broken this out
as a separate diff from the removal of FileData.
Reviewed By: bolinfest
Differential Revision: D5723064
fbshipit-source-id: b73452dfb4edf88b57fef1ad604bb2bde93bacc1
Summary: These don't exist any more, so remove them
Reviewed By: bolinfest
Differential Revision: D5722861
fbshipit-source-id: 7db112dfab1dfdcf517452b314bd912ec8760bd1
Summary:
Added new tool to report stat information of EdenFs like fuse counters, Memory counters, latencies, Inode status for all the mount points etc.
eden stat : Prints the general information about eden like list of mount points, loaded unloaded and materialized inodes in each mount point. Also this reports how well periodic unload job is doing by reporting the number of unloaded inodes by periodic job.
eden stat io : Prints how many number of calls made to a system call in Edenfs.
eden stat memory : returns the memory stat for edenfs.
eden stat latency : reports the latencies of system calls in Edenfs.
Reviewed By: bolinfest
Differential Revision: D5660345
fbshipit-source-id: 97a1c2b83a6d8df0cd1b82c4d54b52d7ebd126bd
Summary:
This test was supposed to be a part of D5627411 but it was causing strange behaviour so was brought to a separate diff for further investigation.
After investigating, the test didn't pass because the UnloadedInodeData struct only contained the name of the file, not the path to it. The fix for this was to implement a way to get the relative path of the file even after the inode is unloaded.
Reviewed By: simpkins
Differential Revision: D5646929
fbshipit-source-id: f166398a651e8aea49da7e4474a5ad7fde2eaa4e
Summary:
1.Modified `TreeInode::unloadChildrenNow()` to return number of inodes that have been unloaded.
2.Modified `EdenServiceHandler::unloadInodeForPath()` to return number of inodes that are unloaded.
Reviewed By: simpkins
Differential Revision: D5627539
fbshipit-source-id: 4cdb0433dced6bf101158b9e6f8c35de67d9abbe
Summary:
When Hg tells the `dirstate` to `clear()`, we should also clear out any data we
have on the server for the `Dirstate`.
As it stands, the way we subclass `dirstate`, it does not appear like `clear()`
should be called, in practice, though one thing that could call it is
`hg debugrebuilddirstate`. It is probably good for us to have an RPC lying
around that we can use to reset the `Dirstate.`
Reviewed By: wez
Differential Revision: D5675298
fbshipit-source-id: 38926cfd93f4f83e4c28910f812a693cb32e423a
Summary:
Previously, we were overloading `hgSetDirstateTuple()` to also make it possible
to delete an entry from the internal `hgDirstateTuples` map. Now we have an
explicit method to do this, which enables us to remove some hacks/TODOs.
Reviewed By: simpkins
Differential Revision: D5665170
fbshipit-source-id: bc0753d4990c8966fd9e6c50b29a954d5023292f
Summary: Modified `TreeInode::unloadChildrenNow` such that inodes are unloaded whose age is greater than a specific age.
Reviewed By: simpkins
Differential Revision: D5526137
fbshipit-source-id: 91e2364d55e31befedcf43d98c26467e1a472ef9
Summary:
Update all of the code using ThreadLocal<EdenStats> to pass in a non-default
Tag parameter to the ThreadLocal template.
A non-default tag parameter is required to use the accessAllThreads() method on
the ThreadLocal object. We need to use accessAllThreads() to perform stats
aggregation correctly. Currently the EdenServer code is only performing
aggregation for stats in the FunctionScheduler.
This diff only updates the ThreadLocal<EdenStats> type, but does not contain
any behavior changes. I will fix the stats aggregation in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D5657268
fbshipit-source-id: bc4b6f56324eb8d3052c023fd3f6f64f99b1d4e0
Summary:
Updated time stamps of TreeInode accurately on mkdir,rmdir,mknode,symlink,create,unlink and readdir.
updated the `TreeInode::getattr` function to return in-memory timestamps.
Reviewed By: simpkins
Differential Revision: D5568183
fbshipit-source-id: c36f7fb767cd4342aab5cc983eea56e37cd2077e
Summary: Removed creation time from `FileInode::state` which was used for getting timestamps of files that are not materialized.Now that we added timestamps to file inodes and tree inodes we no longer need this.
Reviewed By: wez
Differential Revision: D5552761
fbshipit-source-id: 6013b1f694045e08ada7bd64114c4f2e52848fef
Summary:
updating atime,ctime,mtime of FileInode on read, write and setattr system calls.
modified `FileInode::stat` function to return accurate inmemory timestamps.
Reviewed By: simpkins
Differential Revision: D5552666
fbshipit-source-id: 86d446f72908663f8db509b7b789d9f35d17df3a
Summary:
Added `TreeInode::setInodeAttr` a helper function used in `InodeBase::setattr`. Also,added `InodeBase::setAtime` ,`InodeBase::setMtime` and implemented them in `FileInode` and `TreeInode`.
Moved updating timestamp logic to `InodeBase::setattr` from `FileInode::setInodeAttr` and `TreeInode::setInodeAttr`.
Reviewed By: simpkins
Differential Revision: D5545422
fbshipit-source-id: 597cfabb3062166a058cf32776acb50a1bc0c61c
Summary: Removed `FileInode::setattr` from `FileInode` and added a helper function `setInodeAttr` to perform FileInode or TreeInode specific setattr operations in `InodeBase::setattr`. This diff contains implementation of setattr for FileInode i.e for files, will add setattr implementation for directories in another diff.
Reviewed By: simpkins
Differential Revision: D5544968
fbshipit-source-id: 089491d07a603e111966987ef390b6e597aba28c
Summary:
We're seeing that this is always set to true for eden,
which is causing buck to run slower than it should.
To make this work correctly, I've augmented our journal data structure
so that it can track create, change and remove events for the various
paths.
I've also plumbed rename events into the journal.
This requires a slightly more complex merge routine, so I've refactored the two
call sites that were merging in slightly different contexts so that they can
now share the same guts of the merge routine. Perhaps slightly
counterintuitive in the merge code is that we merge a record from the past into
the state for now and this is a bit backwards compared to how people think.
I've expanded the eden integration test to check that we don't mix up
create/change/removes for the same path in a given window.
On the watchman side, we use the presence of the filename in the createdPaths
set as a hint that the file is new. In that case we will set the watchman
`ctime` (which is not the state ctime but is really the *created clock time*)
to match the current journal position if the file is new, or leave it set
to 0 if the file is not known to be new. This will cause the `is_new`
flag to be set appropriately by the code in `watchman/query/eval.cpp`;
if the sequence is 0 then it should never be set to true. Otherwise (when
the file was in the `createPaths` set) it will be set to the current journal
position and this will be seen as newer than the `since` constraint on
the query and cause the file to show as `new`.
Reviewed By: bolinfest
Differential Revision: D5608538
fbshipit-source-id: 8d78f7da05e5e53110108aca220c3a97794f8cc2
Summary:
1. Added a new structure `InodeBase::InodeTimestamps` to wrap atime,ctime,mtime together. This new structure helps in avoiding usage of `struct stat` for timestamps.
2. Modified function `Overlay::openFile` ,`Overlay::updateTimestampToHeader`, `Overlay::deserializeOverlayDir`, `Overlay::parseHeader` to use this new structure for timestamps instead of `struct stat`. Also, modified code in places where this change is being affected.
3. Added new helper methods `FileInode::setattrTimes` and `TreeInode::setattrTimes` to set timestamps in FileInode and TreeInode during setattr. Implementation of setattr for FileInode and TreeInode is in the diffs stacked above this diff.
4. Replaced atime, ctime, mtime in `FileInode::State`, `TreeInode::Dir` to `FileInode::State::timeStamps` and `TreeInode::State::timeStamps`. Made other necessary changes to support this change.
Reviewed By: simpkins
Differential Revision: D5596854
fbshipit-source-id: 2786b7b695508a62fdf8f7829f1ce76054b61c52
Summary: Currently we have two functions `FileInode::setattr` and `FileInode::setAttr` which are used to set given attributes to a `FileInode`. Merged both the functions in to one function called `FileInode::setattr` and removed `FileInode::setAttr`.
Reviewed By: wez
Differential Revision: D5538490
fbshipit-source-id: ec241fad25d6e4694865e5fc3c0a3500e4838bdd
Summary:
Added a new function `InodeBase::updateOverlayHeader` and implemented `FileInode::updateOverlayHeader` and `TreeInode::updateOverlayHeader` to update inmemory timestamps to overlay header when an inode is unreferenced.
Added helper functions in `Overlay` class to read and update timestamps in to the overlay file. Also,modified `Overlay::loadOverlayDir` to read and populate timestamps from overlay header in to treeinode.
Modified constructor of `FileInode::state` to read timestamps from overlay file and to populate inode timestamps.
Added test case to check if time stamps are updated and read correctly on remount.
Fixed a lint warning in TARGETS file
Reviewed By: simpkins
Differential Revision: D5535429
fbshipit-source-id: f6b758f70101c65d316a35101aacc9a3363f7aed
Summary:
This fixes EdenServer::unmount() to actually wait for all EdenMount cleanup
to complete, and fixes unmountAll() to return a Future that correctly waits for
all mount points to be cleaned up.
Previously `unmount()` waited for the mount point to be unmounted from the
kernel, but did not wait for EdenMount shutdown to complete. Previously
EdenMount shutdown was not triggered until the last reference to the
shared_ptr<EdenMount> was released. This often happened in the FUSE channel
thread that triggered the mountFinished() call--it would still hold a
reference to this pointer, and would not release it until after
mountFinished() returns. As a result, when the main thread was shutting down,
`main()` would call `unmountAll()`, and then return soon after it completed.
Some FUSE channel threads may still be running at this point, still performing
`EdenMount` shutdown while the main thread was exiting. This could result in
crashes and deadlocks as shutdown tried to access objects already destroyed by
the main thread.
With this change `EdenMount::shutdown()` is triggered explicitly during
`mountFinished()`, and `unmount()` will not complete until this finishes.
The `EdenMount` object may still exist at this point, and could still be
deleted by the FUSE channel thread, but the deletion now only requires freeing
the memory and does not require accessing other data that may have been cleaned
up by the main thread.
We should still clean up the FUSE channel thread handling in the future, to
make sure these threads are joined before the main thread exits. However, that
cleanup can wait until a separate diff. Ideally I would like to move more of
the mount and unmount logic from EdenServer and EdenServiceHandler and put that
code in EdenMount instead.
Reviewed By: bolinfest
Differential Revision: D5541318
fbshipit-source-id: 470332478357a85c314bc40458373cb0f827f62b
Summary: Changed the initialization of last checkout time in Tree inode constructor by grabbing a lock on `TreeInode::contents_`
Reviewed By: simpkins
Differential Revision: D5524857
fbshipit-source-id: e86ee7d986ca0c280ba156ba9146d6d1f9fa722e
Summary:
In testing, we discovered a case where concurrent Hg operations in EdenFS would
//sometimes// fail with `ECONNREFUSED` when trying to read
`<eden-mount>/.eden/socket`.
This was very confusing, as the standard reasons for `ECONNREFUSED` did not seem
to apply:
- We verified that Eden had not crashed.
- We verified that Eden's UNIX domain socket had a sufficiently large backlog
(1024) such that we should not be at risk of exhausting it with two simultaneous
Hg commands.
It turned out that there was a race condition in our `readlink()` command, which
could cause `<eden-mount>/.eden/socket` to resolve to the wrong path. Failing to
connect to this path manifested itself as an `ECONNREFUSED` error.
It turned out that `readlink()` used `FileInode::readAll()`, which was
performing an `lseek()` followed by a `read()` on a file descriptor while the
descriptor was protected by a //read lock// instead of a //write lock//. Because
the `lseek()` causes a state change, it needs to be protected by a write lock.
Changing the type of the lock fixed the issue.
The only other caller of `FileInode::readAll()` appears to be in
`TreeInode::loadGitIgnoreThenDiff()`, so presumably this fixes a possible race
condition there, as well.
Reviewed By: simpkins
Differential Revision: D5533001
fbshipit-source-id: 86cf84c45463b2ae194d6f46909ea67c0f71d065
Summary:
Make a few minor tweaks to the strace-style logging added in D5464387.
- Call the log categories "eden.strace.<mount_path>" instead of
"eden/strace<mount_path>". The folly logging library uses '.' to separate
nodes in the log category hierarchy, so this puts all of the strace messages
under the "eden.strace" category, which it itself part of the "eden"
category. The "<mount_path>" will contain slashes inside it, but slashes are
not treated specially in log category names.
- Rename `EdenMount::getLogger()` to `EdenMount::getStraceLogger()` since this
logger should be used only for strace-style events, and not for general log
messages for this mount point.
Reviewed By: bolinfest
Differential Revision: D5515245
fbshipit-source-id: 9d833d9fbff47c6a57a7afefeae92755ff0e28b7
Summary: Added atime,ctime,mtime for tracking timestamps for directories inmemory and initialized them to the last checkout time during the creation of TreeInode.
Reviewed By: bolinfest
Differential Revision: D5440950
fbshipit-source-id: 639cf1ce6722f80dde35d33849aa712aa30301a8
Summary:
Added a new data member lastCheckoutTime to EdenMount class to store the time when checkout operation is performed. Also added a method to get the last checkout time which internally returns the lastCheckoutTime in EdenMount class.
Added new fields atime,mtime,ctime in FileInode::State structure to keep track of timestamps in memory. Initialzed timestamps in FileInode::State constructor by calling getLastCheckOutTime from EdenMount class.
Still have to add timestamp tracking for directories and have to initialize them with lastCheckout time.This probably will be done in a seperate diff.
Reviewed By: bolinfest
Differential Revision: D5437682
fbshipit-source-id: e3d6b1bc0c2192538dd3b0d9a6017ceb3ca0843d
Summary:
Moved all the member functions from `FileData` class to `FileInode` class
and made `FileInode` methods independent of shared `FileData` object.
Removed `FileData.h` and `FileData.cpp` files as they are not needed anymore.
Modified functions `FileInode::getSHA1()` and `FileInode::isSameAsFast` and
modified few testcases which are currently using `FileData` class and made
sure that all the test cases are passing.
Reviewed By: bolinfest
Differential Revision: D5430128
fbshipit-source-id: 3e8e6c490e92e4e602355e4ce39b67c450ec53f8
Summary:
D5483953 added a check to ensure that getMode() can only be called on entries
that do not have a loaded inode. However, a few places in the code were still
calling getMode() on tree entries without checking if the inode was loaded or
not.
These crashes were caught in the integration tests run by `buck test eden/...`,
but were not caught by sandcastle tests on the original diff since sandcastle
only runs the eden unit tests, and not the integration tests.
All of these locations only needed to check the file type, which is safe to do
even if the inode is loaded, since the file type portion of the mode can never
change on an existing inode. Only the permissions bits are unsafe to access
once an inode has been loaded (since we need to ask the inode itself for the
latest permissions bits).
I updated these call sites to stop using getMode() and instead use functions
that explicitly check only the file type bits.
Reviewed By: bolinfest
Differential Revision: D5501256
fbshipit-source-id: c989dab2fdacb5b9cdecb6f5101795298f57e78b
Summary:
Before this commit, TreeInode::Entry was a struct which had two
private members: mode and inode. In this commit,
1. TreeInode::Entry was changed from struct to class.
2. Appropriate getters and setters were introduced for the public members to
make them private.
3. Existing code accessing the public members directly was modified to use
the getters instead.
4. A couple of TODOs were added to address Overlay::saveOverlayDir()'s access
of child inode information.
Reviewed By: simpkins
Differential Revision: D5483953
fbshipit-source-id: 50d526731e193a3a4a32742bd4d49deb9ee6b432
Summary: This uses `clang-tidy` to comment out unused parameters (in functions, methods and lambdas) in fbcode. Cases that the tool failed to handle are fixed manually.
Reviewed By: igorsugak
Differential Revision: D5454343
fbshipit-source-id: 5dee339b4334e25e963891b519a5aa81fbf627b2
Summary:
Log messages to an eden.strace category.
This allows us to enable/disable based on the mount_path. For example:
./buck-out/gen/eden/cli/cli.par daemon -F -- --logging eden/strace/data/users/ekent/eden-NEW=DBG7
Thus, we are using a category, rather than filename (default)
Reviewed By: bolinfest
Differential Revision: D5464387
fbshipit-source-id: 6a54badd6bb806cfcda1742fd970073d91303396
Summary:
1.Refactored FileInode and FileData code.
2.Moved some data members of file data into struct State inside file inode
3.Refactoring code such that FileData and FileInode classes are eventually moved to FileInode class.
Reviewed By: simpkins
Differential Revision: D5414427
fbshipit-source-id: cf24721a65541ddfdec7ead4a035de4da3fd5bb5
Summary:
- Updated makeShared() to return a folly::Future<std::shared_ptr<EdenMount>> instead of just a std::shared_ptr<EdenMount>.
- Updated callers to use the returned future (get() for now)
- Refactor makeShared() and the EdenMount constructor to avoid blocking.
Reviewed By: simpkins
Differential Revision: D5424088
fbshipit-source-id: f026a3a3e4abb3593bafda76673e12c55da26322
Summary: Added header to the files that are materialized .
Reviewed By: simpkins
Differential Revision: D5387990
fbshipit-source-id: 1d551d674a39e01d6314d8bb3308e7bea3e669fc
Summary:
This updates the TreeInode code to remove the redundant materialized flag.
A TreeInode should have a Tree Hash if and only if it is dematerialized, so
there is no need for an extra `materialized` boolean.
This diff also fixes an issue in TreeInode::saveOverlayPostCheckout() where it
was not correctly informing it's parent TreeInode of the change if it moved
from one dematerialized state to another (with a different TreeInode hash).
This fixes the code to correctly call `parent->childDematerialized()` when it
needs to inform the parent that it now refers to a different source control
hash.
Reviewed By: wez
Differential Revision: D5336629
fbshipit-source-id: b4d86ecdef2f5faefbc243a09d869c02384ae95c
Summary:
Add a command to deserialize and display information about files in the
overlay. This can be used to help debug the current state of files in the
overlay.
Reviewed By: wez
Differential Revision: D5332014
fbshipit-source-id: 25b01579df33aa9f1926c0144e9f03aa8ece38fc
Summary:
1.Added a new method to create header.
2.Added header to the overlay files of directories.
3.Added test class OverlayTest for Overlay related tests.
Reviewed By: simpkins
Differential Revision: D5335134
fbshipit-source-id: 31f59e7af70a3eeae6350261ded5d8b1bec2b9d0
Summary:
This is mostly needed because of the `folly:file` target split, but there are a few other that had changes that weren't reflected.
This also re-enables auto-deps for `multifeed/aggregator/processor/TARGETS` as it works again.
Reviewed By: yfeldblum
Differential Revision: D5362875
fbshipit-source-id: f9d2793249c0bfe644d0504f19839c108648ea3b
Summary:
[Folly] Remove `construct_in_place`.
It is an alias for `in_place`, and unnecessary.
Reviewed By: Orvid
Differential Revision: D5362354
fbshipit-source-id: 39ce07bad87185e506fe7d780789b78060008182
Summary:
Update the Overlay class to hold a lock on the info file inside the overlay for
as long as it exists.
This just adds an extra layer to help ensure that two separate edenfs processes
are not operating on the same overlay directory at the same time. We already
hold a lock on the .eden directory itself, which generally should be sufficient
protection, since overlay directories are always scoped to a .eden directory.
However, adding a per-Overlay lock will help ensure that we do not have issues
where we try to open an Overlay directory twice, particularly when remounting
an existing mount point that we just closed.
Differential Revision: D5326023
fbshipit-source-id: eb8b213225b8d6905a982db0bfac73a17d1bd246
Summary:
Add a basic unit test that adds a new file, remounts the mount point, and
confirms the materialized contents are loaded from the overlay correctly on
remount.
Differential Revision: D5326024
fbshipit-source-id: d2446030802cc4afe5af09460d590ccf8c43e525
Summary:
Add a main() function to the indoes unit test, to allow a `--logging` flag to
control log levels.
I will eventually update our default test `main()` function to include this
flag, but for now adding a custom `main()` for the inodes test is the simplest
fix.
Reviewed By: wez
Differential Revision: D5326022
fbshipit-source-id: 36f497658fdb21639408fc599cf75908b9c9acb3
Summary: It doesn't need to exist anymore
Reviewed By: yfeldblum
Differential Revision: D5318746
fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
Summary:
this code was present to help in some benchmarking
in the very early prototype code. I noticed that this was
still lingering when reviewing a diff the other day.
Remove it!
Reviewed By: simpkins
Differential Revision: D5334263
fbshipit-source-id: 9ecccf1f9922b4c8f46b2529da665e2fdf11ab7a
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.
A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from. The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.
Reviewed By: wez
Differential Revision: D5316000
fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.
Reviewed By: simpkins
Differential Revision: D5277870
fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
Summary:
Update eden to log via the new folly logging APIs rather than with glog.
This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else. (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)
Reviewed By: wez
Differential Revision: D5290783
fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
Summary:
Now that the non-future versions of these APIs have been removed, rename
getBlobFuture() to getBlob(), and getTreeFuture() to getTree()
Reviewed By: wez
Differential Revision: D5295690
fbshipit-source-id: 30dcb88854b23160692b9cd83a632f863e07b491
Summary:
Remove the blocking getBlob() API.
There were a few call sites in FileData still using this blocking API. For now
I simply updated them to use getBlobFuture() and make a blocking get() call on
the returned future. These call sites already had TODO comments documenting
the blocking behavior.
I plan to rename getBlobFuture() to getBlob() in a subsequent diff.
Reviewed By: wez
Differential Revision: D5295726
fbshipit-source-id: 748fd7a140b9b59da339a330071f732bba38cb35
Summary:
Remove the ObjectStores.h and .cpp files. These files contained helper
functions for looking up Tree and TreeEntry objects based on a deep relative
path.
These functions are no longer used anywhere anymore (the code that performs
deep lookups always does inode lookups instead). Additionally, this file is
the only place still using some of the blocking, non-Future-based APIs for
object lookups. Deleting these files will let us remove these deprecated
blocking APIs.
Reviewed By: wez
Differential Revision: D5295691
fbshipit-source-id: 89229827305490eba4d3a581954a90c5cf63238d
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
Summary:
This is generated by applying clang-tidy `-checks=modernize-use-override` to all the c++ code in project eden.
It enforces the use of the keywords `virtual`, `verride` and `final` in a way compliant to the style guide.
Reviewed By: igorsugak
Differential Revision: D5108807
fbshipit-source-id: 596f2d73f1137de350114416edb1c37da5423ed5
Summary: The `nodiscard` attribute was added to standard C++ in C++17. Prefer the standard formulation when it is available; otherwise, use `__attribute__((__warn_unused_result__))` on compilers that support the GNU extensions, and `_Check_return_` on MSVC. The old `FOLLY_WARN_UNUSED_RESULT` is now expands to `FOLLY_NODISCARD`.
Reviewed By: yfeldblum
Differential Revision: D5105137
fbshipit-source-id: 9aa22e81cd9f0b89f9343433aeae3ba365227ccb
Summary:
These files are defining flags, so they should import gflags.
This import being missing causes open source build to fail.
Reviewed By: bolinfest
Differential Revision: D4992246
fbshipit-source-id: 61eb35b43ae6f9ff88bd34a9ebc8db6d80ac9d08
Summary:
Update Dirstate::remove() to directly call TreeInode::unlink() instead of going
through the EdenDispatcher. Internal code should be able to use TreeInode
directly without going through the FUSE Dispatcher. This prevents us from
having to look up the TreeInode object by inode number again.
This also fixes TreeInode::unlink() and TreeInode::rmdir() to perform FUSE
cache invalidation correctly. EdenDispatcher used to do this for unlink()
calls, but it was missing for rmdir() calls.
Reviewed By: wez
Differential Revision: D4968832
fbshipit-source-id: 51fda5de196392c640436ca6ad7d8333e6799db9
Summary:
This replaces the shouldFileBeDeletedByHgRemove() function with a much simpler
and safer check.
It turns out this code was more complicated than it needed to be: it looked up
the inode in question and confirmed that it still existed, even though it's
calling function had already previously looked up the inode object.
I believe this deletes the last place in the code that still assumes that an
InodeBase object must be loaded if the inode has previously been materialized.
This also removes one of the last remaining places that directly holds the
TreeInode contents_ lock outside of TreeInode.cpp.
Reviewed By: bolinfest
Differential Revision: D4968834
fbshipit-source-id: a37bc0d7889eb81cca5b045cbc82732a53ef53d4
Summary:
This updates the Dirstate::addAll() code to use EdenMount::diff() internally,
instead of getStatusForExistingDirectory().
This lets us delete getStatusForExistingDirectory() and several other helper
functions that it used. The logic in getStatusForExistingDirectory() was based
on the incorrect assumption that files can only be modified from their expected
source control state if they are materialized. This could result in incorrect
results in a variety of cases (particularly after renames, or "hg reset"
operations). The EdenMount::diff() code does not have this problem.
That said, this new addAll() code does still have some performance issues--it
currently does a full tree diff for each input path. We should ideally fix
this in the future to only diff the necessary subtree for each path. However,
in the short term this trade-off seems worth being able to delete all of this
older, buggy code. diff() should be cheap enough in most cases that this won't
be a major problem unless a large number of paths are given as input.
Reviewed By: bolinfest
Differential Revision: D4968835
fbshipit-source-id: 1834aa98a26dcaa0e1c06c7ac25c57944fa1b5f7
Summary:
Make sure we use pointers to const GitIgnoreStack objects when possible.
Also change the code to store the root GitIgnoreStack object for a diff()
operation in the DiffContext object, instead of having EdenMount::diff()
allocate it separately on the heap. This will make it easier to consolidate a
bit more logic in the DiffContext class in the future. In particular, I
suspect we will want a version of diff() that only works on one portion of the
tree. Putting more of the functionality in DiffContext will make it slightly
easier to share code between the full-mount diff and the subtree diff
functions. This also happens to save a memory allocation for now.
Reviewed By: wez
Differential Revision: D4968833
fbshipit-source-id: 1dc33b3d44cdf00e93b22d810c3a736d27c13638
Summary:
Update the eden thrift APIs to support multiple thrift parents, and update the
hg extension to use the new thrift APIs.
Reviewed By: bolinfest
Differential Revision: D4949139
fbshipit-source-id: 98a7bd17ccad7be6a429df34ecfaf3fe7ae0b39a
Summary:
This updates the ClientConfig and EdenMount code to support storing two parent
commits.
This changes the on-disk SNAPSHOT file contents add an 8-byte header that
includes a file identifier and a file format version number, followed by up to
two commit hashes. The code currently can read either the old or new format
from the SNAPSHOT file. We should be able to drop the code for reading the old
format fairly soon if we want, though.
This diff only updates the ClientConfig and EdenMount code, and does not yet
update the thrift APIs or the eden mercurial extension yet. I will update the
rest of the code in a subsequent diff.
Reviewed By: bolinfest, wez
Differential Revision: D4943917
fbshipit-source-id: cf456e67b845aa0cf8b45c822985cb932df107b4
Summary: Add some unit tests that check the behavior of setattr() and getattr().
Reviewed By: bolinfest, wez
Differential Revision: D4941215
fbshipit-source-id: 6691f51aea742d0159a16112ca291546a8adc928
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout(). The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.
This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().
Reviewed By: bolinfest
Differential Revision: D4935943
fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
Summary:
Update the gitignore code so that patterns ending in a trailing slash correctly
match only directories. Fortunately we have file type information available
during TreeInode::computeDiff(), so getting the correct file type information
does not add any extra overhead. The older ignore checking code in
Dirstate.cpp can also do a reasonable job of getting file type information.
That code should also be removed at some point in the future, and use the
TreeInode logic for computing ignore status.
Reviewed By: bolinfest
Differential Revision: D4901326
fbshipit-source-id: 1222c8142876c91e1b80ec937ec84c0c28737224
Summary:
The temporary Hash object doesn't live long enough for the copy into
the string, so make an explicit temporary to extend its lifetime.
Reviewed By: simpkins
Differential Revision: D4900457
fbshipit-source-id: 948d919b929285c30ee1f9f0111cba20b60d4dc6
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.
Reviewed By: bolinfest, wez
Differential Revision: D4889053
fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
Summary:
Fix a subtle crash during checkout when handling newly added entries that
already exist in the working directory: CheckoutAction passed the entry name to
checkoutUpdateEntry() as a PathComponentPiece. However, this
PathComponentPiece could refer to the entry name owned by newScmEntry_, and it
also passed newScmEntry_ into checkoutUpdateEntry() as an rvalue reference.
As a result, if the string data was stored invalidated by the move the name
would no longer be valid when checkoutUpdateEntry() tried to use it.
This bug is triggered by doing an "hg update --clean", where a file added in
the destination commit already exists on disk, and has an entry name of 23
characters or less. (The 23 character limit is fbstring's upper bound on
small string optimizations, where it will store the string data inline in the
object, causing it to be invalidated on move.)
This also fixes a crash in a VLOG() statement when the verbose log level for
TreeInode.cpp was set to 4 or greater.
Reviewed By: bolinfest
Differential Revision: D4882544
fbshipit-source-id: 917ede6eeae2224aaa0724b8b30324f3c3a5c924
Summary:
The intent is to provide a way to locate the SNAPSHOT file
for tools that want to have a very fast way to figure out the commit
id without making any RPCs or subprocess invocations.
Reviewed By: simpkins
Differential Revision: D4824176
fbshipit-source-id: 5adca225d9984146852dad1e83de0d903848c1e5
Summary:
This simplifies the InodePtrImpl class by removing support for pointer-to-const
Inode objects. We don't currently use pointer-to-const Inode objects, and I
don't really anticipate that we will ever need this.
The primary motivating factor behind this diff is a recent change in clang that
breaks the existing code: clang PR31606 prevents children classes from
inheriting template constructor instantiations from their parent which take a
reference to the parent class type as the only argument.
While this could have been fixed by explicitly defining the necessary
constructors in the child class (or simply by overriding newPtrLocked() and
newPtrFromExisting() to return the subclass type), dropping support for
pointer-to-const allows us to simplify the code and resolve this issue.
Reviewed By: wez
Differential Revision: D4825526
fbshipit-source-id: 999de352788b13818b7f23788bb34686e193cd5d
Summary:
This diff fixes FileData::stat() so that we report reasonable timestamp values
on non-materialized files, rather than always leaving them as 0. We set the
timestamps to the time that we created the FileInode. This ensures that
timestamps are updated correctly when files are modified by a checkout
operation.
Note that for materialized files the code reports the timestamp of the overlay
file. This diff does not modify that behavior. However, this behavior is
incorrect, as the overlay file timestamps are not updated by a FUSE client
opening, modifying, then closing a file (since we keep the underyling overlay
file handle open, and don't close it).
In the future we'll need to implement our own tracking of atime, mtime, and
ctime values. We should probably store these in a header inside the overlay
file. For now, this diff is a stop-gap measure that ensures we at least update
non-materialized file timestamps correctly on checkouts.
Reviewed By: bolinfest
Differential Revision: D4765632
fbshipit-source-id: 478da6441e213cdfe830f1c5129212182ce4eeb0
Summary:
This diff adds a new "eden debug" command, with a few subcommands:
- "eden debug inode": Report information about currently loaded inodes
- "eden debug tree": Report data about source control Tree objects
- "eden debug blobmeta": Report metadata about source control Blob objects
- "eden debug blob": Report the contents of source control Blob objects
This diff also includes the thrift APIs in edenfs to support these commands.
Reviewed By: bolinfest
Differential Revision: D4760528
fbshipit-source-id: 7dc2bd6e0e952ba6fb61702c672fb9417d603ffa
Summary:
This updates all of the references to gtest and gmock with googletest.
The change is mechanilcal, generated with the following one-liner:
```lang=bash
hg grep -lwE '(gtest|gmock)' 'glob:**/TARGETS' | grep -v '^third-party-buck' | xargs perl -pi -e '
$gt=qr!(["'"'"'])gtest\g1!;
(
s!$gt(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gtest\1\),!g or
s!$gt((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gt,?\s*\),?!\(\1googletest\1, None, \1gtest\1\),!g or
s!$gt,?!\(\1googletest\1, None, \1gtest\1\),!g
) unless /(name|type) *=/;
$gm=qr!(["'"'"'])gmock\g1!;
(
s!$gm(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gmock\1\),!g or
s!$gm((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gm,?\s*\),?!\(\1googletest\1, None, \1gmock\1\),!g or
s!$gm,?!\(\1googletest\1, None, \1gmock\1\),!g
) unless /(name|type) *=/;
'
```
Reviewed By: meyering
Differential Revision: D4643237
fbshipit-source-id: fda7f41760c7e44254231df87634631c343e6355
Summary:
Tweaks the stats stuff so that we can name the histograms and export them with p50, p90, p99 percentiles.
This is made a bit ugly because the fb303 stats code isn't yet open source, so
there's a moderately ugly preprocessor directive that we assume to be true for
our internal build. We'll need to force that to be 0 for the open source build
until we open the stats code and make it available.
Reviewed By: bolinfest
Differential Revision: D4801868
fbshipit-source-id: 643909e63bd4a74b2cfa580be131f65c5673bc94
Summary:
this is the bare minimum to support creating unix domain sockets.
We only support using mknod to create a unix socket; other uses will yield an error.
I've added an rdev field as a sibling of the existing mode field that we track,
as that is the additional parameter that we need to track as part of the
special file node.
Special file nodes are tracked in the overlay as empty files.
Reviewed By: bolinfest
Differential Revision: D4774099
fbshipit-source-id: 0824b7e509063faa8bede7aff82a7c51930c4f83
Summary:
This is an initial pass at implementing "hg status" using the new EdenMount and
TreeInode diff() code.
More work still needs to be done to clean things up here, but this makes
"hg status" more correct in the face of modified files and directories that are
not modified, and vice-versa.
In particular, the following still needs to be cleaned up in future diffs:
- Update the "hg status" code to more correctly process user directives for
paths that did not appear in the diff output. This will probably be
simplified some by correctly updating the user directive list on checkout and
resetCommit() operations. It may also make things simpler if we store the
user directives in a hierarchical path map, rather than in a flat list.
- Update the addAll() code to also use the new diff logic, rather than the
getStatusForExistingDirectory() function.
- Clean up the locking some. I think it may be worth changing the code to use
a single lock to manage both the current commit ID for the mount and the
dirstate user directives. We probably should hold this lock for the duration
of a diff operation. The diff operation should also return the commit ID
that it is based against.
Reviewed By: wez
Differential Revision: D4752281
fbshipit-source-id: 6a6d7e2815321b09c75e95aa7460b0da41931dc0
Summary:
It's not really magic because we don't have a virtual directory
inode base any more. Instead, we mkdir and populate it at mount time.
What is slightly magical about it is that we give it some special powers:
* We know the inode number of the eden dir and prevent unlink operations
on it or inside it.
* The .eden dir is present in the contents of the root inode and will
show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
magic `.eden` inode number. This means that it is possible to `stat`
and consume the `.eden` directory from any directory inside the eden
mount, even though it won't show up in `readdir` for those child dirs.
The contents of the `.eden` dir are:
* `socket` - a symlink back to the unix domain socket that our thrift
server is listening on. This means that it is a simple
`readlink(".eden/socket")` operation to discover both whether a directory
is part of an eden mount and how to talk to the server.
* `root` - a symlink back to the root of this eden mount. This allows
using `readlink(".eden/root")` as a simple 1-step operation to find
the root of an eden mount, and avoids needing to walk up directory
by directory as is the common pattern for locating `.hg` or `.git`
dirs.
Reviewed By: simpkins
Differential Revision: D4637285
fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
Summary:
Update the EdenMount::diff() code to completely skip .hg and .eden directories.
This was implemented through the GitIgnore code, and adding a new HIDDEN status
The .hg and .eden directories are similar to ignored directories, but we never
want to report any information for them even when listIgnored is true.
This also changes the GitIgnore code to so that GitIgnoreStack computes the
basename for each path once, and passes it down into the underlying match code,
so we don't have to re-compute it for each pattern that needs the basename.
Reviewed By: wez
Differential Revision: D4751917
fbshipit-source-id: ec22d62f31a3ce9ef998a8ccb4413f72f1d7a487
Summary:
This updates the diff() logic to process gitignore rules.
The bulk of the existing TreeInode::diff() was moved into a
TreeInode::computeDiff() function. TreeInode::diff() now loads the gitignore
rules for this directory before calling computeDiff(). The argument list for
computeDiff() was now getting slightly unwieldy, so I created a new DiffContext
class to hold some of the arguments that are the same for the duration of the
entire diff operation. This resulted in some rather mechanical changes through
the computeDiff() and DeferredDiffEntry code to update the argument lists.
Reviewed By: wez
Differential Revision: D4744690
fbshipit-source-id: c4981d9f49878b522e0b3faf35de837739066f3c
Summary:
Add a unit test that modifies a file and then does a forced checkout to revert
back to the original commit state.
This also includes some pretty-printers to make the tests print
CheckoutConflict objects nicely if any test assertions fail.
Reviewed By: bolinfest
Differential Revision: D4727779
fbshipit-source-id: b31dc90be28d17ed0f2a4076c96a08ba82588190
Summary:
This begins adding an EdenMount::diff() method, which walks through the inode
tree and reports differences from the current source control tree state.
My intent is to eventually update the Dirstate code to use this diff()
function, and to remove the existing getModifiedDirectories() logic. The
getModifiedDirectories() code is currently incorrect, as it reports
materialized directories, which is not the same as directories that are
modified from the current source control state.
The ignore processing logic is not currently implemented in EdenMount::diff(),
but it should be relatively straightforward to add in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D4726048
fbshipit-source-id: ad0bb3b5d72bb3830f60fc2b2e56a81217c35353
Summary:
Update the TreeInode and CheckoutAction to correctly be able to remove
directories during checkout operations. This handles both removing directories
that are newly empty as a result of the checkout, and directories that are
replaced with a file or symlink.
This also refactors the rmdir/unlink code a bit so we can share some of its
logic for doing directory removal during the checkout operation.
Reviewed By: bolinfest
Differential Revision: D4708607
fbshipit-source-id: 3b9dd9cc3536845dad0d9278e3c5d3ac1ed04570
Summary:
Synchronize access to the CheckoutContext conflicts_ list when recording
conflicts or errors during a checkout operation.
In theory it is possible for the checkout operation to be proceeding
simultaneously in multiple threads. At the moment this does not happen because
the BackingStore implementations currently block, and always return
immediately-ready Future objects in the same thread. However, once they are
updated to return Future objects that actually complete asynchronously in other
threads the checkout work will continue in these other threads.
Reviewed By: bolinfest
Differential Revision: D4708605
fbshipit-source-id: 885b0114d97dadcacf64652f795a7d3846412f11
Summary:
Add a test that exercises a checkout operation that creates a brand new
subdirectory.
Reviewed By: bolinfest
Differential Revision: D4693994
fbshipit-source-id: 70f43f67c2cdb10bd854f51d5ba2e61c45976975
Summary:
Unfortunately it looks like gcc won't support the [[nodiscard]] attribute until
the gcc-7.x release. (The functionality is available in gcc-4.8 and later, but
using the name [[gnu::warn_unused_result]] instead.)
This changes the code to use the FOLLY_WARN_UNUSED_RESULT macro from
folly/Portability.h instead. (This expands to older non-standard __attribute__
syntax for gcc and clang, and _Check_return_ on Windows.)
Reviewed By: bolinfest
Differential Revision: D4704184
fbshipit-source-id: d5c13630ea611a2f43080a501add42ce9fda6789
Summary:
Add new ASSIGN_PARENT_INODE and ASSIGN_INODE types to LoadBehavior, to exercise
additional code paths during checkout operations. This also cleans up the
overall handling of the LoadBehavior enum.
Reviewed By: bolinfest
Differential Revision: D4694548
fbshipit-source-id: e8359cfdadb1fabb1a0d07bc57a1a1610ed9ba57
Summary:
Refactor FileInode and FileData so that they no longer store the
TreeInode::Entry that refers to this file. The TreeInode::Entry is owned by
the parent TreeInode and is not safe to access without holding the TreeInode
contents_ lock, which the FileInode code never did. Additionally, once a
FileInode is unlocked the TreeInode::Entry is destroyed. Previously the
FileInode code would retain its pointer to this now invalid data, and could
still dereference it.
In the future I think it might be worth refactoring the FileInode/FileData
split a bit more, but this at least addresses the issues around invalid
accesses to entry_.
Reviewed By: bolinfest
Differential Revision: D4688058
fbshipit-source-id: 17d5d0c4e756afc6e3c4bb279cb1bb5231f3134f
Summary:
Remove the TreeInode::entry_ member variable. This was a pointer to the
TreeInode::Entry object representing this inode in the parent TreeInode.
However, it isn't safe for a child Inode to access the TreeInode::Entry owned
by its parent. This Entry object is supposed to be protected by the parent
TreeInode's contents_ lock, and the child inode never held this when accessing
the Entry.
This updates the TreeInode code to never use entry_ any more. I will work on
removing FileInode::entry_ in a subsequent diff.
This did require changing the overlay materialization code some. Previously
the checkout code directly modified entry_ to inform the parent TreeInode if
the child was materialized or not. Now we have to more explicitly tell the
parent TreeInode about changes to each child's materialization state. With
this change we now keep the overlay data on disk more consistent during the
checkout process. We should now always have overlay data for children inodes
on disk whenever the parent thinks they are materialized. We may end up
writing out TreeInode overlay data frequently during the checkout process,
however. We might be able to improve this in the future. For now it seemed
simpler and safer to just update the data frequently to make sure it stays
consistent.
Reviewed By: bolinfest
Differential Revision: D4688055
fbshipit-source-id: d08a14b9304ac49826f0897a0b53281177d9d9f4
Summary:
Update the code so that we no longer immediately remove overlay data about a
file or directory when the inode is unlinked. The file can continue being used
after it was unlinked if there are open file handles or other internal
references to it.
This updates the code to instead wait to remove the overlay data until the
InodeBase object is unloaded.
This does not fully fix all the outstanding issues with using inodes after they
have been unlinked, because the FileInode and TreeInode still use the entry_
pointer that now points to a bogus TreeInode::Entry.
Reviewed By: bolinfest
Differential Revision: D4688056
fbshipit-source-id: a9e1c1a9562590b8b81237057bfc4ef77795807f
Summary:
This changes a couple aspects of the materialization process:
- Write out child files in the overlay before their parent. If we throw an
exception or crash partway through this process, it is better to have the
child not marked materialized in its parent, rather than to have the parent
indicate that the child is materialized but to not actually have any overlay
data for the child. In the former case we will still load the correct child
data the next time we need to, but in the latter case we would fail to load
the child correctly.
- Hold the rename lock while materializing our parent directories, to ensure
that we materialize the correct parent, and it cannot change while we are
trying to perform the materialization.
- Make the parent responsible for modifying the child's TreeInode::Entry. This
data is owned by the parent and is protected by the parent's contents_ lock.
The child really shouldn't ever be directly accessing this data without
holding the parent's contents_ lock.
Reviewed By: bolinfest
Differential Revision: D4688057
fbshipit-source-id: 2662f79cb7d7febb086f4e0888a3d96a580c4bfa
Summary:
This fixes the checkout code to always recurse into directories with conflicts,
and accurately list all individual files with conflicts.
Previously the checkout code always avoided recursing into unloaded directory
inodes that were affected by the checkout. Since the inode was not loaded we
can easily replace it without recursing into it. Unfortunately this resulted
in an inaccurate conflict list--we would report the directory itself as a
conflict. This is insufficient for our mercurial extension, which needs to
know the specific list of files with conflicts, so it can ask mercurial to
perform conflict resolution on them.
Reviewed By: bolinfest
Differential Revision: D4672373
fbshipit-source-id: ff400a4a6e25b8b0754c66cb067a69d553a5c98b
Summary:
Loading the child inode is necessary to avoid race conditions with other
threads that may be currently trying to load the child inode in question.
This updates the code so that rmdir() and unlink() now share most of their
implementation.
Reviewed By: bolinfest
Differential Revision: D4664110
fbshipit-source-id: 871ca7b3c22819b24bc2639d9550bb45f6541f50
Summary:
Replace the one very basic checkout unit test with several tests that
more thoroughly test file modification changes. More tests for
directory changes and file type changes still need to be added, but this is a
first step for now. I will continue adding more tests in upcoming diffs.
Reviewed By: wez
Differential Revision: D4641595
fbshipit-source-id: 443a68e4e3c673fe687446f3469f035b4f48f1b3
Summary:
Add a method to reset the current commit without changing the working
directory state, similar to "hg reset" and "git reset --soft".
This also adds a new EXPECT_FILE_INODE() check for use in the unit tests,
and FileInode::getPermissions() and FileData::readFull() methods to support
this check.
Reviewed By: wez
Differential Revision: D4641476
fbshipit-source-id: 1e516774fe8e292a8d82cc2c354619374a3abe37
Summary:
The FakeTreeBuilder class in D4609587 provides a lot of the same functionality
as the TestMountBuilder class, but is not restricted to being used only at
mount initialization time. (It also provides more powerful functionality for
controlling the ordering of events when loading data from the ObjectStore.)
This switches all of the existing tests to use FakeTreeBuilder rather than
TestMountBuilder, and removes the TestMountBuilder class.
One difference is that FakeTreeBuilder adds data to the FakeBackingStore, while
TestMountBuilder previously put data directly into the LocalStore. This
shouldn't really make much difference for the behavior of the code. Putting
data in the FakeBackingStore gives us more control over when data is available
during load operations, and exercises slightly more of the normal ObjectStore
code path.
Reviewed By: wez
Differential Revision: D4641288
fbshipit-source-id: ca2b45bf69bd373848c12a7b739b286aabe6aa9b
Summary:
Add a FakeTreeBuilder class to make it easier to create Tree structures in the
FakeBackingStore using path names. Previously the test code had to manually
chain together Trees and Blobs. FakeTreeBuilder lets callers call setFile()
with a path name to define the file layout. The files that were defined then
get converted into Tree and Blob data when finalize() is called.
This is similar to the existing TestMountBuilder::addFile() API. However,
TestMountBuilder only populates the LocalStore. FakeTreeBuilder populates
the FakeBackingStore instead. The FakeBackingStore class allows tests to have
much more control about when objects appear ready from the backing store,
allowing control over the order in which Futures are fulfilled during the test.
Reviewed By: bolinfest
Differential Revision: D4609587
fbshipit-source-id: 9642d17daf0dc10f08901e51552bbb3c3e150b53
