Summary:
Add a new utils/SystemError.h header with helper functions to check if a
`std::system_error` contains an errno value.
Most of the code in Eden previously only checked for `std::system_category`
when looking for errno values. `std::generic_category` is the correct category
to use for errno exceptions, but folly/Exception.h incorrectly throws them as
`std::system_category` today. This change makes Eden treat either error type
as errno values for now.
Reviewed By: yfeldblum
Differential Revision: D7329999
fbshipit-source-id: 67a3c3ea10371c53a2e34236b7575deac4cbd53a
Summary:
I noticed from the logs that this was funky:
```
C0131 01:55:28.870926 24027 Bug.cpp:55] EDEN_BUG at eden/fs/inodes/InodeMap.cpp425: !!BUG!! InodeMap::save() called with 104957 inodes still loaded; they must all (except the root) have been unloaded for this to succeed!
```
Reviewed By: chadaustin
Differential Revision: D7310995
fbshipit-source-id: 8c7840bbd60e3e5e815c8cef0d50bf4021e89baf
Summary:
In some cases we could call `delete` with the wrong size in
`UnixSocket::SendQueueDestructor` when `__cpp_sized_deallocation` is available.
In particular, if the input message data contained some empty buffers in the
IOBuf chain we would allocate room for these elements when initially performing
the allocation in `createSendQueueEntry()`, but we would skip over them in the
`SendQueueEntry` constructor, so `iovCount` did not include them. The
`SendQueueDestructor` code used `iovCount` to calculate how much space had been
allocated, and so it would undercount the amount of allocated space in this
case.
This updates `createSendQueueEntry()` to also avoid allocating an iovec entry
for the empty buffers, so that `iovCount` should always accurately reflect how
many entries were originally allocated.
Reviewed By: chadaustin
Differential Revision: D7190579
fbshipit-source-id: 422cc737f146adeb1c133b9f3b500038e05bad10
Summary:
This was an in-person code review item I forgot to enable
with the original diff.
Reviewed By: wez
Differential Revision: D7016624
fbshipit-source-id: 91d729772aa3c0b476f6bf8f6ee7e46cdac54626
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
Most uses of `size_t` in `eden` are unqualified, but a few are qualified.
As discussed ad nauseum in
https://stackoverflow.com/questions/5813700/difference-between-size-t-and-stdsize-t
it is totally safe to use unqualified `size_t` with all compilers/platforms.
Since this saves 5 chars per use, and to improve uniformity, I ran:
```
$ find ~/fbsource/fbcode/eden -type f \
| egrep '\.(h|cpp)$' \
| xargs sed -i 's/std::size_t/size_t/g'
```
Reviewed By: chadaustin
Differential Revision: D7021980
fbshipit-source-id: da268e62a9a93d2a5168a40b6878795ae7516b7f
Summary:
Introduces a persistent non-durable storage mechanism backed
by a memory-mapped file with fixed-length records.
Reviewed By: simpkins
Differential Revision: D6877217
fbshipit-source-id: 0ddacb4137cfe43e67c822dce4064356cdf515b5
Summary: This API seems like it should be const, as it does not modify the clock.
Reviewed By: chadaustin, zhupanov
Differential Revision: D6869719
fbshipit-source-id: c8bf4ccab34538b59e6baeedd0b0ff88b328236e
Summary:
This is the spiritual successor to D3302706 which originally
wanted to solve this by adding a python extension. That would prove
to be too painful for the opensource build so it was shelved.
We now need to be able to run our tests in an environment that doesn't
have the `attr` rpm installed so this is a good time to fix this
in a more portable way.
This diff adds a little wrapper around the functions that we already
have for consuming extended attribute information and augments them
with another to list attributes.
The utility emits output in json format and is intended to be fed
directly into the helper functions we have in `fs.py`.
Reviewed By: chadaustin
Differential Revision: D6851182
fbshipit-source-id: 3d1d1a351f2e01405645d45658d1c8bc61a659a4
Summary:
Whilst chatting with simpkins we realized that we lost
the handshake portion of the takeover protocol during a refactor.
The handshake is important for a couple of reasons:
1. It prevents unmounting and loosing all the mounts in the case
that sometime decides to netcat or otherwise connect to the
socket
2. It gives us an opportunity to short circuit any heavy lifting
if we know that it will be impossible to succeed.
3. It allows us to rollback to earlier builds with older versions
of the takeover protocol.
This diff adds a little bit of machinery to enable passing a set of supported
takeover protocol version numbers. The intent is to retain support for
the two of these at a time; any time we change the encoding/protocol
for takeover we'll bump the version number and add supporting code
to handle the new format, retaining support for the prior version.
Retaining the ability to handle the prior version allows us to downgrade
to an earlier build gracefully if/when the need arises.
I opted to do this here rather than by bumping the `kProtocolID`
constant in `UnixSocket.h` becase we're not really changing the
lowest level of the protocol; just the takeover specific portions.
I haven't actually changed the takeover serialization in this diff,
but do have some work on that happening in D6733406; that diff will
be amended to take advantage and demonstrate how this versioning
scheme works.
A key thing to note about the implementation of this diff is that
the client sends the version number to the server, but doesn't
add any explicit version encoding in the response we receive.
This is deliberate and allows us to upgrade prior builds to
this new scheme. I'll add a more definitive check for this
situation when I actually rev the format in the following diff.
Reviewed By: simpkins
Differential Revision: D6743065
fbshipit-source-id: c991cebfee918daad098105ca6bcfef76374c0ff
Summary:
In my benchmarks, using fbstring over std::string for PathMap is
worthwhile. There are a lot of entries stored in PathMap and the fact
that fbstring is a word smaller than std::string adds up.
Differential Revision: D6820204
fbshipit-source-id: 1ac34a81828891eb8125b13a074de6ed9715edad
Summary:
Switch from glog to folly logging. Also derive from folly::ColdClass to
indicate that the `EdenBug` class is only ever created on unexpected code
paths.
Reviewed By: wez
Differential Revision: D6609613
fbshipit-source-id: 33621c7bafc946df245224da9b6a559e3dfd58d6
Summary:
Add new classes that help send data, file descriptors, and credential
information over unix domain sockets.
UnixSocket provides a low-level, raw callback API, similar to that provided by
the classes in folly/io/async/. UnixSocketFuture is a slightly higher-level
wrapper class that provides a Future-based API on top of this. I expect that
most places in eden will probably use the UnixSocketFuture API, but callers
that repeatedly wait for new messages (like the privhelper server) may want to
use the raw callback API instead.
This will help simplify several places in eden that communicate over unix
domain sockets. Both the privhelper code and the takeover code needs to send
file descriptors over unix domain sockets. They currently each use their own
separate message handling logic for this. We currently communicate with the
hg_import_helper.py script over a pipe, but this could easily be switched to
use this new UnixSocket class as well.
Reviewed By: wez
Differential Revision: D6494981
fbshipit-source-id: 80bd7f06e5b884fc4148162e1a8a3b478acce209
Summary: Drive-by cleanup. `CompareString` is not used anywhere and `std::less<>` would be better anyway.
Reviewed By: wez
Differential Revision: D6655104
fbshipit-source-id: 0fee22172c93335493400e6317d92f9e1e77b40e
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
Add a new function that attempts to normalize a path with `realpath()`, but
falls back to `canonicalPath()` if that fails. This lets us attempt to resolve
symlinks if possible, but still perform best-effort normalization if that
fails.
Reviewed By: chadaustin
Differential Revision: D6527493
fbshipit-source-id: 9137d517452ca7fd825852c1f60ade07f1ee78fa
Summary:
Add EdenCPUThreadPool and UnboundedQueueThreadPool types to make it clearer
that it's always okay for prefetch, deferred diff entry, and hg import to
shuttle work back to the main thread pool.
This diff changes no behavior - it just makes some invariants explicit.
Reviewed By: wez, simpkins
Differential Revision: D6504117
fbshipit-source-id: 3400ad55c00b3719ecba31807fd992442f622cd9
Summary:
Introduce a Clock seam. This will allow us to write tests around
ctime, mtime, and atime logic.
Reviewed By: wez
Differential Revision: D6392543
fbshipit-source-id: 1721d76d2364b135b4ef5c078ef60f7f8526259e
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.
Reviewed By: chadaustin
Differential Revision: D6301759
fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
Summary:
Add EXPECT_THROW_RE() and EXPECT_THROW_ERRNO() macros to folly/test/TestUtils.h
These allow more precise checks than the basic EXPECT_THROW() macro provided as
part of gtest.
These macros are being moved into folly from Facebook's eden repository
(https://github.com/facebookexperimental/eden)
This will allow us to use them in folly tests and in other projects that depend
on folly.
Reviewed By: yfeldblum
Differential Revision: D6301760
fbshipit-source-id: 1f434fb5bc9b7859f763171264fb0b2e1b4bda62
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary: this is all non-hphp includes that are going in container/
Reviewed By: mzlee, yfeldblum
Differential Revision: D6121745
fbshipit-source-id: b024bde8835fc7f332686793d75eb8e71591c912
Summary:
Add a helper class for waiting on socket I/O using an EventBase and fulfilling
a folly::Future when I/O is ready.
Reviewed By: bolinfest
Differential Revision: D6003899
fbshipit-source-id: ba9c1be425b102066eb585320fee495e4180e776
Summary:
Add a new helper class for working with cmsghdr structs.
This updates the PrivHelperConn code to use this new class, and I will also use
this in the upcoming code for implementing graceful mount point takeover on
edenfs restart.
Reviewed By: bolinfest, wez
Differential Revision: D6003897
fbshipit-source-id: 634e9d8f1dc73010c9e9336872cf1270a344bdd2
Summary: Modified `TreeInode::unloadChildrenNow` such that inodes are unloaded whose age is greater than a specific age.
Reviewed By: simpkins
Differential Revision: D5526137
fbshipit-source-id: 91e2364d55e31befedcf43d98c26467e1a472ef9
Summary:
This makes several improvements to the hg import tester script:
- If no --edenDir flag is specified, initialize a new temporary directory to
keep the RocksDB data store.
- Add a `--rocksdb_options_file` flag to allow controlling the options used for
the RocksDB store.
- Add an --import_type flag to allow explicitly selecting if we should test
the flat manifest or tree manifest import code.
- Add a --flat_import_file flag, to allow testing a pre-generated flat manifest
input data file, rather than retrieving the data from mercurial. This allows
benchmarking only the C++ import code, and eliminating the python portion of
the import. The input file can be generated by running
`hg_import_helper.py --manifest <revision>`
Reviewed By: wez
Differential Revision: D5541732
fbshipit-source-id: 340af4fea872412248d41453792b2179f0afa466
Summary:
This adds a new UserInfo class to store the UID, GID, as well as the username
and home directory. This moves the determineUid() and determineGid() functions
from main.cpp into this new class, and makes the logic somewhat smarter now.
In addition to looking up the UID and GID, we now look up the username. This
information is not used yet, but will be used in an upcoming diff to set
supplementary groups.
This also stores the home directory in the UserInfo class. The home directory
is usually necessary to find the user's ~/.edenrc file. Computing it as part
of UserInfo makes the most sense since we will likely have already looked up
the user's passwd entry.
Reviewed By: bolinfest
Differential Revision: D5501252
fbshipit-source-id: 1cb4be9f6c1493de4362da3393034e78bedd9db2
Summary:
This updates the eden/fs/utils/TARGETS file so that PathFuncs.h and
PathFuncs.cpp are in their own separate library. The existing "utils" library
depends on it, but other users can now depend on PathFuncs.h without pulling in
the other code in utils.
I plan to make some of the privhelper code depend on PathFuncs.h in an upcoming
diff, and this makes it so that privhelper will not need to pull in the rest of
the utils code.
Reviewed By: bolinfest
Differential Revision: D5501253
fbshipit-source-id: 804067785ed663e223977d1c84ebcfc28b3f1642
Summary: This uses `clang-tidy` to comment out unused parameters (in functions, methods and lambdas) in fbcode. Cases that the tool failed to handle are fixed manually.
Reviewed By: igorsugak
Differential Revision: D5454343
fbshipit-source-id: 5dee339b4334e25e963891b519a5aa81fbf627b2
Summary: It doesn't need to exist anymore
Reviewed By: yfeldblum
Differential Revision: D5318746
fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.
A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from. The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.
Reviewed By: wez
Differential Revision: D5316000
fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
Summary:
Update eden to log via the new folly logging APIs rather than with glog.
This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else. (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)
Reviewed By: wez
Differential Revision: D5290783
fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
Summary:
It is an error to not provide mode argument when open is
called with O_CREAT.
Reviewed By: oebeling
Differential Revision: D5020517
fbshipit-source-id: 87665ceb16ebdff797f8bf3d33bea082191e3121
Summary:
The error messages for various failure cases in EXPECT_THROW_RE() and
EXPECT_THROW_ERRNO() were missing a space after the statement being executed.
Reviewed By: bolinfest
Differential Revision: D4941175
fbshipit-source-id: e10b4bff7db027acc5cc7ad5c12af248d55c7cd8
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.
Reviewed By: bolinfest, wez
Differential Revision: D4889053
fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
