Summary:
Add methods to UnixSocket and FutureUnixSocket to attach and detach from an
EventBase. This makes it possible to construct a UnixSocket object without
having an EventBase yet and then attach it to an EventBase later.
Reviewed By: bolinfest
Differential Revision: D8053423
fbshipit-source-id: c4de00166dbc0e075b4e4cd81c3dd5b377ea9a52
Summary:
I'd misunderstood the point of SharedMutex's upgrade locks -
unless they're used in rare paths, they don't allow for increased
concurrency. This diff and D7885245 remove all of Eden's ulocks,
replacing them with a helper which checks once with an rlock held, and
if the check fails, switches to a wlock (and checks again).
Reviewed By: yfeldblum
Differential Revision: D7886046
fbshipit-source-id: 545bb0dbb4898cbb71412efc6222ef12e4ee374e
Summary:
Update the folly::Init code to define a `--logging` command line flag, and call
`folly::initLoggingOrDie()` with the value of this command line during
initialization.
This is similar to the existing code that initializes the glog library.
(Programs can use both glog and folly logging together in the same program, and
I expect that many programs will do so as parts get converted to folly::logging
and parts remain using glog.)
Reviewed By: yfeldblum
Differential Revision: D7827344
fbshipit-source-id: 8aa239fbad43bc0b551cbe40cad7b92fa97fcdde
Summary:
I got tired of typing PathComponentPiece{"..."} in tests so here are
some operator literals.
Reviewed By: simpkins
Differential Revision: D7956732
fbshipit-source-id: 85d9f3fd725853a54da9e70fc659bd7eb9e0862c
Summary:
simpkins was curious how data format migrations would be handled in
the upcoming InodeTable. This diff implements the bulk of the logic
which is largely at the MappedDiskVector level. The existing file
format supported record version negotiation and this diff hooks it up
with some type-level operations.
Reviewed By: simpkins
Differential Revision: D7836249
fbshipit-source-id: 00e36bc67068c7524956e908b3872c80a79241c0
Summary:
I frequently find myself forgetting how to make the compiler see that
non-moved-from EDEN_BUG's destructor is noreturn, so add a simple
throwException function to it.
Reviewed By: simpkins
Differential Revision: D7834182
fbshipit-source-id: f279b9ca24f90efb4ad3ac318606dbd2dd002665
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
Update the UnixSocket code to limit the number of iovecs that we send at one
time to folly::kIovMax. (This constant is set from from IOV_MAX on platforms
that provide this setting.)
The thrift serializer code emits data into 16kB chunks, so if we had many
megabytes of data to send we could end up with thousands of iovecs. The kernel
would fail the send with EMSGSIZE in this case.
Reviewed By: chadaustin
Differential Revision: D7665147
fbshipit-source-id: b7a60238d3fca973604b9037f22883cae80891ab
Summary:
The takeover data for a fully-loaded monorepo was 20 MB. Two
monorepos was 40 MB, exceeding the previous 32 MiB limit and causing
takeover to fail. Bump to 512 MiB to minimize the chance of
allocating too much given bogus data while still supporting large
numbers of checkouts.
Reviewed By: simpkins
Differential Revision: D7661637
fbshipit-source-id: 1dc54055e7ac3ce0f6cdbf4a7e5d372c2674dfee
Summary:
UnixSocket traverses the IOBuf chain twice. Refactor that
into a common function because the next diff caps the size of
individual iovecs.
Reviewed By: simpkins
Differential Revision: D7659062
fbshipit-source-id: 88b7d63669d8189b96434c38a6e499ed3b5ebbe6
Summary:
Verify Eden handles looking up an inode by number after graceful
restart and checkout.
Reviewed By: simpkins
Differential Revision: D7346263
fbshipit-source-id: 876b4837708da9ac31f72c06e7defc797fe126f3
Summary:
Fix the code to generate exceptions based on an errno error using
std::generic_category rather than std::system_category.
Reviewed By: yfeldblum
Differential Revision: D7329997
fbshipit-source-id: 3fe257bbbc7a631c801f31120592c8bdbc25c8bf
Summary:
Add a new utils/SystemError.h header with helper functions to check if a
`std::system_error` contains an errno value.
Most of the code in Eden previously only checked for `std::system_category`
when looking for errno values. `std::generic_category` is the correct category
to use for errno exceptions, but folly/Exception.h incorrectly throws them as
`std::system_category` today. This change makes Eden treat either error type
as errno values for now.
Reviewed By: yfeldblum
Differential Revision: D7329999
fbshipit-source-id: 67a3c3ea10371c53a2e34236b7575deac4cbd53a
Summary:
I noticed from the logs that this was funky:
```
C0131 01:55:28.870926 24027 Bug.cpp:55] EDEN_BUG at eden/fs/inodes/InodeMap.cpp425: !!BUG!! InodeMap::save() called with 104957 inodes still loaded; they must all (except the root) have been unloaded for this to succeed!
```
Reviewed By: chadaustin
Differential Revision: D7310995
fbshipit-source-id: 8c7840bbd60e3e5e815c8cef0d50bf4021e89baf
Summary:
In some cases we could call `delete` with the wrong size in
`UnixSocket::SendQueueDestructor` when `__cpp_sized_deallocation` is available.
In particular, if the input message data contained some empty buffers in the
IOBuf chain we would allocate room for these elements when initially performing
the allocation in `createSendQueueEntry()`, but we would skip over them in the
`SendQueueEntry` constructor, so `iovCount` did not include them. The
`SendQueueDestructor` code used `iovCount` to calculate how much space had been
allocated, and so it would undercount the amount of allocated space in this
case.
This updates `createSendQueueEntry()` to also avoid allocating an iovec entry
for the empty buffers, so that `iovCount` should always accurately reflect how
many entries were originally allocated.
Reviewed By: chadaustin
Differential Revision: D7190579
fbshipit-source-id: 422cc737f146adeb1c133b9f3b500038e05bad10
Summary:
This was an in-person code review item I forgot to enable
with the original diff.
Reviewed By: wez
Differential Revision: D7016624
fbshipit-source-id: 91d729772aa3c0b476f6bf8f6ee7e46cdac54626
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
Most uses of `size_t` in `eden` are unqualified, but a few are qualified.
As discussed ad nauseum in
https://stackoverflow.com/questions/5813700/difference-between-size-t-and-stdsize-t
it is totally safe to use unqualified `size_t` with all compilers/platforms.
Since this saves 5 chars per use, and to improve uniformity, I ran:
```
$ find ~/fbsource/fbcode/eden -type f \
| egrep '\.(h|cpp)$' \
| xargs sed -i 's/std::size_t/size_t/g'
```
Reviewed By: chadaustin
Differential Revision: D7021980
fbshipit-source-id: da268e62a9a93d2a5168a40b6878795ae7516b7f
Summary:
Introduces a persistent non-durable storage mechanism backed
by a memory-mapped file with fixed-length records.
Reviewed By: simpkins
Differential Revision: D6877217
fbshipit-source-id: 0ddacb4137cfe43e67c822dce4064356cdf515b5
Summary: This API seems like it should be const, as it does not modify the clock.
Reviewed By: chadaustin, zhupanov
Differential Revision: D6869719
fbshipit-source-id: c8bf4ccab34538b59e6baeedd0b0ff88b328236e
Summary:
This is the spiritual successor to D3302706 which originally
wanted to solve this by adding a python extension. That would prove
to be too painful for the opensource build so it was shelved.
We now need to be able to run our tests in an environment that doesn't
have the `attr` rpm installed so this is a good time to fix this
in a more portable way.
This diff adds a little wrapper around the functions that we already
have for consuming extended attribute information and augments them
with another to list attributes.
The utility emits output in json format and is intended to be fed
directly into the helper functions we have in `fs.py`.
Reviewed By: chadaustin
Differential Revision: D6851182
fbshipit-source-id: 3d1d1a351f2e01405645d45658d1c8bc61a659a4
Summary:
Whilst chatting with simpkins we realized that we lost
the handshake portion of the takeover protocol during a refactor.
The handshake is important for a couple of reasons:
1. It prevents unmounting and loosing all the mounts in the case
that sometime decides to netcat or otherwise connect to the
socket
2. It gives us an opportunity to short circuit any heavy lifting
if we know that it will be impossible to succeed.
3. It allows us to rollback to earlier builds with older versions
of the takeover protocol.
This diff adds a little bit of machinery to enable passing a set of supported
takeover protocol version numbers. The intent is to retain support for
the two of these at a time; any time we change the encoding/protocol
for takeover we'll bump the version number and add supporting code
to handle the new format, retaining support for the prior version.
Retaining the ability to handle the prior version allows us to downgrade
to an earlier build gracefully if/when the need arises.
I opted to do this here rather than by bumping the `kProtocolID`
constant in `UnixSocket.h` becase we're not really changing the
lowest level of the protocol; just the takeover specific portions.
I haven't actually changed the takeover serialization in this diff,
but do have some work on that happening in D6733406; that diff will
be amended to take advantage and demonstrate how this versioning
scheme works.
A key thing to note about the implementation of this diff is that
the client sends the version number to the server, but doesn't
add any explicit version encoding in the response we receive.
This is deliberate and allows us to upgrade prior builds to
this new scheme. I'll add a more definitive check for this
situation when I actually rev the format in the following diff.
Reviewed By: simpkins
Differential Revision: D6743065
fbshipit-source-id: c991cebfee918daad098105ca6bcfef76374c0ff
Summary:
In my benchmarks, using fbstring over std::string for PathMap is
worthwhile. There are a lot of entries stored in PathMap and the fact
that fbstring is a word smaller than std::string adds up.
Differential Revision: D6820204
fbshipit-source-id: 1ac34a81828891eb8125b13a074de6ed9715edad
Summary:
Switch from glog to folly logging. Also derive from folly::ColdClass to
indicate that the `EdenBug` class is only ever created on unexpected code
paths.
Reviewed By: wez
Differential Revision: D6609613
fbshipit-source-id: 33621c7bafc946df245224da9b6a559e3dfd58d6
Summary:
Add new classes that help send data, file descriptors, and credential
information over unix domain sockets.
UnixSocket provides a low-level, raw callback API, similar to that provided by
the classes in folly/io/async/. UnixSocketFuture is a slightly higher-level
wrapper class that provides a Future-based API on top of this. I expect that
most places in eden will probably use the UnixSocketFuture API, but callers
that repeatedly wait for new messages (like the privhelper server) may want to
use the raw callback API instead.
This will help simplify several places in eden that communicate over unix
domain sockets. Both the privhelper code and the takeover code needs to send
file descriptors over unix domain sockets. They currently each use their own
separate message handling logic for this. We currently communicate with the
hg_import_helper.py script over a pipe, but this could easily be switched to
use this new UnixSocket class as well.
Reviewed By: wez
Differential Revision: D6494981
fbshipit-source-id: 80bd7f06e5b884fc4148162e1a8a3b478acce209
Summary: Drive-by cleanup. `CompareString` is not used anywhere and `std::less<>` would be better anyway.
Reviewed By: wez
Differential Revision: D6655104
fbshipit-source-id: 0fee22172c93335493400e6317d92f9e1e77b40e
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
Add a new function that attempts to normalize a path with `realpath()`, but
falls back to `canonicalPath()` if that fails. This lets us attempt to resolve
symlinks if possible, but still perform best-effort normalization if that
fails.
Reviewed By: chadaustin
Differential Revision: D6527493
fbshipit-source-id: 9137d517452ca7fd825852c1f60ade07f1ee78fa
Summary:
Add EdenCPUThreadPool and UnboundedQueueThreadPool types to make it clearer
that it's always okay for prefetch, deferred diff entry, and hg import to
shuttle work back to the main thread pool.
This diff changes no behavior - it just makes some invariants explicit.
Reviewed By: wez, simpkins
Differential Revision: D6504117
fbshipit-source-id: 3400ad55c00b3719ecba31807fd992442f622cd9
Summary:
Introduce a Clock seam. This will allow us to write tests around
ctime, mtime, and atime logic.
Reviewed By: wez
Differential Revision: D6392543
fbshipit-source-id: 1721d76d2364b135b4ef5c078ef60f7f8526259e
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.
Reviewed By: chadaustin
Differential Revision: D6301759
fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
Summary:
Add EXPECT_THROW_RE() and EXPECT_THROW_ERRNO() macros to folly/test/TestUtils.h
These allow more precise checks than the basic EXPECT_THROW() macro provided as
part of gtest.
These macros are being moved into folly from Facebook's eden repository
(https://github.com/facebookexperimental/eden)
This will allow us to use them in folly tests and in other projects that depend
on folly.
Reviewed By: yfeldblum
Differential Revision: D6301760
fbshipit-source-id: 1f434fb5bc9b7859f763171264fb0b2e1b4bda62
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary: this is all non-hphp includes that are going in container/
Reviewed By: mzlee, yfeldblum
Differential Revision: D6121745
fbshipit-source-id: b024bde8835fc7f332686793d75eb8e71591c912
Summary:
Add a helper class for waiting on socket I/O using an EventBase and fulfilling
a folly::Future when I/O is ready.
Reviewed By: bolinfest
Differential Revision: D6003899
fbshipit-source-id: ba9c1be425b102066eb585320fee495e4180e776
Summary:
Add a new helper class for working with cmsghdr structs.
This updates the PrivHelperConn code to use this new class, and I will also use
this in the upcoming code for implementing graceful mount point takeover on
edenfs restart.
Reviewed By: bolinfest, wez
Differential Revision: D6003897
fbshipit-source-id: 634e9d8f1dc73010c9e9336872cf1270a344bdd2
Summary: Modified `TreeInode::unloadChildrenNow` such that inodes are unloaded whose age is greater than a specific age.
Reviewed By: simpkins
Differential Revision: D5526137
fbshipit-source-id: 91e2364d55e31befedcf43d98c26467e1a472ef9
Summary:
This makes several improvements to the hg import tester script:
- If no --edenDir flag is specified, initialize a new temporary directory to
keep the RocksDB data store.
- Add a `--rocksdb_options_file` flag to allow controlling the options used for
the RocksDB store.
- Add an --import_type flag to allow explicitly selecting if we should test
the flat manifest or tree manifest import code.
- Add a --flat_import_file flag, to allow testing a pre-generated flat manifest
input data file, rather than retrieving the data from mercurial. This allows
benchmarking only the C++ import code, and eliminating the python portion of
the import. The input file can be generated by running
`hg_import_helper.py --manifest <revision>`
Reviewed By: wez
Differential Revision: D5541732
fbshipit-source-id: 340af4fea872412248d41453792b2179f0afa466
Summary:
This adds a new UserInfo class to store the UID, GID, as well as the username
and home directory. This moves the determineUid() and determineGid() functions
from main.cpp into this new class, and makes the logic somewhat smarter now.
In addition to looking up the UID and GID, we now look up the username. This
information is not used yet, but will be used in an upcoming diff to set
supplementary groups.
This also stores the home directory in the UserInfo class. The home directory
is usually necessary to find the user's ~/.edenrc file. Computing it as part
of UserInfo makes the most sense since we will likely have already looked up
the user's passwd entry.
Reviewed By: bolinfest
Differential Revision: D5501252
fbshipit-source-id: 1cb4be9f6c1493de4362da3393034e78bedd9db2
Summary:
This updates the eden/fs/utils/TARGETS file so that PathFuncs.h and
PathFuncs.cpp are in their own separate library. The existing "utils" library
depends on it, but other users can now depend on PathFuncs.h without pulling in
the other code in utils.
I plan to make some of the privhelper code depend on PathFuncs.h in an upcoming
diff, and this makes it so that privhelper will not need to pull in the rest of
the utils code.
Reviewed By: bolinfest
Differential Revision: D5501253
fbshipit-source-id: 804067785ed663e223977d1c84ebcfc28b3f1642