Summary:
Drop all the remaining instances of `IOBufQueue::clear()`.
(Note: this ignores all push blocking failures!)
Reviewed By: yfeldblum
Differential Revision: D32941992
fbshipit-source-id: c7d983e215a78014ffda74a944b628e1886c35c8
Summary:
static_assert sizeof doesn't show the actual size in clang or gcc, so
implement a CheckSize template.
Reviewed By: xavierd
Differential Revision: D32998926
fbshipit-source-id: 432fbf19eae5b4572e5a4908f9007b7242a3ae69
Summary:
I was using this to debug graceful restart locally, and had it at a high debug
level. DBG1 is a bit verbose this should probably be DBG7 with the rest of the
debug logging I added.
Reviewed By: genevievehelsel
Differential Revision: D32902321
fbshipit-source-id: 4b7db7cbe96db8ef5b1035932c06a1de264f8502
Summary:
We've seen several reports of users getting confused by the output of `df` and
thinking that EdenFS was using 100+ GBs of disk space. In reality, EdenFS is
simply reporting the stats from the underlying filesystem, and thus isn't a
true representation of the disk space usage.
To avoid this confusion, let's simply pretend that EdenFS doesn't use any disk
space. This isn't exactly true as the overlay and some internal datastructures
are using space already, but `eden du` is the right way to see these.
Reviewed By: fanzeyi
Differential Revision: D32536715
fbshipit-source-id: ec9b27084340d2e69cc5a14d6aa71def5fb4465d
Summary:
Provide a no-arg constructor, for C++20
This avoids the following errors:
```
eden/fs/nfs/rpc/Server.h:234:63: error: no matching constructor for initialization of 'facebook::eden::RpcTcpHandler::State'
explicit StateWrapper(folly::EventBase* evb) : evb_{evb}, state_{} {}
^ ~~
eden/fs/nfs/rpc/Server.h:221:5: note: candidate constructor not viable: requires single argument 'state', but no arguments were provided
State(const State& state) = delete;
^
eden/fs/nfs/rpc/Server.h:223:5: note: candidate constructor not viable: requires single argument 'state', but no arguments were provided
State(State&& state) = delete;
```
Differential Revision: D32521305
fbshipit-source-id: 70d53db2093b41b58d4abc0b20389b0169b7a2e3
Summary:
For restart for NFS we now have the pieces to transfer nfs mounts.
The last piece of restart is transferring the mountd socket. This extracts the
socket from the current mountd and provides it to the takeover server, as well
as implements reading the mountd socket from the takeover client and starting
the mountd with this server socket.
Reviewed By: xavierd
Differential Revision: D31802786
fbshipit-source-id: 1727ea0913bfe1dcb02fdbe27c8cc3b9d62a8464
Summary:
For restart for NFS we now have the pieces for the nfs channel to
perform takeover and for the takeover code to transfer the data.
This commit plumbs everything through so that we are able to stop and nfs
channel on takeover. this means extracting the connected socket and plubming it
to the takeover server.
Reviewed By: xavierd
Differential Revision: D31357367
fbshipit-source-id: 91973f7c149ae9209e85aa40111f41e7753db63f
Summary:
For restart for NFS we now have the pieces for the nfs channel to
perform takeover and for the takeover code to transfer the data.
This commit plumbs everything through so that we are able to start and nfs
mount during takeover. This means starting the NFS channel from an existing
connected socket.
Reviewed By: xavierd
Differential Revision: D31802788
fbshipit-source-id: 9034c1cc93e6bb64c09a203d265cf3bdedd1e160
Summary:
We are working towards supporting graceful restart on NFS so that NFS users will
have this available when we start the migration.
This commit is teaching the Server at the core of the nfsd and mountd to
support takeoverShutdown and takeoverStartup. This means stopping processing
requests when we call takover shutdown and passing the sockets (both server
socket and connected sockets) up the stack.
For now we are only supporting takeover once everything has been initialized
and not during mount or unmount.
This means for the nfsd we will only care about the one connected socket which
is being used to communicate with the kernel. we will pass through the
shutdown promise for the channel. For the mountd we only care about the server
socket which we will return from the takeoverShutdown method itself.
Then we also add being able to initialize itself from a server socket or
a connected socket.
Reviewed By: xavierd
Differential Revision: D31802790
fbshipit-source-id: 06ef4da48cd864bfa20ea0fb98485792255d98cd
Summary:
AsyncServerSocket works like this:
1) a socket is accepted from the fd
2) the socket is pushed into the socket queue
3) the connectionAccepted() callback is called with the socket is dequeued
Things happening before connectionAccepted() are like black box to us currently. For example the time taken for a socket to sit in the socket queue is completely unknown. Adding this AcceptInfo into the callback could allow us to pass more information to downstream for instrumenting.
This diff only pass timestamp of before the socket is being pushed into the queue.
Reviewed By: yfeldblum
Differential Revision: D31633582
fbshipit-source-id: 66072955318388c2410b48313d3d54078edce478
Summary: This adds inode number to NFS trace event so that we can use it in ActivityRecorder to show the filename of the FS request.
Reviewed By: xavierd
Differential Revision: D30849770
fbshipit-source-id: 580faf5fccb1a225399d9aec843e23eae1874e87
Summary: This adds the support for FS events logging for NFS. For context, each type of event is assigned a sampling group that determines its sampling rate. In TraceBus subscription callback, events are sent to `FsEventLogger` to be sampled and logged through `HiveLogger`.
Reviewed By: xavierd
Differential Revision: D30843863
fbshipit-source-id: 65394d31b1197efd69c7fd4c1b24562f5abd5785
Summary: As title. `RequsetContext` allows us to track metrics such as latency and count.
Reviewed By: genevievehelsel
Differential Revision: D29835813
fbshipit-source-id: 6b85fc8f11923f530fce6d871fa2253db21bfa98
Summary: We already have AccessType for FUSE, this adds the same categorization for NFS. This allows us to easily filter events in trace stream and ActivityRecorder.
Reviewed By: chadaustin
Differential Revision: D29771074
fbshipit-source-id: a437f0693f9062fb2df3b6f618a9d8860a05df12
Summary: This diff connects the trace bus with trace fs command.
Reviewed By: chadaustin
Differential Revision: D29367135
fbshipit-source-id: f9217b286c1a21805d70b21282c10d4ad722a391
Summary: If the handler throws without returning a future, the finish event will not be published. This diff adds the `LiveRequest` to publish finish event in its destructor.
Reviewed By: chadaustin
Differential Revision: D29332452
fbshipit-source-id: 880a4b67ba47b737063a3955c9f4bdbf605f1a43
Summary:
This diff adds tracking for nfs, based on the implementation of that of `FuseChannel`.
Also commented below few questions on the implementation before working on integrating to `strace`.
Reviewed By: chadaustin
Differential Revision: D29279126
fbshipit-source-id: de6bb36dfbe2f550a91f2bf254616bbc639c0c3d
Summary: NfsTaskQueue can be made more generic to be shared across the codebase, so this makes it its own target in `eden/fs/utils` w/ the name EdenTaskQueue.
Reviewed By: xavierd
Differential Revision: D29244762
fbshipit-source-id: 78348f2ff8fa66bc801aefe7d6b3905e0da278e8
Summary:
Previously if you `sudo umount -f fbsource-nfs` then try to
`eden mount fbsource-nfs`, the mount will fail because the EdenMount already
exists and is still running.
Let's properly unmount our selfs on a force unmount like we do for fuse.
There are two potential ways to detect a fource unmount: the UMNT call to the
mount deamon or the socket to the nfsd closing. The UMNT call is unreliable
(on Linux we do not get the UNMT call on `umount -l`), so this diff pursues the
socket closing option.
When the nfsd socket is closed we trigger the EdenMount unmounting process if
this has not already started.
Reviewed By: xavierd
Differential Revision: D28329482
fbshipit-source-id: 5df8f3eb818a92536095195f1b3a9e412394fbf6
Summary:
For non-UTF8 names, the PathComponent constructor would raise an exception, and
since that exception wasn't caught by the handler itself, it would bubble up to
the RPC server and a generic "server IO error" would be sent back to the
client. Since non-UTF8 names aren't a server error, but an invalid argument, we
should instead return a different error.
Unfortunately, EILSEQ isn't an error that an NFS server can return, instead
let's use EINVAL as the argument is clearly invalid.
Reviewed By: chadaustin
Differential Revision: D28482032
fbshipit-source-id: b59044f1a76f7eac79e2df07356a0aeafa22e3c5
Summary:
The modeToNfsMode simply didn't consider all the mode bits to be translated to
the proper NFS mode bits. It now does.
Reviewed By: chadaustin
Differential Revision: D28459428
fbshipit-source-id: d879fb1be2085e44110ba552bc47d2770637fc86
Summary:
An NFS client caches the attributes of files to avoid having to request these
very frequently. What this means is that a file changed by another client (or
by the server itself) may take some time to be reflected on the client, that
time depends on the attribute caching configuration of the mount point.
For EdenFS, files can changed in 2 ways:
- Either it is changed by the user via the mount point,
- Or the user runs an `hg update`
For the first one, the client will simply update its attributes appropriately,
but for the second one, the cached attributes will only be updated when the
user does opens the file, any calls to stat prior will return the old
attributes. Since EdenFS runs on the same host, we can force the attributes
caches to be discarded by simply issuing an open call on the file that changed.
Reviewed By: chadaustin
Differential Revision: D28456482
fbshipit-source-id: 91022d35a33e436c47d94403d0c139992f880cf9
Summary:
Now that makeImmediateFutureWith exists, we can simply use it instead of
constructing a ImmediateFuture<folly::Unit> and calling thenValue on it.
Reviewed By: chadaustin
Differential Revision: D28518059
fbshipit-source-id: 0041cf863fb32efab274f11c77c76109ca9b454f
Summary:
Now that both implementation are using ImmediateFuture, we can move the
ImmediateFuture one layer up.
Reviewed By: kmancini
Differential Revision: D28302479
fbshipit-source-id: 3c2c164a90ffb42a0e7da8528f976af34fc87315
Summary:
With the Mountd code being converted to ImmediateFuture, we can now convert
Nfsd3 to use ImmediateFuture too. For now, this isn't expected to perform
better due to the InodeMap still using on folly::Future, which forces the
ImmediateFuture code to store a SemiFuture. A future change will look at
switching the InodeMap to ImmediateFuture to solve this.
Reviewed By: kmancini
Differential Revision: D28297422
fbshipit-source-id: 8b85103657e877b0f102130f2117bbe60598ed52
Summary:
As a first towards converting the code to use ImmediateFuture more broadly,
let's start with a small self contained part of the code.
This is mostly a sed except for the dispatchRpc method that needs to call
.semi().via().
Reviewed By: kmancini
Differential Revision: D28297421
fbshipit-source-id: e706e91fc8f132d4ef742ae98af9bb8304e0bf36
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
macOS supports NFS servers that can be reached via a unix socket as a way to
improve performance by reducing the TCP cost. To support this, let's first
allow the socket to bind to to be passed to the RpcServer, and then pass it
through to the privhelper code.
Reviewed By: kmancini
Differential Revision: D28261423
fbshipit-source-id: 78c60aac26353d1da76a67897429b964332df8b3
Summary:
folly::via is a Future API, and thus it creates one, which requires allocating
it and then attaching it to the Executore. Since the code to dispatch a request
isn't Future based, we don't need to use folly::via, and we can simply add the
lambda to the Executor directly. This removes expensive memory allocations from
the EventBase.
Reviewed By: kmancini
Differential Revision: D27976674
fbshipit-source-id: 8fa9724a94ba69b071ab894cdbbad0d33733c098
Summary:
Neither macOS, nor Linux are sending multi-fragment requests to the NFS server.
Since supporting these means calling into memmove, which can be expensive for
large requests, let's just remove support for them for now. If somehow macOS
and/or Linux start sending these, the XCHECK(isLast) will catch this and we can
fix the code by then.
Reviewed By: kmancini
Differential Revision: D27976671
fbshipit-source-id: 77c758b2bb36517d22d5b637e6f0ebf84cc19e5b
Summary:
The EventBase is single threaded, and for heavily concurrent client workflows,
it could see a lot activity, thus every cycle saved can be used to drive more
client requests. The construction of the IOBuf doesn't need to be done while in
the EventBase, thus let's build it outside.
Reviewed By: kmancini
Differential Revision: D27976670
fbshipit-source-id: c6c015ef26df1dcb3fc0c5f179e474bafbd71fac
Summary:
Passing 64 to preallocate means that the AsyncSocket code will issue reads of
64 bytes, even though the IOBufQueue has significantly more space available. We
can thus pass a bigger size to preallocate to reduce both the cost of
allocation, and the syscall cost. For heavily concurrent client code, this will
allow us to read more than one request per syscall.
The careful reader may have noticed that for very small requests the code may
reallocate more often that it should as it will always reallocate when falling
under 4KB. This is likely to not be an issue in practice.
Reviewed By: kmancini
Differential Revision: D27976672
fbshipit-source-id: 4c7e3aecc4763ab20854f3c466ce0872332f9b77
Summary:
These are various cleanups that should make the code easier to read, there is
no behavior changes.
Reviewed By: kmancini
Differential Revision: D27976673
fbshipit-source-id: 470eb628ca75bf1712a93c6e9aa3a27c3f314d01
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
Summary: Since HeaderClientChannel now accepts a transport unique_ptr there's no need to have this deleter exposed outside of HeaderClientChannel.
Reviewed By: iahs
Differential Revision: D27729209
fbshipit-source-id: 064b03afdfe567b6df6437348596f0f6f97f6aaf
Summary:
This allows unix sockets to be created in the mount. This will allow Buck to
run properly as it tries to create sockets in the repository.
Reviewed By: kmancini
Differential Revision: D27690406
fbshipit-source-id: 5725d68bdda12f3a5882ce48b6bdd02b14cdece4
Summary: This merely adds the types for the procedure
Reviewed By: kmancini
Differential Revision: D27690405
fbshipit-source-id: b94fb03658cabaece4166c29135c5fdf9a613d3c
Summary:
This is roughly the same logic as the UNLINK one with the only difference being
in the handling of "." and "..".
Reviewed By: kmancini
Differential Revision: D27684716
fbshipit-source-id: 86a95c38e6c783bc3a45c0a8b000d0210b6dd0b8
Summary:
This merely adds the types needed for the RMDIR procedure. Implementation will
follow.
Reviewed By: genevievehelsel
Differential Revision: D27684736
fbshipit-source-id: 84f5a4f3dc805e7893853b0de1dc19cb01c1319f
Summary:
To get the size in bytes, we need to multiply the quantity with the block size,
not with itself.
Reviewed By: genevievehelsel
Differential Revision: D27690857
fbshipit-source-id: 7d7ca767881b1118fc24befed230a63f342bc911
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
Non-list optional data can be present in some XDR description, let's special
case it so the intent is clear when declaring XDR datastructures.
Reviewed By: fanzeyi
Differential Revision: D27306768
fbshipit-source-id: 9d4d18bf8deff16f859c6d28a2579341dac8ee6f
Summary:
After receiving a network packet, it's possible that more than one fragment
were received as part of it. We thus need to service all of them before
returning.
This would typically be seen when running `rg` in the repository, which would
cause hangs due to some requests not being serviced as they would stay in the
iobuf queue until a new packet was received.
Reviewed By: kmancini
Differential Revision: D27194038
fbshipit-source-id: 3d81c797b5be7d0466d4acad7208f6a82593b4ca
Summary:
Computing the length of an iobuf chain can be expensive due to having to walk
its entirety. Thanksfully, IOBufQueue can cache the total length when data is
appended to it, which makes computing the length a constant operation.
Reviewed By: kmancini
Differential Revision: D27194037
fbshipit-source-id: af659c162ada61f2796bf407f419f5f15e918c02
Summary:
By moving the work to a background threadpool, we can more quickly go back to
servicing incoming NFS requests and thus allow more work to be done
concurrently. This would allow tools like ripgrep to being able to use multiple
cores to search in the code base.
Reviewed By: genevievehelsel
Differential Revision: D27194040
fbshipit-source-id: 7f1775ddaaa7eaf8776a06d05951cb936cd3fbb5
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
Summary:
This should have been added in D27243075 (5a150e125a) but I forgot to run `hg add` and it
was thus not added...
Reviewed By: fanzeyi
Differential Revision: D27279169
fbshipit-source-id: 69807cc05fef33f51b2a491b66c2e8aeb7136deb
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
