Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
Reviewed By: kmancini
Differential Revision: D35851539
fbshipit-source-id: d42b12c77ddeacb4777f2e360fdebd1cbbc1bd6d
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
Reviewed By: xavierd
Differential Revision: D35573251
fbshipit-source-id: 5339212de0e617df13277ffcb5a67a535263847d
Summary:
When we try to switch to platform 10 we get this compile error:
```
eden/fs/nfs/Nfsd3.cpp:1378:26:
error: call to consteval function
'fmt::basic_format_string<char, const unsigned int &>::basic_format_string<const char *, 0>'
is not a constant expression
return fmt::format(fmtString, std::get<typename T::TrueVariant>(val.v));
```
The problem is that fmtString either needs to be a `basic_format_string` or
needs to be a constexpr so that we can cast it to `basic_format_string` through
various means.
Not totally sure why this worked pre c++ 20, but C++20 introduced some stricter
type checking on the fmt library or constexpr stuff in general i guess.
https://github.com/fmtlib/fmt/issues/2438 outlines the possible fixes
`fmt::runtime` looks like the best way.
Reviewed By: vitaut
Differential Revision: D35756658
fbshipit-source-id: 7b65ccf0719a964ab1f2fc1aed72744e72374d20
Summary:
TL;DR: File invalidation after checkout is broken on NFS macOS. This proposes a
fix.
Previously, to invalidate things on NFS we opened and closed all the parent
directories of any files/directories that changed during a checkout operation.
This worked on Linux because all open calls result in some request into the
EdenFS daemon (usually a getattr I think). The returned response from this
would show that the directory had update timestamps. So the open would see the
parent directories in their updated state. This would trigger the NFS client to
clear it's caches for that directory and their recursive children to preserve
CTO. CTO or close-to-open consistency guarantees that if on process observed a
file in state A and closed the file, then another process opened that same file
, it will see the file in state A or a later state.
macOS does not seem to do CTO.
It seems that most of the "read" filesystem calls can basically be served from
cache on NFS. Only writes seem to be guaranteed to make it into eden.
So instead of using a "read" filesystem call to trigger invalidation, we need to
use a write one.
I tried opening things in write mode, but the problem is that we need to
invalidate directories (to ensure the entry information is updated) and
directories can not be opened in write mode.
I tried writing 0 bytes to the files themselves that have changed, but this
empty write is short circuited somewhere in the kernel.
The only filesystem call that can be a noop, and seems to trigger a call into
eden is chmod. We are not really working off any guarantees any more, but
it seems to work on both Linux and macOS in my testing so far and is better
than our current broken state of invalidation.
So now to invalidate we chmod parent directories that have changed with their
current mode. Note that this could get extra tricky if we were mixing updating
directory permissions with invalidating them. We would need to ensure we chmod
with the most up to date mode bits. However, because we do not update
permissions anywhere that we also invalidate (checkout does not know how to
update directory permissions) things are a little simpler.
It's important that the chmod can not complete until it seems an updated view of
the parent directory. So we have to be careful about locking here. (Although
that hasn't really changed since we switched from open to chmod.)
One thing that does change is that since chmod is technically a write, it could
cause extra materialization that would be bad for checkout and status
performance. To prevent these problems I have updated setattr to not materialize
a directory when the attribute change is a noop which it should be in our
invalidation technique unless another client modified the directory in the
meantime in which case the directory should be modified anyways. This would
mean that we could end up wiping out clients changes to permissions in the
working copy during checkout. but this matches non eden checkout behavior. So I
think this is ok.
Reviewed By: xavierd
Differential Revision: D35435764
fbshipit-source-id: 196c4995b130b595f9582204237e726e5212993f
Summary: This code has enough risk of a copy paste error that it deserves a unit test.
Reviewed By: chadaustin
Differential Revision: D35161787
fbshipit-source-id: 5691d13a74a0f059dfd6a93ea2852dca8399a165
Summary:
We have seen that sometimes the a client sends us nfsv4 requests with the
nfsv3 version on it.
We fail to parse those.
It's likely the kernel that is sending us that event, but technically any client
could connect to us, so we don't know that for sure.
Let's make sure we log if we see more than one client connected to us. so that
we can confirm if this is the kernel or not.
We should probably also not let non kernel clients connect to us. But I have not
figured out how to do that. tbd
Reviewed By: fanzeyi
Differential Revision: D35448531
fbshipit-source-id: e0810c8961c18b305b80bb874ae4f6aee9583d07
Summary:
In the NFS protocol, the client does not completely resolve permissions based
on the access bits in the mode field of stat.
Clients make a special call into NFS, ACCESS, to check if a certain user has
permissions to perform some operation on a file or directory.
Access was perviously unimplemented for NFS and returned open permissions.
This fully implements the access procedure.
Reviewed By: xavierd
Differential Revision: D34632815
fbshipit-source-id: eda39e02c12c189cd1fc3a32dbe864f00d2c3458
Summary:
If an NFS client sends us an improperly formatted request, EdenFS crashes.
This is dangerous because any process on the machine could send us an NFS
request and this could crash eden.
Eden should be resilient to badly formatted requests/requests it fails to
parse.
This will make debugging failed requests harder as errors will be less
obvious to the user, but it will also allow users to keep using their mount
after a failed request.
Reviewed By: xavierd
Differential Revision: D34981438
fbshipit-source-id: 107de2324b1dc145bd426398614ee76b72c5c446
Summary:
A EdenFS messenger dogfooder on macOS is hitting our check here that file
handles be a certain size.
This likely indicates that we have a big somewhere in parsing that is causing
us to parse something that is not a file handle as a file handle. Its also
possible this request could be malformed.
Eventually we should not trust clients enough to allow them to crash eden with
a malformed request, but the first step to fixing this bug is understanding
wether the request that crashes us is malformed or not.
So first I am adding some extra logging info to this xcheck so that we can
determine the cause. I will follow up with a solution to prevent clients
from crashing EdenFS.
Reviewed By: xavierd
Differential Revision: D34969740
fbshipit-source-id: 783ca1147f6c9d9c86996836d14263a0a79d6b7d
Summary:
Refactor the existing notification class so it uses the new Notifier interface. The notifier interface will be used to set up notifications/notification trays on different platforms.
The notifier interface is not fully completed yet. More methods will be added in a future diff.
As of now, we only have 1 Notifier, the command notifier. I hope to remove this notifier once we implement notifiers for all platforms that we support.
Reviewed By: chadaustin
Differential Revision: D34291809
fbshipit-source-id: a2a67af2683f1f88781428e8d88191f49e100e96
Summary:
When ran in the destructor, the class inheriting from RequestContext would have
been destroyed, and thus the vtable for the RequestContext would no longer from
the child class. In practice that meant that the PID of all requests was always
a nullopt, making `eden minitop` and others completely useless.
To solve this, we can simply create a shared_ptr with a custom deleter that
calls finishRequest first before calling the destructor.
Reviewed By: genevievehelsel
Differential Revision: D34160296
fbshipit-source-id: 7f6d9bfa36bc3db256cfe0f61478c22a44028001
Summary:
In the case where the shared_ptr holding a RequestContext is copied, some stats
may still be updated after the request completes. Thus it's best to call
finishRequest in the destructor to make sure the stats are properly updated.
Reviewed By: chadaustin
Differential Revision: D33724775
fbshipit-source-id: 5e4b247d44bdbeb322f25316dc98798e551cc43b
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary: Instead of taking a uint64_t, we'll take in an InodeNumber. This allows us to avoid the conversion from InodeNumber -> uint64_t -> InodeNumber.
Reviewed By: chadaustin, xavierd
Differential Revision: D33135685
fbshipit-source-id: c9e4317b1f5f97ad924dce5da322860c91c64115
Summary: don't include this header if we're on Windows
Reviewed By: xavierd
Differential Revision: D33111628
fbshipit-source-id: 8bf6c7db4f023f7c348f1e1c439c15856acbf476
Summary: In the next diff, we need to use statToPostOpAttr() in another file. We will move these utility functions so we can access them from any file.
Reviewed By: xavierd
Differential Revision: D32771540
fbshipit-source-id: f4d6e2819da3ef66248134337f98eb0a938b6edb
Summary: This method is needed so that the NfsDispatcher layer can fill in missing stat data for each DirList entry.
Reviewed By: kmancini, xavierd
Differential Revision: D32770593
fbshipit-source-id: 2510adf8b8738259d5e6b8714c6492fd4f75f5d3
Summary:
We account for args.dircount in this implementation (dircount is the max number of bytes we can use for holding directory information)
We *don't* account for args.maxcount. We should probably do that.
Reviewed By: kmancini, xavierd
Differential Revision: D32767072
fbshipit-source-id: c07d0f3aec5a13ed2bb3fa6960ea65e95438dff3
Summary:
The current readdir implementation uses NfsDirList to hold readdir results. Our new readdirplus implementation should use the same struct. To do this, we will add the ability to use an entryplus3 XdrList instead of an entry3 XdrList.
We also add a new constructor so we can specify which XdrList to initialize when we create an NfsDirList.
Reviewed By: xavierd
Differential Revision: D32708688
fbshipit-source-id: e2b324dd1dffc8620be89dca471043aba9b167a5
Summary:
Drop all the remaining instances of `IOBufQueue::clear()`.
(Note: this ignores all push blocking failures!)
Reviewed By: yfeldblum
Differential Revision: D32941992
fbshipit-source-id: c7d983e215a78014ffda74a944b628e1886c35c8
Summary:
static_assert sizeof doesn't show the actual size in clang or gcc, so
implement a CheckSize template.
Reviewed By: xavierd
Differential Revision: D32998926
fbshipit-source-id: 432fbf19eae5b4572e5a4908f9007b7242a3ae69
Summary:
I was using this to debug graceful restart locally, and had it at a high debug
level. DBG1 is a bit verbose this should probably be DBG7 with the rest of the
debug logging I added.
Reviewed By: genevievehelsel
Differential Revision: D32902321
fbshipit-source-id: 4b7db7cbe96db8ef5b1035932c06a1de264f8502
Summary:
We've seen several reports of users getting confused by the output of `df` and
thinking that EdenFS was using 100+ GBs of disk space. In reality, EdenFS is
simply reporting the stats from the underlying filesystem, and thus isn't a
true representation of the disk space usage.
To avoid this confusion, let's simply pretend that EdenFS doesn't use any disk
space. This isn't exactly true as the overlay and some internal datastructures
are using space already, but `eden du` is the right way to see these.
Reviewed By: fanzeyi
Differential Revision: D32536715
fbshipit-source-id: ec9b27084340d2e69cc5a14d6aa71def5fb4465d
Summary:
Provide a no-arg constructor, for C++20
This avoids the following errors:
```
eden/fs/nfs/rpc/Server.h:234:63: error: no matching constructor for initialization of 'facebook::eden::RpcTcpHandler::State'
explicit StateWrapper(folly::EventBase* evb) : evb_{evb}, state_{} {}
^ ~~
eden/fs/nfs/rpc/Server.h:221:5: note: candidate constructor not viable: requires single argument 'state', but no arguments were provided
State(const State& state) = delete;
^
eden/fs/nfs/rpc/Server.h:223:5: note: candidate constructor not viable: requires single argument 'state', but no arguments were provided
State(State&& state) = delete;
```
Differential Revision: D32521305
fbshipit-source-id: 70d53db2093b41b58d4abc0b20389b0169b7a2e3
Summary:
For restart for NFS we now have the pieces to transfer nfs mounts.
The last piece of restart is transferring the mountd socket. This extracts the
socket from the current mountd and provides it to the takeover server, as well
as implements reading the mountd socket from the takeover client and starting
the mountd with this server socket.
Reviewed By: xavierd
Differential Revision: D31802786
fbshipit-source-id: 1727ea0913bfe1dcb02fdbe27c8cc3b9d62a8464
Summary:
For restart for NFS we now have the pieces for the nfs channel to
perform takeover and for the takeover code to transfer the data.
This commit plumbs everything through so that we are able to stop and nfs
channel on takeover. this means extracting the connected socket and plubming it
to the takeover server.
Reviewed By: xavierd
Differential Revision: D31357367
fbshipit-source-id: 91973f7c149ae9209e85aa40111f41e7753db63f
Summary:
For restart for NFS we now have the pieces for the nfs channel to
perform takeover and for the takeover code to transfer the data.
This commit plumbs everything through so that we are able to start and nfs
mount during takeover. This means starting the NFS channel from an existing
connected socket.
Reviewed By: xavierd
Differential Revision: D31802788
fbshipit-source-id: 9034c1cc93e6bb64c09a203d265cf3bdedd1e160
Summary:
We are working towards supporting graceful restart on NFS so that NFS users will
have this available when we start the migration.
This commit is teaching the Server at the core of the nfsd and mountd to
support takeoverShutdown and takeoverStartup. This means stopping processing
requests when we call takover shutdown and passing the sockets (both server
socket and connected sockets) up the stack.
For now we are only supporting takeover once everything has been initialized
and not during mount or unmount.
This means for the nfsd we will only care about the one connected socket which
is being used to communicate with the kernel. we will pass through the
shutdown promise for the channel. For the mountd we only care about the server
socket which we will return from the takeoverShutdown method itself.
Then we also add being able to initialize itself from a server socket or
a connected socket.
Reviewed By: xavierd
Differential Revision: D31802790
fbshipit-source-id: 06ef4da48cd864bfa20ea0fb98485792255d98cd
Summary:
AsyncServerSocket works like this:
1) a socket is accepted from the fd
2) the socket is pushed into the socket queue
3) the connectionAccepted() callback is called with the socket is dequeued
Things happening before connectionAccepted() are like black box to us currently. For example the time taken for a socket to sit in the socket queue is completely unknown. Adding this AcceptInfo into the callback could allow us to pass more information to downstream for instrumenting.
This diff only pass timestamp of before the socket is being pushed into the queue.
Reviewed By: yfeldblum
Differential Revision: D31633582
fbshipit-source-id: 66072955318388c2410b48313d3d54078edce478
Summary: This adds inode number to NFS trace event so that we can use it in ActivityRecorder to show the filename of the FS request.
Reviewed By: xavierd
Differential Revision: D30849770
fbshipit-source-id: 580faf5fccb1a225399d9aec843e23eae1874e87
Summary: This adds the support for FS events logging for NFS. For context, each type of event is assigned a sampling group that determines its sampling rate. In TraceBus subscription callback, events are sent to `FsEventLogger` to be sampled and logged through `HiveLogger`.
Reviewed By: xavierd
Differential Revision: D30843863
fbshipit-source-id: 65394d31b1197efd69c7fd4c1b24562f5abd5785
Summary: As title. `RequsetContext` allows us to track metrics such as latency and count.
Reviewed By: genevievehelsel
Differential Revision: D29835813
fbshipit-source-id: 6b85fc8f11923f530fce6d871fa2253db21bfa98
Summary: We already have AccessType for FUSE, this adds the same categorization for NFS. This allows us to easily filter events in trace stream and ActivityRecorder.
Reviewed By: chadaustin
Differential Revision: D29771074
fbshipit-source-id: a437f0693f9062fb2df3b6f618a9d8860a05df12
Summary: This diff connects the trace bus with trace fs command.
Reviewed By: chadaustin
Differential Revision: D29367135
fbshipit-source-id: f9217b286c1a21805d70b21282c10d4ad722a391
Summary: If the handler throws without returning a future, the finish event will not be published. This diff adds the `LiveRequest` to publish finish event in its destructor.
Reviewed By: chadaustin
Differential Revision: D29332452
fbshipit-source-id: 880a4b67ba47b737063a3955c9f4bdbf605f1a43
Summary:
This diff adds tracking for nfs, based on the implementation of that of `FuseChannel`.
Also commented below few questions on the implementation before working on integrating to `strace`.
Reviewed By: chadaustin
Differential Revision: D29279126
fbshipit-source-id: de6bb36dfbe2f550a91f2bf254616bbc639c0c3d
Summary: NfsTaskQueue can be made more generic to be shared across the codebase, so this makes it its own target in `eden/fs/utils` w/ the name EdenTaskQueue.
Reviewed By: xavierd
Differential Revision: D29244762
fbshipit-source-id: 78348f2ff8fa66bc801aefe7d6b3905e0da278e8
Summary:
Previously if you `sudo umount -f fbsource-nfs` then try to
`eden mount fbsource-nfs`, the mount will fail because the EdenMount already
exists and is still running.
Let's properly unmount our selfs on a force unmount like we do for fuse.
There are two potential ways to detect a fource unmount: the UMNT call to the
mount deamon or the socket to the nfsd closing. The UMNT call is unreliable
(on Linux we do not get the UNMT call on `umount -l`), so this diff pursues the
socket closing option.
When the nfsd socket is closed we trigger the EdenMount unmounting process if
this has not already started.
Reviewed By: xavierd
Differential Revision: D28329482
fbshipit-source-id: 5df8f3eb818a92536095195f1b3a9e412394fbf6
Summary:
For non-UTF8 names, the PathComponent constructor would raise an exception, and
since that exception wasn't caught by the handler itself, it would bubble up to
the RPC server and a generic "server IO error" would be sent back to the
client. Since non-UTF8 names aren't a server error, but an invalid argument, we
should instead return a different error.
Unfortunately, EILSEQ isn't an error that an NFS server can return, instead
let's use EINVAL as the argument is clearly invalid.
Reviewed By: chadaustin
Differential Revision: D28482032
fbshipit-source-id: b59044f1a76f7eac79e2df07356a0aeafa22e3c5
Summary:
The modeToNfsMode simply didn't consider all the mode bits to be translated to
the proper NFS mode bits. It now does.
Reviewed By: chadaustin
Differential Revision: D28459428
fbshipit-source-id: d879fb1be2085e44110ba552bc47d2770637fc86
Summary:
An NFS client caches the attributes of files to avoid having to request these
very frequently. What this means is that a file changed by another client (or
by the server itself) may take some time to be reflected on the client, that
time depends on the attribute caching configuration of the mount point.
For EdenFS, files can changed in 2 ways:
- Either it is changed by the user via the mount point,
- Or the user runs an `hg update`
For the first one, the client will simply update its attributes appropriately,
but for the second one, the cached attributes will only be updated when the
user does opens the file, any calls to stat prior will return the old
attributes. Since EdenFS runs on the same host, we can force the attributes
caches to be discarded by simply issuing an open call on the file that changed.
Reviewed By: chadaustin
Differential Revision: D28456482
fbshipit-source-id: 91022d35a33e436c47d94403d0c139992f880cf9
Summary:
Now that makeImmediateFutureWith exists, we can simply use it instead of
constructing a ImmediateFuture<folly::Unit> and calling thenValue on it.
Reviewed By: chadaustin
Differential Revision: D28518059
fbshipit-source-id: 0041cf863fb32efab274f11c77c76109ca9b454f
Summary:
Now that both implementation are using ImmediateFuture, we can move the
ImmediateFuture one layer up.
Reviewed By: kmancini
Differential Revision: D28302479
fbshipit-source-id: 3c2c164a90ffb42a0e7da8528f976af34fc87315
Summary:
With the Mountd code being converted to ImmediateFuture, we can now convert
Nfsd3 to use ImmediateFuture too. For now, this isn't expected to perform
better due to the InodeMap still using on folly::Future, which forces the
ImmediateFuture code to store a SemiFuture. A future change will look at
switching the InodeMap to ImmediateFuture to solve this.
Reviewed By: kmancini
Differential Revision: D28297422
fbshipit-source-id: 8b85103657e877b0f102130f2117bbe60598ed52
Summary:
As a first towards converting the code to use ImmediateFuture more broadly,
let's start with a small self contained part of the code.
This is mostly a sed except for the dispatchRpc method that needs to call
.semi().via().
Reviewed By: kmancini
Differential Revision: D28297421
fbshipit-source-id: e706e91fc8f132d4ef742ae98af9bb8304e0bf36
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
macOS supports NFS servers that can be reached via a unix socket as a way to
improve performance by reducing the TCP cost. To support this, let's first
allow the socket to bind to to be passed to the RpcServer, and then pass it
through to the privhelper code.
Reviewed By: kmancini
Differential Revision: D28261423
fbshipit-source-id: 78c60aac26353d1da76a67897429b964332df8b3
Summary:
folly::via is a Future API, and thus it creates one, which requires allocating
it and then attaching it to the Executore. Since the code to dispatch a request
isn't Future based, we don't need to use folly::via, and we can simply add the
lambda to the Executor directly. This removes expensive memory allocations from
the EventBase.
Reviewed By: kmancini
Differential Revision: D27976674
fbshipit-source-id: 8fa9724a94ba69b071ab894cdbbad0d33733c098
Summary:
Neither macOS, nor Linux are sending multi-fragment requests to the NFS server.
Since supporting these means calling into memmove, which can be expensive for
large requests, let's just remove support for them for now. If somehow macOS
and/or Linux start sending these, the XCHECK(isLast) will catch this and we can
fix the code by then.
Reviewed By: kmancini
Differential Revision: D27976671
fbshipit-source-id: 77c758b2bb36517d22d5b637e6f0ebf84cc19e5b
Summary:
The EventBase is single threaded, and for heavily concurrent client workflows,
it could see a lot activity, thus every cycle saved can be used to drive more
client requests. The construction of the IOBuf doesn't need to be done while in
the EventBase, thus let's build it outside.
Reviewed By: kmancini
Differential Revision: D27976670
fbshipit-source-id: c6c015ef26df1dcb3fc0c5f179e474bafbd71fac
Summary:
Passing 64 to preallocate means that the AsyncSocket code will issue reads of
64 bytes, even though the IOBufQueue has significantly more space available. We
can thus pass a bigger size to preallocate to reduce both the cost of
allocation, and the syscall cost. For heavily concurrent client code, this will
allow us to read more than one request per syscall.
The careful reader may have noticed that for very small requests the code may
reallocate more often that it should as it will always reallocate when falling
under 4KB. This is likely to not be an issue in practice.
Reviewed By: kmancini
Differential Revision: D27976672
fbshipit-source-id: 4c7e3aecc4763ab20854f3c466ce0872332f9b77
Summary:
These are various cleanups that should make the code easier to read, there is
no behavior changes.
Reviewed By: kmancini
Differential Revision: D27976673
fbshipit-source-id: 470eb628ca75bf1712a93c6e9aa3a27c3f314d01
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
Summary: Since HeaderClientChannel now accepts a transport unique_ptr there's no need to have this deleter exposed outside of HeaderClientChannel.
Reviewed By: iahs
Differential Revision: D27729209
fbshipit-source-id: 064b03afdfe567b6df6437348596f0f6f97f6aaf
Summary:
This allows unix sockets to be created in the mount. This will allow Buck to
run properly as it tries to create sockets in the repository.
Reviewed By: kmancini
Differential Revision: D27690406
fbshipit-source-id: 5725d68bdda12f3a5882ce48b6bdd02b14cdece4
Summary: This merely adds the types for the procedure
Reviewed By: kmancini
Differential Revision: D27690405
fbshipit-source-id: b94fb03658cabaece4166c29135c5fdf9a613d3c
Summary:
This is roughly the same logic as the UNLINK one with the only difference being
in the handling of "." and "..".
Reviewed By: kmancini
Differential Revision: D27684716
fbshipit-source-id: 86a95c38e6c783bc3a45c0a8b000d0210b6dd0b8
Summary:
This merely adds the types needed for the RMDIR procedure. Implementation will
follow.
Reviewed By: genevievehelsel
Differential Revision: D27684736
fbshipit-source-id: 84f5a4f3dc805e7893853b0de1dc19cb01c1319f
Summary:
To get the size in bytes, we need to multiply the quantity with the block size,
not with itself.
Reviewed By: genevievehelsel
Differential Revision: D27690857
fbshipit-source-id: 7d7ca767881b1118fc24befed230a63f342bc911
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
Non-list optional data can be present in some XDR description, let's special
case it so the intent is clear when declaring XDR datastructures.
Reviewed By: fanzeyi
Differential Revision: D27306768
fbshipit-source-id: 9d4d18bf8deff16f859c6d28a2579341dac8ee6f
Summary:
After receiving a network packet, it's possible that more than one fragment
were received as part of it. We thus need to service all of them before
returning.
This would typically be seen when running `rg` in the repository, which would
cause hangs due to some requests not being serviced as they would stay in the
iobuf queue until a new packet was received.
Reviewed By: kmancini
Differential Revision: D27194038
fbshipit-source-id: 3d81c797b5be7d0466d4acad7208f6a82593b4ca
Summary:
Computing the length of an iobuf chain can be expensive due to having to walk
its entirety. Thanksfully, IOBufQueue can cache the total length when data is
appended to it, which makes computing the length a constant operation.
Reviewed By: kmancini
Differential Revision: D27194037
fbshipit-source-id: af659c162ada61f2796bf407f419f5f15e918c02
Summary:
By moving the work to a background threadpool, we can more quickly go back to
servicing incoming NFS requests and thus allow more work to be done
concurrently. This would allow tools like ripgrep to being able to use multiple
cores to search in the code base.
Reviewed By: genevievehelsel
Differential Revision: D27194040
fbshipit-source-id: 7f1775ddaaa7eaf8776a06d05951cb936cd3fbb5
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
Summary:
This should have been added in D27243075 (5a150e125a) but I forgot to run `hg add` and it
was thus not added...
Reviewed By: fanzeyi
Differential Revision: D27279169
fbshipit-source-id: 69807cc05fef33f51b2a491b66c2e8aeb7136deb
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
Summary:
In a bunch of places, the code assumes that an EdenMount is associated with a
Fuse channel. With NFS, that's no longer the case, thus let's make sure to
check the return value of mount->getFuseChannel(). In the case where it will
make sense to have something for NFS, I've either added an EDEN_BUG, or a TODO,
so we can come back to it later.
Reviewed By: chadaustin
Differential Revision: D26836431
fbshipit-source-id: c061b8f20199e5af3139a5003827f184f6eac8d4
Summary:
The NFS readdir turns out to be pretty similar to the FUSE one, with a couple
of differences. For one, it only populates the directory entry name, it also
puts a limit on the total size of the serialized result, including all the
NFS/XDR overhead.
It is not specified if the . and .. entries need to be returned, but since the
NFS spec is usually pretty explicit about these and makes it clear that this is
for the most part a client burden, I didn't add these. I may have to revisit
this later when I get to manually browse a repository.
Since the READDIR RPC doesn't populate any filehandle, the client will have to
issue a LOOKUP RPC for each entries, potentially leading to some
inefficiencies. A future diff will implement the READDIRPLUS to fix these.
Reviewed By: chadaustin
Differential Revision: D26802310
fbshipit-source-id: b821b57021d0c2dca33427975b1acd665173bc5c
Summary:
This simplifies a handful of tests and will make writing the READDIR RPC a bit
less magic when computing the amount of memory needed per entry.
Reviewed By: chadaustin
Differential Revision: D26802312
fbshipit-source-id: fc66cb68f721ed34c8f9879cdda2cd8db6ed8daa
Summary: This merely adds the types for the READDIR RPC.
Reviewed By: chadaustin
Differential Revision: D26802313
fbshipit-source-id: 634ff9b3f97dc4dba56d225c1fb9eae0a94c02d5
Summary:
When creating the .hg directory, Mercurial issues a SYMLINK RPC, thus let's
support it.
Reviewed By: kmancini
Differential Revision: D26785005
fbshipit-source-id: a760d55e6117cc3725444c604e3e4036f4a317b2
Summary:
As it's name implies, this RPC is used to rename files. It's not clear whether
all the error cases that the spec specifies are properly covered, but future
tests can uncover this.
Reviewed By: kmancini
Differential Revision: D26771235
fbshipit-source-id: cad1065a5277e2ab169dd34c7d485d6a4cdd4b76
Summary:
This is pattern that is repeated in several functions, let's only have one
function to make the code easier to read.
Reviewed By: kmancini
Differential Revision: D26771236
fbshipit-source-id: 64a68e90eafcea85f850374751ae7bf34f98f118
Summary: This merely adds the types needed for the RENAME RPC.
Reviewed By: kmancini
Differential Revision: D26771238
fbshipit-source-id: 7b9db7b46ffba2d7a906d0e2b60e24df0b5b055d
Summary:
The SETATTR RPC allows for changing various attributes of the file, like it's
mode, uid, gid, etc. The one piece of the NFS RFC that isn't implemented is
that NFS allows for a client to pass a ctime to the server that it needs to
check prior to setting the attributes. This is done to avoid concurrent
operations on the file conflicting with each other. This is not implemented for
now as Mercurial appears to not be using it.
Reviewed By: kmancini
Differential Revision: D26760073
fbshipit-source-id: 3474665fcf1b089ef6f7de4a6c45a26ef324240e
Summary:
This merely adds the types needed for the SYMLINK RPC, the implementation will
follow.
Reviewed By: kmancini
Differential Revision: D26737273
fbshipit-source-id: 4ed3029304fe64892e88bc05a64b4b3b19fd5460
Summary: This is merely adding the type, the implementation will follow.
Reviewed By: kmancini
Differential Revision: D26737271
fbshipit-source-id: 42de7873b271a2bf9499f1274bca50f23dc1016b
Summary:
Both READDIR and READDIRPLUS RPC (and other) are using lists as a way to have
an unsized array of values. From a behavior perspective, this is similar to an
array of XdrOptionalValue but without it being prefixed by a size, and with an
additional last element being empty.
To simplify writing these RPC, let's add this XdrList type.
Reviewed By: kmancini
Differential Revision: D26729816
fbshipit-source-id: 8d14bbc6f0513aac51d65625e5751cedc2071a0b
Summary:
These aren't tied to the RPC protocol but more to how XDR works, therefore it
makes more sense to move these to the Xdr.h file. It also enables tests to use
it instead of having to write the serialization by hand.
Reviewed By: chadaustin
Differential Revision: D26729815
fbshipit-source-id: 227299b94e2f6fa5ecc747a8d5705f0931a7d1f8
Summary: This merely adds the types for the SETATTR RPC.
Reviewed By: kmancini
Differential Revision: D26707362
fbshipit-source-id: 89cbf7a501cf4e13878cd84b6c36691d2a08594f
Summary:
Now that a large part of the RPC is implemented, being able to see what
arguments are being passed in makes it convenient for debugging.
Reviewed By: kmancini
Differential Revision: D26707363
fbshipit-source-id: 676adef996bf584140b324df1419b139f4600c91
Summary:
This reads the data from the file. One of the optimization that isn't done in
this diff would be to detect the EOF without having read return a 0-bytes long
buffer. For now this isn't done, but more plumbing in the FileInode would be
required to achieve this.
Reviewed By: kmancini
Differential Revision: D26704138
fbshipit-source-id: 82e6d4994dd5dc46372778c10d9f68f7ca1e3a87
Summary: This merely adds the types for the READ RPC.
Reviewed By: kmancini
Differential Revision: D26704137
fbshipit-source-id: a803fc9082cbec6ce0c6b9db2bfede55bf478df1
Summary:
While creating the .hg directory, Mercurial seems to want to remove some files
too, thus let's thread this to the inode layer to allow Mercurial to continue.
Reviewed By: kmancini
Differential Revision: D26699272
fbshipit-source-id: e6d283b92e18962698c0df3c6361adfbd0c50223
Summary: This is merely adding the types needed to implement the REMOVE procedure.
Reviewed By: kmancini
Differential Revision: D26694441
fbshipit-source-id: 736d06ed7bfae40f827fbec6017e069fe11d7464
Summary:
Since EdenFS doesn't support hardlinks, this is merely collecting the attribute
of the file before always failing.
Reviewed By: kmancini
Differential Revision: D26682014
fbshipit-source-id: 8c39170147023a03e87ed7a0fae66f9184b4f690
Summary:
The RPC simply queries various filesystem attributes, we merely forward what
statfs on the overlayfs gives us.
Reviewed By: kmancini
Differential Revision: D26681613
fbshipit-source-id: 5b94d05cafff8d77390fe60a3b5cf1dc3e022f42
Summary: This merely adds the types for the RPC.
Reviewed By: kmancini
Differential Revision: D26681615
fbshipit-source-id: d092cf0b6b5bb7435702d125b5c6ea7ee68356dc
Summary:
This simply writes the data passed in to the inode. Note that the current
implementation has a protocol violation since it doesn't sync the data written
to disk but advertise to the client that it did. This is completely wrong from
a data consistency guarantee, but is probably fine for now. Once the code
becomes closer to being production ready, this will be changed to honor the
client asks.
Reviewed By: kmancini
Differential Revision: D26681614
fbshipit-source-id: 82ad7a141be3bbe365363b1f6692ae62f253423f
Summary:
The Appender API doesn't allow us to simply append an IOBuf chain to it forcing
the entire chain to be copied to it. For the most part, this isn't an issue,
but to reduce the overhead of the READ NFS procedure, this will be a nice
benefit.
For the most part, this diff is a codemod, with a couple of places that needed
to be manually fixed. These were in the rpc/Server.{cpp,h} and
rpc/StreamClient.{cpp,h}.
Reviewed By: genevievehelsel
Differential Revision: D26675988
fbshipit-source-id: 04feef8623fcddd02ff7aea0b68a17598ab1d0f8
Summary:
The ser, de and roundtrip are duplicated 3 times in the various tests, let's
move them into a common library.
Reviewed By: chadaustin
Differential Revision: D26675989
fbshipit-source-id: 1da0bc33429795a889b72c76fa18e5fa3cb7df6f
Summary: This merely adds the various types for the WRITE RPC.
Reviewed By: chadaustin
Differential Revision: D26671895
fbshipit-source-id: 8409c8a1f90e97478aed7c9f19b881c46234b539
Summary:
For the READ/WRITE RPC calls, copying data in and out of an IOBuf chain can be
fairly expensive, to avoid this overhead, we can simply clone the data out of
the IOBuf chain directly, saving on the cost of copy.
Since the code uses a folly::io::Appender that doesn't support adding an IOBuf
to it, we still pay the cost of copying data to it, switching to
folly::io::QueueAppender may solve this.
Reviewed By: chadaustin
Differential Revision: D26671896
fbshipit-source-id: 0161f04cb820bf27ef66fdef6b4a1ce4eb778b96
