Summary:
Non-list optional data can be present in some XDR description, let's special
case it so the intent is clear when declaring XDR datastructures.
Reviewed By: fanzeyi
Differential Revision: D27306768
fbshipit-source-id: 9d4d18bf8deff16f859c6d28a2579341dac8ee6f
Summary:
After receiving a network packet, it's possible that more than one fragment
were received as part of it. We thus need to service all of them before
returning.
This would typically be seen when running `rg` in the repository, which would
cause hangs due to some requests not being serviced as they would stay in the
iobuf queue until a new packet was received.
Reviewed By: kmancini
Differential Revision: D27194038
fbshipit-source-id: 3d81c797b5be7d0466d4acad7208f6a82593b4ca
Summary:
Computing the length of an iobuf chain can be expensive due to having to walk
its entirety. Thanksfully, IOBufQueue can cache the total length when data is
appended to it, which makes computing the length a constant operation.
Reviewed By: kmancini
Differential Revision: D27194037
fbshipit-source-id: af659c162ada61f2796bf407f419f5f15e918c02
Summary:
By moving the work to a background threadpool, we can more quickly go back to
servicing incoming NFS requests and thus allow more work to be done
concurrently. This would allow tools like ripgrep to being able to use multiple
cores to search in the code base.
Reviewed By: genevievehelsel
Differential Revision: D27194040
fbshipit-source-id: 7f1775ddaaa7eaf8776a06d05951cb936cd3fbb5
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
Summary:
This should have been added in D27243075 (5a150e125a) but I forgot to run `hg add` and it
was thus not added...
Reviewed By: fanzeyi
Differential Revision: D27279169
fbshipit-source-id: 69807cc05fef33f51b2a491b66c2e8aeb7136deb
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary: For the prefetch-profile feature, added a matching enable option to go with the existing disable option. Enable temporarily enables the prefetch-profile feature for cases where a user has already used disable.
Reviewed By: kmancini
Differential Revision: D27191711
fbshipit-source-id: cf9cb7d4054e9fcee66bdb5f205562ec79c2757e
Summary:
When copying files from Finder to EdenFS, Finder appears to be issuing a couple
of getxattr calls to EdenFS, however having these being denied by osxfuse
causes the copy to fail, resulting in a subpar experience.
Reviewed By: fanzeyi
Differential Revision: D27137328
fbshipit-source-id: 66bcc2e316d832385da05f0768da9f82ab36a374
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary: It's best to test for a regular file by comparing dtype_t, not mode_t directly.
Reviewed By: chadaustin
Differential Revision: D26851735
fbshipit-source-id: 62d3833a53748ea9b45794ec6390d158e888c241
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
Summary:
In a bunch of places, the code assumes that an EdenMount is associated with a
Fuse channel. With NFS, that's no longer the case, thus let's make sure to
check the return value of mount->getFuseChannel(). In the case where it will
make sense to have something for NFS, I've either added an EDEN_BUG, or a TODO,
so we can come back to it later.
Reviewed By: chadaustin
Differential Revision: D26836431
fbshipit-source-id: c061b8f20199e5af3139a5003827f184f6eac8d4
Summary:
In the case where EdenFS would fail to initialize very early, the optional
server would be empty, and thus trying to dereference it would lead to EdenFS
crashing with no good error message. Let's simply test for this.
Reviewed By: genevievehelsel
Differential Revision: D27162907
fbshipit-source-id: a078b9995a94e5a86cf2893cbebfe7f5ca6d064e
Summary:
The NFS readdir turns out to be pretty similar to the FUSE one, with a couple
of differences. For one, it only populates the directory entry name, it also
puts a limit on the total size of the serialized result, including all the
NFS/XDR overhead.
It is not specified if the . and .. entries need to be returned, but since the
NFS spec is usually pretty explicit about these and makes it clear that this is
for the most part a client burden, I didn't add these. I may have to revisit
this later when I get to manually browse a repository.
Since the READDIR RPC doesn't populate any filehandle, the client will have to
issue a LOOKUP RPC for each entries, potentially leading to some
inefficiencies. A future diff will implement the READDIRPLUS to fix these.
Reviewed By: chadaustin
Differential Revision: D26802310
fbshipit-source-id: b821b57021d0c2dca33427975b1acd665173bc5c
Summary:
This simplifies a handful of tests and will make writing the READDIR RPC a bit
less magic when computing the amount of memory needed per entry.
Reviewed By: chadaustin
Differential Revision: D26802312
fbshipit-source-id: fc66cb68f721ed34c8f9879cdda2cd8db6ed8daa
Summary: This merely adds the types for the READDIR RPC.
Reviewed By: chadaustin
Differential Revision: D26802313
fbshipit-source-id: 634ff9b3f97dc4dba56d225c1fb9eae0a94c02d5
Summary:
Looking at the spec, READDIRPLUS appears to be more complex to implement than
READDIR, for now, let's force the use of READDIR. Future changes will have to
implement READDIRPLUS as that will likely be a perf improvement.
Reviewed By: chadaustin
Differential Revision: D26802311
fbshipit-source-id: cb784d74507e6c2c2ba4dc0aebe69cfcd69db40b
Summary:
This type is very specific to Fuse, let's make it obvious. The readdir method
has also been renamed as it is also very specific to Fuse.
Reviewed By: chadaustin
Differential Revision: D26802309
fbshipit-source-id: c2acdfd1c0006935c59b685fcda729e1bef88928
Summary:
When creating the .hg directory, Mercurial issues a SYMLINK RPC, thus let's
support it.
Reviewed By: kmancini
Differential Revision: D26785005
fbshipit-source-id: a760d55e6117cc3725444c604e3e4036f4a317b2
Summary:
This is mostly just copying /usr/include/linux/fuse.h from my devserver and
updating some flags in FuseChannel to display the new flags.
Reviewed By: chadaustin
Differential Revision: D27144667
fbshipit-source-id: 4854c6edd4c793ca707db26fecd11e2a3e9d7b75
Summary:
Some of our subprocess calls are running into dylib errors. The cause looks to
be related to our environment variables. We already have environment hygenics
for buck, so lets borrow this to use elsewhere.
This is to fix prefetch profile fetching on mac, but I ran into another error
when testing `eden du --clean`.
Reviewed By: genevievehelsel
Differential Revision: D27135268
fbshipit-source-id: 3955ddefc5e9ff60e966f63f7dc65ef737186464
Summary: Free the statement cache before trying to release the database. This is causing issues in `eden rm`.
Reviewed By: chadaustin
Differential Revision: D27104645
fbshipit-source-id: 74f4c4e73b048f58b83a28c2d280e5cb1c767048
Summary:
suppress the deprecated errors.
found something similar in here: https://stackoverflow.com/questions/1902021/suppressing-is-deprecated-when-using-respondstoselector
example failure:
https://www.internalfb.com/intern/buck/build/a3b550b8-4099-4f27-8975-5bfffd6447e5/
```
eden/fs/inodes/test/OverlayTest.cpp:730:1: error:
'InstantiateTestCase_P_IsDeprecated' is deprecated: INSTANTIATE_TEST_CASE_P is deprecated, please use INSTANTIATE_TEST_SUITE_P [-Werror,-Wdeprecated-declarations]
INSTANTIATE_TEST_CASE_P(
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/gtest-param-test.h:507:38:
note: expanded from macro 'INSTANTIATE_TEST_CASE_P'
static_assert(::testing::internal::InstantiateTestCase_P_IsDeprecated(), \
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/internal/gtest-internal.h:1209:1: note: 'InstantiateTestCase_P_IsDeprecated' has been explicitly marked deprecated here
GTEST_INTERNAL_DEPRECATED(
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/internal/gtest-port.h:2215:59:
note: expanded from macro 'GTEST_INTERNAL_DEPRECATED'
#define GTEST_INTERNAL_DEPRECATED(message) __attribute__((deprecated(message)))
```
Reviewed By: mzlee
Differential Revision: D27037957
fbshipit-source-id: b12cc500441c9ed4ed72825475c57047fb0c2076
Summary: This diff replaces `SqliteStatement` in `TreeOverlayStore` with the new `PersistendSqliteStatement`.
Reviewed By: chadaustin
Differential Revision: D26656314
fbshipit-source-id: dbbfeddd52d3be2a1b5d3b6121d3694c41841fd5
Summary:
This diff makes `SqliteStatement` to construct persistent statement by default.
This change should be harmless if the caching assumption is not true. If the statement is not cached by the caller, it's probably on a non-critical path.
Reviewed By: chadaustin
Differential Revision: D26639365
fbshipit-source-id: 26af6ccdba036dc0fccf774e9ffb6bb2a014c5ff
Summary: This diff implements the newly added methods to `IOverlay` so we can build & test `SqliteOverlay` on devserver.
Reviewed By: chadaustin
Differential Revision: D25283647
fbshipit-source-id: 50d0c2dd47cd8f74c860a131ffad1d6342a86ea2
Summary:
Prefetch profiles has been tested to work on windows. Prefetching will be a little slower
on windows as process creation is more expensive, but it is still functional, and should
be ready for us to gather some early numbers on windows sandcastle.
Reviewed By: xavierd
Differential Revision: D27016292
fbshipit-source-id: ed0c8b732b1dfb8bf0a301ae5e76469d4e59f8d2
Summary:
We don't have data on how often this repo-corrupted event occurs, so
begin logging it.
Reviewed By: xavierd
Differential Revision: D26968638
fbshipit-source-id: 95e62a595f74f22fdec8a60ebb0671e53cbe199e
Summary:
It's always annoyed me that HgProxyHash has a constructor which knows
how to load itself from a LocalStore. Add an explicit load() function,
and clean up some other stuff about the class while I'm in there.
Reviewed By: xavierd
Differential Revision: D26769231
fbshipit-source-id: f0ea9f16c3f1fbcd3d4361bcc34845901094b282
Summary: adds a new subcommand `disable` which will update the user's local eden config with a flag to disable prefetching
Reviewed By: kmancini
Differential Revision: D26964514
fbshipit-source-id: 043c89a5848f827377744ebc46fe93c431efee3e
Summary:
Since an EdenFS mount can only be mounted in a single place, we don't need to
support the NLM protocol, thus let's inform Linux to not use it.
Reviewed By: kmancini
Differential Revision: D26771237
fbshipit-source-id: bc534a938ad6348bf10a553c6c4f283f043da4f5
Summary:
As it's name implies, this RPC is used to rename files. It's not clear whether
all the error cases that the spec specifies are properly covered, but future
tests can uncover this.
Reviewed By: kmancini
Differential Revision: D26771235
fbshipit-source-id: cad1065a5277e2ab169dd34c7d485d6a4cdd4b76
Summary:
This is pattern that is repeated in several functions, let's only have one
function to make the code easier to read.
Reviewed By: kmancini
Differential Revision: D26771236
fbshipit-source-id: 64a68e90eafcea85f850374751ae7bf34f98f118
Summary: This merely adds the types needed for the RENAME RPC.
Reviewed By: kmancini
Differential Revision: D26771238
fbshipit-source-id: 7b9db7b46ffba2d7a906d0e2b60e24df0b5b055d
Summary:
The SETATTR RPC allows for changing various attributes of the file, like it's
mode, uid, gid, etc. The one piece of the NFS RFC that isn't implemented is
that NFS allows for a client to pass a ctime to the server that it needs to
check prior to setting the attributes. This is done to avoid concurrent
operations on the file conflicting with each other. This is not implemented for
now as Mercurial appears to not be using it.
Reviewed By: kmancini
Differential Revision: D26760073
fbshipit-source-id: 3474665fcf1b089ef6f7de4a6c45a26ef324240e
Summary: A simple debug command to help check ProjectedFS File State on disk
Reviewed By: chadaustin
Differential Revision: D26918172
fbshipit-source-id: 098724fbb0ca1e8eb6798b3ce669a43e2ea576ee
Summary:
This teaches eden top to read input from the command line which we will use to
switch between help and main page as well as manipulate the process table.
Similar to the python version of eden top q exits. For now we spit back out all
of the other input we get.
Reviewed By: fanzeyi
Differential Revision: D26892622
fbshipit-source-id: 30039d2c2475c036c500c6cb1055c0b2d739691a
Summary:
This adds a subcommand to the rust cli for top command. This sets us up to
start moving over the eden top functionality.
Reviewed By: fanzeyi
Differential Revision: D26892621
fbshipit-source-id: 1d0525291fbad1e5ea97ca22691eacb3fe42ffaf
Summary:
On EdenFS startup, it would always print the following warning:
W0304 16:32:36.103828 642480 EdenConfig.cpp:424] Ignoring unknown section in eden config: /etc/eden/edenfs.rc, key: prefetch-profiles
This is due to that particular config not being specified in EdenConfig.h. By
adding it to EdenConfig.h, the warning disappear.
Reviewed By: genevievehelsel
Differential Revision: D26834504
fbshipit-source-id: 409de118f015226f839cce3ff79a4a2d5b9b43a3
Summary:
Fast-track some tricky-to-avoid getxattr requests from the kernel
itself. It would be best to avoid the context switch entirely, but in
lieu of that, at least don't log anything or enter our more expensive
FUSE handling path.
This is follow-up to D24039130 (37f030ba72), which apparently did not completely
eliminate POSIX ACL lookups.
Reviewed By: xavierd
Differential Revision: D26803589
fbshipit-source-id: 18cba8e3ffc45516e6458872e408ed29a562c7a8
Summary:
Taking a fuse_setattr_in arg means that it can only be used on FUSE, and while
FUSE has been the only backend on UNIX for a while, this is changing with NFS
being added. Thus, we need to find a solution that isn't tied to FUSE.
The solution is fairly simple, let's simply have a struct with optional fields
that needs changing, FUSE and NFS will set these to what needs changing.
Reviewed By: chadaustin
Differential Revision: D26742213
fbshipit-source-id: 16e3e8cdd22d88ace43485d9e3744280de1ee8ad
Summary:
As its name implies FuseDispatcher::Attr is purely FUSE related and thus
shouldn't affect the inodes. It also removes a blocker to using setattr for
NFS.
Reviewed By: chadaustin
Differential Revision: D26742214
fbshipit-source-id: 41a67af0c948d969d5a427f24718de5b134980da
Summary:
This merely adds the types needed for the SYMLINK RPC, the implementation will
follow.
Reviewed By: kmancini
Differential Revision: D26737273
fbshipit-source-id: 4ed3029304fe64892e88bc05a64b4b3b19fd5460
Summary: This is merely adding the type, the implementation will follow.
Reviewed By: kmancini
Differential Revision: D26737271
fbshipit-source-id: 42de7873b271a2bf9499f1274bca50f23dc1016b
Summary:
Both READDIR and READDIRPLUS RPC (and other) are using lists as a way to have
an unsized array of values. From a behavior perspective, this is similar to an
array of XdrOptionalValue but without it being prefixed by a size, and with an
additional last element being empty.
To simplify writing these RPC, let's add this XdrList type.
Reviewed By: kmancini
Differential Revision: D26729816
fbshipit-source-id: 8d14bbc6f0513aac51d65625e5751cedc2071a0b
Summary:
These aren't tied to the RPC protocol but more to how XDR works, therefore it
makes more sense to move these to the Xdr.h file. It also enables tests to use
it instead of having to write the serialization by hand.
Reviewed By: chadaustin
Differential Revision: D26729815
fbshipit-source-id: 227299b94e2f6fa5ecc747a8d5705f0931a7d1f8
Summary: This merely adds the types for the SETATTR RPC.
Reviewed By: kmancini
Differential Revision: D26707362
fbshipit-source-id: 89cbf7a501cf4e13878cd84b6c36691d2a08594f
Summary:
Now that a large part of the RPC is implemented, being able to see what
arguments are being passed in makes it convenient for debugging.
Reviewed By: kmancini
Differential Revision: D26707363
fbshipit-source-id: 676adef996bf584140b324df1419b139f4600c91
Summary:
This reads the data from the file. One of the optimization that isn't done in
this diff would be to detect the EOF without having read return a 0-bytes long
buffer. For now this isn't done, but more plumbing in the FileInode would be
required to achieve this.
Reviewed By: kmancini
Differential Revision: D26704138
fbshipit-source-id: 82e6d4994dd5dc46372778c10d9f68f7ca1e3a87
Summary: This merely adds the types for the READ RPC.
Reviewed By: kmancini
Differential Revision: D26704137
fbshipit-source-id: a803fc9082cbec6ce0c6b9db2bfede55bf478df1
Summary:
While creating the .hg directory, Mercurial seems to want to remove some files
too, thus let's thread this to the inode layer to allow Mercurial to continue.
Reviewed By: kmancini
Differential Revision: D26699272
fbshipit-source-id: e6d283b92e18962698c0df3c6361adfbd0c50223
Summary:
All the callers of these methods have a real context in hand, let's use it
instead of creating a null one.
Reviewed By: genevievehelsel
Differential Revision: D26699271
fbshipit-source-id: 9fb268c9b3194d7e951e30ab5e90c4f2f0388e81
Summary: This is merely adding the types needed to implement the REMOVE procedure.
Reviewed By: kmancini
Differential Revision: D26694441
fbshipit-source-id: 736d06ed7bfae40f827fbec6017e069fe11d7464
Summary:
Since EdenFS doesn't support hardlinks, this is merely collecting the attribute
of the file before always failing.
Reviewed By: kmancini
Differential Revision: D26682014
fbshipit-source-id: 8c39170147023a03e87ed7a0fae66f9184b4f690
Summary:
The RPC simply queries various filesystem attributes, we merely forward what
statfs on the overlayfs gives us.
Reviewed By: kmancini
Differential Revision: D26681613
fbshipit-source-id: 5b94d05cafff8d77390fe60a3b5cf1dc3e022f42
Summary: This merely adds the types for the RPC.
Reviewed By: kmancini
Differential Revision: D26681615
fbshipit-source-id: d092cf0b6b5bb7435702d125b5c6ea7ee68356dc
Summary:
This simply writes the data passed in to the inode. Note that the current
implementation has a protocol violation since it doesn't sync the data written
to disk but advertise to the client that it did. This is completely wrong from
a data consistency guarantee, but is probably fine for now. Once the code
becomes closer to being production ready, this will be changed to honor the
client asks.
Reviewed By: kmancini
Differential Revision: D26681614
fbshipit-source-id: 82ad7a141be3bbe365363b1f6692ae62f253423f
Summary:
The Appender API doesn't allow us to simply append an IOBuf chain to it forcing
the entire chain to be copied to it. For the most part, this isn't an issue,
but to reduce the overhead of the READ NFS procedure, this will be a nice
benefit.
For the most part, this diff is a codemod, with a couple of places that needed
to be manually fixed. These were in the rpc/Server.{cpp,h} and
rpc/StreamClient.{cpp,h}.
Reviewed By: genevievehelsel
Differential Revision: D26675988
fbshipit-source-id: 04feef8623fcddd02ff7aea0b68a17598ab1d0f8
Summary:
The ser, de and roundtrip are duplicated 3 times in the various tests, let's
move them into a common library.
Reviewed By: chadaustin
Differential Revision: D26675989
fbshipit-source-id: 1da0bc33429795a889b72c76fa18e5fa3cb7df6f
Summary: This merely adds the various types for the WRITE RPC.
Reviewed By: chadaustin
Differential Revision: D26671895
fbshipit-source-id: 8409c8a1f90e97478aed7c9f19b881c46234b539
Summary:
For the READ/WRITE RPC calls, copying data in and out of an IOBuf chain can be
fairly expensive, to avoid this overhead, we can simply clone the data out of
the IOBuf chain directly, saving on the cost of copy.
Since the code uses a folly::io::Appender that doesn't support adding an IOBuf
to it, we still pay the cost of copying data to it, switching to
folly::io::QueueAppender may solve this.
Reviewed By: chadaustin
Differential Revision: D26671896
fbshipit-source-id: 0161f04cb820bf27ef66fdef6b4a1ce4eb778b96
Summary:
The CREATE RPC is intended to create regular files on the server. It supports 3
ways of creating them: unchecked, guarded and exclusive. The exclusive mode
allows for a client to being able to re-transmit the creation and for it to not
fail when that happens. The way this is implemented requires the server to
store a cookie passed in by the client and on retransmission to compare it in
order to not fail the file creation. This complexity may not be warranted and
needed in the case of EdenFS, and the spec allows for a server to not support
this operation, thus EdenFS claims to not support it.
The 2 other modes are more traditional, guarded will simply fail if the file
already exist, while unchecked succeed in that same situation. It's not
entirely clear in the RFC what the behavior is supposed to be in the unchecked
case, and whether the attributes needs to be applied. For now, the error is
just ignored, a warning message is logged and a success is returned in this
case. In the future, setting the file attribute (in particular the mode and
size) may be necessary.
Reviewed By: kmancini
Differential Revision: D26654766
fbshipit-source-id: c2e8142b8a4ff756a868e5859fdda4b07e53bddf
Summary: This is unused, and it has a hard dependency on Fuse, so let's remove it.
Reviewed By: chadaustin
Differential Revision: D26742212
fbshipit-source-id: 091556be39e599512d34920503083d03d4c5a0c2
Summary:
This diff adds a gflag option to turn on `TreeOverlay`. As this diff, this
option only works on Windows.
Reviewed By: xavierd
Differential Revision: D25223471
fbshipit-source-id: ccab4c85cf4b08980e67814238457a8eb83af3dc
Summary:
This diff adds `TreeOverlay` that implements the `IOverlay` interface based on
`TreeOverlayStore`. At this point this class does not do much other than
forwarding calls to `TreeOverlayStore`. This layer will become necessary in the
future when we add file handling support to `TreeOverlay`.
Reviewed By: xavierd
Differential Revision: D25223473
fbshipit-source-id: 02d03081ba10c82123fe69c35dbc70f865c8370d
Summary:
This diff adds `TreeOverlayStore` that is used in following diffs to store tree
data in SQLite database. This class encapsulates the SQL statments needed for
storing trees.
Reviewed By: chadaustin
Differential Revision: D24083787
fbshipit-source-id: 61c08617fe50e7eac94c614948f5ba5355dba71a
Summary:
We don't need to care if a directory from overlay exists or just empty. This saves the overlay implementation from keeping track of what directories are stored and which are not.
This diff changes `Overlay::loadOverlayDir` to return `DirContents` instead of `std::optional`. The same method on `IOverlay` remain unchanged to give the backing overlay implementation the freedom to choose.
Reviewed By: chadaustin
Differential Revision: D25507406
fbshipit-source-id: f7edcc55485fabeedfe11e9f269eea15a3cc32ad
Summary:
This diff adds more logging to SQLite so we can see what statements are being
executed in logs. Setting `eden.fs.sqlite=DBG9` will show every SQL statement
being executed and bound values.
Reviewed By: chadaustin
Differential Revision: D25223472
fbshipit-source-id: ed089b7ec112c75d9f7bc63d9fe53f0ec2bd6420
Summary:
This diff introduces `transaction()` to `SqliteDatabase` to make managing SQLite transaction easier.
Usage:
```
db->transaction([](auto& txn) {
SqliteStatement(txn, "SELECT ...");
SqliteStatement(txn, "INSERT ...");
});
```
It will automatically roll back when exception happens, and commit when the closure finishes.
Reviewed By: chadaustin
Differential Revision: D25102179
fbshipit-source-id: 5a4ced8c6eb8016e15b8132be2b5abcc9760e2f9
Summary: This diff adds an alias `SqliteDatabase::Connection` to make it less mouthful when we want to reference the type of the SQLite connection we created
Reviewed By: xavierd
Differential Revision: D25386777
fbshipit-source-id: a6a58d522200fe0bf35ac52f6dad16063c0389d0
Summary:
To support the new _Relational Overlay_, we need to know what is actually being
modified by overlay. This diff adds two method `addChild` and `removeChild` to
IOverlay that supports passing such information to the actual implementation.
Reviewed By: chadaustin
Differential Revision: D24153544
fbshipit-source-id: 04e5f4c231ff9b5fe0ee5be082b1a75d78758bb4
Summary:
This is needed for the next diff in the stack to build on Windows. We never
build anything form `./inodes/overlay` on Windows.
Reviewed By: xavierd
Differential Revision: D24258241
fbshipit-source-id: e1c5234433a1ec92419dcb090c8949800efa605f
Summary:
This diff introduce `IOverlay` that provides a shared interface between
`FSOverlay` and `SqliteOverlay`. This allows us to share more code between
POSIX and Windows, and prepare for the new SQLite-backed Overlay
implementation.
Note as of this diff `IOverlay` is only at a very initial version. The
interface is far from complete and still need a few iteration to change.
Reviewed By: chadaustin
Differential Revision: D23941454
fbshipit-source-id: 69cce9d3852f21b2d19f528e7e05fb183538ace9
Summary:
`folly::format` is a problematic API because it returns a `Formatter` object that may store references to arguments as its comment warns:
```
* Formatter class.
*
* Note that this class is tricky, as it keeps *references* to its lvalue
* arguments (while it takes ownership of the temporaries), and it doesn't
* copy the passed-in format string. Thankfully, you can't use this
* directly, you have to use format(...) below.
```
This has negative safety and performance (encourages reuse of the same formatter) implications because contrary to what the comment says you can use the object directly since it's returned from `folly::format`. For example
```
auto f = folly::format(std::string("{}"), 42);
f.str();
```
is a UB with no compile-time diagnostic (only caught by ASAN).
It's also unnecessary because the `Formatter` object is usually converted to string straight away via `str()` or written to an output stream. Reusing the formatter doesn't make much sense either because the expensive part is formatting, not capturing arguments.
This diff deprecates `folly::format` suggesting `fmt::format` as a potential replacement. `fmt::format` doesn't have the above problem because arguments are always in scope. It also has the following advantages:
* Better compile times.
* Compile-time format string checks.
* Compatibility with C++20 `std::format`.
* Better performance, particularly with format string compilation which eliminates format string processing overhead altogether.
* Smaller binary footprint.
Also remove `folly::writeTo` which is no longer used and the `format_nested_fbstrings` benchmark which is identical to `format_nested_strings` since there is no `fbstr()` any more.
Reviewed By: yfeldblum
Differential Revision: D26391489
fbshipit-source-id: f0309e78db0eb6d8c22b426d4cc333a17c53f73e
Summary:
Remove a bit of boilerplate in the definition of subcommands, and pave
the way for sub-sub commands.
Reviewed By: xavierd
Differential Revision: D26416574
fbshipit-source-id: 1477505cab6c3de2c7ac14fe862d366a707d155b
Summary:
This diff allows the user to specify a merge function in the shape of `merge(&mut T, T);` to customizing merging behavior.
This is used in the next diff to merge the `HashMap` that catches all unknown fields.
Reviewed By: xavierd
Differential Revision: D26415382
fbshipit-source-id: 10801ddc7d41b82d8d165a8b63466aa9b92baaab
Summary: With this diff we can start to handle concrete error types, return correct exit code for Thrift IO errors for example.
Reviewed By: xavierd
Differential Revision: D26412948
fbshipit-source-id: f598ed2d9187126509c149f9d6293025d0f39968
Summary:
Xavier reported seeing a pile of redundant checkout events in his
journal, which caused Watchman to issue a series of no-op, sequential hg
status calls, which caused Buck to time out waiting for Watchman.
We're not sure what paths cause Mercurial to issue no-op
checkOutRevision Thrift calls, but we can certainly filter them out
inside of the journal.
Reviewed By: fanzeyi
Differential Revision: D26699828
fbshipit-source-id: f738b53bbafa4027dd70322d64413246bb5bb828
Summary: This simply adds the types to implement the CREATE RPC call.
Reviewed By: kmancini
Differential Revision: D26654767
fbshipit-source-id: 5972102aebb3b06c6838a28f3a191304efc7c945
Summary:
The MKDIR RPC is the first RPC that modifies the working copy. It is only
partially implemented due to the parent directory attributes not being
collected just yet. Implementing these will be essential for good caching on
the client side. For simplicity reason, this isn't done just yet.
Reviewed By: kmancini
Differential Revision: D26644056
fbshipit-source-id: f99add63a2ce5789a9aa81cefe4d04c2f4a741ed
Summary: This adds all the types necessaryt to implement the MKDIR RPC.
Reviewed By: kmancini
Differential Revision: D26644057
fbshipit-source-id: cfbcf3f0a7212433eba67f2266d5f7bcc2906a88
Summary:
This is a fairly common pattern in the NFS code, and we can reduce code
duplication by automatically implementing the XdrTrait.
Reviewed By: kmancini
Differential Revision: D26639683
fbshipit-source-id: 6e9cb14ce1538834224017b8ef88955e563d6723
Summary: There are no correct uses of the catch-all overload of `folly::exceptionStr`. Every use is a bug of some kind, leading to the impression that this overload ought to be removed.
Differential Revision: D26539622
fbshipit-source-id: dc2ca0781ea02f1327a334bb1fe2e533fa46d1b3
Summary:
If multiple requests to fetch a tree come in at the same time fast enough (where the first request hasn't had the chance to retrieve the data from Mercurial and save it in the cache), we will request to download the tree multiple times. If the tree is not in the hgcache, this means we will make an extra round trip to the server. There is no limit to how many concurrent requests can be made, meaning we could make a large amount of round trips to the server for the same tree.
This adds a tracking mechanism in which we track in progress tree fetches, and if we get a request that is already in the queue or being processed, we just return a future that will be fulfilled by the first request, instead of placing this duplicate request in the queue as well.
This also makes sure if we get a duplicate request, but the duplicate request has a higher priority than the request already in the queue, we will update the priority of the request in the queue.
Reviewed By: chadaustin
Differential Revision: D26355499
fbshipit-source-id: 8d3192cf0f5628c650715f4597c92fc8c9238650
Summary:
This is a long standing bug whereby if you made the terminal too thin,
the formatting code would crash.
This fix just prevents underflow in the precision format string (and
alignment).
Reviewed By: fanzeyi
Differential Revision: D26673865
fbshipit-source-id: 945e6f227c19962ac0926e117600aa9d6a990305
Summary:
The NFS RPC description is filled with variant over nfsstat3 with only 2 cases:
NFS3_OK, and the default one. I was feeling bad having to write the same code
again and again for the deserialize part of the XdrTrait, and thus came up with
a template solution that enables us to remove all of the boilerplate code.
The only drawback is that this moves the variant one additional layer down, and
that means an extra layer of brace when initializing it. From what I know,
this is caused by the -Wmissing-braces warning, maybe we could remove that
warning?
Reviewed By: kmancini
Differential Revision: D26627163
fbshipit-source-id: ab56dee42273f180b2edf4f529221a15154ac2fb
Summary:
"value not present in store" is an unhelpful error message once
serialized through Thrift etc. Provide more useful context when a key
is missing.
Reviewed By: fanzeyi
Differential Revision: D26678102
fbshipit-source-id: 514ac2fe580d1dd7c67fc20c89b75e5d8121c329
Summary:
Previously, when an argument was just a plan filehandle, I just bypassed adding
the structure and just read the handle directly from the stream. kmancini
correctly pointed out that this is a bit confusing, thus let's add the arg
struct everywhere to cut that confusion.
Reviewed By: kmancini
Differential Revision: D26649568
fbshipit-source-id: bc4d6e519450f228d151a3de7acdfc1002c43b38
Summary:
This is merely a forward to the readlink FileInode method. This allows the
.eden directory and its symlinks to be read properly.
Reviewed By: kmancini
Differential Revision: D26619947
fbshipit-source-id: d5f8ef81ce3000c0f0a2f47007affe2c0292ef31
Summary: This adds the various types necessary to implement the READLINK RPC.
Reviewed By: kmancini
Differential Revision: D26610437
fbshipit-source-id: a64644d0499381678170da38886b5dff1c1b9270
Summary:
The LOOKUP RPC is a bit tricky as we need to special case the "." and ".."
names to represent the current and parent directories. Obtaining the parent
directory is inherently racy from the client perspective, and thus EdenFS won't
try to hold the renameLock in this case.
Reviewed By: kmancini
Differential Revision: D26597789
fbshipit-source-id: 9c8dcaf7db84f8c09f7505cab7afed48df79b754
Summary:
Migrate away from streaming Thrift Python, and instead implement `eden
trace hg` in C++, which will eventually let us support Windows.
This also fixes an unexplained crash in `eden trace hg` on macOS.
Reviewed By: xavierd
Differential Revision: D25515377
fbshipit-source-id: 1916aa7f09df37e9dee3479a858932c724f8aed4
Summary: This diff sets up the ability for us to track requests as they are shuffled around in the `std::vector<> queue`. Since the queue is a vector, and since it is sorted everytime a new element is added or removed, we cannot keep track of elements in the queue with indices or references. Instead, we will store the requests in a shared_ptr so we can maintain a pointer to the request no matter where the request is moved around to.
Reviewed By: kmancini
Differential Revision: D26355907
fbshipit-source-id: d714d689963106a4f495221dbcfcbab758ffc7b2
Summary:
NFS clients will act differently depending on what kind of error is returned
from an RPC. A NFS3ERR_NOENT in response to a LOOKUP RPC may for instance
populate a negative lookup cache to avoid re-querying the same file again.
Thus, we need to translate the various errors with their corresponding
nfsstat3 values.
Reviewed By: genevievehelsel
Differential Revision: D26597788
fbshipit-source-id: d22a552196a378d6d7e9cd374e63bb8bcf436ce4
Summary:
It's silly to use `eden prefetch --no-prefetch` to efficiently glob
for filenames. Introduce an `eden glob` command which resolves a glob
relative to the current working directory.
Reviewed By: genevievehelsel
Differential Revision: D25450358
fbshipit-source-id: 45d6dc870d21510e51d5662c75e80385886899fc
Summary:
I want to use EdenError from some code outside of service/, so move
EdenError into utils.
Reviewed By: genevievehelsel
Differential Revision: D25447438
fbshipit-source-id: 2d1ddfa379238369679e84708518a9ba106f76b9
Summary:
As useful as `eden strace` can be, it's hard to correlate inode
numbers to their filenames via the mkdir/create/lookup request that
returned the inode. Add logging for result codes.
Reviewed By: kmancini
Differential Revision: D26287958
fbshipit-source-id: dae4b56513831185b038546f238938b0d21a6bad
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
Summary:
LOOKUP is the next RPC that EdenFS needs to implement, let's start by adding
the RPC types.
Reviewed By: kmancini
Differential Revision: D26562919
fbshipit-source-id: 3a4ce1a039b6405df9ff94757963aa46acf56d92
Summary:
The ACCESS RPC merely tries to see if a client can perform some action on a
file/directory. Until we have a better story around checking the RPC
credentials and passing it around, let's just return the requested access
rights as-is.
Reviewed By: kmancini
Differential Revision: D26562918
fbshipit-source-id: 468e038cc063c289b5554f3faa6a7f99be2731e4
Summary: This merely add the types necessary to implement the ACCESS RPC
Reviewed By: kmancini
Differential Revision: D26562920
fbshipit-source-id: 2f3238e723e97a54896fabb026adf9b92ab4fcd3
Summary:
Take 2 of fixing the lifetime issues of the RpcServer, this time, the
RpcAcceptCallback needs to live for longer than RpcServer itself. This is due
to the acceptStopped callback being called after RpcServer is being destroyed.
Since the RpcServer needs to be destroyed in the EventBase thread, the
acceptStopped callback is delayed, causing it to be invoked after the RpcServer
has freed its memory.
To solve this, we simply need to make the RpcAcceptCallback a delayed
destructor, and hold a reference to itself that goes away once the
acceptStopped callback is called.
Reviewed By: kmancini
Differential Revision: D26556644
fbshipit-source-id: 32cd80f1664649c1ad96f88c4eec6dd2ed6e8c12
Summary:
Now that the privhelper has the necessary code to unmount an NFS mount, let's
do it. With this in place, this enables `eden mount` and `eden unmount` for NFS
mounts. Unfortunately, it appears that during unmount, releasing the Nfsd3
object triggers a use-after-free in the rpc code
Reviewed By: kmancini
Differential Revision: D26500109
fbshipit-source-id: a210761f818b28b1eb0044f7314a0e2b9017848c
Summary:
This is merely fixing a typo that I made previous where I reused the prjfs
request timeout instead of adding a new one.
Reviewed By: kmancini
Differential Revision: D26500110
fbshipit-source-id: b9bf0cbc0d74866cdc2471f126751d6d8e514e21
Summary:
Now that `mount(2)` complete, we can start plumbing the rest of the mount code
to accomodate for NFS. Trying to find a common ground between Prjfs, FUSE and
Prjfs is harder than I wish it would be and thus I wasn't able to find a
satisfatory solution. For now, I went with a std::variant that stores either a
FuseChannel, or the Nfsd3. In the future once Nfs is stabilized and we have a
good understanding of how it differs (and where it doesn't), EdenMount would
probably need a good refactoring.
Reviewed By: kmancini
Differential Revision: D26500111
fbshipit-source-id: f02a2eaf8890261f150d7cdd2cdfd134aa62c2aa
Summary:
Comments affecting runtime behavior disturbs me, so use explicit
structopt annotations on help text. And move those annotations to the
command implementations.
Reviewed By: fanzeyi
Differential Revision: D26412452
fbshipit-source-id: 066dfdd1c54254bae4bd2af65031487b7a1094da
Summary: clap/structopt adds a -V flag to every subcommand by default. Disable that.
Reviewed By: fanzeyi
Differential Revision: D26412093
fbshipit-source-id: 03a0320fd15444f700b359f5ed0ca8c40b10ae1c
Summary:
I don't want the fallback when testing, so add an environment variable
for bypassing it.
Reviewed By: fanzeyi
Differential Revision: D26411754
fbshipit-source-id: f2aea82bf3e79db11e72ad5f5ce33513cfc2f05b
Summary:
Rather than offloading dealing with unexpected errors to each
subcommand, allow them to propogate out. Subcommands can still be
responsible for handling expected "errors" like EdenFS not running.
Reviewed By: fanzeyi
Differential Revision: D26411186
fbshipit-source-id: 4e1c5fb1d7bed495e3e22cca44d3f84f7f4c7f14
Summary:
We should be consistent about how we render mode_t across all of the
FUSE requests.
Reviewed By: singhsrb
Differential Revision: D26286527
fbshipit-source-id: aadf247c0621be79525c4df2da2940c02ee27719
Summary:
By passing the argument to the channel, we can make it so that the NFS code
correctly replies to whether it is case sensitive or not.
Reviewed By: kmancini
Differential Revision: D26500112
fbshipit-source-id: 2988eae403ff3648b50a1a8f0c978be2828ba568
Summary:
By moving the createChannel outside of the EdenMount class, we no longer need
the channelType alias, so let's do it.
Reviewed By: kmancini
Differential Revision: D26500113
fbshipit-source-id: f992ed2aaac5d692d316d06340bf9b219a2d7006
Summary: The RPC just translate the various `struct stat` fields into the fattr3 ones.
Reviewed By: kmancini
Differential Revision: D26389793
fbshipit-source-id: 48c6e109d5d2cb62cab096114c37314d7833035f
Summary:
Similarly to what is done for FUSE and ProjectedFS, the dispatcher is the glue
that sits in between the protocol specific bits and the inodes layer.
For now, this only implements "getattr" but it will be filled overtime as more
RPC can be answered properly.
Reviewed By: kmancini
Differential Revision: D26389795
fbshipit-source-id: 19cf3457feec2ebc100e632cb28c20b11fdde26d
Summary: This merely adds the various datastructures needed to implement GETATTR.
Reviewed By: kmancini
Differential Revision: D26389794
fbshipit-source-id: bda557a21386483449c18aa9e52f4f626b73e69f
Summary:
Timeseries is memory intensive and not really required in the current context
it is being used.
Reviewed By: chadaustin
Differential Revision: D26315632
fbshipit-source-id: ee51c3ad8bef6fce152aa787c8c4602f0b499f92
Summary: This implements the same logic as the `edenfsctl uptime`.
Reviewed By: fanzeyi
Differential Revision: D26412789
fbshipit-source-id: ebcf5f0b4767025ea210f7e9c69116b79436d5d0
Summary:
In Python, no passed in timeout means a 3s connection timeout, let's do the
same in Rust.
Reviewed By: chadaustin
Differential Revision: D26407991
fbshipit-source-id: ad2919e2cb72e5a113499d7e036ae285ecf9ae34
Summary: You can use `instance.get_config()` to get access to global EdenFS configurations
Reviewed By: chadaustin
Differential Revision: D26407350
fbshipit-source-id: 022cc59fd86b2711c15cfd781872465c6ada9081
Summary:
This diff adds `edenfs-config` for loading EdenFS configurations from various
locations.
Reviewed By: xavierd
Differential Revision: D26391272
fbshipit-source-id: 3d34da98b2d530e13cdd831d3dc204e44304c486
Summary:
**Problem:** EdenFS has the classic hierarchical configuration design. We load from `/etc/eden/edenfs.rc` first, then `/etc/eden/conf.d/*` then `$HOME/.edenrc`. The latter can overwrite the former. At the end we have a complete view of the configurations.
`serde` is great, but it can't give us the information of whether it generated a field from `serde(default)` or from de-serialization. So we can't just deserialize then merge the configuration files. We need that information, and nor serde should give us that functionality.
`stack-config` is created to load configurations with serde. It automatically generates code with the intermediate data structure and taking care of merging of multiple configuration files. It derives a data type based on the original struct but wrap each field with `Option<T>` and mark it with Serde. Then it generates the code to merge and build the final concrete configuration data type.
It does not care what kind of data format the configuration file is, as long as it can deserialize into the generate optional type, it accepts it.
Example, say I have a file with this structure:
```
#[derive(StackConfig)]
struct Config {
path: String,
}
```
Then this crate will generate:
```
mod __stack_config_private {
#[derive(serde::Deserialize)]
pub(super) struct ConfigOpt {
path: Option<String>
}
...
}
struct ConfigLoader {
layers: Vec<__stack_config_private::ConfigOpt>,
}
impl ConfigLoader {
fn new() { ... }
fn load(&mut self, layer: __stack_config_private::ConfigOpt> { ... }
fn build(self) -> Config { ... }
}
```
See `examples/parse.rs` for usage.
Reviewed By: xavierd
Differential Revision: D26377547
fbshipit-source-id: 1e07d9867742913fd76ed4f765160f0035a2f2a3
Summary: This diff sets up debug logging for EdenFS CLI with tracing.
Reviewed By: chadaustin
Differential Revision: D26354205
fbshipit-source-id: bcc89fe3eaf4c7ae7642b8c20fd74fd3ea6dd4ee
Summary:
While I was fixing the cli_test I realized that the rest of the tests didn't
enforce strict type checking, let's make sure it is enabled so `pyre -l eden`
works for all the tests now and in the future.
Reviewed By: chadaustin
Differential Revision: D26356267
fbshipit-source-id: 4f026b6f96c410115a6a38d772f8e06ab977293b
Summary:
The default filesystem on macOS (APFS) is case-insensitive, but EdenFS has so
far been case-sensitive except on Windows. Some of the native tools (Unity for
instance), expect macOS to always be case-insensitive, and is thus breaking due
to that.
The safe behavior would be to have EdenFS behave exactly the same as APFS: be
case insensitive. For now, to avoid breaking users this will be done on new
mounts only, and once fully validated, this will be made the default and forced
on.
Reviewed By: chadaustin
Differential Revision: D26356269
fbshipit-source-id: 96ca331d8c9726213520dff3e3563019d0400a95
Summary:
This diff adds a Rust wrapper that will forward any command it fails to parse
to the companion binary `edenfsctl.real` at the same directory. This will allow
us to rewrite some of the subcommands in Rust while keeps the original
functionality.
With `EDENFSCTL_SKIP_RUST` environment variable set, it will skip the argument
parsing process in case of any emergency situation.
Reviewed By: xavierd
Differential Revision: D25758959
fbshipit-source-id: 0fbc69d0e0733ad2608622417be93dc7db2d6fa6
Summary:
My understanding of C++ not being perfect, I thought that these 2 fields would
be default initialized, but it turns out this is wrong, and the values of these
would be whatever is on the stack at the time a non-initialized variant is
used. The XdrSerialize variant test exercise this case and fails when optimized
due to that, this change fixes it.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D26371695
fbshipit-source-id: 489cac846b51508809783bb577c7a97c4e6c060a
Summary:
Instead of having one "Dispatcher" type that the various backend overload,
let's simply have a per-mount type dispatcher type. The previous model worked
fine when EdenFS supported only one way of mounting a repository, but with NFS
coming, unix platform will support both FUSE and NFS, making the Dispatcher
overload nonsensical.
As a behavioral change, the dispatcher lifetime and ownership is changed a bit.
It used to live for the duration of the EdenMount object, but is now tied to
the channel lifetime, as it is now owned by it.
Reviewed By: kmancini
Differential Revision: D26329477
fbshipit-source-id: 3959b90a4909e3ab0898caa308f54686f59a943c
Summary:
As I'm modifying the definition for CheckoutConfig, I'm heavily relying on
`pyre` to tell me about the places I need to fix, but this test not being type
checked means that pyre misses it.
Reviewed By: chadaustin
Differential Revision: D26356268
fbshipit-source-id: 2139c0053b182e656074de3117ee1950e519f19a
Summary: The PATHCONF call gather various POSIX attribute for the passed in file-handle.
Reviewed By: kmancini
Differential Revision: D26270239
fbshipit-source-id: 96a46c58402ce169cacc789c1df3f05f969dcd33
Summary:
The FSINFO procedure is used to collect information about this mount point and
its limits. For now, I've just used random values for most of these, I'll come
back to it later to figure out the ones that makes sense.
Reviewed By: kmancini
Differential Revision: D26269947
fbshipit-source-id: e5fdd961ea7772926208916de0179721930daeab
Summary: Instead of using a plan DBG7, let's use the strace logger to log the RPC calls.
Reviewed By: kmancini
Differential Revision: D26269946
fbshipit-source-id: e8ab7acc658463b9f0f1d4c420f63b2e0c88f46a
Summary: These will be needed when implementing FSINFO.
Reviewed By: kmancini
Differential Revision: D26268588
fbshipit-source-id: 8160f00e988a088769ad682a6de062f6c0443c73
Summary:
Variant are default constructed with the first variant alternative, which means
that when deserialization would not deserialize anything, the first variant
would be initialized. This would mean that deserialize(serialize(myVariant))
would be different from myVariant, which is unexpected.
Reviewed By: kmancini
Differential Revision: D26268586
fbshipit-source-id: ebfd7f9e473509b2a16acdbb6914e828c67c5005
Summary:
With the PrivHelper now being able to try to NFS mount, we can just plug it in
the EdenMount code to actually attempt to mount EdenFS via NFS.
The mount will expectedly fail due to the FSINFO RPC not being implemented just
yet, but that will be the next step.
Reviewed By: kmancini
Differential Revision: D26266143
fbshipit-source-id: a44ae98af76b55a0c24c89e766c072c1a2e1b4fd
Summary:
This will be the main entry point for everything related to the various NFS
programs. For now, it simply starts an nfsd program when registering a new
mount point and returns the ports that mount(2) should use to connect to it.
Reviewed By: kmancini
Differential Revision: D26204060
fbshipit-source-id: 115d2d206109499d5b49c529ebf1f739fc364c23
Summary:
Using the current EventBase can lead to unexpected behavior when the RpcServer
is created on the wrong thread. For instance, if it is created on a thread
pool, and another future is running on that same thread, the RpcServer won't be
able to accept and service new connections, potentially leading to deadlocks.
To avoid this, EdenFS's EventBase is excplicitely passed in, this will ensure
that the RpcServer is created on the proper thread.
Reviewed By: kmancini
Differential Revision: D26266145
fbshipit-source-id: 23211e3aa200c32d2f6fbbfd9ae6fb307896a873
Summary:
MSVC is broken as it doesn't understand the various macros used to generate
XdrTrait for specific types, and I can't figure out a way to make that work.
Even if I provide dummy version of these, the buck build breaks due to glog
trying to redefine the ERROR symbol that some Microsoft headers contain. For
now, let's just ifdef it out since it's unlikely that we're going to be using
NFS on Windows anytime soon.
Reviewed By: singhsrb
Differential Revision: D26293519
fbshipit-source-id: bbaf325c7d1f1688327708360244797a6d48179e
Summary:
Since mounting EdenFS via NFS requires the same privilege as mounting EdenFS
with FUSE, let's re-use the PrivHelper infrastructure that FUSE already uses.
The macOS bit isn't yet implemented as developing on Linux is for now easier,
the mount args are also overly complicated on macOS and not well documented.
Reviewed By: fanzeyi
Differential Revision: D26255629
fbshipit-source-id: 295261dd40442fe7e0f9439c4f4c25e0d50211a3
Summary: This enables the repository to be mounted via NFS, and not FUSE.
Reviewed By: chadaustin
Differential Revision: D26229827
fbshipit-source-id: 5af5a47ebe5f1dd54df7707bf57d9b7476921f29
Summary:
Accumulate every commit transition in the journal, rather than
treating every merged range of journal deltas as having a "from"
commit and a "to" commit. This makes it possible for Watchman to
accurately report the set of changed files across an ABA situation or
rapid `hg next` and `hg prev` operations.
Reviewed By: genevievehelsel
Differential Revision: D26193478
fbshipit-source-id: 8b54b9d5bcefa1811008a3b6e9c3aa25a69471ca
Summary:
For now, they all pretend to not be available, except the null one which does
nothing as per the RFC.
Reviewed By: genevievehelsel
Differential Revision: D26159846
fbshipit-source-id: 8d0f43f6bacc5c5a93e883e527769cb7a3b6e22b
Summary: That way the NFS procedure won't clash with it.
Reviewed By: genevievehelsel
Differential Revision: D26159845
fbshipit-source-id: 22ce07326f9ec42aa9d44352ae5bb71368337c03
Summary:
For now, this only registers itself against rpcbind and always reply that the
procedure is unavailable. In the future, this will service all the procedures
and forward them to a Dispatcher.
Reviewed By: genevievehelsel
Differential Revision: D26159844
fbshipit-source-id: 21908f1333ed41b3eea3fb5ce19c8e68391df103
Summary:
We do have logs for what we receive, let's do the same for what we are sending
back.
Reviewed By: genevievehelsel
Differential Revision: D26152811
fbshipit-source-id: 8e605f78a8c849f3bd65b70be51617fc058330ff
Summary:
In the NFS spec, the fhandle3 is defined as an opaque byte array, and thus its
size must preceed its content. Let's also move it to an NfsdRpc.h as this type
will be predominantly used by the Nfsd RPC program.
Reviewed By: chadaustin
Differential Revision: D26152812
fbshipit-source-id: 0cc37325078a2c7b58551eaa5177436b21e03838
Summary:
Now that we have the list of available mount points, we can properly reply to
the MNT RPC call.
Reviewed By: kmancini
Differential Revision: D26152810
fbshipit-source-id: f364292ead321689fb20f858aee45efdfbd7287d
Summary:
For a proper NFS server, rpcbind is used to query and collect the port/protocol
of the mountd and nfs programs at mount time. In EdenFS, the mount.nfs command
will be run by EdenFS directly, against itself, and we can manually specify
what the ports are for both the mountd and nfs programs. Thus we can use these
directly to avoid polluting the rpcbind database, and potentially conflicting
with a real NFS server.
This will be particularly useful when testing EdenFS as this allows multiple
tests to run at the same time.
Reviewed By: kmancini
Differential Revision: D26140416
fbshipit-source-id: 6b8aa22e69f580a9d6f6edf775d268809a909118
Summary:
Once everything is plugged in, EdenFS will first register the mount point with
Mountd, and then execute the mount.nfs CLI against itself to mount the
repository with the OS.
For now, this does the first part by simply informing Mountd of a new mount
point during mount.
Reviewed By: kmancini
Differential Revision: D26139990
fbshipit-source-id: e5097b8e90c032c8c7bdd2cd03b69695074d7874
Summary:
When NFS is enabled, we will need to have one mountd RPC program running to
be able to serve the mount requests, let's make sure that we have one by adding
it to the ServerState.
On top of the per-mount protocol setting added previously, I've also added a
global setting to make sure that we're not advertising ourself as an NFS server
before the code is more functional.
Reviewed By: genevievehelsel
Differential Revision: D26139991
fbshipit-source-id: 0d9e02618c931a0401394888ea8f3680974966b5
Summary:
As the NFS bits are integrated into the rest of EdenFS, we need to make sure
the code compiles properly. The only major change was the removal of the
//common dependencies as these aren't available to OSS builds.
Reviewed By: genevievehelsel
Differential Revision: D26181750
fbshipit-source-id: 44f8ff335e543ab282f40fa7f8d28fc19cf4581c
Summary:
On macOS, EdenFS will soon be exposed as an NFS server. To ease the transition,
let's have a per-mount config to decide whether this repository should be
mounted via FUSE or NFS
Reviewed By: genevievehelsel
Differential Revision: D26139992
fbshipit-source-id: d28b0e2327c1101f3555ab0bbc9a5b03d82fd7ff
Summary:
The portmap protocol is limited to IPv4, while the rpcbind not only does
support IPv4 and IPv6, but also unix sockets. While we still bind only to an
ipv4, this should allow us to support IPv6 in the future.
Reviewed By: kmancini
Differential Revision: D26114218
fbshipit-source-id: 46e4dfe60a9c1ddbccbb586bf4e3d24012fd26ee
Summary:
As I'm going through the various NFS specs, it becomes clear that variant are
widely used, and the macros to generate them is awkward and making working with
them harder than it should be. Instead writing their deserialization would be
more flexible.
While doing this, I also revamped how serialization/deserialization is done by
using a trait that contains 2 static function: serialize and deserialize, new
types are expected to specialize the XdrTrait with their own
serialization/deserialization.
Reviewed By: kmancini
Differential Revision: D26160026
fbshipit-source-id: 951bdd20620cc5715e780e99b64e15397688570d
Summary: `hg gc` is deprecated and does nothing anymore. Lets just remove this section talking about running `hg gc`
Reviewed By: chadaustin
Differential Revision: D26178391
fbshipit-source-id: 2bb16bbac38465a524ed50454d548564324d35bf
Summary:
The folly::async bits are a bit complicated to get right in particular around
its lifetime management. The solution that is used in our codebase is to make
use of a folly::DelayedDestruction and the DestructorGuard. As long as a
DestructorGuard is alive, the object cannot be destroyed, and thus we can
simply hold a guard in the Reader, and make sure that a guard is held during
the future callbacks.
The only thing to be aware of is that the DestructorGuard is not thread safe,
but since all the callbacks and futures are running in the context of the
single threaded event base, this is not an issue.
Reviewed By: chadaustin
Differential Revision: D26105288
fbshipit-source-id: 4c7b91eca2bc1cae66d8d1724931b20340740574
Summary:
When the RPC server starts, it registers itself against the global rpcbind
daemon to allow RPC clients to reach it. However, it never unregisters itself
when it is destructed, leading to the rpcbind daemon having outdated
information. It also prevents a future RPC server for the same program and
version to register itself.
To fix this, we can simply unregister on destruction. The MountdUtil binary is
also fixed to be able to interrupt it with ctrl+c.
Reviewed By: chadaustin
Differential Revision: D26105289
fbshipit-source-id: c9c3d552ec04a78025d6ff715a425f9ec3aa1328
Summary:
In NFSv3, mounting is not part of the NFS program but is done in a separate
program. This adds the basic scaffolding to reply to the NULL procedure for the
mount protocol. Later, this will be enhanced by having `eden mount` first
register a mount point to Mountd, then trigger a `mount.nfs` against localhost
where Mountd will be able to reply properly.
Similarly to PortMapUtil, I've added a small binary that allows for a "mountd"
process to be spawned. Note that since the code doesn't yet answer properly to
the mount procedure, nfusr closes the connection and retries, causing mountd to
crash due to RpcServer being unsafe on error, this will be fixed separately.
Reviewed By: chadaustin
Differential Revision: D26033504
fbshipit-source-id: 27f90d9072a93460a3a383aadde38e50801e3e87
Summary:
The NFS protocol is comprised of several different RPC "programs": the mount,
nlm and nfs programs. Since all of these 3 needs to independently register with
rpcbind, let's have some common scafolding to read/write from the socket and to
simplify the implementation of these programs.
This code was written by wez
Reviewed By: chadaustin
Differential Revision: D25986691
fbshipit-source-id: 15c5fdc68323fd964ed79aa06392a83bf964ab4a
Summary:
The portmap protocol allows for service discovery and registration against the
per-host rpcbind daemon. An NFS server will need to register against it to be
mountable.
The portmap_util binary is here for testing purposes and will not be used in
EdenFS.
This code was written by wez.
Reviewed By: kmancini
Differential Revision: D25986694
fbshipit-source-id: 1eee7238fdf70c8c4937e685da91ad08d46befe4
Summary:
Built on top of XDR, the Remote Procedure Call (RPC) protocol allows for
structured client/server communication. NFS being built on top of this
protocol, this adds some basic infrastructure and types defined in the RPC RFC.
A basic client is also being added.
This code was written by wez.
Reviewed By: chadaustin
Differential Revision: D25986693
fbshipit-source-id: a5feffcb22607bcd2c7fa2cede1b70dd8aa48caf
Summary:
Logging all these throttling notifications is not necessary. There can
sometimes be big batches of fetches (like 100s of K). Lets reduce this by a
factor of 1000.
Note we also would like to add logging of what process triggered these fetches
what endpoint they use etc. This will help us identify the workflows causing it,
so we could address them or skip aux data fetching in these code paths.
But this requires some fiddling with ObjectFetchContext and the logging
code, so its gonna take a bit longer :(
Reviewed By: genevievehelsel
Differential Revision: D25505654
fbshipit-source-id: e7c40164db86fadf4baf0afd0c52879e0cb2568b
Summary:
With certain compiler flags:
```
#include <atomic>
struct ImportPriority {
ImportPriority() {}
};
std::atomic<ImportPriority> f() {
throw "";
}
```
Fails with with:
```
libgcc/include/c++/trunk/atomic:194:7: error: exception specification of explicitly defaulted default constructor does not match the calculated one
atomic() noexcept = default;
^
Demo.cpp:15:29: note: in instantiation of template class 'std::atomic<ImportPriority>' requested here
std::atomic<ImportPriority> f() {
^
1 error generated.
```
Reading the C++ spec, constructors default to noexcept, but it's not clear if that's true when they have a body. There are flags you can set that make this compile, but let's be maximally compatible and add `noexcept`.
Reviewed By: fanzeyi
Differential Revision: D26076451
fbshipit-source-id: 2f63256377fb31fd7867d7b03e7572e033f72dfc
Summary:
The FbWhoAmI background thread throws an exception when /etc/fbwhoami
cannot be opened, which terminates the process. This happens to people
in various spurious situations on corp. Since EdenFS doesn't care, let
FbWhoAmI return empty string values in that case.
Reviewed By: genevievehelsel
Differential Revision: D26010530
fbshipit-source-id: b0e9294e45f6fc2f7547982692de8ff09939405b
