Summary:
This simply writes the data passed in to the inode. Note that the current
implementation has a protocol violation since it doesn't sync the data written
to disk but advertise to the client that it did. This is completely wrong from
a data consistency guarantee, but is probably fine for now. Once the code
becomes closer to being production ready, this will be changed to honor the
client asks.
Reviewed By: kmancini
Differential Revision: D26681614
fbshipit-source-id: 82ad7a141be3bbe365363b1f6692ae62f253423f
Summary:
The Appender API doesn't allow us to simply append an IOBuf chain to it forcing
the entire chain to be copied to it. For the most part, this isn't an issue,
but to reduce the overhead of the READ NFS procedure, this will be a nice
benefit.
For the most part, this diff is a codemod, with a couple of places that needed
to be manually fixed. These were in the rpc/Server.{cpp,h} and
rpc/StreamClient.{cpp,h}.
Reviewed By: genevievehelsel
Differential Revision: D26675988
fbshipit-source-id: 04feef8623fcddd02ff7aea0b68a17598ab1d0f8
Summary:
The ser, de and roundtrip are duplicated 3 times in the various tests, let's
move them into a common library.
Reviewed By: chadaustin
Differential Revision: D26675989
fbshipit-source-id: 1da0bc33429795a889b72c76fa18e5fa3cb7df6f
Summary: This merely adds the various types for the WRITE RPC.
Reviewed By: chadaustin
Differential Revision: D26671895
fbshipit-source-id: 8409c8a1f90e97478aed7c9f19b881c46234b539
Summary:
For the READ/WRITE RPC calls, copying data in and out of an IOBuf chain can be
fairly expensive, to avoid this overhead, we can simply clone the data out of
the IOBuf chain directly, saving on the cost of copy.
Since the code uses a folly::io::Appender that doesn't support adding an IOBuf
to it, we still pay the cost of copying data to it, switching to
folly::io::QueueAppender may solve this.
Reviewed By: chadaustin
Differential Revision: D26671896
fbshipit-source-id: 0161f04cb820bf27ef66fdef6b4a1ce4eb778b96
Summary:
The CREATE RPC is intended to create regular files on the server. It supports 3
ways of creating them: unchecked, guarded and exclusive. The exclusive mode
allows for a client to being able to re-transmit the creation and for it to not
fail when that happens. The way this is implemented requires the server to
store a cookie passed in by the client and on retransmission to compare it in
order to not fail the file creation. This complexity may not be warranted and
needed in the case of EdenFS, and the spec allows for a server to not support
this operation, thus EdenFS claims to not support it.
The 2 other modes are more traditional, guarded will simply fail if the file
already exist, while unchecked succeed in that same situation. It's not
entirely clear in the RFC what the behavior is supposed to be in the unchecked
case, and whether the attributes needs to be applied. For now, the error is
just ignored, a warning message is logged and a success is returned in this
case. In the future, setting the file attribute (in particular the mode and
size) may be necessary.
Reviewed By: kmancini
Differential Revision: D26654766
fbshipit-source-id: c2e8142b8a4ff756a868e5859fdda4b07e53bddf
Summary: This is unused, and it has a hard dependency on Fuse, so let's remove it.
Reviewed By: chadaustin
Differential Revision: D26742212
fbshipit-source-id: 091556be39e599512d34920503083d03d4c5a0c2
Summary:
This diff adds a gflag option to turn on `TreeOverlay`. As this diff, this
option only works on Windows.
Reviewed By: xavierd
Differential Revision: D25223471
fbshipit-source-id: ccab4c85cf4b08980e67814238457a8eb83af3dc
Summary:
This diff adds `TreeOverlay` that implements the `IOverlay` interface based on
`TreeOverlayStore`. At this point this class does not do much other than
forwarding calls to `TreeOverlayStore`. This layer will become necessary in the
future when we add file handling support to `TreeOverlay`.
Reviewed By: xavierd
Differential Revision: D25223473
fbshipit-source-id: 02d03081ba10c82123fe69c35dbc70f865c8370d
Summary:
This diff adds `TreeOverlayStore` that is used in following diffs to store tree
data in SQLite database. This class encapsulates the SQL statments needed for
storing trees.
Reviewed By: chadaustin
Differential Revision: D24083787
fbshipit-source-id: 61c08617fe50e7eac94c614948f5ba5355dba71a
Summary:
We don't need to care if a directory from overlay exists or just empty. This saves the overlay implementation from keeping track of what directories are stored and which are not.
This diff changes `Overlay::loadOverlayDir` to return `DirContents` instead of `std::optional`. The same method on `IOverlay` remain unchanged to give the backing overlay implementation the freedom to choose.
Reviewed By: chadaustin
Differential Revision: D25507406
fbshipit-source-id: f7edcc55485fabeedfe11e9f269eea15a3cc32ad
Summary:
This diff adds more logging to SQLite so we can see what statements are being
executed in logs. Setting `eden.fs.sqlite=DBG9` will show every SQL statement
being executed and bound values.
Reviewed By: chadaustin
Differential Revision: D25223472
fbshipit-source-id: ed089b7ec112c75d9f7bc63d9fe53f0ec2bd6420
Summary:
This diff introduces `transaction()` to `SqliteDatabase` to make managing SQLite transaction easier.
Usage:
```
db->transaction([](auto& txn) {
SqliteStatement(txn, "SELECT ...");
SqliteStatement(txn, "INSERT ...");
});
```
It will automatically roll back when exception happens, and commit when the closure finishes.
Reviewed By: chadaustin
Differential Revision: D25102179
fbshipit-source-id: 5a4ced8c6eb8016e15b8132be2b5abcc9760e2f9
Summary: This diff adds an alias `SqliteDatabase::Connection` to make it less mouthful when we want to reference the type of the SQLite connection we created
Reviewed By: xavierd
Differential Revision: D25386777
fbshipit-source-id: a6a58d522200fe0bf35ac52f6dad16063c0389d0
Summary:
To support the new _Relational Overlay_, we need to know what is actually being
modified by overlay. This diff adds two method `addChild` and `removeChild` to
IOverlay that supports passing such information to the actual implementation.
Reviewed By: chadaustin
Differential Revision: D24153544
fbshipit-source-id: 04e5f4c231ff9b5fe0ee5be082b1a75d78758bb4
Summary:
This is needed for the next diff in the stack to build on Windows. We never
build anything form `./inodes/overlay` on Windows.
Reviewed By: xavierd
Differential Revision: D24258241
fbshipit-source-id: e1c5234433a1ec92419dcb090c8949800efa605f
Summary:
This diff introduce `IOverlay` that provides a shared interface between
`FSOverlay` and `SqliteOverlay`. This allows us to share more code between
POSIX and Windows, and prepare for the new SQLite-backed Overlay
implementation.
Note as of this diff `IOverlay` is only at a very initial version. The
interface is far from complete and still need a few iteration to change.
Reviewed By: chadaustin
Differential Revision: D23941454
fbshipit-source-id: 69cce9d3852f21b2d19f528e7e05fb183538ace9
Summary:
`folly::format` is a problematic API because it returns a `Formatter` object that may store references to arguments as its comment warns:
```
* Formatter class.
*
* Note that this class is tricky, as it keeps *references* to its lvalue
* arguments (while it takes ownership of the temporaries), and it doesn't
* copy the passed-in format string. Thankfully, you can't use this
* directly, you have to use format(...) below.
```
This has negative safety and performance (encourages reuse of the same formatter) implications because contrary to what the comment says you can use the object directly since it's returned from `folly::format`. For example
```
auto f = folly::format(std::string("{}"), 42);
f.str();
```
is a UB with no compile-time diagnostic (only caught by ASAN).
It's also unnecessary because the `Formatter` object is usually converted to string straight away via `str()` or written to an output stream. Reusing the formatter doesn't make much sense either because the expensive part is formatting, not capturing arguments.
This diff deprecates `folly::format` suggesting `fmt::format` as a potential replacement. `fmt::format` doesn't have the above problem because arguments are always in scope. It also has the following advantages:
* Better compile times.
* Compile-time format string checks.
* Compatibility with C++20 `std::format`.
* Better performance, particularly with format string compilation which eliminates format string processing overhead altogether.
* Smaller binary footprint.
Also remove `folly::writeTo` which is no longer used and the `format_nested_fbstrings` benchmark which is identical to `format_nested_strings` since there is no `fbstr()` any more.
Reviewed By: yfeldblum
Differential Revision: D26391489
fbshipit-source-id: f0309e78db0eb6d8c22b426d4cc333a17c53f73e
Summary:
Remove a bit of boilerplate in the definition of subcommands, and pave
the way for sub-sub commands.
Reviewed By: xavierd
Differential Revision: D26416574
fbshipit-source-id: 1477505cab6c3de2c7ac14fe862d366a707d155b
Summary:
This diff allows the user to specify a merge function in the shape of `merge(&mut T, T);` to customizing merging behavior.
This is used in the next diff to merge the `HashMap` that catches all unknown fields.
Reviewed By: xavierd
Differential Revision: D26415382
fbshipit-source-id: 10801ddc7d41b82d8d165a8b63466aa9b92baaab
Summary: With this diff we can start to handle concrete error types, return correct exit code for Thrift IO errors for example.
Reviewed By: xavierd
Differential Revision: D26412948
fbshipit-source-id: f598ed2d9187126509c149f9d6293025d0f39968
Summary:
Xavier reported seeing a pile of redundant checkout events in his
journal, which caused Watchman to issue a series of no-op, sequential hg
status calls, which caused Buck to time out waiting for Watchman.
We're not sure what paths cause Mercurial to issue no-op
checkOutRevision Thrift calls, but we can certainly filter them out
inside of the journal.
Reviewed By: fanzeyi
Differential Revision: D26699828
fbshipit-source-id: f738b53bbafa4027dd70322d64413246bb5bb828
Summary: This simply adds the types to implement the CREATE RPC call.
Reviewed By: kmancini
Differential Revision: D26654767
fbshipit-source-id: 5972102aebb3b06c6838a28f3a191304efc7c945
Summary:
The MKDIR RPC is the first RPC that modifies the working copy. It is only
partially implemented due to the parent directory attributes not being
collected just yet. Implementing these will be essential for good caching on
the client side. For simplicity reason, this isn't done just yet.
Reviewed By: kmancini
Differential Revision: D26644056
fbshipit-source-id: f99add63a2ce5789a9aa81cefe4d04c2f4a741ed
Summary: This adds all the types necessaryt to implement the MKDIR RPC.
Reviewed By: kmancini
Differential Revision: D26644057
fbshipit-source-id: cfbcf3f0a7212433eba67f2266d5f7bcc2906a88
Summary:
This is a fairly common pattern in the NFS code, and we can reduce code
duplication by automatically implementing the XdrTrait.
Reviewed By: kmancini
Differential Revision: D26639683
fbshipit-source-id: 6e9cb14ce1538834224017b8ef88955e563d6723
Summary: There are no correct uses of the catch-all overload of `folly::exceptionStr`. Every use is a bug of some kind, leading to the impression that this overload ought to be removed.
Differential Revision: D26539622
fbshipit-source-id: dc2ca0781ea02f1327a334bb1fe2e533fa46d1b3
Summary:
If multiple requests to fetch a tree come in at the same time fast enough (where the first request hasn't had the chance to retrieve the data from Mercurial and save it in the cache), we will request to download the tree multiple times. If the tree is not in the hgcache, this means we will make an extra round trip to the server. There is no limit to how many concurrent requests can be made, meaning we could make a large amount of round trips to the server for the same tree.
This adds a tracking mechanism in which we track in progress tree fetches, and if we get a request that is already in the queue or being processed, we just return a future that will be fulfilled by the first request, instead of placing this duplicate request in the queue as well.
This also makes sure if we get a duplicate request, but the duplicate request has a higher priority than the request already in the queue, we will update the priority of the request in the queue.
Reviewed By: chadaustin
Differential Revision: D26355499
fbshipit-source-id: 8d3192cf0f5628c650715f4597c92fc8c9238650
Summary:
This is a long standing bug whereby if you made the terminal too thin,
the formatting code would crash.
This fix just prevents underflow in the precision format string (and
alignment).
Reviewed By: fanzeyi
Differential Revision: D26673865
fbshipit-source-id: 945e6f227c19962ac0926e117600aa9d6a990305
Summary:
The NFS RPC description is filled with variant over nfsstat3 with only 2 cases:
NFS3_OK, and the default one. I was feeling bad having to write the same code
again and again for the deserialize part of the XdrTrait, and thus came up with
a template solution that enables us to remove all of the boilerplate code.
The only drawback is that this moves the variant one additional layer down, and
that means an extra layer of brace when initializing it. From what I know,
this is caused by the -Wmissing-braces warning, maybe we could remove that
warning?
Reviewed By: kmancini
Differential Revision: D26627163
fbshipit-source-id: ab56dee42273f180b2edf4f529221a15154ac2fb
Summary:
"value not present in store" is an unhelpful error message once
serialized through Thrift etc. Provide more useful context when a key
is missing.
Reviewed By: fanzeyi
Differential Revision: D26678102
fbshipit-source-id: 514ac2fe580d1dd7c67fc20c89b75e5d8121c329
Summary:
Previously, when an argument was just a plan filehandle, I just bypassed adding
the structure and just read the handle directly from the stream. kmancini
correctly pointed out that this is a bit confusing, thus let's add the arg
struct everywhere to cut that confusion.
Reviewed By: kmancini
Differential Revision: D26649568
fbshipit-source-id: bc4d6e519450f228d151a3de7acdfc1002c43b38
Summary:
This is merely a forward to the readlink FileInode method. This allows the
.eden directory and its symlinks to be read properly.
Reviewed By: kmancini
Differential Revision: D26619947
fbshipit-source-id: d5f8ef81ce3000c0f0a2f47007affe2c0292ef31
Summary: This adds the various types necessary to implement the READLINK RPC.
Reviewed By: kmancini
Differential Revision: D26610437
fbshipit-source-id: a64644d0499381678170da38886b5dff1c1b9270
Summary:
The LOOKUP RPC is a bit tricky as we need to special case the "." and ".."
names to represent the current and parent directories. Obtaining the parent
directory is inherently racy from the client perspective, and thus EdenFS won't
try to hold the renameLock in this case.
Reviewed By: kmancini
Differential Revision: D26597789
fbshipit-source-id: 9c8dcaf7db84f8c09f7505cab7afed48df79b754
Summary:
Migrate away from streaming Thrift Python, and instead implement `eden
trace hg` in C++, which will eventually let us support Windows.
This also fixes an unexplained crash in `eden trace hg` on macOS.
Reviewed By: xavierd
Differential Revision: D25515377
fbshipit-source-id: 1916aa7f09df37e9dee3479a858932c724f8aed4
Summary: This diff sets up the ability for us to track requests as they are shuffled around in the `std::vector<> queue`. Since the queue is a vector, and since it is sorted everytime a new element is added or removed, we cannot keep track of elements in the queue with indices or references. Instead, we will store the requests in a shared_ptr so we can maintain a pointer to the request no matter where the request is moved around to.
Reviewed By: kmancini
Differential Revision: D26355907
fbshipit-source-id: d714d689963106a4f495221dbcfcbab758ffc7b2
Summary:
NFS clients will act differently depending on what kind of error is returned
from an RPC. A NFS3ERR_NOENT in response to a LOOKUP RPC may for instance
populate a negative lookup cache to avoid re-querying the same file again.
Thus, we need to translate the various errors with their corresponding
nfsstat3 values.
Reviewed By: genevievehelsel
Differential Revision: D26597788
fbshipit-source-id: d22a552196a378d6d7e9cd374e63bb8bcf436ce4
Summary:
It's silly to use `eden prefetch --no-prefetch` to efficiently glob
for filenames. Introduce an `eden glob` command which resolves a glob
relative to the current working directory.
Reviewed By: genevievehelsel
Differential Revision: D25450358
fbshipit-source-id: 45d6dc870d21510e51d5662c75e80385886899fc
Summary:
I want to use EdenError from some code outside of service/, so move
EdenError into utils.
Reviewed By: genevievehelsel
Differential Revision: D25447438
fbshipit-source-id: 2d1ddfa379238369679e84708518a9ba106f76b9
Summary:
As useful as `eden strace` can be, it's hard to correlate inode
numbers to their filenames via the mkdir/create/lookup request that
returned the inode. Add logging for result codes.
Reviewed By: kmancini
Differential Revision: D26287958
fbshipit-source-id: dae4b56513831185b038546f238938b0d21a6bad
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
