Summary:
This diff adds `TreeOverlayStore` that is used in following diffs to store tree
data in SQLite database. This class encapsulates the SQL statments needed for
storing trees.
Reviewed By: chadaustin
Differential Revision: D24083787
fbshipit-source-id: 61c08617fe50e7eac94c614948f5ba5355dba71a
Summary:
We don't need to care if a directory from overlay exists or just empty. This saves the overlay implementation from keeping track of what directories are stored and which are not.
This diff changes `Overlay::loadOverlayDir` to return `DirContents` instead of `std::optional`. The same method on `IOverlay` remain unchanged to give the backing overlay implementation the freedom to choose.
Reviewed By: chadaustin
Differential Revision: D25507406
fbshipit-source-id: f7edcc55485fabeedfe11e9f269eea15a3cc32ad
Summary:
This diff adds more logging to SQLite so we can see what statements are being
executed in logs. Setting `eden.fs.sqlite=DBG9` will show every SQL statement
being executed and bound values.
Reviewed By: chadaustin
Differential Revision: D25223472
fbshipit-source-id: ed089b7ec112c75d9f7bc63d9fe53f0ec2bd6420
Summary:
This diff introduces `transaction()` to `SqliteDatabase` to make managing SQLite transaction easier.
Usage:
```
db->transaction([](auto& txn) {
SqliteStatement(txn, "SELECT ...");
SqliteStatement(txn, "INSERT ...");
});
```
It will automatically roll back when exception happens, and commit when the closure finishes.
Reviewed By: chadaustin
Differential Revision: D25102179
fbshipit-source-id: 5a4ced8c6eb8016e15b8132be2b5abcc9760e2f9
Summary: This diff adds an alias `SqliteDatabase::Connection` to make it less mouthful when we want to reference the type of the SQLite connection we created
Reviewed By: xavierd
Differential Revision: D25386777
fbshipit-source-id: a6a58d522200fe0bf35ac52f6dad16063c0389d0
Summary:
To support the new _Relational Overlay_, we need to know what is actually being
modified by overlay. This diff adds two method `addChild` and `removeChild` to
IOverlay that supports passing such information to the actual implementation.
Reviewed By: chadaustin
Differential Revision: D24153544
fbshipit-source-id: 04e5f4c231ff9b5fe0ee5be082b1a75d78758bb4
Summary:
This is needed for the next diff in the stack to build on Windows. We never
build anything form `./inodes/overlay` on Windows.
Reviewed By: xavierd
Differential Revision: D24258241
fbshipit-source-id: e1c5234433a1ec92419dcb090c8949800efa605f
Summary:
This diff introduce `IOverlay` that provides a shared interface between
`FSOverlay` and `SqliteOverlay`. This allows us to share more code between
POSIX and Windows, and prepare for the new SQLite-backed Overlay
implementation.
Note as of this diff `IOverlay` is only at a very initial version. The
interface is far from complete and still need a few iteration to change.
Reviewed By: chadaustin
Differential Revision: D23941454
fbshipit-source-id: 69cce9d3852f21b2d19f528e7e05fb183538ace9
Summary:
`folly::format` is a problematic API because it returns a `Formatter` object that may store references to arguments as its comment warns:
```
* Formatter class.
*
* Note that this class is tricky, as it keeps *references* to its lvalue
* arguments (while it takes ownership of the temporaries), and it doesn't
* copy the passed-in format string. Thankfully, you can't use this
* directly, you have to use format(...) below.
```
This has negative safety and performance (encourages reuse of the same formatter) implications because contrary to what the comment says you can use the object directly since it's returned from `folly::format`. For example
```
auto f = folly::format(std::string("{}"), 42);
f.str();
```
is a UB with no compile-time diagnostic (only caught by ASAN).
It's also unnecessary because the `Formatter` object is usually converted to string straight away via `str()` or written to an output stream. Reusing the formatter doesn't make much sense either because the expensive part is formatting, not capturing arguments.
This diff deprecates `folly::format` suggesting `fmt::format` as a potential replacement. `fmt::format` doesn't have the above problem because arguments are always in scope. It also has the following advantages:
* Better compile times.
* Compile-time format string checks.
* Compatibility with C++20 `std::format`.
* Better performance, particularly with format string compilation which eliminates format string processing overhead altogether.
* Smaller binary footprint.
Also remove `folly::writeTo` which is no longer used and the `format_nested_fbstrings` benchmark which is identical to `format_nested_strings` since there is no `fbstr()` any more.
Reviewed By: yfeldblum
Differential Revision: D26391489
fbshipit-source-id: f0309e78db0eb6d8c22b426d4cc333a17c53f73e
Summary:
Remove a bit of boilerplate in the definition of subcommands, and pave
the way for sub-sub commands.
Reviewed By: xavierd
Differential Revision: D26416574
fbshipit-source-id: 1477505cab6c3de2c7ac14fe862d366a707d155b
Summary:
This diff allows the user to specify a merge function in the shape of `merge(&mut T, T);` to customizing merging behavior.
This is used in the next diff to merge the `HashMap` that catches all unknown fields.
Reviewed By: xavierd
Differential Revision: D26415382
fbshipit-source-id: 10801ddc7d41b82d8d165a8b63466aa9b92baaab
Summary: With this diff we can start to handle concrete error types, return correct exit code for Thrift IO errors for example.
Reviewed By: xavierd
Differential Revision: D26412948
fbshipit-source-id: f598ed2d9187126509c149f9d6293025d0f39968
Summary:
Xavier reported seeing a pile of redundant checkout events in his
journal, which caused Watchman to issue a series of no-op, sequential hg
status calls, which caused Buck to time out waiting for Watchman.
We're not sure what paths cause Mercurial to issue no-op
checkOutRevision Thrift calls, but we can certainly filter them out
inside of the journal.
Reviewed By: fanzeyi
Differential Revision: D26699828
fbshipit-source-id: f738b53bbafa4027dd70322d64413246bb5bb828
Summary: This simply adds the types to implement the CREATE RPC call.
Reviewed By: kmancini
Differential Revision: D26654767
fbshipit-source-id: 5972102aebb3b06c6838a28f3a191304efc7c945
Summary:
The MKDIR RPC is the first RPC that modifies the working copy. It is only
partially implemented due to the parent directory attributes not being
collected just yet. Implementing these will be essential for good caching on
the client side. For simplicity reason, this isn't done just yet.
Reviewed By: kmancini
Differential Revision: D26644056
fbshipit-source-id: f99add63a2ce5789a9aa81cefe4d04c2f4a741ed
Summary: This adds all the types necessaryt to implement the MKDIR RPC.
Reviewed By: kmancini
Differential Revision: D26644057
fbshipit-source-id: cfbcf3f0a7212433eba67f2266d5f7bcc2906a88
Summary:
This is a fairly common pattern in the NFS code, and we can reduce code
duplication by automatically implementing the XdrTrait.
Reviewed By: kmancini
Differential Revision: D26639683
fbshipit-source-id: 6e9cb14ce1538834224017b8ef88955e563d6723
Summary: There are no correct uses of the catch-all overload of `folly::exceptionStr`. Every use is a bug of some kind, leading to the impression that this overload ought to be removed.
Differential Revision: D26539622
fbshipit-source-id: dc2ca0781ea02f1327a334bb1fe2e533fa46d1b3
Summary:
If multiple requests to fetch a tree come in at the same time fast enough (where the first request hasn't had the chance to retrieve the data from Mercurial and save it in the cache), we will request to download the tree multiple times. If the tree is not in the hgcache, this means we will make an extra round trip to the server. There is no limit to how many concurrent requests can be made, meaning we could make a large amount of round trips to the server for the same tree.
This adds a tracking mechanism in which we track in progress tree fetches, and if we get a request that is already in the queue or being processed, we just return a future that will be fulfilled by the first request, instead of placing this duplicate request in the queue as well.
This also makes sure if we get a duplicate request, but the duplicate request has a higher priority than the request already in the queue, we will update the priority of the request in the queue.
Reviewed By: chadaustin
Differential Revision: D26355499
fbshipit-source-id: 8d3192cf0f5628c650715f4597c92fc8c9238650
Summary:
This is a long standing bug whereby if you made the terminal too thin,
the formatting code would crash.
This fix just prevents underflow in the precision format string (and
alignment).
Reviewed By: fanzeyi
Differential Revision: D26673865
fbshipit-source-id: 945e6f227c19962ac0926e117600aa9d6a990305
Summary:
The NFS RPC description is filled with variant over nfsstat3 with only 2 cases:
NFS3_OK, and the default one. I was feeling bad having to write the same code
again and again for the deserialize part of the XdrTrait, and thus came up with
a template solution that enables us to remove all of the boilerplate code.
The only drawback is that this moves the variant one additional layer down, and
that means an extra layer of brace when initializing it. From what I know,
this is caused by the -Wmissing-braces warning, maybe we could remove that
warning?
Reviewed By: kmancini
Differential Revision: D26627163
fbshipit-source-id: ab56dee42273f180b2edf4f529221a15154ac2fb
Summary:
"value not present in store" is an unhelpful error message once
serialized through Thrift etc. Provide more useful context when a key
is missing.
Reviewed By: fanzeyi
Differential Revision: D26678102
fbshipit-source-id: 514ac2fe580d1dd7c67fc20c89b75e5d8121c329
Summary:
Previously, when an argument was just a plan filehandle, I just bypassed adding
the structure and just read the handle directly from the stream. kmancini
correctly pointed out that this is a bit confusing, thus let's add the arg
struct everywhere to cut that confusion.
Reviewed By: kmancini
Differential Revision: D26649568
fbshipit-source-id: bc4d6e519450f228d151a3de7acdfc1002c43b38
Summary:
This is merely a forward to the readlink FileInode method. This allows the
.eden directory and its symlinks to be read properly.
Reviewed By: kmancini
Differential Revision: D26619947
fbshipit-source-id: d5f8ef81ce3000c0f0a2f47007affe2c0292ef31
Summary: This adds the various types necessary to implement the READLINK RPC.
Reviewed By: kmancini
Differential Revision: D26610437
fbshipit-source-id: a64644d0499381678170da38886b5dff1c1b9270
Summary:
The LOOKUP RPC is a bit tricky as we need to special case the "." and ".."
names to represent the current and parent directories. Obtaining the parent
directory is inherently racy from the client perspective, and thus EdenFS won't
try to hold the renameLock in this case.
Reviewed By: kmancini
Differential Revision: D26597789
fbshipit-source-id: 9c8dcaf7db84f8c09f7505cab7afed48df79b754
Summary:
Migrate away from streaming Thrift Python, and instead implement `eden
trace hg` in C++, which will eventually let us support Windows.
This also fixes an unexplained crash in `eden trace hg` on macOS.
Reviewed By: xavierd
Differential Revision: D25515377
fbshipit-source-id: 1916aa7f09df37e9dee3479a858932c724f8aed4
Summary: This diff sets up the ability for us to track requests as they are shuffled around in the `std::vector<> queue`. Since the queue is a vector, and since it is sorted everytime a new element is added or removed, we cannot keep track of elements in the queue with indices or references. Instead, we will store the requests in a shared_ptr so we can maintain a pointer to the request no matter where the request is moved around to.
Reviewed By: kmancini
Differential Revision: D26355907
fbshipit-source-id: d714d689963106a4f495221dbcfcbab758ffc7b2
Summary:
NFS clients will act differently depending on what kind of error is returned
from an RPC. A NFS3ERR_NOENT in response to a LOOKUP RPC may for instance
populate a negative lookup cache to avoid re-querying the same file again.
Thus, we need to translate the various errors with their corresponding
nfsstat3 values.
Reviewed By: genevievehelsel
Differential Revision: D26597788
fbshipit-source-id: d22a552196a378d6d7e9cd374e63bb8bcf436ce4
Summary:
It's silly to use `eden prefetch --no-prefetch` to efficiently glob
for filenames. Introduce an `eden glob` command which resolves a glob
relative to the current working directory.
Reviewed By: genevievehelsel
Differential Revision: D25450358
fbshipit-source-id: 45d6dc870d21510e51d5662c75e80385886899fc
Summary:
I want to use EdenError from some code outside of service/, so move
EdenError into utils.
Reviewed By: genevievehelsel
Differential Revision: D25447438
fbshipit-source-id: 2d1ddfa379238369679e84708518a9ba106f76b9
Summary:
As useful as `eden strace` can be, it's hard to correlate inode
numbers to their filenames via the mkdir/create/lookup request that
returned the inode. Add logging for result codes.
Reviewed By: kmancini
Differential Revision: D26287958
fbshipit-source-id: dae4b56513831185b038546f238938b0d21a6bad
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
Summary:
LOOKUP is the next RPC that EdenFS needs to implement, let's start by adding
the RPC types.
Reviewed By: kmancini
Differential Revision: D26562919
fbshipit-source-id: 3a4ce1a039b6405df9ff94757963aa46acf56d92
Summary:
The ACCESS RPC merely tries to see if a client can perform some action on a
file/directory. Until we have a better story around checking the RPC
credentials and passing it around, let's just return the requested access
rights as-is.
Reviewed By: kmancini
Differential Revision: D26562918
fbshipit-source-id: 468e038cc063c289b5554f3faa6a7f99be2731e4
Summary: This merely add the types necessary to implement the ACCESS RPC
Reviewed By: kmancini
Differential Revision: D26562920
fbshipit-source-id: 2f3238e723e97a54896fabb026adf9b92ab4fcd3
Summary:
Take 2 of fixing the lifetime issues of the RpcServer, this time, the
RpcAcceptCallback needs to live for longer than RpcServer itself. This is due
to the acceptStopped callback being called after RpcServer is being destroyed.
Since the RpcServer needs to be destroyed in the EventBase thread, the
acceptStopped callback is delayed, causing it to be invoked after the RpcServer
has freed its memory.
To solve this, we simply need to make the RpcAcceptCallback a delayed
destructor, and hold a reference to itself that goes away once the
acceptStopped callback is called.
Reviewed By: kmancini
Differential Revision: D26556644
fbshipit-source-id: 32cd80f1664649c1ad96f88c4eec6dd2ed6e8c12
Summary:
Now that the privhelper has the necessary code to unmount an NFS mount, let's
do it. With this in place, this enables `eden mount` and `eden unmount` for NFS
mounts. Unfortunately, it appears that during unmount, releasing the Nfsd3
object triggers a use-after-free in the rpc code
Reviewed By: kmancini
Differential Revision: D26500109
fbshipit-source-id: a210761f818b28b1eb0044f7314a0e2b9017848c
Summary:
This is merely fixing a typo that I made previous where I reused the prjfs
request timeout instead of adding a new one.
Reviewed By: kmancini
Differential Revision: D26500110
fbshipit-source-id: b9bf0cbc0d74866cdc2471f126751d6d8e514e21
Summary:
Now that `mount(2)` complete, we can start plumbing the rest of the mount code
to accomodate for NFS. Trying to find a common ground between Prjfs, FUSE and
Prjfs is harder than I wish it would be and thus I wasn't able to find a
satisfatory solution. For now, I went with a std::variant that stores either a
FuseChannel, or the Nfsd3. In the future once Nfs is stabilized and we have a
good understanding of how it differs (and where it doesn't), EdenMount would
probably need a good refactoring.
Reviewed By: kmancini
Differential Revision: D26500111
fbshipit-source-id: f02a2eaf8890261f150d7cdd2cdfd134aa62c2aa
Summary:
Comments affecting runtime behavior disturbs me, so use explicit
structopt annotations on help text. And move those annotations to the
command implementations.
Reviewed By: fanzeyi
Differential Revision: D26412452
fbshipit-source-id: 066dfdd1c54254bae4bd2af65031487b7a1094da
