Summary:
As I'm looking through the code, it took me a bit of time to fully grasp what
the commented code was for, document it for future readers.
Reviewed By: chadaustin
Differential Revision: D30399723
fbshipit-source-id: bdf448b725192d7541b1d7de7e043ff97700dbce
Summary: This is never called, no need to keep it around.
Reviewed By: chadaustin
Differential Revision: D30399722
fbshipit-source-id: bbc169141b58976031fcae224f24ea23897c6f21
Summary:
If a wrong type is passed in, an exception would be thrown at runtime, while
the static_assert would fire at compiled time, reducing the time to find the
error for the developer.
Reviewed By: chadaustin
Differential Revision: D30398255
fbshipit-source-id: fd021f96063565f83c55a9bf3f175bf879afa6ed
Summary: This diff teaches eden doctor to generate a warning when user has SQLite overlay repo on disk and asking them to migrate.
Reviewed By: chadaustin
Differential Revision: D30345721
fbshipit-source-id: 95796ca77979f034904b87e3a38f149baddd720a
Summary:
This is a re-submit of D29915585 (6c5c7055ce), I've merely fixed the bug that it introduced,
thus the credit goes to markbt. Below is the original commit message:
If Mercurial asks EdenFS to update to a commit that it has just created, this
can cause a long delay while EdenFS tries to import the commit.
EdenFS needs to get the trees out of the Hg data store. But these also won't
know about the new trees until the data store is refreshed or synced.
To fix this, call the refresh method on the store if we fail to find the tree,
and try again. To make this work, we must first only look locally. To keep
things simple, we only do this for the root tree.
However, currently indexedlogdatastore doesn't actually do anything when you
ask it to refresh.
To fix these, we call flush(), which actually does a sync operation and loads
the latest data from disk, too.
Reviewed By: chadaustin
Differential Revision: D30387805
fbshipit-source-id: 3fdbd27b306f03df53b68a0bcc5ee5dc140326bb
Summary:
The SmartPlatform service that queries for a user's most used directories allows optional parameters of: os, startTime, endTime, and sandcastleAlias instead of user. This diff extends the current predictive prefetch option which queries based on the current user, mount repository, and a default numResults, to allow specification of all parameters including the optional ones.
If a user and/or repo is not specified these are determined from the server state and mount, respectively. If numResults is not specified, a default value is used (predictivePrefetchProfileSize, currently 10,000).
For sandcastle aliases, we check if the SANDCASTLE_ALIAS environment variable is set, and if so, use the value as a parameter. If a sandcastle alias is specified, the smartservice will ignore the user and query based on the alias, otherwise a user is assumed.
Differential Revision: D30160507
fbshipit-source-id: 174797f0a6f840bb33f669c8d1bb61d76ff7a309
Summary:
When a user reports a slow EdenFS and a high network traffic is suspected, the
lack of Thrift stats makes it hard to fully validate this. Thus, let's collect
some stats and put them in the rage.
The collected stats are the exact same ones that `eden top` uses.
Reviewed By: chadaustin
Differential Revision: D30355746
fbshipit-source-id: 519a8e2c8b0c458daecdcc0813a8d7416d5362d6
Summary:
In the case where the path to the mount has symlinks, EdenFS would only accept
the path to it that was specified at mount time, even though another path may
refer to the same directory.
To solve this, we can simply normalize paths in all the Thrift endpoint to make
sure that EdenFS always refers to a mount point under its non-symlinked path.
Reviewed By: chadaustin
Differential Revision: D30320515
fbshipit-source-id: e578d059a3b1307d6b24c4b9bdb1ceb3b534c460
Summary:
Take 2. This time with measured improvement (3x).
With WAL mode enabled in SQLite. The bottleneck focuses at flush time. WPA shows a lot of time are spent at flushing changes into the WAL file.
Setting `synchronous` to `OFF` in SQLite will make SQLite to not wait for flushing changes onto the disk. This saves us time and compromises with correctness only when the system crashes.
From SQLite documentation:
> With synchronous OFF (0), SQLite continues without syncing as soon as it has handed data off to the operating system. If the application running SQLite crashes, the data will be safe, but the database might become corrupted if the operating system crashes or the computer loses power before that data has been written to the disk surface. On the other hand, commits can be orders of magnitude faster with synchronous OFF.
https://sqlite.org/pragma.html#pragma_synchronous
Reviewed By: chadaustin
Differential Revision: D30314601
fbshipit-source-id: 87f6cd05149045313678328bb3fc7dd10e894e37
Summary:
Whenever a file is modified in an EdenFS mount and a watchman subscription is
active, watchman will be nodified and will issue a getFilesChangedSince Thrift
call. In order to do that, Watchman ends up always re-creating a new connection
to EdenFS, causing the .eden/socket or .eden/config to be re-read in order to
find EdenFS's socket.
For workloads with heavy write traffic to EdenFS, this readlink/read can add
up. On Windows, writing ~2.5GB worth of data lead Watchman to read over 650MB
worth of data from the .eden/config!
Reviewed By: kmancini
Differential Revision: D29508654
fbshipit-source-id: 60440d645340bc4fe16ea9618d7a5080740e4d87
Summary: The journal stream is disconnected at Watchman shutdown, which is the expected behavior. This changes the log level to INFO.
Reviewed By: chadaustin
Differential Revision: D30231657
fbshipit-source-id: 94909daeba786b1bed7497e4a21ffcfc52d6d9cb
Summary:
This moves some Prjfs logic into the channel code, which allows for
de-duplicating a bit of code. This will also make a subsequent change in the
rename code easier to do.
Differential Revision: D30023970
fbshipit-source-id: 7efa6dcc4318213e9d266932527b5a56edacefd7
Summary: Due to lifetime issues of FetchContext& w.r.t. background prefetching, we can just create the helper in `_globFiles` and use that to maintain lifetime
Reviewed By: xavierd
Differential Revision: D30175224
fbshipit-source-id: b2fccb76f9d4011139e80bd5bc52c40bbab08b94
Summary: Added a Thrift method that tells EdenFS to prefetch files from a user's most used directories using an endpoint that talks to the edenfs/edenfs_service SmartPlatform service to get the directory list. The default number of directories is set to 10,000.
Reviewed By: genevievehelsel
Differential Revision: D29909976
fbshipit-source-id: bfb1a411d50d7355ff604de5bc090a9e2c3100a0
Summary: This was an oversight. With this edenfsctl should be able to run on Windows and talk with EdenFS daemon via UNIX domain socket.
Differential Revision: D30089140
fbshipit-source-id: abe12aeee06d91f7a7323a6042d2b69c31963bbb
Summary:
Original commit changeset: 34fe02ddf580
We are seeing reports of EdenFS only showing partial directories to our user. Local testing shows this commit seems to be causing the issue. Reverting it for now.
Reviewed By: kmancini
Differential Revision: D30136949
fbshipit-source-id: 7fcc81506c132055a5b639a383b8c9be68118dc5
Summary:
This newly added config controls the default behavior of `eden clone` and
whether a new clone will use NFS or FUSE. This is intended to facilitate the
transition to NFS from FUSE on macOS.
Differential Revision: D30110056
fbshipit-source-id: ea6b493aa803297952b46434f6d11d8edf58e40b
Summary: As title. This was added as `thrift-py3` doesn't support CMake well but we can workaround by importing from py.
Reviewed By: xavierd
Differential Revision: D30078300
fbshipit-source-id: 277866f8b226f164b5e30231aa10b59c0aba5807
Summary:
This adds the options to `eden stats` for collecting only fast stats and printing in JSON.
`eden stats` can be slow especially due to collecting fb303 counters and private bytes. An example use case of this new lightweight endpoint is that Buck can poll it to display Eden related info in its cli (see [post](https://fb.workplace.com/groups/132499338763090/permalink/210396380973385/) for context).
Reviewed By: xavierd
Differential Revision: D29687041
fbshipit-source-id: a663e71231527c5dfb822acbf238af0ac6ce4a00
Summary: Currently we have to manually save the id returned from `start_recording`. After this, we can simply ask for the list of all active recorder sessions.
Reviewed By: genevievehelsel
Differential Revision: D30056117
fbshipit-source-id: 7fd69b70e7b04fcd0b3724f4ee16c5e5e86badaf
Summary:
This improves buck build(and likely other tools) execution time significantly on the cold cache, with eden api enabled(e.g. no import helper)
I have noticed during the tests that we don't urilize network well when accessing files not in cache.
Adding some instrumentation shows that we only do up to 8 parallel fetches on mercurial side.
This seem to be limited by number of hg queue workers, simply boosting this number from 8->32 improves performance a lot, and it is still manageable number of threads
We probably need some better solution to allow for greater parallelism.
With this diff time for `buck build fb4a` on the cold cache is reduced **from 121 to 95 minutes(-30%)**.
'Parsing buck files' stage is reduced **from 07:16 min to 05:53 min(-25%)**.
During buck build we get to actual parallelism of about 16 fetches which is limited by number of buck threads.
Bumping number of buck threads with `-j 32` further reduces build time to **54 minutes (-60% from original)**
Reviewed By: xavierd
Differential Revision: D30025720
fbshipit-source-id: 6bcae1f353a0d31d8ce632ccb991c2a02f3136fc
Summary:
* `FuseRequestContext` currently uses `fuseHeader_.opcode` to get the cause detail.
* However, `fuseHeader_.opcode` is set to 0 by `stealReq` to indicate that the request has been released.
* This lead to `getCauseDetail` returning `<unknown>` if we call it after the request is released (e.g. in the clean up future).
This adds the `opcode_` member so that `getCauseDetail` always return the correct cause detail regardless of whether the request has been released. This also removes `examineReq` as it's not used.
Reviewed By: xavierd
Differential Revision: D30050518
fbshipit-source-id: 0c3e4d31e59a5b5fd87967d6c976b573692a2af9
Summary:
Per comment in D30017261 (e9c039ab4a)
> [...] we can get rid of this ESTALE check. That dates back to when edenfs had a bug that returned ESTALE when reading the .eden directory sometimes.
Reviewed By: xavierd
Differential Revision: D30024979
fbshipit-source-id: 645097c8f689c916245845561fd3d824ff7df8b4
Summary: This is unused, no need to keep it around.
Reviewed By: genevievehelsel
Differential Revision: D30046503
fbshipit-source-id: 1d20d9b4ce672d5d79410203807dbc93b4bce31a
Summary: For `future_` endpoints, we wrap the final return statement with `wrapFuture(std::move(helper), ...` to ensure we keep the ThriftLogHelper alive through the whole call.
Differential Revision: D30018980
fbshipit-source-id: 2c63fe5d7b4504912cc46a32ca04f16e98b0805f
Summary: `eden mount` currently swallows all but one type of exceptions, which resulted in cli reporting false positive mount success when the thrift endpoint throws. The try-catch was added in D21934538 (51df752a46) to provide clear error message for a particular windows error and we can just re-raise so that the other exceptions are propagated.
Reviewed By: chadaustin
Differential Revision: D30017261
fbshipit-source-id: 6a8a44330a90275b3c044301ff644dce0d6dee13
Summary:
Today the globing code is not able to parse `\` as a path separator
which causes an error like this:
```
facebook.eden.ttypes.EdenError: Invalid glob (attempt to construct a PathComponent from a string containing a directory separator: tools\arcanist): tools\arcanist
```
Many other tools like hg files still use these separators, so it would be
nice if we could support `\`.
This change will mean that Windows globs will not support special characters.
But they were already broken so we can not have any users successfully using
them today.
Ideally we will not have to support special charcters on Windows because it is
icky, But I left a note in the code about how we could do this, if its ever
needed in the future
Reviewed By: chadaustin
Differential Revision: D29971586
fbshipit-source-id: 32f54986e801b2aa3b3b1c90c650bd45531e8c0a
Summary:
Copying PathPiece onto futures is a recipe for a use-after-free due to the
underlying string being likely freed when the future executes, let's instead
use plain Path and have the caller move/copy them onto the futures.
Differential Revision: D29989957
fbshipit-source-id: 2e6cebf8c9dfc8f55df8edbc410c2a7f681b8f22
Summary: Boost is not happy when it encounters UNIX domain socket file on Windows. Let's detect the file type manually instead of using boost.
Reviewed By: xavierd
Differential Revision: D29974282
fbshipit-source-id: e11558abdbc565014189ae763a5b3fb5486d38d7
Summary:
During a diff operation, files that have a case change should not reported as
changed for case insensitive mount. This is a follow up to D29150552 (37ccaa9231) where the
TreeInode code was taught to ignore case changes for case insensitive mounts.
Reviewed By: kmancini
Differential Revision: D29952464
fbshipit-source-id: e5fa8eccc1c1984a680838fd8589a0278989d9d0
Summary:
While not strictly necessary, we should be providing the correct `Key` for
trees when fetching them from the Hg datapack store. Pass the `path` through
to the backing store so it can be used to construct the `Key` for the tree.
Reviewed By: DurhamG
Differential Revision: D29933214
fbshipit-source-id: d9631ea37b5ffa3f7051112d12cf3161c7c677ef
Summary:
If Mercurial asks EdenFS to update to a commit that it has just created, this
can cause a long delay while EdenFS tries to import the commit.
EdenFS needs to get the trees out of the Hg data store. But these also
won't know about the new trees until the data store is refreshed or synced.
To fix this, call the refresh method on the store if we fail to find the tree,
and try again. To make this work, we must first only look locally. To keep
things simple, we only do this for the root tree.
However, currently indexedlogdatastore doesn't actually do anything when you
ask it to refresh.
To fix these, we call `flush()`, which actually does a `sync` operation and
loads the latest data from disk, too.
Reviewed By: DurhamG
Differential Revision: D29915585
fbshipit-source-id: 34fe02ddf5804be425d9cabe1a56069f22e5e4d4
Summary:
If Mercurial asks EdenFS to update to a commit that it has just created, this
can cause a long delay while EdenFS tries to import the commit.
EdenFS needs to resolve the commit to a root manifest. It does this via the
import helper, but the import helper won't know about the commit until it is
restarted, which takes a long time.
To fix this, we add an optional "root manifest" parameter to the checkout or
reset parents thrift calls. This allows the Mercurial client to inform EdenFS
of the root manifest that it already knows about, allowing EdenFS to skip this
step.
Reviewed By: chadaustin
Differential Revision: D29845604
fbshipit-source-id: 61736d84971cd2dd9a8fdaa29a1578386246e4bf
Summary:
There is no need to obtain the PrjfsChannel twice, especially if the first one
is unused.
Reviewed By: fanzeyi
Differential Revision: D29977555
fbshipit-source-id: 56428eae84a6abd8689b4f997173e0aa1501aede
Summary:
TSAN complains that pipes_ is read and written in different threads without
proper synchronization. To avoid this, we can simply close the FileDescriptor
without removing it from the pipes map as this achieve the same result: it
notifies the reader that the endpoint is closed.
Differential Revision: D29924043
fbshipit-source-id: be92630799bb5c78089dbe85f9c2f02f937300bc
Summary: This affects all platforms but more noticeable on Mac that tons of 100% printed (e.g. P409794954), probably due to some weirdness with cursor.
Reviewed By: fanzeyi
Differential Revision: D29922276
fbshipit-source-id: 987f6b9ef5a8a4ab738aa6edbd617184bbcb2d1c
Summary: As title. `RequsetContext` allows us to track metrics such as latency and count.
Reviewed By: genevievehelsel
Differential Revision: D29835813
fbshipit-source-id: 6b85fc8f11923f530fce6d871fa2253db21bfa98
Summary:
On Windows, there a commonly occuring issue where a checkout operation would
crash EdenFS as a conflict is being added for an unlinked inode, thus
triggering the XCHECK in the addConflict method.
From looking at the code, the comment that claims that inodes cannot be
unlinked during checkout isn't entirely accurate: EdenFS will unlink inodes
during checkout when their content changed. The code itself should properly
remove the unlinked inode from its parent TreeInode and thus I haven't fully
figured out the exact series of event that leads to a conflict being added for
an unlinked inode. Since the asumption from the comment is invalid, it should
be safe to not assert that the inode shouldn't be unlinked and use
InodeBase::getUnsafePath instead of InodeBase::getPath
Reviewed By: kmancini
Differential Revision: D29241901
fbshipit-source-id: 4239df576b3cbf716fb336fd4d6542939337a297
Summary:
In some cases, the code needs to have access to the path for an inode even if
that inode is unlinked. In such situation, neither getPath nor getLogPath are
suitable, thus let's introduce a getUnsafePath, which is intended for these
handful of places.
The only known use case for such method is when adding conflicts during checkouts.
Reviewed By: genevievehelsel
Differential Revision: D29241902
fbshipit-source-id: 7756a95813d6fd5e471538cf82d29604dd5b8e5e
Summary:
This adds debug commands for ActivityRecorder:
```
eden debug start_recording --output-dir <DIR>
* stdout: the id of the profile
eden debug stop_recording --unique <ID>
* stdout: the output file path
```
Users can record multiple profiles concurrently. Each profile is identified by the timestamp when it started.
Reviewed By: genevievehelsel
Differential Revision: D29666359
fbshipit-source-id: 487ca67de77378a8141bc4ac46b9abd1375ffd23
