Summary: Introduces an `eden logs` command to read a large chunk (1M by default) file into a paste. This is also added to the the eden rage report to get more insight into systems in which we cannot log in to view the logs (like laptops).
Reviewed By: kmancini
Differential Revision: D24146812
fbshipit-source-id: 991f1595b974eb01f77e86559a8413b0b09a24a4
Summary:
The LookupProcessor class is built with the purpose of iterating all the inodes
in the passed in path. However, the LookupProcessor object may outlive the
lifetime of the path, and thus we need to build an iterator on the copied path,
not on the argument.
Differential Revision: D24581874
fbshipit-source-id: b66dc007920b7adad5272bf56d3034acb211fec6
Summary:
When built with buck, cmake isn't available, switch to a method of echoing that
should be available on every system: powershell.
Reviewed By: genevievehelsel
Differential Revision: D24493404
fbshipit-source-id: 5986b846a211be59f1057b1e7ecd924f3f7380a5
Summary:
We've had a handful of reports of access denied errors seen during an `hg
update`, ignore them due to:
```
// TODO(T78476916): The access denied are coming from
// PrjMarkDirectoryAsPlaceholder recursively calling into EdenFS, which
// is denied by the BAIL_ON_RECURSIVE_CALL macro.
//
// In theory this means that EdenFS is invalidating a directory that
// isn't materialized, ie: doing useless work. Despite having a negative
// performance impact, this doesn't affect correctness, so ignore for now.
//
// A long term fix will need to not issue invalidation on directories
// that aren't materialized.
```
Reviewed By: wez
Differential Revision: D24544622
fbshipit-source-id: 4f421836b6d8c7b527e860dcf55103e0e239e3ae
Summary: Fix some build warnings that show up on macOS.
Reviewed By: wez
Differential Revision: D24436572
fbshipit-source-id: bd300110df79fe9d259c8f5ce2b1085cb85a0cac
Summary:
Sometimes you only want to trace writes into the EdenFS checkout, so
add the ability to run `eden strace` with `--reads`, `--writes`, or
both in order to see only those events.
Reviewed By: wez
Differential Revision: D24468539
fbshipit-source-id: a1b3c730987cf86ce3d39952c6a5e9c5edaddfc2
Summary:
Before replacing the existing debugInodeStatus changes, make some
small refactorings to help me understand it.
Reviewed By: xavierd
Differential Revision: D24299970
fbshipit-source-id: 3b60aa8365bb9a45e12357374282ba837ccc36eb
Summary:
Add a function that makes it easy to traverse EdenFS's view of the
traversed-so-far filesystem.
Reviewed By: xavierd
Differential Revision: D24299962
fbshipit-source-id: 275c849846bf45a2e80780411d60266b961d825b
Summary:
Now that Linux/macOS startup no longer uses fork it becomes trivial to share
the same code with Windows. This improves Windows startup in several different ways:
- `edenfsctl start` now displays the status of the start process in the console
- `edenfsctl start` no longer has an arbitrary timeout after which it reports
a timeout even though EdenFS is still starting up.
This also kills a bunch of Windows specific code that is no longer needed.
Reviewed By: fanzeyi
Differential Revision: D24393690
fbshipit-source-id: 28100aec96da81c92d5b592353edceed332e2364
Summary:
Previously, when that code was ported on Windows, paths separator were
converted from '\' to '/' when a wide string was provided, all the other paths
were treated as is.
The main issue with this strategy is that not all paths can be converted, the
non-stored ones for instance are immutable, which leads to some subtle bugs
down the line. For instance, the paths: "Z:/foo/bar/baz" and "Z:\foo/bar\baz"
would not be equal as the path separator isn't the same, but both of these are
actually the same path underneath.
To solve this, this diff first introduce a Windows path separator, and then
modifies the path comparison functions to ignore the path separator and only
compare the components.
I'm definitively not a fan of the pattern I use for searching for both / and \
in paths, suggestions are welcome for how to improve that.
Reviewed By: chadaustin
Differential Revision: D24376980
fbshipit-source-id: 0702bf775c7c3937b2138abd5a63d339ac80aaed
Summary:
The path iterator functions are skipping sanity checking of paths as they
assume that the path has already been validated. Unfortunately, that isn't the
case as the only sanity checking we are doing is on the beginning and end of
the string that is passed in. Notably, `RelativePathPiece{"./foo"}` would not
fail the sanity checking even though it it's not made of valid components.
Reviewed By: chadaustin
Differential Revision: D24439804
fbshipit-source-id: 852b3a1180b185cb0bfb96bf5bcdc98b231f32c5
Summary:
As the string is created with MAX_PATH wide characters, a lot of them are going
to stay nul. A future diff adds a sanity check to the PathComponent that
validates that it doesn't contain any nul bytes, which chokes on these paths.
Reviewed By: chadaustin
Differential Revision: D24479715
fbshipit-source-id: e51c4b1c53a3f375664c0c0a1e325ebe302b0cd3
Summary: Comment out the tests that do not compile on Windows.
Reviewed By: fanzeyi
Differential Revision: D24458438
fbshipit-source-id: 14bd4803ccd68ea6e02f95f868b8a801161518ea
Summary: This merely filters out the files that don't compile on mode/win.
Reviewed By: chadaustin
Differential Revision: D24456917
fbshipit-source-id: d314ca2936e064133f2fc6a6199b38d17f2d9ef7
Summary:
FUSE_ACCESS was incorrectly categorized as an 'other' FUSE request
type. Categorize it as 'read'.
Reviewed By: wez
Differential Revision: D24437458
fbshipit-source-id: 5ea0a7d5a0fd57b9230f37674fd2568a4917bab5
Summary:
I noticed while playing with `eden strace` that, on my Linux machine,
running `ls -lA` would cause a pile of FUSE_GETXATTR requests for
`system.posix_acl_access` and `system.posix_acl_default`.
FUSE does not cache getxattr yet, but the kernel can cache ACLs if we
pretend to support them. Thus, advertise FUSE_POSIX_ACL support, but
continue returning ENOSYS for all setxattr calls.
Reviewed By: wez
Differential Revision: D24039130
fbshipit-source-id: a2e69c43b728fb51fb1d1b41ca9d31eba8efaa19
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.
If there are any pending requests at the time `eden strace` is run,
they are printed first.
Reviewed By: wez
Differential Revision: D24038978
fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
Summary:
Buck-out can contain some files that we wont have the ability to read on mac os
(like .Spotlight-V100 and .Trashes). This means `eden du` will error out on
reading them.
We handle FileNotFound but not permission errors, lets treat them the same.
Reviewed By: genevievehelsel
Differential Revision: D24396252
fbshipit-source-id: 31e2ecfeff1783ae9c3479f59335d95a0d9b6de7
Summary:
The communication protocol between edenfs and watchman is that edenfs
advertises that "something in the mount has changed!" and it's up to
watchman to call getFilesChangedSince or getCurrentJournalPosition to
inspect what has changed.
This diff avoids redundantly notifying "something in the mount has
changed!" between the initial notification and actually reading the
journal contents. This way, a Watchman subscription cannot (*) fall
behind, and memory usage stays approximately constant during heavy
write traffic.
Hopefully, this will prevent the kernel from thinking the edenfs mount
is a slow IO device under high memory pressure.
* Technically, if there were a multitude of subscribeStreamTemporary
streams, and all but one was not calling getFilesChangedSince, it's
possible that notifications could back up on the stuck
subscriptions.
Reviewed By: wez
Differential Revision: D24090247
fbshipit-source-id: 6561c13e847b749c093adab75250df474d3210f9
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.
Reviewed By: kmancini
Differential Revision: D24036420
fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
Summary:
Provide an API that allows a FuseChannel TraceBus subscriber to
request population of rendered argument strings. This is not on by
default to avoid the cost in the common case that there are no
subscribers that use the field.
Reviewed By: kmancini
Differential Revision: D24036018
fbshipit-source-id: ff12252de3777606360efc75814149931637aa9b
Summary:
Provide a standard argument renderer per HandlerEntry so it can be
used outside of the strace logger.
Reviewed By: kmancini
Differential Revision: D24035882
fbshipit-source-id: f3f1d918b1a7c937777e2c5fc0de981b5ca3286c
Summary:
Instead of logging in the Dispatcher, move strace logging to
FuseChannel where it can be standardized for all FUSE request types.
Reviewed By: wez
Differential Revision: D24035838
fbshipit-source-id: c84d8c27b62f9944e2d26a35a7ed7bbbeeb5bf0e
Summary:
Watchman, the only consumer of subscribeStreamTemporary today, does
not use the published JournalPosition values. Instead, it waits a bit,
and calls either getCurrentJournalPosition or
getFilesChangedSince. This is a good thing, because write traffic to a
filesystem can easily exceed the rate at which Watchman (or even
streaming Thrift!) can consume it, spiking memory usage and degrading
performance.
The next diff in this stack will take advantage of this clarified
communication protocol to reduce edenfs memory usage under heavy write
traffic.
Reviewed By: wez
Differential Revision: D24089811
fbshipit-source-id: 49de6be24a78934888162a577a554d50e2fff355
Summary:
GNU `df` (and any other coreutil that relies on gnulib's ME_REMOTE
flag) detects remote filesystems with some heuristics. One of which is
whether the device type contains a colon. Since edenfs is a remote
filesystem, include a colon, so it's properly detected as such.
Reviewed By: genevievehelsel
Differential Revision: D24174015
fbshipit-source-id: 4b1f2f49c6eee6e690a6f570924274060a66eee7
Summary:
Chef writes root-owned temporary files in /etc/eden/config.d. If
edenfsctl attempts to read those configs at this time, it will
fail. Just skip all hidden files.
Reviewed By: kmancini
Differential Revision: D24377953
fbshipit-source-id: 765b7f096b62e6749e186adc3e90bbc7814079c8
Summary:
InodeBase::getPath is not deterministic under concurrent
renames. Instead, thread the paths down through the recursion so
they're consistent.
Reviewed By: genevievehelsel
Differential Revision: D24297724
fbshipit-source-id: 3e8664ce2bc5159e464d1d76ed37294c4eac1709
Summary:
On Windows, computing the sha1 of a materialized file requires opening up the
file in the working copy, as the file is cached there. Interestingly, this
potentially means that for computing the sha1 of a file, EdenFS may receive a
callback from ProjectedFS about that file or a parent directory. At this point,
EdenFS just refuses to serve this callback, as doing so may trigger an infinite
loop, or simply deadlocks. While this may sound weird, recursive callbacks are
not expected, as this signify that EdenFS view of the working copy doesn't
match what it actually is.
To close the loop, and from a code perspective, this means that computing the
sha1 of a file can fail and can throw an exception. Unfortunately, the code
didn't reflect this fact and exceptions were simply ignored, when that happens
during a checkout operation, this can leave the working copy in a weird state,
further agravating the mismatch between EdenFS view of the working copy, and
what it actually is.
Reviewed By: wez
Differential Revision: D24282048
fbshipit-source-id: 745af03189fe345150f0b1792ee1b37a1b8fb0d4
Summary:
While on Linux these can't fail (or, to be more precise: it doesnt' matter),
they can on Windows. One such exemple is when a user lock a file and triggers
an update that modifies this file. The invalidation will fail, and thus the
update should keep track of that file not being updated properly.
Previously, the invalidation would raise an exception, but that proved to be
the wrong approach as some state would need to be rolled back which the
exception didn't help in. For that, let's just return a Try and make sure that
we handle all the cases properly.
Reviewed By: chadaustin
Differential Revision: D24163672
fbshipit-source-id: ac881984138eefa65c053478a160e2a653fd3fdf
Summary: Add option to redirect the eden rage output to stderr
Reviewed By: chadaustin
Differential Revision: D24210031
fbshipit-source-id: 736af81de6a41e49c2459a1cd8124123d6928323
Summary:
There's a bug in Thrift-py3 streaming support, where interrupting
iterating over an async stream leaves Thrift objects in a broken
state. Futures get dropped (and warnings are printed to the console)
but the destructors hang. Don't even try to garbage collect the heap
in that case.
Reviewed By: genevievehelsel
Differential Revision: D24032229
fbshipit-source-id: 5f16667fe6cfd27de1b39cc2974028729e18b214
Summary: Thrift-py3 supports streaming, so give the new client access to APIs from streamingeden.thrift.
Reviewed By: wez
Differential Revision: D24032144
fbshipit-source-id: 44f350b5cfa943154084b8d64f6c696e315e6b88
Summary: Eden mounts are identified by paths, not by strings, so fix the Thrift signature.
Reviewed By: wez
Differential Revision: D23774513
fbshipit-source-id: c0fb82c48eee5ce4e8fbffef5623f9016ef76e40
Summary:
Adds a subcommand to `eden debug prefetch_profile` to fetch all the files
for a list of profiles or all the active profiles for a checkout.
These profiles (files) can contain lists of file names, or globs of files, eden
will be able to resolve them since this uses the existing prefetch code.
I opted to put this command under `eden debug prefetch_profile` instead
of `eden prefetch` since the command may change to call into eden with out
the list of files (letting eden resolve the active profiles and names). Then
it will no longer resemble prefetch and so long term it may be better homed
here.
Reviewed By: genevievehelsel
Differential Revision: D23771483
fbshipit-source-id: 12af81d40cc495efd381e3c3a2df645d72053ccd
Summary:
Before I make further changes to the Journal, improve the comments and
refactor a few small things.
Reviewed By: kmancini
Differential Revision: D24089530
fbshipit-source-id: de9da2c1e6b1c87b6587781cfa55ae7cc4085eeb
Summary:
Scanning through the functionality provided in ServiceFrameworkLight,
it looks like none of it really applies to the edenfs daemon, so break
the dependency entirely. Removing this complexity would have prevented
the regression where we stopped recording Thrift call statistics.
This should also improve our build times a bit, and maybe resource
consumption.
Reviewed By: genevievehelsel
Differential Revision: D24094784
fbshipit-source-id: fefd1a648c7ecba8484296527ff8100269c176b6
Summary: Add a reliable, lightweight TraceBus class for publishing events to a background thread. Subscribers can be registered for observing events or computing telemetry about them.
Reviewed By: wez
Differential Revision: D23404525
fbshipit-source-id: 3539466421b0821ffb918ea862168d3cccd19b15
Summary: To be more Windows compatable, we should move away from using `sh -c`. We don't use shell=True because that is susceptible to shell injection. As long as we don't close stdin until we're done, using Popen raw should be fine.
Reviewed By: xavierd
Differential Revision: D24151425
fbshipit-source-id: c0bcc883af948491862e8ce0cee56bcbe98e04f1
Summary:
Similarly to how we could try invalidating a file that isn't cached, we could
also be trying to invalidate a file whose path isn't cached. Both are
legitimate, and thus we need to ignore both.
Reviewed By: chadaustin
Differential Revision: D24125225
fbshipit-source-id: e8abe5cde5aa3602bb48258abb64aa0cdf60241d
Summary:
Thrift represents `binary` data type as `std::string` in C++. This method will
help us to convert `Hash` into a byte string.
Reviewed By: xavierd
Differential Revision: D24083621
fbshipit-source-id: ae50088db7727d98ca11a017f82b71e942217a17
Summary:
This diff adds a new constructor to `SqliteDatabase` to allow creation of
in-memory SQLite database. This can come in handy in testing.
Reviewed By: xavierd
Differential Revision: D24083579
fbshipit-source-id: ad6dd8b1c20392a882c1f164ef1f8af2f0ba11f8
Summary:
This allows `edenfsctl debug processfetch` to display what processes triggered
some IO in EdenFS which will be useful to debug rogue processes walking the
entire repo.
Reviewed By: chadaustin
Differential Revision: D23997665
fbshipit-source-id: 7d92755d0068a4b1819eb0c84b30cbdaa24296f7
Summary:
This will enable to gather a bit more debugging regarding what processes are
fetching data. The one missing bit on Windows is to collect the process name,
for now, a "NOT IMPLEMENTED" placeholder is put in place.
Reviewed By: wez
Differential Revision: D23946258
fbshipit-source-id: 9f7642c7b9207c5b48ffff0f4eb0333af00bc7d5
Summary: HostInfoProperties is allocated for every HostInfo and is accessed on every request. There's no reason this should be a unique_ptr, and the pointer indirection is expensive.
Reviewed By: jmswen
Differential Revision: D24009296
fbshipit-source-id: 2034d1c6e61e0dec51ca6ac7bd14ab12e74966d4
Summary:
The SpawnedProcess tests were failing on my macOS machine because pwd
and getcwd returned slightly different paths. Normalize them before
comparing.
Reviewed By: genevievehelsel
Differential Revision: D24094634
fbshipit-source-id: aacf802280b1dd1de19797604bfe359d7e60cbf8
