Summary:
There's a bug in Thrift-py3 streaming support, where interrupting
iterating over an async stream leaves Thrift objects in a broken
state. Futures get dropped (and warnings are printed to the console)
but the destructors hang. Don't even try to garbage collect the heap
in that case.
Reviewed By: genevievehelsel
Differential Revision: D24032229
fbshipit-source-id: 5f16667fe6cfd27de1b39cc2974028729e18b214
Summary: Thrift-py3 supports streaming, so give the new client access to APIs from streamingeden.thrift.
Reviewed By: wez
Differential Revision: D24032144
fbshipit-source-id: 44f350b5cfa943154084b8d64f6c696e315e6b88
Summary: Eden mounts are identified by paths, not by strings, so fix the Thrift signature.
Reviewed By: wez
Differential Revision: D23774513
fbshipit-source-id: c0fb82c48eee5ce4e8fbffef5623f9016ef76e40
Summary:
Adds a subcommand to `eden debug prefetch_profile` to fetch all the files
for a list of profiles or all the active profiles for a checkout.
These profiles (files) can contain lists of file names, or globs of files, eden
will be able to resolve them since this uses the existing prefetch code.
I opted to put this command under `eden debug prefetch_profile` instead
of `eden prefetch` since the command may change to call into eden with out
the list of files (letting eden resolve the active profiles and names). Then
it will no longer resemble prefetch and so long term it may be better homed
here.
Reviewed By: genevievehelsel
Differential Revision: D23771483
fbshipit-source-id: 12af81d40cc495efd381e3c3a2df645d72053ccd
Summary:
Before I make further changes to the Journal, improve the comments and
refactor a few small things.
Reviewed By: kmancini
Differential Revision: D24089530
fbshipit-source-id: de9da2c1e6b1c87b6587781cfa55ae7cc4085eeb
Summary:
Scanning through the functionality provided in ServiceFrameworkLight,
it looks like none of it really applies to the edenfs daemon, so break
the dependency entirely. Removing this complexity would have prevented
the regression where we stopped recording Thrift call statistics.
This should also improve our build times a bit, and maybe resource
consumption.
Reviewed By: genevievehelsel
Differential Revision: D24094784
fbshipit-source-id: fefd1a648c7ecba8484296527ff8100269c176b6
Summary: Add a reliable, lightweight TraceBus class for publishing events to a background thread. Subscribers can be registered for observing events or computing telemetry about them.
Reviewed By: wez
Differential Revision: D23404525
fbshipit-source-id: 3539466421b0821ffb918ea862168d3cccd19b15
Summary: To be more Windows compatable, we should move away from using `sh -c`. We don't use shell=True because that is susceptible to shell injection. As long as we don't close stdin until we're done, using Popen raw should be fine.
Reviewed By: xavierd
Differential Revision: D24151425
fbshipit-source-id: c0bcc883af948491862e8ce0cee56bcbe98e04f1
Summary:
Similarly to how we could try invalidating a file that isn't cached, we could
also be trying to invalidate a file whose path isn't cached. Both are
legitimate, and thus we need to ignore both.
Reviewed By: chadaustin
Differential Revision: D24125225
fbshipit-source-id: e8abe5cde5aa3602bb48258abb64aa0cdf60241d
Summary:
Thrift represents `binary` data type as `std::string` in C++. This method will
help us to convert `Hash` into a byte string.
Reviewed By: xavierd
Differential Revision: D24083621
fbshipit-source-id: ae50088db7727d98ca11a017f82b71e942217a17
Summary:
This diff adds a new constructor to `SqliteDatabase` to allow creation of
in-memory SQLite database. This can come in handy in testing.
Reviewed By: xavierd
Differential Revision: D24083579
fbshipit-source-id: ad6dd8b1c20392a882c1f164ef1f8af2f0ba11f8
Summary:
This allows `edenfsctl debug processfetch` to display what processes triggered
some IO in EdenFS which will be useful to debug rogue processes walking the
entire repo.
Reviewed By: chadaustin
Differential Revision: D23997665
fbshipit-source-id: 7d92755d0068a4b1819eb0c84b30cbdaa24296f7
Summary:
This will enable to gather a bit more debugging regarding what processes are
fetching data. The one missing bit on Windows is to collect the process name,
for now, a "NOT IMPLEMENTED" placeholder is put in place.
Reviewed By: wez
Differential Revision: D23946258
fbshipit-source-id: 9f7642c7b9207c5b48ffff0f4eb0333af00bc7d5
Summary: HostInfoProperties is allocated for every HostInfo and is accessed on every request. There's no reason this should be a unique_ptr, and the pointer indirection is expensive.
Reviewed By: jmswen
Differential Revision: D24009296
fbshipit-source-id: 2034d1c6e61e0dec51ca6ac7bd14ab12e74966d4
Summary:
The SpawnedProcess tests were failing on my macOS machine because pwd
and getcwd returned slightly different paths. Normalize them before
comparing.
Reviewed By: genevievehelsel
Differential Revision: D24094634
fbshipit-source-id: aacf802280b1dd1de19797604bfe359d7e60cbf8
Summary:
Formats a subset of opted-in Python files in fbsource.
Black formatting was applied first, which is guaranteed
safe as the AST will not have changed during formatting.
Pyfmt was then run, which also includes import sorting.
The changes from isort were manually reviewed, and
some potentially dangerous changes were reverted,
and the directive was added to those
files. A final run of pyfmt shows no more changes to
be applied.
Reviewed By: zertosh
Differential Revision: D24101830
fbshipit-source-id: 0f2616873117a821dbc6cfb6d8e4f64f4420312b
Summary: Since this is only used in the manifest target, fold it into it.
Reviewed By: DurhamG
Differential Revision: D24062629
fbshipit-source-id: c3241b53bde7abba8a80a2945661d1a24b7e3034
Summary:
Less #ifdef is always better, since normalizing a path also works on
non-Windows, there is no reason to special case Windows here.
Reviewed By: fanzeyi
Differential Revision: D24020603
fbshipit-source-id: 114dae3bd9a4743230f4c82d219ff74ffc9379c7
Summary:
In c++, to zero-initialize, empty braces is sufficient for a non-class type.
{0} will only zero-initialize the first field, which is not the intent here.
Reviewed By: wez
Differential Revision: D24068481
fbshipit-source-id: 2de87da983a05f25e0222bf5338533a7b96fb36a
Summary:
This is no longer about datapack, but only about parsing manifest entries, thus
renaming.
Reviewed By: DurhamG
Differential Revision: D24062634
fbshipit-source-id: 5c52b784d20437e87012dd4bc6cb13d879da9cb9
Summary:
This code is effectively unused. The only bit still relevant is that EdenFS
still depends on the Manifest class to parse a manifest.
Reviewed By: DurhamG
Differential Revision: D24037723
fbshipit-source-id: 901ae2ffc8960a95ec655a2e14d79afb8d32dcab
Summary:
It was used as a fallback path only for a while now. Since Mercurial has
support CMD_CAT_TREE for quite some time, let's get rid of it.
Reviewed By: fanzeyi
Differential Revision: D24037004
fbshipit-source-id: 69887e6d8508419a22d68d062c78676aacba3b24
Summary:
If the data isn't found in the Rust one, it can't be found in the non-Rust one.
Since the non-Rust one will issue a filesystem rescan, this is a fairly
expensive operation which shows up in strobelight when trying to walk the
entire repo with: `rg --files`.
There is one last place that still use the non-Rust stores and that's as a
fallack for when Mercurial doesn't support CMD_CAT_TREE. Since this has been
supported for a bit, I'll make a followup change to completely get rid of the
non-Rust stores.
Reviewed By: fanzeyi
Differential Revision: D24035451
fbshipit-source-id: acd9741a16f3786796d329a4cddfe4ee435bcad9
Summary:
On Windows, the du command line application doesn't exist, thus we cannot use
it. Instead, we can simply re-implement the du functionality in Python and use
that on all platforms.
Reviewed By: chadaustin
Differential Revision: D24030269
fbshipit-source-id: e86c1bcdeac7eeca70201f6fde48c20ef7e305a6
Summary:
There is no good reason for this function to be in a header, let's move it to
StringConv.cpp
Reviewed By: genevievehelsel
Differential Revision: D24000882
fbshipit-source-id: 5bb6bc3b9ef37232d38b8e35da693e12c0453ea1
Summary:
Replace it by simply using std::chrono facilities to achieve the exact same
behavior.
Reviewed By: chadaustin
Differential Revision: D24027637
fbshipit-source-id: d22eefec5d408ead0b4cfaa20e716f4c10cce0b5
Summary:
There is no reason not to populate this on Windows, so let's do it to remove a
handful of #ifdef.
Reviewed By: chadaustin
Differential Revision: D24003912
fbshipit-source-id: 380b37d6d1a91f13cdc711c0055a5437f1184c20
Summary:
stats local-store hardcodes the column names and has diverged from the
source of truth in KeySpace.h. Bring it back up to date. Ideally,
there would be a Thrift call to retrieve the column information, but
genevievehelsel might end up replacing this stuff soon.
Reviewed By: genevievehelsel
Differential Revision: D24027548
fbshipit-source-id: 5e5195585025081969865f8d3651e742f721ea09
Summary:
For the longest time, invalidation errors were simply ignored, and while some
of them are valid to ignore, others just aren't and should be surfaced. One
exemple of a valid error is when a file is opened with no sharing, attempts to
invalidate the file would fail, but not inform the users, who would then be
surprised that the file wouldn't be updated on disk properly.
Instead of ignoring all errors, I've special cased the ones that we may be
expecting to receive, and will treat all the others as errors. A future diff
will attempt to properly not update the parent TreeInode so that a failed
invalidation would then be visible during an `hg update`.
Reviewed By: genevievehelsel
Differential Revision: D23980547
fbshipit-source-id: 2e67bfe817951fd0b497214a6474ff28b953b6e6
Summary: The logs would never show what directory was changed to be a placeholder.
Reviewed By: fanzeyi
Differential Revision: D23977719
fbshipit-source-id: b1f7f539457956510638a31ca3e49c4cee641fd8
Summary: This is not needed, we can use the PrjfsChannel class directly where needed.
Reviewed By: chadaustin
Differential Revision: D23946259
fbshipit-source-id: eafcd38c0927fa282d62ada0986a7ef8b612174b
Summary:
Prefetch profile can now be used in automation to register profiles and to fetch
them. Since it will be used out side of the team it seems like this should be
moved out from under debug.
Reviewed By: genevievehelsel
Differential Revision: D23773337
fbshipit-source-id: b7f646ec202653233ed82528380d91f583a8341e
Summary:
This will allow us (and maybe users) to remove a users active profiles
easily when investigating issues. (And also will be useful for testing.)
Reviewed By: genevievehelsel
Differential Revision: D23517409
fbshipit-source-id: d57e595770b5a60726e2efe9cbf21c89dd694cc0
Summary:
This will allow tools to activate prefetch profiles relavant to their workflow.
Eventually this should also cause eden to prefetch the profile for the current
commit, but for now it just activates the profile.
Reviewed By: genevievehelsel
Differential Revision: D23517412
fbshipit-source-id: 172a844ca00bf3bcf14f4b94c25e8256e516b737
Summary:
This will allow us (and maybe users) to see a users active profiles
easily when investigating issues. (And also will be useful for testing.)
Reviewed By: genevievehelsel
Differential Revision: D23517410
fbshipit-source-id: 7a592f7b39c887600307a4f08668ca1ee9b982bf
Summary:
Assuming the format for active prefetch profiles as discussed here:
https://fb.quip.com/v8yFAzeGx8TU
This allows the cli to read in and write out active profiles, allowing the cli
to manipulate them in the following changes. This will allow tools to activate
profiles useful for that workflow. Additionally, it will allow us (and maybe
users) to see/remove a users active profiles easily when investigating issues.
Reviewed By: genevievehelsel
Differential Revision: D23517411
fbshipit-source-id: 8ab18a1aeccf8dcdd04aca826d61070fa0aedd33
Summary:
It looks like clang in mode/win choke on the code otherwise: P143369452. It's
definitively not clear to me why. Since fmt::format should pretty much
similarly to folly::to<std::string>, just use that instead.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D23884978
fbshipit-source-id: f4a7e47169c8b78011340ec0a41871c9fa4b7181
Summary:
This stack updates eden to be able to check all of the locations that able
users certificate may reside.
There can be multiple places where a cert may reside (we cant always
definitively choose one place to look based on the platform). Thus we
need to be able to configure multiple locations for certs in our eden
config.
This makes the switch over in eden from using one place for the client
cert to use and using the first available client cert from a list.
NOTE: most of this diff is fixing unit tests take a look at `EdenConfig.h` and `EdenConfig.cpp` first
Reviewed By: wez
Differential Revision: D23359939
fbshipit-source-id: 44beecce3ef098a734dbd7c5eb3fa5f0aad6b50b
Summary:
These shows up on Windows when building with mode/win, silencing them is easy,
so let's do it.
Reviewed By: wez
Differential Revision: D23871727
fbshipit-source-id: 7d7ea9504c397b72903e98967188a5295f2f1040
Summary:
These don't compile on Windows, and in order to get mode/win to compile, we
need to avoid compiling their contents. Ideally, we could do that in the
TARGETS files with the select statement, but that's not available in fbcode.
Thus, we do the next best thing: ifdef the file entirely.
Reviewed By: wez
Differential Revision: D23871728
fbshipit-source-id: b4d9df6503eaa008e649afd7bdc665cd37a9585d
Summary:
*shakes fist at C++ copy-constructors*
We weren't guaranteeing that case-insensitive status was being propagated on
copies or moves, which meant that eg: `lookup("workspace")` would be treated as
case-sensitive when mounted case insensitively.
Reviewed By: xavierd
Differential Revision: D23857218
fbshipit-source-id: 67e33a8455a0a85e5885389b5bb38b20ef043894
Summary: This can be included in all the supported platforms, thus don't #ifdef it.
Reviewed By: wez
Differential Revision: D23858664
fbshipit-source-id: e61da33a97d87cbfab5bb96d2bdaa865d2c01801
Summary:
This enables autodeps, and brings us one step closer to building EdenFS with
Buck on Windows.
Reviewed By: fanzeyi
Differential Revision: D23857794
fbshipit-source-id: c8587a6f7b9e4d9575a62f592c1d0737dff2a8f0
Summary:
Now that prjfs/EdenDispatcher is no longer directly tied to ProjectedFS (sort
of, this is still a bit WIP), we can move it out of the prjfs directory onto
the inodes one. This allows breaking the circular dependency cycle mentioned in
the previous diff where prjfs and inodes depend on each other.
For now, this is merely a copy/paste of the code enclosed in big #ifdef _WIN32,
we might be able to do better later, but for now this is properly good enough.
Reviewed By: wez
Differential Revision: D23857539
fbshipit-source-id: 77c620bac1656d01d7daee4dbf8b10694a589751
Summary:
If we are to look at the dependency graph for EdenFS on Windows, we would
notice that the eden_prjfs target depends on eden_inodes, and vice versa,
causing a cycle. While CMake is perfectly happy with that, Buck doesn't like
that. The solution to removing this cycle is to move the code that needs the
dependency to eden_inodes into the eden_inodes target, and that's the
EdenDispatcher. However, since PrjfsChannel needs to hold a dispatcher to call
into it, it needs to know the methods exposed by the dispatcher. To achieve
this, a simple abstract class is added, this is the same as what is done for
FUSE.
Reviewed By: wez
Differential Revision: D23857540
fbshipit-source-id: c495c67d43724f648e5ffa17776e4d5d4513698a
Summary: The Windows code is now free of sformat, and only depends on fmt::format.
Reviewed By: fanzeyi
Differential Revision: D23786940
fbshipit-source-id: 4c9d4a8c30e5a3d5eb7971cdc3139539c5893a8d
Summary:
Now that all the logic is in PrjfsChannel.cpp, this header no longer needs to
be included. This helps in making this file as free as possible from Windows
specific bits which will allow it to be moved into the inodes top level
directory.
Reviewed By: fanzeyi
Differential Revision: D23786941
fbshipit-source-id: a4a3a47ea00808ca7a839709068efab06436167f
Summary:
Most of the non-tests callers have an ObjectFetchContext available, let's use
it instead of a static null context. This will help in understanding why some
files/trees are being fetched.
Reviewed By: chadaustin
Differential Revision: D23745752
fbshipit-source-id: b2e145d9e559bde0542adbc5b20ff56ccfc59ece
