Summary:
update thrift interface to include a function that returns attributes for a given list of files.
to support this, we add several structures that define the functions parameters and return type.
- FileAttributes: a "bitmask" enum. Each attribute is a power of 2 to enable bitwise ORing attributes together to decide which attributes to fetch from files.
- FileAttributeData: struct to hold the attributes of an individual file
- FileAttributeDataOrError: union that is either a FileAttributeData or EdenError thrown when accessing the file attributes
- GetAttributesFromFilesParameters: self explanatory.
- GetAttributesFromFilesResult: self explanatory. Map of file -> FileAttributeDataOrError
- getAttributesFromFile(): actual function that will be implemented in EdenServiceHandler.cpp
Reviewed By: chadaustin
Differential Revision: D32155168
fbshipit-source-id: 6dd768f3bcee1e32a2362df231feebfb1297dbf6
Summary:
Cast EDEN_MICRO to something fmt::format can handle: char* (from std::basic_string<char8_t>); but that means this is no longer a literal, so cannot concat. Move to its own fmt argument.
This avoids the following errors:
eden/fs/service/EdenServiceHandler.cpp:227:60: error: invalid operands to binary expression ('basic_ostream<char>' and 'std::basic_string<char8_t>')
Differential Revision: D32486517
fbshipit-source-id: c799bdd73be1b46cc938b2076ebc78d0bdb3dded
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
reinterpret_cast another u8"..." literal to const char*
This avoids the following errors:
```
stderr: In file included from eden/fs/utils/test/Utf8Test.cpp:9:
In file included from folly/portability/GTest.h:32:
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1545:11: error: invalid operands to binary expression ('char8_t const[4]' and 'const std::basic_string<char>')
if (lhs == rhs) {
~~~ ^ ~~~
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1572:12: note: in instantiation of function template specialization 'testing::internal::CmpHelperEQ<char8_t [4], std::basic_string<char>>' requested here
return CmpHelperEQ(lhs_expression, rhs_expression, lhs, rhs);
^
eden/fs/utils/test/Utf8Test.cpp:43:3: note: in instantiation of function template specialization 'testing::internal::EqHelper::Compare<char8_t [4], std::basic_string<char>, nullptr>' requested here
EXPECT_EQ(u8"\uFFFD", ensureValidUtf8("\xff"));
```
Differential Revision: D32484896
fbshipit-source-id: b271292eb121e0e183bec56ab51fc6acb791b052
Summary:
On Windows, directory removal notifications can be received out of order, and
sometimes duplicated. The next diff will rework the way EdenFS handle
notifications on Windows by looking at the on-disk state and applying it to the
inode hierarchy. In some cases, an entire directory hierarchy will need to be
removed which the added function will make it easier to do.
Reviewed By: chadaustin
Differential Revision: D32329806
fbshipit-source-id: ad9af389c993420184c0e59dd3d25188adddb77a
Summary: Have constructor do init() instead. I can't think of any valid reason to have this API other than confusing everyone who decides to use it.
Reviewed By: praihan
Differential Revision: D32302228
fbshipit-source-id: 136e0d2919291acd60be953337309cd006bd674b
Summary: Sha1 computation show up in many profilers, we can skip this step entirely if we are using metadata from hg.
Reviewed By: xavierd
Differential Revision: D32396312
fbshipit-source-id: 928f8458a91bba4d18da6730d3fc244eb5499489
Summary:
This diff allows eden to fetch blob metadata from hg cache.
Combined with previous diffs in stack, it allows to entirely skip scs metadata import, saving whole additional network round trip for operations like ls -lr or parsing buck files.
This diff adds hg:use-aux-metadata config that controls this behaviour.
When config is set, we instruct hg backing store to fetch file aux metadata along with get tree request.
Later when blob metadata is requested we are checking hg cache to get the metadata.
For trees that were imported prior to this config we fallback to reading scs metadata cache.
In terms of performance effect, this diff reduces traffic on laptops during listing/stating files significantly, because x2p client sends significant amount of extra data.
It does not improve end to end speed significantly though, because scs queries run in parallel with tree request.
Before
```
ls -lR fbsource.eden/fbcode/admarket 0.97s user 8.02s system 3% cpu 4:47.42 total
Traffic usage: 77Mb 25Mb down, 52Mb up
```
After
```
ls -lR fbsource.eden/fbcode/admarket 0.98s user 8.19s system 3% cpu 4:33.65 total
Traffic usage: 40 Mb, 22Mb down, 18Mb up
```
On laptops with worse network this likely to improve speed too, because some networks have much lower uplink latency
Additionally, this allows to efficiently use metadata on windows (where previously entire blob needed to be downloaded in order to calculate blob size)
Reviewed By: xavierd
Differential Revision: D32371710
fbshipit-source-id: 8ac962ea0136bc3bc4e30d0bf31692e65c35c43f
Summary:
When unwrapping Try it is convenient to be able to build an ImmediateFuture
directly from the .exception() call.
Reviewed By: chadaustin
Differential Revision: D32333774
fbshipit-source-id: 0066ca7eddc4a57010f60974c234b57691b73d77
Summary:
For some reason, in some cases the compiler wants to use the folly::collectAll
function instead of the ImmediateFuture one. Adding the namespace solves this
issue.
Reviewed By: chadaustin
Differential Revision: D32329807
fbshipit-source-id: 22ee9b20cf33e5718c9bb0270dd93f3f857b0967
Summary:
Previously it raised
``` raise TTransportException(
type=TTransportException.NOT_OPEN, message="eden not running"
)
```
even if it wasn't thrift and it wasn't eden.
Now it will raise OSError (as regular sockets do), and translate it to TTransportException if it's used as a thrift socket
Reviewed By: DurhamG
Differential Revision: D32171878
fbshipit-source-id: 290368acf8def5aec812abc88c23a946e7a0f1f4
Summary:
So that I can re-use it in the next diff.
I considered putting this in a util/ subfolder but when doing the `use` imports, this caused issues since there was another crate called util and would require renaming all those use statements.
Reviewed By: fanzeyi
Differential Revision: D32331290
fbshipit-source-id: 78f9bf94bcf6426f1f0f23761061161e3e392832
Summary: As fanzeyi pointed out, the Rust compiler can correctly infer that this code is unreachable (since compiler warnings pop up) and we don't need to explicitly write unreachable.
Reviewed By: fanzeyi
Differential Revision: D32295516
fbshipit-source-id: d8562fab7a4d012db5fe91c4947558531e512144
Summary:
This is because the save / restore cursor position sometimes behaves differently depending on the command that I run minitop with (e.g., it sometimes will re-print the first row repeatedly if the saved cursor position was somehow wrong).
I also explored calculating how much to truncate by getting the screen size, but this was simpler.
We can add this back in if we switch to an "alternate screen".
Reviewed By: fanzeyi
Differential Revision: D32289941
fbshipit-source-id: 3edda78ab2aa82958f6a40efd03e302520f84f4b
Summary:
There is a race in here. We read the config, then mutate it, then put the new
config in place. However, if something else comes and mutates the config
between the time where we read it and write it again, we'll clobber its
changes.
Concretely, if you're running lots of `eden clone` or `eden rm` concurrently,
then the current mechanism will sometimes "forget" some directories or add back
directories you removed, which will e.g. break `eden list` (and cause those
commands to sometimes fail as well).
This diff fixes that by wrapping the critical section with a file lock.
Reviewed By: chadaustin, xavierd
Differential Revision: D32251727
fbshipit-source-id: 9eee57c7a824286bb4f0e7911e5ccd9d769c6daa
Summary:
There was confusion in previous diff with some proxy hash interface taking optional<> and some taking nullable pointer, resulting in cretion of optional with null pointer inside.
To avoid this confusion I am just going to use optional<> everywhere.
Reviewed By: chadaustin
Differential Revision: D32324008
fbshipit-source-id: 2f0df1f50d032dd0c8823a1671a69ff74435eb3b
Summary:
On takeover, the new EdenFS sends the current mount point over to the
privhelper, and due to being a suid root process, the privhelper needs to
double check that the given mount point is valid before accepting it. In order
to do this, a `stat` is ran on the mount point.
Unfortunately, this `stat` call may need to perform a lookup and getattr FUSE
calls, but since EdenFS hasn't yet initialized FUSE, it may hang, leading to
EdenFS not being able to restart.
To fix this, we can simply start FUSE prior to advertising the mount point to
the privhelper, this way the lookup and getattr FUSE calls can be answered.
Reviewed By: kmancini
Differential Revision: D32322661
fbshipit-source-id: 4ea1e44ce362e04ed7eebaff1f2a8f381f8bff60
Summary:
Do not hide cursor because when you stop the process, it leaves the cursor as disabled.
Even when this was disabled, you could still type anyway. Better to use alternate methods to disable this.
Reviewed By: fanzeyi
Differential Revision: D32254638
fbshipit-source-id: ce76b8943aea5f15eb35a263afc1b67593c2ba12
Summary:
I noticed that I'm hitting this assertion. Likely because the running process changes what commands it's running. Let's remove this assertion.
This is the same as the python `eden top`'s behavior where they just continue to aggregate and never update the `cmd`.
Reviewed By: fanzeyi
Differential Revision: D32173253
fbshipit-source-id: 72bc4710e585fa800d7d96c04fa9a88f13a8557d
Summary:
The QueuedImmediateExecutor can have surprising effect, let's instead move the
futures to the global CPU executor, but only if the ImmediateFuture isn't
ready, ie: when there is still some computation to be performed. The use of
folly::detachOnGlobalCPUExecutor also makes it obvious that the intent is to
run the future elsewhere in the background.
Reviewed By: chadaustin
Differential Revision: D32155651
fbshipit-source-id: 8ae5be48230ff44c58a5697d3824825eeb8235dc
Summary:
In order to compute the sha1 of materialized files in the working copy, EdenFS
on Windows will open and read the file out of the working copy directly. In the
case where EdenFS is wrong about the state of the file, this can lead to a
ProjectedFS callback to be triggered which may need to take the same lock that
the sha1 computation holds. This obviously results in a deadlock and to protect
against this, EdenFS would bail early when detecting recursive callbacks.
With read-only callbacks having been switched to only take into account the
underlying Mercurial data, and not the inode state, these callbacks will never
attempt to take any inode locks. This also means that the deadlock described
above can no longer happen: the ObjectStore layer will never attempt to read
anything from the working copy, thus recursive callbacks are impossible. These
checks are thus unecessary.
In theory, these checks could stay in place as a safety measure, but EdenFS
will soon need to list directory entry and call PrjGetOnDiskFileState in
response to file/directory creation/removal to handle the case where these can
happen out of order.
Reviewed By: genevievehelsel
Differential Revision: D32149411
fbshipit-source-id: 73b228bb8c73416b38f7f2042c25964e1aef3c4e
Summary:
We already had a collectAllSafe for argument packs, let's have a similar
function when the argument is a vector.
Reviewed By: chadaustin
Differential Revision: D32323573
fbshipit-source-id: 2cfa3da565a7d9bb88746e3cc6b10d1d8890c05e
Summary:
This config has been enabled for many months now, let's simply assume it is set
to true on Windows.
Reviewed By: genevievehelsel
Differential Revision: D32192585
fbshipit-source-id: 76829a031f859594a5139dd09a6479cabeda2089
Summary:
This diff introduces new format for serializing trees in local store.
Unlike currently used git serialization format, new format supports variable length object ids.
Long term this will be replaced with using mercurial as a source of truth, but for now we need this to roll out proxy hash replacement.
Reviewed By: chadaustin
Differential Revision: D31998935
fbshipit-source-id: 9aacfbea631e75c0ea1421094a0b1ae255adb04a
Summary: If toml values are maps, we were overriding the previous values without merging them. This makes `eden-config` to incorrectly discard values that should have been kept.
Reviewed By: genevievehelsel
Differential Revision: D32288269
fbshipit-source-id: e9c9f6a1aed3b0de719a046e56344cab763a3e7e
Summary: I wasn't sure why I didn't make them a test when I wrote them. Let's make them a test and continuously run them to make sure `stack-config`'s implementation is correct.
Differential Revision: D32288267
fbshipit-source-id: 988f65993de0947ce408a6809b32b3ece5e9b3a3
Summary: When debugging, it's nice to know what the kernel thinks the mount table is.
Reviewed By: xavierd
Differential Revision: D32288215
fbshipit-source-id: d4c15b16689f20846a45f682c4089e2fa8250661
Summary:
In ProjectedFS terminology, the "backing store" refers to the state of files
and directories in the source control backend. For the longest time, EdenFS
assumed that the Inodes hierarchy was the "backing store", but this lead to
subtle bugs. For instance, renaming a non-hydrated placeholder would render the
file unreadable due to the read callback from ProjecedFS callback using the old
file name, not the new one
(https://github.com/microsoft/ProjFS-Managed-API/issues/68). It also meant that
files just created would always get a tombstone written on disk when being
removed as ProjectedFS queries EdenFS when removing a file via the access
callback. Returning true from this callback means that this file is present in
the backing store and thus ProjectedFS needs to create a tombstone on disk.
In order to fix these issues, EdenFS needs not to query the Inodes hierarchy,
but the Tree hierarchy. The latter matches with the expected behavior from
ProjectedFS.
For modification callbacks, these are by design acting on the Inode hierarchy
and thus are unchanged.
Reviewed By: chadaustin
Differential Revision: D32022639
fbshipit-source-id: bef18a0709a7eff9ef0a48a90126d1339be0023e
Summary:
Update tracing-subscriber crate to 0.3.1. Also add some features that David Barsky suggested.
Also update tracing-appender since 0.1.x is incompatible.
Reviewed By: zsol
Differential Revision: D32175169
fbshipit-source-id: ad714db1c1c67b508be1c97d3dc960dd7d809cc6
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
add an explicit constructor, now required for C++20, see https://en.cppreference.com/w/cpp/language/default_constructor
This avoids the following errors:
eden/fs/utils/test/ImmediateFutureTest.cpp:335:17: error: no matching constructor for initialization of 'Uncopyable'
# Context:
The cost of delaying a platform upgrade is now estimated at about $3M/week.
For the preceding one, it was measured at [**$1M/week**](https://fburl.com/jfmpsqmy).
Differential Revision: D32250064
fbshipit-source-id: 8c6990489cafb09629c03e56bdbae92395d6f85f
Summary:
This diff introduces config store:embed-proxy-hashes.
When this config is set, we store HgId directly into ObjectId, instead of using proxy hash object.
This allows to bypass proxy hash rocks db storage when reading files.
**Compatibility notes**
This diff is compatible with previous versions unless store:embed-proxy-hashes config is set.
Once config is set, new ObjectId format is used and serialized into inodes. Once this is done previous versions of eden fs won't be able to read overlay inodes created by this version.
This means we need to be careful with setting this config - once set we won't be able to roll back eden fs version easily, it will basically require re-creating eden checkout.
Inodes created prior to this config being set will remain written in old format, only when new inode is written is when new format is used.
**Git tree format issue**
We use git tree serialization format in the LocalStore to serialize trees.
This format assumes 20-byte hashes and is not compatible with variable length ObjectId.
In this diff we bypass this issue by not storing trees into local store. This seem ok in terms of correctness, because tree information can always be fetched from mercurial.
However, this seem to impose performance penalty on some work loads (see below).
We can solve this by either introducing new format that supports var length object id(short term), or by getting rid of tree cache and efficiently getting the data directly from mercurial(long term).
**Performance numbers**
Hot file access time is reduced by 50%:
```
$ fsprobe.sh run cat.targets
Before:
lat: 0.2331 ms, qps: 4, dur: 28.697384178s, 123092 files, 217882490 bytes, 1641 errors, rate 7.59 Mb/s
After:
lat: 0.1611 ms, qps: 6, dur: 19.835917353s, 123092 files, 217882490 bytes, 1641 errors, rate 10.98 Mb/s
```
However, we do not see improvement with arc focus, most likely due to bypassing tree serialization, so we will need to figure out that issue.
We can still merge this diff and see if enabling this feature on other workloads like sandcastle is benefitical.
Reviewed By: chadaustin
Differential Revision: D31777929
fbshipit-source-id: fc4b678477d0737c9f242968f0be99ed04f4f58a
Summary:
Hide the cursor and disable the line wrap
Some issues I've still noticed:
- If you resize the window to be smaller it messes things up (you can resize the terminal to be larger and it works fine)
- Sometimes it will decide to continuously print instead of overwriting the old text. Not sure why exactly this happens. It happens more often when I run `buck run` rather than the underlying binary.
- You can still type even though the cursor is disabled
I suspect to address this, I need to rely on the `event` module in the `crossterm` crate.
Reviewed By: fanzeyi
Differential Revision: D32114319
fbshipit-source-id: 037dc1b7e4fbba6e624b00210b4cb07b8d0644c9
Summary:
This diff gets access counts and prints them out in a simple table laid out with tabs.
Still a lot to do, but wanted to get this code out there to get feedback earlier. Some things I need to do:
- format last access time
- sort by last access time
- laying out in a nicer table (I plan on using comfy-table: https://fburl.com/code/k353fp6i)
- updating every `refresh_rate` seconds (I plan on using cross-term: https://fburl.com/code/ahuoqy7t)
Reviewed By: fanzeyi
Differential Revision: D32000885
fbshipit-source-id: 6fc60d13e9519ed45ad6e18628013f9b3ac88bc7
Summary:
The makeImmediateFuture is merely a glorified try/catch, and thus the entirety
of the readdir callback would run immediately, and would block on every size
sequentially. This is inefficient and may cause files to be fetched one at a time.
By reworking the inner logic, we can collect all the sizes beforehand prior to
servicing the callback, allowing for sizes to be fetched concurrently, and for
completing the callback asynchronously.
Reviewed By: chadaustin
Differential Revision: D31782916
fbshipit-source-id: d6315347492e969ffa79037dc2a4f275f4b95a8d
Summary:
Since the CWD wasn't specified when gathering the Buck version, the buck
version returned by `buck --version-fast` could be different from the one in
use in the subdirectory where `buck kill` would be called. This would lead to
Buck failing to stop itself, which would interrupt `eden stop`.
Reviewed By: kmancini
Differential Revision: D32193032
fbshipit-source-id: cbe4983df785b4439645037f9d42084cf745a423
Summary:
The original bug has been fixed for over a year now, let's remove the
workaround.
Reviewed By: kmancini
Differential Revision: D32193033
fbshipit-source-id: 29de7a52e47f8fe070b45b785756b4c93e42151c
Summary: The same as the option that is available in `eden top`.
Reviewed By: fanzeyi
Differential Revision: D31863321
fbshipit-source-id: 3571cc4866fc43e5f43dc5f390cf737b33a7c0b4
Summary:
C++20 changed the rules around char8_t. Cast to char* as
necessary. Also, fix references to some removed std::allocator
associated types.
Reviewed By: fanzeyi, meyering
Differential Revision: D32155762
fbshipit-source-id: dc72e5bc58e8c08612ea1a1eb2568861701287c6
Summary: It is conflicts with CTIME from libc when sys/ioctl.h is imported
Reviewed By: johnny90
Differential Revision: D32017643
fbshipit-source-id: 0346674a7a215524ccaf7cce0e5dfafb5ea9f288
