Summary:
Up to now, Windows had to have its own version of folly::{readFile, writeFile,
writeFileAtomic} as these only operate on `char *` path, which can only
represent ascii paths on Windows. Since the Windows version is slightly
different from folly, this forced the code to either ifdef _WIN32, or use the
folly version pretending that it would be OK. The Windows version was also
behaving slightly differently from folly. For instance, where folly would
return a boolean to indicate success, on Windows we would throw an exception.
To simplify our code, add type safety and unify both, we can implement our own
wrappers on top of either folly or Windows APIs.
We still have some code that uses folly::readFile but these should only be
dealing with filedescriptors. As a following step, we may want to have our own
File class that wraps a file descriptor/HANDLE so we can completely remove all
uses of folly::readFile.
Reviewed By: wez
Differential Revision: D23037325
fbshipit-source-id: 2b9a026f3ee6220ef55097abe649b23e38d9fe91
Summary:
Instead of modifying the existing APIs and marking all callsites as deprecated in one diff, I am going to take the "add and remove" approach, where I will add the deprecated version of methods first, then mark all callsites, and finally remove the existing ones. This provides some backward compatibility without breaking things.
This diff is to add deprecated getSelection APIs to ServiceSelectorCache.
Differential Revision: D22981269
fbshipit-source-id: 6e3025e7f7df6ee7f9e1cba9dc036ca84adbe49a
Summary:
The StringConv.h header contains many functions to convert from Windows paths
to Eden's path (and vice versa) to workaround the fact that Eden's path don't
support wide strings that Windows uses. Let's simply add support for these wide
strings in PathFuncs so we can greatly simplify all the call sites. Instead of
calling "edenToWinName(winstr)", "PathComponent(winstr)" is both more
descriptive and more idiomatic.
To be fair, I'm not entirely a fan of the approach taken in this diff, as this
adds Windows specific code to PathFuncs.h, but I feel that the benefit is too
big to not do that.
Reviewed By: chadaustin
Differential Revision: D23004523
fbshipit-source-id: 3a1507e398a66909773251907db01e06603b91dd
Summary:
Previously, fetch heavy event's cmdline was delimited by '\x00' when logged to Scuba. (for example: `grep--color=auto-rtest.`)
Now we replace \x00 with a space, so command name and args will be separated by space. ( `grep --color=auto -r test .` )
Reviewed By: kmancini
Differential Revision: D22772868
fbshipit-source-id: 4ab42e78c7bc786767eee3413b9586739a12e8ac
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: yfeldblum
Differential Revision: D22631599
fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
Summary:
While there isn't a direct equivalent to the unix RSS and VSZ, the working set
size and the page file usage can be used instead, they roughly have the same
meaning and that's probably good enough.
I've also moved the private bytes into the MemoryStats as it's a memory stat
too.
Reviewed By: chadaustin
Differential Revision: D22081006
fbshipit-source-id: d597d523eee21e7651a14e60b1fd6dc152a61185
Summary:
Currently when we are resolving the full command line for a client pid, we only
read the first 256 bytes of the command.
This means that some commands will be truncated, this has come up in some
of our recently added logs. This ups the buffer size so that we can
hopefully get the full command line.
The longer term solution would be to implement the something fancier mentioned
in the comment in the code copied below, but also has drawbacks as mentioned.
> // Could do something fancy if the entire buffer is filled, but it's better
// if this code does as few syscalls as possible, so just truncate the
// result
Reviewed By: wez
Differential Revision: D22436219
fbshipit-source-id: 80a9aecfe148aa3e333ca480c6a8cb8b9c5c86f2
Summary:
We have seen that eden will unexpectedly fetch data, we want to know why.
This adds the plumbing to interact with edens current logging to be able to
log when eden fetches data from the server and what caused eden to do this
fetch. Later changes will use the classes created here to log the cause of data
fetches.
Reviewed By: chadaustin
Differential Revision: D22051013
fbshipit-source-id: 27d377d7057e66f3e7a304cd7004f8aa44f8ba62
Summary:
Add some functions for validating and producing valid UTF-8 to be used
in an upcoming diff.
Reviewed By: fanzeyi
Differential Revision: D21890510
fbshipit-source-id: b25144a34f1df91c72e8ed776b1ee7c1d68344c8
Summary:
- Added uptime field to DaemonInfo thrift struct
- Created startTime member variable in EdenServer
- Made appropriate refactoring changes to EdenMain and EdenServer
- Changed main.py and util.py to use the new uptime value
Reviewed By: genevievehelsel
Differential Revision: D21471140
fbshipit-source-id: 8868de667dfb95de93e3e71b90c0412fb3825388
Summary:
In glibc, pthread cancellation support adds two atomic CAS operations
to each "cancellation point" syscall (see pthreads(7)). This includes
read() and write(). We can avoid that overhead by disabling pthread
cancellation at the start of the FUSE worker threads.
This saves two CAS operations (~40 ns) in the critical FUSE request
processing loop.
Reviewed By: simpkins
Differential Revision: D21469690
fbshipit-source-id: 7f28a2a8e831006351657981e901dc572c58cf48
Summary:
All of these were simply NOT_IMPLEMENTED on Windows, but the code compiles
and doesn't break any existing tests. The underlying called functions might
have been implemented already, or are NOT_IMPLEMENTED, either way, this reduces
the amount of `#ifdef _WIN32`.
Reviewed By: chadaustin
Differential Revision: D21405622
fbshipit-source-id: bdc2de41d6a57e1c0b532e76eeb2c0c86180d558
Summary:
This brings it closer to folly::writeFile which should help in avoiding ifdef
whenever we want to use it.
Reviewed By: wez
Differential Revision: D21319020
fbshipit-source-id: 80fbf7fba671b18b5ef68375910e1a2a8869f590
Summary:
Google Benchmark is easier to use, has more built-in functionality,
and more accurate default behavior than Folly Benchmark, so switch
EdenFS to use it.;
Reviewed By: simpkins
Differential Revision: D20273672
fbshipit-source-id: c90c49878592620a83d2821ed4bc75c20e599a75
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
Enable the unit tests under eden/fs/utils on Windows.
This does comment out a few tests related to `AbsolutePath` that are broken on
Windows. The AbsolutePath constructor does not enforce that the input path is
actually absolute. Additionally the `canonicalPath()` function ends up doing
weird things and returning paths that start with `/` followed by a drive
letter (e.g., `/C:/foo`). These issues should ideally be addressed in
subsequent diffs.
Reviewed By: xavierd
Differential Revision: D21239886
fbshipit-source-id: ef08d62353ba83b96d9fd79bd4636f4a0f961373
Summary:
Previously, for logging the value of unexpected enum values, we wrote
`static_cast<int>`. Given enumerations can have any integral type
backing them, this was somewhat inaccurate. Instead, introduce an
explicit enumValue function which returns a value of the appropriate
underlying type.
Reviewed By: genevievehelsel
Differential Revision: D20975176
fbshipit-source-id: 0bb5b0d2f68f8fe9d68e4c6a847d59ae0997d0df
Summary:
It turns out that macOS, FreeBSD, and Linux all agree on the values
for dtype. If we assume dtype is just the high nibble of mode_t
shifted right by 12, then the Windows CRT agrees too. So hardcode the
values in eden.thrift and add appropriate static_asserts. This opens
the possibility of simply static_cast'ing the numeric values in
Watchman.
Reviewed By: simpkins
Differential Revision: D20975101
fbshipit-source-id: 354ffcbdf3f1d5f8b1715abf1026eaea429d16cf
Summary:
Add variants of the helpers that return by value rather
than by reference so that we can use those on Windows (which
doesn't embed a `timespec` in a `struct stat`).
Reviewed By: simpkins
Differential Revision: D20562242
fbshipit-source-id: e4769fccb40229765bbf99d0967708cc864db6c3
Summary:
This diff updates the EdenDispatcher to fetch/update the FS info in EdenMount which is backed by InodeTree. This helps to store the FS state in the Inode structure, which are used for source control status and update operations.
Also added a custom formatter definition for RelativePathPiece to make it easy to log relative paths.
Reviewed By: simpkins
Differential Revision: D20480861
fbshipit-source-id: b4bf1da3eeebeaee46a4a187eea9193302182068
Summary: This diff is merging the Windows version of MountTest with the POSIX version. The merged MountTest is build and tested later in this stack with the Merged EdenMount.
Reviewed By: simpkins
Differential Revision: D20480864
fbshipit-source-id: 65e9402f1b03c81166835a6a605053a1bf011ddc
Summary:
This updates the top-level CMakeLists.txt file to compute package version
information, and expose this to C++ code in `eden-config.h`, and to Python
code in a new `eden/config.py` module.
Previously we exposed an `EDEN_VERSION` macro for the C++ code in
`eden-config.h`, but this was not initialized or used anywhere. Now the
top-level CMakeLists.txt file computes appropriate version information and
exposes the package name, version, release, commit ID, and build time in these
configuration files.
The version selection logic in CMakeLists.txt based largely on the code that
wez wrote for watchman in D20636833.
Reviewed By: wez
Differential Revision: D21000164
fbshipit-source-id: db1a1035f1eefec058bbad558d35e113005e454e
Summary:
Build and run the fuse privhelper unit tests, as well as the `drop_privs`
helper program used by some of the integration tests.
Reviewed By: wez
Differential Revision: D21004425
fbshipit-source-id: 650e0729909f4753095e19fba4f01c02d516713b
Summary:
Some of our types were vulnerable to the issue described in
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1008r0.pdf so
make all deleted default constructors explicit.
Reviewed By: simpkins
Differential Revision: D21008976
fbshipit-source-id: 5b21923f25121dabf4bb0ea55f94536fb3532e6b
Summary: Missed cleaning up the other definition in D19956272.
Reviewed By: chadaustin
Differential Revision: D20199689
fbshipit-source-id: 9a00cb7b640f71edc034632e354fb61112a74bc1
Summary: Windows SDK doesn't define dirent. Defining it here for adding Inodes support on Edenfs on Windows.
Reviewed By: simpkins
Differential Revision: D19956272
fbshipit-source-id: 1bdf9a7563c194fe38008741b09668242ffa64ee
Summary: On Windows we will verify both Windows and POSIX path separators, because we have both types of paths.
Reviewed By: simpkins
Differential Revision: D19562772
fbshipit-source-id: a6d8280e3bae4f6cd32b1f379bb59e22dd584211
Summary:
folly::collect's Future is completed immediately when any of its input
future is completed with an exception. This makes forking off subtasks
and then joining with folly::collect dangerous when closures don't
hold pointers or references to parameters, including `this`. Introduce
a collectSafe function, which has the same signature as folly::collect
but unconditionally waits until all futures are completed.
Reviewed By: simpkins
Differential Revision: D19529772
fbshipit-source-id: b3b493fcbb0d9058d2d6e5c8a064dfdae3a43e57
Summary: `folly::cold_detail::ColdClass` was marking things (like `folly::Unexpected`) cold, but at the cost of inhibiting the inliner from doing its job. This is leading to bad codegen, which offsets any small wins we mind get for the `cold` attribute.
Reviewed By: yfeldblum
Differential Revision: D19324159
fbshipit-source-id: 7ed431b6c9d6e963c3bf438c707fa6cf6a38bf9d
Summary: Fix some warnings that only show up in the macOS build.
Reviewed By: fanzeyi
Differential Revision: D19053236
fbshipit-source-id: 81f7187b263e0db6a57582677088519f9b97f1d7
Summary:
This splits `EDEN_BUG()` into three separate version. All three crash in
debug mode builds, but in release builds they behave differently:
- `EDEN_BUG()` throws an exception
- `EDEN_BUG_FUTURE(Type)` returns a `folly::Future<Type>` that has been
fulfilled with an exception.
- `EDEN_BUG_EXCEPTION()` returns a `folly::exception_wrapper`.
The main advantage of this is that this allows the compiler to detect that
`EDEN_BUG()` can never return. Previously `EDEN_BUG()` was used for all 3 of
these different cases, and its behavior depended on whether `toException()`
was ever called. As a result we could not easily get the compiler to identify
code paths where we know at compile time that it will never return.
Reviewed By: chadaustin
Differential Revision: D18652103
fbshipit-source-id: 070107c7520f51b05696905fa243de5f8df15958
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.
The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:
```
void test_log() {
FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.
`nm` before:
```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```
`nm` after:
```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```
Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.
```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```
So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.
In addition to binary size improvements, switching to fmt will potentially
* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).
Reviewed By: simpkins
Differential Revision: D15485589
fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
Summary:
I spent hours debugging a deadlock in my code that was caused by
accidental usage of the deprecated Synchronized::operator->. Avoid
that in the future by building Eden deprecation-clean.
Reviewed By: genevievehelsel
Differential Revision: D18354757
fbshipit-source-id: fe74250635f779bb5010f907d57951132b9edea4
Summary:
The Windows build spews a great many warnings. Address some of them by
enabling the unused-exception-parameter warning on Clang/Linux too.
Reviewed By: yfeldblum
Differential Revision: D18178930
fbshipit-source-id: efecb605b84d4f06c8c8411a23d17904bbdff746
Summary:
Add code on macOS to collect the process's resident set size and
virtual size so the memory usage counter shows up.
Reviewed By: wez
Differential Revision: D18142618
fbshipit-source-id: b42206017ace5309b7bc379f042c1cd6e455f574
Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary: EdenFS recently is unable to import data from API Server on devservers. This is due to the missing of service router support during the build time. This diff fixes this issue.
Reviewed By: wez
Differential Revision: D18042201
fbshipit-source-id: 15868d8e81b9403218e481b3fbf0a7ca1279b023
Summary:
We make use of the KERN_PROCARGS2 MIB data that we can
retrieve via `sysctl`.
If we can't retrieve that data then we fall back to libproc as
we were doing previously. From my testing so far it seems like
the main reason for failure is that the target process is a
protected system process.
Reviewed By: chadaustin
Differential Revision: D17724101
fbshipit-source-id: 8de1a978e6f89612bfe247e0fd540d9078f50746
Summary:
This diff removes the logic that consumes the legacy bind
mount list and mounts them on startup. That functionality has been
replaced with the eden redirect command.
Instead of performing the bind mounts in the server, the server will
now run `eden redirect fixup` to apply that configuration.
This diff also changes the behavior of performBindMounts: previously, if the
bind mount setup failed, we would tear down the entire repo mount. Since we're
now spawning an external process, it is much more likely that something might
fail and result in a bad experience, so we no longer bail out in that case:
we'll continue and leave the bind mounts as-is. The user can then use `eden
doctor` or `eden redirect fixup` to sort things out.
Reviewed By: simpkins
Differential Revision: D17236366
fbshipit-source-id: 8b004551a076216f0e5448942f00b5195ee18803
Summary:
D17623550 deleted `ServiceCacheIf::invalidateSelection()` but did not remove
this override, which broke the build for EdenFS's tests.
Reviewed By: wez
Differential Revision: D17638351
fbshipit-source-id: 200ead0f79a2252dc7ac2318745cd98e520737a4
Summary:
On a machine with one core, it's quite possible for the process to
freeze for a second, resulting in a flaky test.
Reviewed By: wez
Differential Revision: D17579535
fbshipit-source-id: a3e158b5895c06bdf560ebecd849a74e4efa591e
