Summary:
This splits `EDEN_BUG()` into three separate version. All three crash in
debug mode builds, but in release builds they behave differently:
- `EDEN_BUG()` throws an exception
- `EDEN_BUG_FUTURE(Type)` returns a `folly::Future<Type>` that has been
fulfilled with an exception.
- `EDEN_BUG_EXCEPTION()` returns a `folly::exception_wrapper`.
The main advantage of this is that this allows the compiler to detect that
`EDEN_BUG()` can never return. Previously `EDEN_BUG()` was used for all 3 of
these different cases, and its behavior depended on whether `toException()`
was ever called. As a result we could not easily get the compiler to identify
code paths where we know at compile time that it will never return.
Reviewed By: chadaustin
Differential Revision: D18652103
fbshipit-source-id: 070107c7520f51b05696905fa243de5f8df15958
Summary:
Now that the fb-mercurial sources are available in the Eden repository, update
our CMake build files to always build them. This moves the build logic from
the centralized `FBMercurialFeatures.cmake` file into `CMakeLists.txt` files
in the appropriate subdirectories.
Reviewed By: chadaustin
Differential Revision: D18588011
fbshipit-source-id: ded9decde5c2ec766aae0bb0f4f5b021d1044a98
Summary:
Update the CMake build to always build the Rust datapack libraries, even on
Windows. This allows us to completely eliminate the `EDEN_HAVE_RUST_DATAPACK`
checks.
Note that I did leave the `EDEN_HAVE_RUST_DATAPACK` macro in place, as we
still do not build the Rust datapack code on Mac when building with Buck.
Reviewed By: fanzeyi
Differential Revision: D18588008
fbshipit-source-id: 1a4c9ceec5372d0e6a7313d2eb87edabd1e60a96
Summary:
Update Eden's top-level CMakeLists.txt file to build the Rust datapack
libraries. Previously these were built by invoking CMake separately inside th
`eden/scm` subdirectory. Now that the code has been combined into a single
location we can use a single CMake invocation to drive the build of both these
components.
The old code did not build the Rust datapack code on Windows, and this diff
does not change that behavior. I'm not aware of any reason to skip building
this on Windows, so I plan to enable building this code on Windows in a
subsequent diff.
Reviewed By: pkaush
Differential Revision: D18588006
fbshipit-source-id: 20f4f0ea9fef8595a9dd35a21115952b2808c824
Summary: Even if blobs have different hashes, they could have the same contents. For example, if between the two revisions being compared, if a file was changed and then later reverted. In that case, the contents would be the same but the blobs would have different hashes. Currently, `getScmStatusBetweenRevisions()` would report false positives in this case. This is also needed so we do not report false positives in `getScmStatus()` when hit this code path
Reviewed By: simpkins
Differential Revision: D18647086
fbshipit-source-id: 66e12648a24fd7e5612eee5e599a5b81c7c5f2d1
Summary:
Annotate some import statements so autodeps won't complain about them, since
it doesn't know where they come from in the build.
Note that `hg_import_helper.py` isn't used any more by the current code, and
should ideally be killed off soon. However, just annotating these import
statements for now in the short term is the easiest way to make autodeps
happy.
Reviewed By: fanzeyi
Differential Revision: D18652104
fbshipit-source-id: 310bc5c63447e3bf4dfd081d7b3930eed4d0498b
Summary:
The `eden/fs/sqlite` code uses `eden/fs/utils/PathFuncs.h`, so add a proper
dependency on the `eden_utils` library.
Reviewed By: pkaush
Differential Revision: D18652105
fbshipit-source-id: b8dffd93971ab5efb71d7bc3c2dcb62249b3915b
Summary:
Record the durations of each phase of checkout so it can be logged to
scribe in a future diff.
Reviewed By: simpkins
Differential Revision: D18632258
fbshipit-source-id: e447976af22cf25cffd640e79b95da2989b07448
Summary: Adding an experimental option to EdenFS so we can disable blob caching if we need.
Reviewed By: chadaustin
Differential Revision: D18441665
fbshipit-source-id: 56751c0199d6658bfbf7ec3865f746a7279324ab
Summary:
We're about to return more information from checkout, so
introduce a CheckoutResult struct.
Reviewed By: genevievehelsel
Differential Revision: D18630820
fbshipit-source-id: 78b68ef3551ca8c9e9968631de554084c333b4b9
Summary: make `future_getScmStatusV2` into `async_tm_getScmStatusV2` so we can short circut the status call if the request is no longer active.
Reviewed By: simpkins
Differential Revision: D18378212
fbshipit-source-id: b2bca014a37d99f7a92478c3e23bde0b874dd13c
Summary: This reads `enforceParents` from a config instead of always assuming true for `getScmStatusV2()`. This will allow a easy kill switch in case throwing errors from this thrift call causes issues with something that calls hg status
Reviewed By: simpkins
Differential Revision: D18258164
fbshipit-source-id: 1ae421a941c01a678d25d5453c771262b03558d0
Summary:
This diff adds the C++ conversation methods that converts the Rust struct introduced in the previous Diff into EdenFS's `Tree`.
We have to keep the RocksDB write batch in for storing proxy hashes.
Reviewed By: wez
Differential Revision: D18365504
fbshipit-source-id: 2a2cd2cfebd9f0ae4e279db608becc5890c5ccdf
Summary: Create a template functions to convert strings with a wide variety of input types like folly::StringPiece, std::string, folly::fbstring and std::string_view to wide char string.
Reviewed By: wez
Differential Revision: D18071832
fbshipit-source-id: 147af97dff929766db83c9477117ec82da42b03e
Summary: We had linking issues with the Boost while compiling with getdeps. I have moved the tests to use std::filesystem instead.
Reviewed By: wez
Differential Revision: D18195608
fbshipit-source-id: 32213a9b22105869eef78724bb3e583c3fd8e708
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.
The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:
```
void test_log() {
FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.
`nm` before:
```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```
`nm` after:
```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```
Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.
```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```
So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.
In addition to binary size improvements, switching to fmt will potentially
* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).
Reviewed By: simpkins
Differential Revision: D15485589
fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
Summary: make the error message returned in the case of out of date parents during a new status call more user friendly and provide possible remediation instructions
Reviewed By: simpkins
Differential Revision: D18328835
fbshipit-source-id: b214f45bb055d008db8b233ddd2a1843332db838
Summary:
Merge the fb-mercurial code into the Eden repository, under the
`eden/scm` subdirectory.
Reviewed By: quark-zju
Differential Revision: D18445774
fbshipit-source-id: fc3307f9937e0c7e1c8f7d03c5102c4fe5dedb10
Summary:
Allocate and hook up a StructuredLogger at startup if the EdenConfig
has the appropriate values set.
Reviewed By: simpkins
Differential Revision: D18071821
fbshipit-source-id: 3996b6644bbf0c16bb3b9950d57a79d4619a1656
Summary:
The `HgDatapackStore.cpp` file currently depends on code not available on Mac,
so do not attempt to build it there.
Reviewed By: fanzeyi
Differential Revision: D18467948
fbshipit-source-id: cce37bb480426e6d972045321f15c1f2aa4457ee
Summary:
I spent hours debugging a deadlock in my code that was caused by
accidental usage of the deprecated Synchronized::operator->. Avoid
that in the future by building Eden deprecation-clean.
Reviewed By: genevievehelsel
Differential Revision: D18354757
fbshipit-source-id: fe74250635f779bb5010f907d57951132b9edea4
Summary:
Throw errors if a read or write is called on the `Overlay` class after `close()` has been called. Adds IO reference counting to know when it is safe to release the overlay lock after calling `close()`
This takes care to catch errors that could be thrown around overlay IO and makes sure to correctly decrement the IO count in the case of errors.
Reviewed By: chadaustin
Differential Revision: D17957360
fbshipit-source-id: af5e3c3f98c77800914d03cd748fb2f1c59ae264
Summary:
If a checkout's SNAPSHOT file disappeared, the mount error given would
be that a Hash had an invalid size. Instead, throw a file not found
error on mount.
Reviewed By: genevievehelsel
Differential Revision: D18381074
fbshipit-source-id: 35282e0990189d4084a2a64330a5733561cf88c2
Summary:
Looking at a log, it wasn't immediately obvious what might have passed
an invalid hash into the Hash constructor. Improve the error message
to make the cause clearer.
Reviewed By: genevievehelsel
Differential Revision: D18380916
fbshipit-source-id: 620b8fa902a87496b87a5aa0ff304e6991585864
Summary: changes `overlay_` to be stored as a `std::shared_ptr` in order to safely create and pass `std::weak_ptr`s of the overlay to OverlayFiles that are being created.
Reviewed By: chadaustin
Differential Revision: D18302732
fbshipit-source-id: 495b613914771714ba8a8d1beb1c7a0cd72b33d4
Summary:
This moves error handling with `OverlayFile` closer to the source of the error with the use of `folly::Expected`.
This is for future IOReference counting but also lets us transform all of these generic `std::system_error`s into InodeErrors (which is a subclass of `std::system_error` that can include inode path in the error message).
Another option for error handling would be to throw folly::checkUnixError inside the `OverlayFile` instead of returning a `folly::Expected`, but I am always hesitant to assume that the user of this class wants the error to be thrown, since they may want to do some other work (cleanup etc) before throwing an error.
Reviewed By: chadaustin
Differential Revision: D18215242
fbshipit-source-id: 5e38fdc7e6efe7d646dd3895932c9b2f26674b36
Summary: This file wrapper is returned instead of a `folly::File` when talking with the `Overlay`. This is basework for supporting closing the `Overlay` class even if it is still in use so file modifications on files given from the `Overlay` can be denied if the `Overlay` is closed.
Reviewed By: chadaustin
Differential Revision: D17973664
fbshipit-source-id: e4b9778b3a830b6310fda4a9e66b459530b9fe17
Summary:
Fix the `getStatInfo()` code to avoid walking all loaded inodes. That
behavior was okay early on when we were first developing EdenFS and this
API wasn't used in many places. However, `eden rage` and `eden stats` both
exercise this code path, and it seems pretty bad to walk all loaded inodes in
order to return some stats.
I removed the counts about number of materialized inodes for now, since I
don't think we're really using this metric much.
Reviewed By: chadaustin
Differential Revision: D17578306
fbshipit-source-id: 55ab0209745869b160e91167d6cff7d95f39a95a
Summary:
We noticed that some operations were surprisingly slow and
the hypothesis was that caching in the kext wasn't working well.
I traced into this with a simple repeated `realpath` of the same file
while running the server with debug logging turned up. That showed
that we were getting two requests from the kernel for each inode in
the path resolution for each invocation of the realpath tool.
I switched to looking at the kext code and found this section to be
a potential issue:
https://github.com/osxfuse/kext/blob/support/osxfuse-3/osxfuse/fuse_internal.h#L711
```
/* XXX: truncation; user space sends us a 64-bit tv_sec */ \
VTOFUD(vp)->attr_valid.tv_sec = (time_t)struct_name ## _get_attr_valid(fuse_out); \
VTOFUD(vp)->attr_valid.tv_nsec = struct_name ## _get_attr_valid_nsec(fuse_out); \
nanouptime(&uptsp_ ## __funct__); \
```
Switching our default TTL to be the maximum possible value that fits
in a signed 32-bit time_t cuts down on the requests from the kernel to
a single lookup for the symlink:
```
V1105 15:56:28.900839 28132386 FuseChannel.cpp:1141] fuse request opcode=1 FUSE_LOOKUP unique=2 len=47 nodeid=14910747 uid=2048904527 gid=1876110778 pid=24546
V1105 15:56:28.900885 28132386 FuseChannel.cpp:1437] FUSE_LOOKUP parent=14910747 name=README
V1105 15:56:28.901111 28143988 FuseChannel.cpp:421] sendRawReply: unique=2 header->len=160 wrote=160
```
and no requests at all for a plain file in the repo.
Reviewed By: simpkins
Differential Revision: D18340767
fbshipit-source-id: caebf051a543c54f7a03852fd2e0abab68448ded
Summary:
This diff makes EdenFS to be able to import manifest from Rust code.
We are only able to import raw manifest data from Rust for now. This code still
send the imported data to the ctreemanifest for parsing. Follow up diff will
fix this.
Reviewed By: chadaustin
Differential Revision: D18097701
fbshipit-source-id: b821a1d595f43a19d9ce9cd6d5ff5174d49e34e6
Summary:
Background: https://fb.workplace.com/groups/sourcecontrolteam/permalink/2440123159442349/?comment_id=2443624539092211
This may or may not be the root cause but this is definitely gonna be causing issues.
Between D17468473 and D17866320, turning on this option will make EdenFS to use the old Rust code that does not strip copyrev data.
Reviewed By: wez
Differential Revision: D18250237
fbshipit-source-id: b3816bd08b8dc3b2a90931829c368a58731c7843
Summary: Calling a command that is not implemented will terminate the edenfs. It was ok for testing but now we have it running on Sandcastle, so will throw Eden error instead.
Reviewed By: chadaustin
Differential Revision: D18145495
fbshipit-source-id: 17e57f426d3cf80c93c2112741a027db8cbc6b8d
Summary:
Testpilot is crashing to parse the JSON with the test names. I didn't debug it any further but looked like didn't like the names and renaming them worked.
Traceback (most recent call last):
File "testinfra/testpilot/testpilot.py", line 616, in pre_main
File "testinfra/testpilot/testpilot.py", line 111, in main
File "testinfra/testpilot/testpilot.py", line 262, in main_impl
File "testinfra/testpilot/testpilot.py", line 259, in main_impl
File "testinfra/testpilot/integration/fbcode/finder.py", line 478, in get_tests
KeyError: 'eden-StateDBNodeTest.fetchDirectoryEntries-getdeps-windows-none-6 - main'
Reviewed By: wez
Differential Revision: D18208976
fbshipit-source-id: 1785aa4b1c42041578350b9c2a7e928059eeb648
Summary:
Add a new thrift API for computing the difference between the working
directory and a given source control commit.
This has the following differences from the old getScmStatus() commit:
- The parameters are accepted in a GetScmStatusParams structure now.
This makes it easier for the server-side C++ implementation to tell which
parameters have actually been specified by the caller. This will make it
easier to extend this API in the future without having to replace it with a
new function call again.
- The return value is a GetScmStatusResult, which includes both the ScmStatus
and the EdenFS version number. This will allow code like `hg status` to get
both the status results and the EdenFS version in a single call, without
needing to make multiple separate thrift calls.
- This new call will return an error if the caller requests the status against
a commit that disagrees with EdenFS's view of the current commit. Because
the individual `hg` command line processes do not perform any
synchronization of their own when reading the working directory parent,
they can often call EdenFS with stale parent information, or while a
checkout is currently in progress. This new behavior will reject the
request with an error, rather than having EdenFS perform a potentially very
expensive status computation when the results probably aren't actually
useful to the caller anyway.
Reviewed By: chadaustin
Differential Revision: D15110218
fbshipit-source-id: ebc2f74dafc090d4fd245de8e4d62e2b086500dd
Summary: Adds a non-optional EdenErrorType struct for EdenError, this can be used in case of special error case handling of errors without error message parsing. Currently this is just passed along and not consumed anywhere in the client, but later in the stack is used for specific retry of checkout if "CHECKOUT_IN_PROGRESS" on the consuming side.
Reviewed By: chadaustin
Differential Revision: D18139917
fbshipit-source-id: b3f2ec4c480fc5246ff2f46d09c436021bad8b61
Summary:
The Windows build spews a great many warnings. Address some of them by
enabling the unused-exception-parameter warning on Clang/Linux too.
Reviewed By: yfeldblum
Differential Revision: D18178930
fbshipit-source-id: efecb605b84d4f06c8c8411a23d17904bbdff746
Summary:
Add code on macOS to collect the process's resident set size and
virtual size so the memory usage counter shows up.
Reviewed By: wez
Differential Revision: D18142618
fbshipit-source-id: b42206017ace5309b7bc379f042c1cd6e455f574
Summary:
add an enforceParent flag to signal if the diff call should only be run against its parent commit. setup work for `getScmStatusV2`
The only callsite for this version of diff() is in the testcase and in `async_tm_getScmStatus`, in which both cases they set `enforceCurrentParent = false`.
Reviewed By: chadaustin
Differential Revision: D17911708
fbshipit-source-id: 361c7586e1452544172cf9e79a0cdf0888154096
Summary:
Introduce a framework that allows recording structured log events
which are encoded as JSON and piped to a configured command line
program.
Reviewed By: pkaush
Differential Revision: D18025183
fbshipit-source-id: ab6b4d510a905a30252f2cff85d107a0d32d149e
Summary: Having the header files in CMake, adds them in Visual Studio project and makes it easier to work in Visual Studio.
Reviewed By: chadaustin
Differential Revision: D18145496
fbshipit-source-id: f6607e0cbf2e6a8d9fc3ad9f8bed90458aec355e
Summary: This diff will create an instance of CurrentState in EdenMount and integrate it with the Prjfs notifications.
Reviewed By: chadaustin
Differential Revision: D18068335
fbshipit-source-id: 5c78949c8b0ad316d2b014e808297f29092ae703
Summary: This diff adds the notifications callbacks from Prjfs. These will be used in the later diffs in the stack.
Reviewed By: chadaustin
Differential Revision: D18068035
fbshipit-source-id: a8e679ef32ae39251e6c1e926feb9cd1b71258e0
Summary: This diff creates a Guid as a unique mount id. This mount id will be used to uniquely identify our repo in the Regdb.
Reviewed By: chadaustin
Differential Revision: D18055798
fbshipit-source-id: 01099d7fdf15c325498e83c8b63637dd1d85d028
Summary: The is the top level interface which will receive all the FS changes and will store them in the RegDB for later use.
Reviewed By: chadaustin
Differential Revision: D17998282
fbshipit-source-id: 7570edcfe0f6bf9164bfa0688665fb7253a21bc6
