Summary: make `future_getScmStatusV2` into `async_tm_getScmStatusV2` so we can short circut the status call if the request is no longer active.
Reviewed By: simpkins
Differential Revision: D18378212
fbshipit-source-id: b2bca014a37d99f7a92478c3e23bde0b874dd13c
Summary: This reads `enforceParents` from a config instead of always assuming true for `getScmStatusV2()`. This will allow a easy kill switch in case throwing errors from this thrift call causes issues with something that calls hg status
Reviewed By: simpkins
Differential Revision: D18258164
fbshipit-source-id: 1ae421a941c01a678d25d5453c771262b03558d0
Summary:
This diff adds the C++ conversation methods that converts the Rust struct introduced in the previous Diff into EdenFS's `Tree`.
We have to keep the RocksDB write batch in for storing proxy hashes.
Reviewed By: wez
Differential Revision: D18365504
fbshipit-source-id: 2a2cd2cfebd9f0ae4e279db608becc5890c5ccdf
Summary: Create a template functions to convert strings with a wide variety of input types like folly::StringPiece, std::string, folly::fbstring and std::string_view to wide char string.
Reviewed By: wez
Differential Revision: D18071832
fbshipit-source-id: 147af97dff929766db83c9477117ec82da42b03e
Summary: We had linking issues with the Boost while compiling with getdeps. I have moved the tests to use std::filesystem instead.
Reviewed By: wez
Differential Revision: D18195608
fbshipit-source-id: 32213a9b22105869eef78724bb3e583c3fd8e708
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.
The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:
```
void test_log() {
FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.
`nm` before:
```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```
`nm` after:
```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```
Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.
```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```
So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.
In addition to binary size improvements, switching to fmt will potentially
* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).
Reviewed By: simpkins
Differential Revision: D15485589
fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
Summary: make the error message returned in the case of out of date parents during a new status call more user friendly and provide possible remediation instructions
Reviewed By: simpkins
Differential Revision: D18328835
fbshipit-source-id: b214f45bb055d008db8b233ddd2a1843332db838
Summary:
Merge the fb-mercurial code into the Eden repository, under the
`eden/scm` subdirectory.
Reviewed By: quark-zju
Differential Revision: D18445774
fbshipit-source-id: fc3307f9937e0c7e1c8f7d03c5102c4fe5dedb10
Summary:
Allocate and hook up a StructuredLogger at startup if the EdenConfig
has the appropriate values set.
Reviewed By: simpkins
Differential Revision: D18071821
fbshipit-source-id: 3996b6644bbf0c16bb3b9950d57a79d4619a1656
Summary:
The `HgDatapackStore.cpp` file currently depends on code not available on Mac,
so do not attempt to build it there.
Reviewed By: fanzeyi
Differential Revision: D18467948
fbshipit-source-id: cce37bb480426e6d972045321f15c1f2aa4457ee
Summary:
I spent hours debugging a deadlock in my code that was caused by
accidental usage of the deprecated Synchronized::operator->. Avoid
that in the future by building Eden deprecation-clean.
Reviewed By: genevievehelsel
Differential Revision: D18354757
fbshipit-source-id: fe74250635f779bb5010f907d57951132b9edea4
Summary:
Throw errors if a read or write is called on the `Overlay` class after `close()` has been called. Adds IO reference counting to know when it is safe to release the overlay lock after calling `close()`
This takes care to catch errors that could be thrown around overlay IO and makes sure to correctly decrement the IO count in the case of errors.
Reviewed By: chadaustin
Differential Revision: D17957360
fbshipit-source-id: af5e3c3f98c77800914d03cd748fb2f1c59ae264
Summary:
If a checkout's SNAPSHOT file disappeared, the mount error given would
be that a Hash had an invalid size. Instead, throw a file not found
error on mount.
Reviewed By: genevievehelsel
Differential Revision: D18381074
fbshipit-source-id: 35282e0990189d4084a2a64330a5733561cf88c2
Summary:
Looking at a log, it wasn't immediately obvious what might have passed
an invalid hash into the Hash constructor. Improve the error message
to make the cause clearer.
Reviewed By: genevievehelsel
Differential Revision: D18380916
fbshipit-source-id: 620b8fa902a87496b87a5aa0ff304e6991585864
Summary: changes `overlay_` to be stored as a `std::shared_ptr` in order to safely create and pass `std::weak_ptr`s of the overlay to OverlayFiles that are being created.
Reviewed By: chadaustin
Differential Revision: D18302732
fbshipit-source-id: 495b613914771714ba8a8d1beb1c7a0cd72b33d4
Summary:
This moves error handling with `OverlayFile` closer to the source of the error with the use of `folly::Expected`.
This is for future IOReference counting but also lets us transform all of these generic `std::system_error`s into InodeErrors (which is a subclass of `std::system_error` that can include inode path in the error message).
Another option for error handling would be to throw folly::checkUnixError inside the `OverlayFile` instead of returning a `folly::Expected`, but I am always hesitant to assume that the user of this class wants the error to be thrown, since they may want to do some other work (cleanup etc) before throwing an error.
Reviewed By: chadaustin
Differential Revision: D18215242
fbshipit-source-id: 5e38fdc7e6efe7d646dd3895932c9b2f26674b36
Summary: This file wrapper is returned instead of a `folly::File` when talking with the `Overlay`. This is basework for supporting closing the `Overlay` class even if it is still in use so file modifications on files given from the `Overlay` can be denied if the `Overlay` is closed.
Reviewed By: chadaustin
Differential Revision: D17973664
fbshipit-source-id: e4b9778b3a830b6310fda4a9e66b459530b9fe17
Summary:
Fix the `getStatInfo()` code to avoid walking all loaded inodes. That
behavior was okay early on when we were first developing EdenFS and this
API wasn't used in many places. However, `eden rage` and `eden stats` both
exercise this code path, and it seems pretty bad to walk all loaded inodes in
order to return some stats.
I removed the counts about number of materialized inodes for now, since I
don't think we're really using this metric much.
Reviewed By: chadaustin
Differential Revision: D17578306
fbshipit-source-id: 55ab0209745869b160e91167d6cff7d95f39a95a
Summary:
We noticed that some operations were surprisingly slow and
the hypothesis was that caching in the kext wasn't working well.
I traced into this with a simple repeated `realpath` of the same file
while running the server with debug logging turned up. That showed
that we were getting two requests from the kernel for each inode in
the path resolution for each invocation of the realpath tool.
I switched to looking at the kext code and found this section to be
a potential issue:
https://github.com/osxfuse/kext/blob/support/osxfuse-3/osxfuse/fuse_internal.h#L711
```
/* XXX: truncation; user space sends us a 64-bit tv_sec */ \
VTOFUD(vp)->attr_valid.tv_sec = (time_t)struct_name ## _get_attr_valid(fuse_out); \
VTOFUD(vp)->attr_valid.tv_nsec = struct_name ## _get_attr_valid_nsec(fuse_out); \
nanouptime(&uptsp_ ## __funct__); \
```
Switching our default TTL to be the maximum possible value that fits
in a signed 32-bit time_t cuts down on the requests from the kernel to
a single lookup for the symlink:
```
V1105 15:56:28.900839 28132386 FuseChannel.cpp:1141] fuse request opcode=1 FUSE_LOOKUP unique=2 len=47 nodeid=14910747 uid=2048904527 gid=1876110778 pid=24546
V1105 15:56:28.900885 28132386 FuseChannel.cpp:1437] FUSE_LOOKUP parent=14910747 name=README
V1105 15:56:28.901111 28143988 FuseChannel.cpp:421] sendRawReply: unique=2 header->len=160 wrote=160
```
and no requests at all for a plain file in the repo.
Reviewed By: simpkins
Differential Revision: D18340767
fbshipit-source-id: caebf051a543c54f7a03852fd2e0abab68448ded
Summary:
This diff makes EdenFS to be able to import manifest from Rust code.
We are only able to import raw manifest data from Rust for now. This code still
send the imported data to the ctreemanifest for parsing. Follow up diff will
fix this.
Reviewed By: chadaustin
Differential Revision: D18097701
fbshipit-source-id: b821a1d595f43a19d9ce9cd6d5ff5174d49e34e6
Summary:
Background: https://fb.workplace.com/groups/sourcecontrolteam/permalink/2440123159442349/?comment_id=2443624539092211
This may or may not be the root cause but this is definitely gonna be causing issues.
Between D17468473 and D17866320, turning on this option will make EdenFS to use the old Rust code that does not strip copyrev data.
Reviewed By: wez
Differential Revision: D18250237
fbshipit-source-id: b3816bd08b8dc3b2a90931829c368a58731c7843
Summary: Calling a command that is not implemented will terminate the edenfs. It was ok for testing but now we have it running on Sandcastle, so will throw Eden error instead.
Reviewed By: chadaustin
Differential Revision: D18145495
fbshipit-source-id: 17e57f426d3cf80c93c2112741a027db8cbc6b8d
Summary:
Testpilot is crashing to parse the JSON with the test names. I didn't debug it any further but looked like didn't like the names and renaming them worked.
Traceback (most recent call last):
File "testinfra/testpilot/testpilot.py", line 616, in pre_main
File "testinfra/testpilot/testpilot.py", line 111, in main
File "testinfra/testpilot/testpilot.py", line 262, in main_impl
File "testinfra/testpilot/testpilot.py", line 259, in main_impl
File "testinfra/testpilot/integration/fbcode/finder.py", line 478, in get_tests
KeyError: 'eden-StateDBNodeTest.fetchDirectoryEntries-getdeps-windows-none-6 - main'
Reviewed By: wez
Differential Revision: D18208976
fbshipit-source-id: 1785aa4b1c42041578350b9c2a7e928059eeb648
Summary:
Add a new thrift API for computing the difference between the working
directory and a given source control commit.
This has the following differences from the old getScmStatus() commit:
- The parameters are accepted in a GetScmStatusParams structure now.
This makes it easier for the server-side C++ implementation to tell which
parameters have actually been specified by the caller. This will make it
easier to extend this API in the future without having to replace it with a
new function call again.
- The return value is a GetScmStatusResult, which includes both the ScmStatus
and the EdenFS version number. This will allow code like `hg status` to get
both the status results and the EdenFS version in a single call, without
needing to make multiple separate thrift calls.
- This new call will return an error if the caller requests the status against
a commit that disagrees with EdenFS's view of the current commit. Because
the individual `hg` command line processes do not perform any
synchronization of their own when reading the working directory parent,
they can often call EdenFS with stale parent information, or while a
checkout is currently in progress. This new behavior will reject the
request with an error, rather than having EdenFS perform a potentially very
expensive status computation when the results probably aren't actually
useful to the caller anyway.
Reviewed By: chadaustin
Differential Revision: D15110218
fbshipit-source-id: ebc2f74dafc090d4fd245de8e4d62e2b086500dd
Summary: Adds a non-optional EdenErrorType struct for EdenError, this can be used in case of special error case handling of errors without error message parsing. Currently this is just passed along and not consumed anywhere in the client, but later in the stack is used for specific retry of checkout if "CHECKOUT_IN_PROGRESS" on the consuming side.
Reviewed By: chadaustin
Differential Revision: D18139917
fbshipit-source-id: b3f2ec4c480fc5246ff2f46d09c436021bad8b61
Summary:
The Windows build spews a great many warnings. Address some of them by
enabling the unused-exception-parameter warning on Clang/Linux too.
Reviewed By: yfeldblum
Differential Revision: D18178930
fbshipit-source-id: efecb605b84d4f06c8c8411a23d17904bbdff746
Summary:
Add code on macOS to collect the process's resident set size and
virtual size so the memory usage counter shows up.
Reviewed By: wez
Differential Revision: D18142618
fbshipit-source-id: b42206017ace5309b7bc379f042c1cd6e455f574
Summary:
add an enforceParent flag to signal if the diff call should only be run against its parent commit. setup work for `getScmStatusV2`
The only callsite for this version of diff() is in the testcase and in `async_tm_getScmStatus`, in which both cases they set `enforceCurrentParent = false`.
Reviewed By: chadaustin
Differential Revision: D17911708
fbshipit-source-id: 361c7586e1452544172cf9e79a0cdf0888154096
Summary:
Introduce a framework that allows recording structured log events
which are encoded as JSON and piped to a configured command line
program.
Reviewed By: pkaush
Differential Revision: D18025183
fbshipit-source-id: ab6b4d510a905a30252f2cff85d107a0d32d149e
Summary: Having the header files in CMake, adds them in Visual Studio project and makes it easier to work in Visual Studio.
Reviewed By: chadaustin
Differential Revision: D18145496
fbshipit-source-id: f6607e0cbf2e6a8d9fc3ad9f8bed90458aec355e
Summary: This diff will create an instance of CurrentState in EdenMount and integrate it with the Prjfs notifications.
Reviewed By: chadaustin
Differential Revision: D18068335
fbshipit-source-id: 5c78949c8b0ad316d2b014e808297f29092ae703
Summary: This diff adds the notifications callbacks from Prjfs. These will be used in the later diffs in the stack.
Reviewed By: chadaustin
Differential Revision: D18068035
fbshipit-source-id: a8e679ef32ae39251e6c1e926feb9cd1b71258e0
Summary: This diff creates a Guid as a unique mount id. This mount id will be used to uniquely identify our repo in the Regdb.
Reviewed By: chadaustin
Differential Revision: D18055798
fbshipit-source-id: 01099d7fdf15c325498e83c8b63637dd1d85d028
Summary: The is the top level interface which will receive all the FS changes and will store them in the RegDB for later use.
Reviewed By: chadaustin
Differential Revision: D17998282
fbshipit-source-id: 7570edcfe0f6bf9164bfa0688665fb7253a21bc6
Summary: StateDirectoryEntry will contain the current state of a file or folder in the prjfs cache and will be totally in memory. In the next few diffs we will bring in the code that will load and store these in the db.
Reviewed By: chadaustin
Differential Revision: D17985967
fbshipit-source-id: 260dca369a3acce10385b04fdb5ff111cfc998be
Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary:
This saves the version number (like 20191014-154333) in the EdenServer. This will be returned to the user in a getScmStatusV2 call later up the stack. This is useful for being able to warn the user if EdenFS is running on an old or known-bad version.
I ended up splitting up `getEdenfsVersion()` and `getEdenfsBuildName()` because I don't think we need to store "edenfs " with every version number in the server / return this from the thrift call of `getScmStatusV2()`. Not sure of the use case of having "edenfs " prepended to the version number returned from the current `getEdenfsBuildName()` so didn't want to change that, but if it seems fine we could remove that prepended edenfs (or if it actually makes sense to return the string with that prepended from the ScmStatus call).
Reviewed By: chadaustin
Differential Revision: D17919528
fbshipit-source-id: 5abdf7851371c4614da862657350a8cc263eb014
Summary: We should handle errors from Rust code gracefully so it does not abort the mounting process.
Reviewed By: chadaustin
Differential Revision: D18096335
fbshipit-source-id: 17cec6f619ea2cc8c1719b0cf97287c815ff6f0a
Summary: EdenFS recently is unable to import data from API Server on devservers. This is due to the missing of service router support during the build time. This diff fixes this issue.
Reviewed By: wez
Differential Revision: D18042201
fbshipit-source-id: 15868d8e81b9403218e481b3fbf0a7ca1279b023
Summary: This diff switches EdenFS from using the `revisionstore` crate into `backingstore`
Reviewed By: chadaustin
Differential Revision: D17866320
fbshipit-source-id: f6de1fe568ae893e4c2555039df6edfb0a450885
Summary: Introduce a new ScribeLogger class that spawns and maintains a process. Log messages are newline-delimited and written to the process's stdin. If the process stops responding or responds too slowly, log messages are dropped.
Reviewed By: pkaush
Differential Revision: D17777215
fbshipit-source-id: c998d10c73fc103122d69ae19c5d84f58b7939d2
Summary:
krallin noted that we were not collecting LFS stats from the import helper.
This diff should get us to run the import helper with those stats enabled, although
we don't yet know how useful those will be in practice.
We originally defaulted to `hg.real` to bypass some startup overhead and make our tests
run faster. We've evolved more control over this in the test harness over time, so I
think we're probably fine to make the runtime non-test default just be the normal hg
entrypoint.
Reviewed By: chadaustin
Differential Revision: D18039164
fbshipit-source-id: 7593cdca9afe30fa2fc928b84ba552dbecfed388