Summary:
We noticed that some operations were surprisingly slow and
the hypothesis was that caching in the kext wasn't working well.
I traced into this with a simple repeated `realpath` of the same file
while running the server with debug logging turned up. That showed
that we were getting two requests from the kernel for each inode in
the path resolution for each invocation of the realpath tool.
I switched to looking at the kext code and found this section to be
a potential issue:
https://github.com/osxfuse/kext/blob/support/osxfuse-3/osxfuse/fuse_internal.h#L711
```
/* XXX: truncation; user space sends us a 64-bit tv_sec */ \
VTOFUD(vp)->attr_valid.tv_sec = (time_t)struct_name ## _get_attr_valid(fuse_out); \
VTOFUD(vp)->attr_valid.tv_nsec = struct_name ## _get_attr_valid_nsec(fuse_out); \
nanouptime(&uptsp_ ## __funct__); \
```
Switching our default TTL to be the maximum possible value that fits
in a signed 32-bit time_t cuts down on the requests from the kernel to
a single lookup for the symlink:
```
V1105 15:56:28.900839 28132386 FuseChannel.cpp:1141] fuse request opcode=1 FUSE_LOOKUP unique=2 len=47 nodeid=14910747 uid=2048904527 gid=1876110778 pid=24546
V1105 15:56:28.900885 28132386 FuseChannel.cpp:1437] FUSE_LOOKUP parent=14910747 name=README
V1105 15:56:28.901111 28143988 FuseChannel.cpp:421] sendRawReply: unique=2 header->len=160 wrote=160
```
and no requests at all for a plain file in the repo.
Reviewed By: simpkins
Differential Revision: D18340767
fbshipit-source-id: caebf051a543c54f7a03852fd2e0abab68448ded
Summary:
This diff makes EdenFS to be able to import manifest from Rust code.
We are only able to import raw manifest data from Rust for now. This code still
send the imported data to the ctreemanifest for parsing. Follow up diff will
fix this.
Reviewed By: chadaustin
Differential Revision: D18097701
fbshipit-source-id: b821a1d595f43a19d9ce9cd6d5ff5174d49e34e6
Summary:
Background: https://fb.workplace.com/groups/sourcecontrolteam/permalink/2440123159442349/?comment_id=2443624539092211
This may or may not be the root cause but this is definitely gonna be causing issues.
Between D17468473 and D17866320, turning on this option will make EdenFS to use the old Rust code that does not strip copyrev data.
Reviewed By: wez
Differential Revision: D18250237
fbshipit-source-id: b3816bd08b8dc3b2a90931829c368a58731c7843
Summary: Calling a command that is not implemented will terminate the edenfs. It was ok for testing but now we have it running on Sandcastle, so will throw Eden error instead.
Reviewed By: chadaustin
Differential Revision: D18145495
fbshipit-source-id: 17e57f426d3cf80c93c2112741a027db8cbc6b8d
Summary:
Testpilot is crashing to parse the JSON with the test names. I didn't debug it any further but looked like didn't like the names and renaming them worked.
Traceback (most recent call last):
File "testinfra/testpilot/testpilot.py", line 616, in pre_main
File "testinfra/testpilot/testpilot.py", line 111, in main
File "testinfra/testpilot/testpilot.py", line 262, in main_impl
File "testinfra/testpilot/testpilot.py", line 259, in main_impl
File "testinfra/testpilot/integration/fbcode/finder.py", line 478, in get_tests
KeyError: 'eden-StateDBNodeTest.fetchDirectoryEntries-getdeps-windows-none-6 - main'
Reviewed By: wez
Differential Revision: D18208976
fbshipit-source-id: 1785aa4b1c42041578350b9c2a7e928059eeb648
Summary:
Add a new thrift API for computing the difference between the working
directory and a given source control commit.
This has the following differences from the old getScmStatus() commit:
- The parameters are accepted in a GetScmStatusParams structure now.
This makes it easier for the server-side C++ implementation to tell which
parameters have actually been specified by the caller. This will make it
easier to extend this API in the future without having to replace it with a
new function call again.
- The return value is a GetScmStatusResult, which includes both the ScmStatus
and the EdenFS version number. This will allow code like `hg status` to get
both the status results and the EdenFS version in a single call, without
needing to make multiple separate thrift calls.
- This new call will return an error if the caller requests the status against
a commit that disagrees with EdenFS's view of the current commit. Because
the individual `hg` command line processes do not perform any
synchronization of their own when reading the working directory parent,
they can often call EdenFS with stale parent information, or while a
checkout is currently in progress. This new behavior will reject the
request with an error, rather than having EdenFS perform a potentially very
expensive status computation when the results probably aren't actually
useful to the caller anyway.
Reviewed By: chadaustin
Differential Revision: D15110218
fbshipit-source-id: ebc2f74dafc090d4fd245de8e4d62e2b086500dd
Summary: Adds a non-optional EdenErrorType struct for EdenError, this can be used in case of special error case handling of errors without error message parsing. Currently this is just passed along and not consumed anywhere in the client, but later in the stack is used for specific retry of checkout if "CHECKOUT_IN_PROGRESS" on the consuming side.
Reviewed By: chadaustin
Differential Revision: D18139917
fbshipit-source-id: b3f2ec4c480fc5246ff2f46d09c436021bad8b61
Summary:
The Windows build spews a great many warnings. Address some of them by
enabling the unused-exception-parameter warning on Clang/Linux too.
Reviewed By: yfeldblum
Differential Revision: D18178930
fbshipit-source-id: efecb605b84d4f06c8c8411a23d17904bbdff746
Summary:
Add code on macOS to collect the process's resident set size and
virtual size so the memory usage counter shows up.
Reviewed By: wez
Differential Revision: D18142618
fbshipit-source-id: b42206017ace5309b7bc379f042c1cd6e455f574
Summary:
add an enforceParent flag to signal if the diff call should only be run against its parent commit. setup work for `getScmStatusV2`
The only callsite for this version of diff() is in the testcase and in `async_tm_getScmStatus`, in which both cases they set `enforceCurrentParent = false`.
Reviewed By: chadaustin
Differential Revision: D17911708
fbshipit-source-id: 361c7586e1452544172cf9e79a0cdf0888154096
Summary:
Introduce a framework that allows recording structured log events
which are encoded as JSON and piped to a configured command line
program.
Reviewed By: pkaush
Differential Revision: D18025183
fbshipit-source-id: ab6b4d510a905a30252f2cff85d107a0d32d149e
Summary: Having the header files in CMake, adds them in Visual Studio project and makes it easier to work in Visual Studio.
Reviewed By: chadaustin
Differential Revision: D18145496
fbshipit-source-id: f6607e0cbf2e6a8d9fc3ad9f8bed90458aec355e
Summary: This diff will create an instance of CurrentState in EdenMount and integrate it with the Prjfs notifications.
Reviewed By: chadaustin
Differential Revision: D18068335
fbshipit-source-id: 5c78949c8b0ad316d2b014e808297f29092ae703
Summary: This diff adds the notifications callbacks from Prjfs. These will be used in the later diffs in the stack.
Reviewed By: chadaustin
Differential Revision: D18068035
fbshipit-source-id: a8e679ef32ae39251e6c1e926feb9cd1b71258e0
Summary: This diff creates a Guid as a unique mount id. This mount id will be used to uniquely identify our repo in the Regdb.
Reviewed By: chadaustin
Differential Revision: D18055798
fbshipit-source-id: 01099d7fdf15c325498e83c8b63637dd1d85d028
Summary: The is the top level interface which will receive all the FS changes and will store them in the RegDB for later use.
Reviewed By: chadaustin
Differential Revision: D17998282
fbshipit-source-id: 7570edcfe0f6bf9164bfa0688665fb7253a21bc6
Summary: StateDirectoryEntry will contain the current state of a file or folder in the prjfs cache and will be totally in memory. In the next few diffs we will bring in the code that will load and store these in the db.
Reviewed By: chadaustin
Differential Revision: D17985967
fbshipit-source-id: 260dca369a3acce10385b04fdb5ff111cfc998be
Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary:
This saves the version number (like 20191014-154333) in the EdenServer. This will be returned to the user in a getScmStatusV2 call later up the stack. This is useful for being able to warn the user if EdenFS is running on an old or known-bad version.
I ended up splitting up `getEdenfsVersion()` and `getEdenfsBuildName()` because I don't think we need to store "edenfs " with every version number in the server / return this from the thrift call of `getScmStatusV2()`. Not sure of the use case of having "edenfs " prepended to the version number returned from the current `getEdenfsBuildName()` so didn't want to change that, but if it seems fine we could remove that prepended edenfs (or if it actually makes sense to return the string with that prepended from the ScmStatus call).
Reviewed By: chadaustin
Differential Revision: D17919528
fbshipit-source-id: 5abdf7851371c4614da862657350a8cc263eb014
Summary: We should handle errors from Rust code gracefully so it does not abort the mounting process.
Reviewed By: chadaustin
Differential Revision: D18096335
fbshipit-source-id: 17cec6f619ea2cc8c1719b0cf97287c815ff6f0a
Summary: EdenFS recently is unable to import data from API Server on devservers. This is due to the missing of service router support during the build time. This diff fixes this issue.
Reviewed By: wez
Differential Revision: D18042201
fbshipit-source-id: 15868d8e81b9403218e481b3fbf0a7ca1279b023
Summary: This diff switches EdenFS from using the `revisionstore` crate into `backingstore`
Reviewed By: chadaustin
Differential Revision: D17866320
fbshipit-source-id: f6de1fe568ae893e4c2555039df6edfb0a450885
Summary: Introduce a new ScribeLogger class that spawns and maintains a process. Log messages are newline-delimited and written to the process's stdin. If the process stops responding or responds too slowly, log messages are dropped.
Reviewed By: pkaush
Differential Revision: D17777215
fbshipit-source-id: c998d10c73fc103122d69ae19c5d84f58b7939d2
Summary:
krallin noted that we were not collecting LFS stats from the import helper.
This diff should get us to run the import helper with those stats enabled, although
we don't yet know how useful those will be in practice.
We originally defaulted to `hg.real` to bypass some startup overhead and make our tests
run faster. We've evolved more control over this in the test harness over time, so I
think we're probably fine to make the runtime non-test default just be the normal hg
entrypoint.
Reviewed By: chadaustin
Differential Revision: D18039164
fbshipit-source-id: 7593cdca9afe30fa2fc928b84ba552dbecfed388
Summary: This makes a `diff()` function definition private inside EdenMount in order to solidify use of default parameters and to consolidate callsites.
Reviewed By: chadaustin
Differential Revision: D17962075
fbshipit-source-id: 2a6d4213f98828f4ebe0ba6db43211a96d1b2bc8
Summary:
The use of futures in the checkout code was a bit confusing. Before I
make further changes, refactor that.
Reviewed By: genevievehelsel
Differential Revision: D17985158
fbshipit-source-id: 26889ee85dc5704bc61347703dae370d4529d67b
Summary: Removing this DCHECK for the searchExpression because we can recieve requests for empty string and is a valid request.
Reviewed By: chadaustin
Differential Revision: D17985604
fbshipit-source-id: 40ffbe153a7ae44e85ed7d7ad7b2e79affb1d529
Summary: Added a multi-byte version of toString() for the guid class. Also renamed the existing toString() to toWString() which returns the widechar string for the GUID.
Reviewed By: chadaustin
Differential Revision: D17985500
fbshipit-source-id: fb3d976e39c0583122e47e3e444cf0f68e6a9544
Summary:
When we implemented FUSE_NO_OPEN_SUPPORT and FUSE_NO_OPENDIR_SUPPORT,
we forgot to remove the FileHandle and FileHandleBase code.
Reviewed By: pkaush
Differential Revision: D17991710
fbshipit-source-id: dfeb26d512f017cef7710929ccff1f6940cf8641
Summary: Removing an import that is not really used.
Reviewed By: pkaush
Differential Revision: D17997050
fbshipit-source-id: 7ba6a869937d51d640c72c22417a26fcf42c6ab3
Summary: refactor current test to enable fault injector for TestMount and remove DECLARE in c++ file.
Reviewed By: chadaustin
Differential Revision: D17882339
fbshipit-source-id: 719fcf082a9a1d1ea7a1fadc561b6f368f1ccdff
Summary: Tracing was not an accurate name for what this directory had become. So rename it to telemetry.
Reviewed By: wez
Differential Revision: D17923303
fbshipit-source-id: fca07e8447d9b9b3ea5d860809a2d377e3c4f9f2
Summary: While reading the config code, I made some minor, behavior-changing improvements.
Reviewed By: genevievehelsel
Differential Revision: D17918029
fbshipit-source-id: e40bce099a9555559c028c701f6fb75a6a4fc6be
Summary: FileMetadata will also store the SCM hash to be used later.
Reviewed By: chadaustin
Differential Revision: D17883243
fbshipit-source-id: 6121ca76f78b05f074799c0f52b568aa7805d8ab
Summary: This class is to simplify working with Windows registry and would be used in the future diffs.
Reviewed By: chadaustin
Differential Revision: D17791021
fbshipit-source-id: 76de156de67be4a007a5254efe48a14bdfefb24d
Summary:
Add a configuration knob that allows environments to opt out of Eden's
custom permission checks on the unix domain socket, instead using the
standard unix domain socket permission rules for the host OS.
Reviewed By: wez
Differential Revision: D17863456
fbshipit-source-id: c60c52891e49cc8027832bd42029cfd52752547b
Summary:
Instead of having accessors for every config setting in EdenConfig,
just expose the ConfigSettings directly.
Reviewed By: fanzeyi
Differential Revision: D17847805
fbshipit-source-id: 8c6c1010c010113cf859677449797ea916f2a2a5
Summary:
I plan to simplify access to EdenConfig by making ConfigSettings
public rather than writing accessors for each setting. To avoid
exposing implementation details, switch to private inheritance.
Reviewed By: wez
Differential Revision: D17847752
fbshipit-source-id: 02ef7afe96f09cc29a54b21bfafbef9234e6f74d
Summary: Removes `TreeDiffer` class and passes `DiffContext` through standalone `TreeDiffer` functions as first argument as per comment on D17400466 for setup for processing gitignores in the `TreeDiffer` codepath. (also this allows for easy implementation of short circut of `future_getScmStatusBetweenRevisions` similar to D17531102)
Reviewed By: chadaustin
Differential Revision: D17717977
fbshipit-source-id: d480d212474bd80aeac9cd9bb901f97562b62b13
Summary:
I found the logic for whether the config should be reloaded
unnecessarily complicated, so reduce it to deciding whether to reload
or not. This removes the need to acquire a write lock in autoreload's
common case.
Reviewed By: wez
Differential Revision: D17847699
fbshipit-source-id: 50fee1aac15cc8f896333c93459fea6510646600
Summary:
To allow downstream consumers to make authentication decisions based
on effective group, store and expose the peer effective group id.
Reviewed By: yfeldblum
Differential Revision: D17844620
fbshipit-source-id: 327437d5b36213e06ca6358266109f96761ac8dc
Summary:
Give ThriftPermissionChecker flexibility by allowing it to access
UserInfo if it needs to. This also gives ThriftPermissionChecker
access to EdenConfig for use in future diffs.
Reviewed By: wez
Differential Revision: D17844453
fbshipit-source-id: 5f7badd084bfbe82bd6b3797ef68daa7b63741b5
