Summary: the request will be passed with the context, so simply check the context at all three "looping" points to see if the request was cancelled. I split this into its own diff due to its unique test plan.
Reviewed By: chadaustin
Differential Revision: D18647091
fbshipit-source-id: ccb7ac765f8f3b0a60e936779cc27b658b9fd8b9
Summary:
In any case in which our Inode is not materialized and we would like to continue to diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
Also, this allows us to remove the RemoveDiffEntry since that only looked at the scmEntry, so that work can also be deferred to the source control tree differ easily.
This avoids loading inodes when we don't need to. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D19292528
fbshipit-source-id: 7b5ec161f7bd65962ae30c49fdad6dbc73954c2a
Summary:
In any case in which our Inode is not materialized and we would like to continue to diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
This reduces the number of inodes loaded by avoiding to load them if we don't need to. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D19292527
fbshipit-source-id: 2dbf6c4130214f581b6cc09e663f7567abab6ef3
Summary:
In any case in which an inode is not materialized and we are computing a diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
This avoids loading inodes when not necessary. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D18647087
fbshipit-source-id: 74632de777f04e2468d10977cc757b8993d439e9
Summary: Pass a GitIgnoreStack* and isIgnored flag through the diff operation. It is used later in the stack when we go down this code path in the TreeInode diffing code. When context->loadFileContentsFromPath() is null, the gitignore loading code will be ignored, used when we do not want to honor a gitignore file. For example, during `getScmStatusBetweenRevisions()`.
Reviewed By: chadaustin
Differential Revision: D18647089
fbshipit-source-id: 20d2abd2ef61669465e134165da5a0ac5e987cca
Summary: Produce a -Wdeprecated-declarations warning in code that uses the old SYNCHRONIZED macros.
Reviewed By: yfeldblum
Differential Revision: D19567088
fbshipit-source-id: b0637c06cbd125d09e1b39eec1e82a71ddca2324
Summary:
In the case that the overlay was closed before the initial
createOverlayFile, prevent an invalid OverlayFile from being accessed,
causing the test to fail.
Reviewed By: genevievehelsel
Differential Revision: D19591636
fbshipit-source-id: 9d961f044c3b68b5c0b2dcd108ff85db6326276d
Summary: I reproduced the KeySpace.cpp ICE on my Windows 10 VM, so bump the MSVC version check. I have not confirmed it does not ICE on MSVC 2019.
Reviewed By: pkaush
Differential Revision: D19562325
fbshipit-source-id: 03cf84e95c709bb2ebe7c3e0055a4f290eb17f8a
Summary: This code is no longer used now that `eden redirect` exists.
Reviewed By: wez
Differential Revision: D19565010
fbshipit-source-id: 4703bc39a024c4c491a83ef5a6e85711a8f7e4d8
Summary:
It's not often obvious to people where edenfs.log is located on the
filesystem, so redirect people to `eden debug log` instead.
Reviewed By: simpkins
Differential Revision: D19560709
fbshipit-source-id: 6b031c80bab2c21dfddba4d7ec37d4e70218ea08
Summary:
Now that loading too many blobs only GCs the blob cache, treat cached
trees as ephemeral again.
Reviewed By: simpkins
Differential Revision: D19358375
fbshipit-source-id: 191427f46276f0f5d024b5bc923a68f7a42044da
Summary:
Instead of clearing every single cached object when the total size
exceeds the ephemeral storage limit, keep a limit per object type and
only clear those that exceed their quota.
Reviewed By: simpkins
Differential Revision: D19358312
fbshipit-source-id: 6918d6f4cc2931aed79a9025d0e0f357ede515e0
Summary:
EdenFS no longer uses the blobsize keyspace. To avoid wasting space,
clear and campact it on startup, along with any keyspaces we
deprecated in the future.
Reviewed By: fanzeyi
Differential Revision: D19354881
fbshipit-source-id: 5285757a0e44ab1080c5f940283e06b17bec811d
Summary:
Indicate getManifestEntry is deprecated so we remember to delete it
down the line.
Reviewed By: genevievehelsel
Differential Revision: D19459793
fbshipit-source-id: 86dd173cfb12c36cf6ba19eaf4be96dfe624c05f
Summary: Simplify the definition and use of KeySpace and move it into its own header.
Reviewed By: simpkins
Differential Revision: D19353441
fbshipit-source-id: ef07677d927a48839b709711388abeb3c1ed9679
Summary: This diff addresses the unreleased mapped pack files in the old datapack code.
Reviewed By: chadaustin
Differential Revision: D19399737
fbshipit-source-id: 86a6254a2939fd69e1ce2b25b8bfcb36b0deb16b
Summary:
As reported by JT, EdenFS may hold the file descriptor of mapped pack files too long even when it is deleted by external processes, thus taking more disk spaces.
This diff fixes this problem by making EdenFS periodically rescan the pack files.
Reviewed By: chadaustin
Differential Revision: D19395439
fbshipit-source-id: 4bfd6a7ac13dceb3099d2704d62b3825433aff4b
Summary:
stderr: eden/fs/inodes/EdenMount.cpp:1122:37:
error: local variable 'unmountError' will be copied despite being thrown by name [-Werror,-Wreturn-std-move]
eden/fs/inodes/EdenMount.cpp:1122:37: note: call 'std::move' explicitly to avoid copying
Reviewed By: simpkins
Differential Revision: D19408659
fbshipit-source-id: 1575104414daaeb06892586492cdb74e2df604c5
Summary: based on comment on D18647089, changes the behavior when a directory is replaced with a file, and then the directory is ignored. The behavior should be that the file is reported as `ADDED`, previously the file would have been reported as `IGNORED`
Reviewed By: simpkins
Differential Revision: D19318796
fbshipit-source-id: 8da1b7c6df182ca591e4aea21f6573ca81790cb4
Summary:
We want to have a shared pool of `HgImporter` across different threads, but that would require `HgImporter` is safe to be *passed* between threads. (This is not making `HgImporter` entirely thread safe.)
However, `HgImporter` currently holds a pointer to the thread local fb303 counter. This diff pushes down `EdenStats` so we only access the thread local stats variable when we need to add counters. This might be a little slower but it does make it safer.
Reviewed By: chadaustin
Differential Revision: D19053250
fbshipit-source-id: 44a897acc90c6042ae22a0417eece39e099ee13f
Summary: The only reason `HgImporter` still holds a pointer to LocalStore is for looking up proxy hashes. We can directly pass in the Mercurial hash and path instead.
Reviewed By: chadaustin
Differential Revision: D19049039
fbshipit-source-id: 45d1e1f363ed73dca447b22e5891495cf9ad202b
Summary: scuba logging for mismatched parent commits, it would be nice to have a count on how often this is encountered so we can pinpoint spikes of this case. I am not sure how helpful having the parent commits would be in the event, my thought there was that it could help see how far away the commits are and if one of the commtis doesn't exist (like in the case of stripping commtis with a cancelled hg split), but I am open to having an empty event as well if storing these strings is too expensive.
Reviewed By: fanzeyi
Differential Revision: D19432378
fbshipit-source-id: fa3e7cd6aef832c4f918f339b781590b7484cfdc
Summary:
There is a bug in Projected FS on Windows 1903 and 1909, which BSODs the system on SetInformation call. Projected FS seems to be dereferencing a nullptr while attempting to check if the file is in the backing store. Providing a separate callback to check if the file exists in the backing store will avoid the buggy code (suggested by Microsoft).
This will enable Eden FS on Windows 1903 and 1909.
Reviewed By: simpkins
Differential Revision: D19377522
fbshipit-source-id: ab5e9bc8e2472f3e689d71d05a560bac86211295
Summary: This is a followup on D18666585.
Reviewed By: chadaustin
Differential Revision: D19377523
fbshipit-source-id: f9ab4011d0978a6934de77d1837a37082603cf52
Summary: We missed updating this in D18669664.
Reviewed By: simpkins
Differential Revision: D19377524
fbshipit-source-id: e23fb039047855454b4f12353bc375b1a2c8d962
Summary: `folly::cold_detail::ColdClass` was marking things (like `folly::Unexpected`) cold, but at the cost of inhibiting the inliner from doing its job. This is leading to bad codegen, which offsets any small wins we mind get for the `cold` attribute.
Reviewed By: yfeldblum
Differential Revision: D19324159
fbshipit-source-id: 7ed431b6c9d6e963c3bf438c707fa6cf6a38bf9d
Summary:
D19135344 broke EdenFS's cmake builds, and our macOS releases haven't
been going out. Rather than pass the same streaming options to the
Thrift compiler in the CMake build, remove deprecated_server_streams
from both builds.
Reviewed By: simpkins
Differential Revision: D19204397
fbshipit-source-id: c22e634ded82a683daff5c92182c8a1eb1ed4d21
Summary: This stack is deprecated in favor of the rocket stack.
Reviewed By: rhodo
Differential Revision: D18123600
fbshipit-source-id: db6e3bf4add9c49cf8f1d2f12f97b207e87c7302
Summary:
Reduce the log level for a few messages to help reduce some less important
clutter when running with an elevated debug log level.
Reviewed By: chadaustin
Differential Revision: D19188773
fbshipit-source-id: 396bb15e119fc12765ecbc707e7a1dadbdd02422
Summary:
Remove some half-baked, unnecessary logic for caching sizes separately
from SHA-1. Eden's backing stores do not support chunking large files
yet, so there's no value in caching content SHA-1 and size
separately. This fixes a scenario where fetching blob size and then
SHA-1 would result in two backing store imports.
Reviewed By: fanzeyi
Differential Revision: D19169096
fbshipit-source-id: dc32f3313e5f4230c06a5bbaa67da7bf0febaba8
Summary:
Troditionally, ObjectStore relied on HgBackingStore writing directly
to LocalStore in order to cache trees. This had the unfortunate side
effect that other backing store implementations did not benefit from
tree caching.
Move tree caching into ObjectStore so all backing stores benefit from
tree caching.
Reviewed By: simpkins, fanzeyi
Differential Revision: D19168211
fbshipit-source-id: b1019591ebb4760cc8b933b9adb82174b8e5fa1f
Summary:
If a GC ran at the same time as HgBackingStore imported a commit, the
import would fail. Rather than round-tripping the tree through the
LocalStore, just return it.
Reviewed By: genevievehelsel
Differential Revision: D19167798
fbshipit-source-id: eddb280def349483289d2e300a58eff3ad7416e1
Summary: There is one instace of `getScmStatusBetweenRevisions` in use - it is used in the eden cli in a hacky way to check if a commit hash is valid. Since this is not used anywhere else in a meaningful way, this replaces that use case with a hg call and depreciates `getScmStatusBetweenRevisions`
Reviewed By: simpkins
Differential Revision: D18690026
fbshipit-source-id: 02bd2c20a0f631ec41116f9fd4e18d14369298ef
Summary:
This diff turns the return type of `BackingStore::getBlob` from `folly::Future` into `folly::SemiFuture` to prevent executor leaks.
This also enable us to remove the need of holding `serverThreadPool` from backing stores.
----
**Changes**
* `ObjectStore` now needs to hold a `folly::Executor::KeepAlive` that is used to turn `SemiFuture`s it gets from backing stores into `Future`.
* Signature changes of the implementations of `BackingStore` class.
* For tests, I chose to use `QueuedImmediateExecutor` in place of `UnboundedQueueExecutor` as it will basically execute tasks inline. I'm concerned introducing thread pool executor in tests may turn tests flaky.
Reviewed By: wez
Differential Revision: D18669664
fbshipit-source-id: 0cae89f365dcf8b345b49d64469a530cf25d4ac5
Summary:
Add a config option to restrict the number of tree prefetches that can run in
parallel. Without this applications that recursively walk a checkout tree can
end up spawning a huge number of asynchronous prefetch operations. These
prefetch operations are quite expensive today, as we currently have to fetch
the full file contents in order to retrieve the file sizes. A recursive
directory walk can end up building up a huge backlog of in-progress tree
prefetches. This will slow down the directory walk, and it can take tens of
minutes to finish processing the prefetch backlog even after the directory
walk is aborted.
Reviewed By: chadaustin
Differential Revision: D19136685
fbshipit-source-id: cdc7a570d016fd7ca81a639cef83380b197acdfc
Summary:
EdenMount offered both getRootTree and getRootTreeFuture. Replace the
former with the latter and adjust remaining callers.
Reviewed By: genevievehelsel
Differential Revision: D18666585
fbshipit-source-id: 961361a1fb545e48db0ece970bb5e6ffbca8efef
Summary:
The last remaining user of ObjectStore::getBlobMetadata was a debug
Thrift call. Remove it and update the Thrift call to use getBlobSize
and getBlobSha1.
Reviewed By: pkaush
Differential Revision: D18663376
fbshipit-source-id: 86baefc9004a07aac4ddf5849870431be04c75f2
Summary: Fix some warnings that only show up in the macOS build.
Reviewed By: fanzeyi
Differential Revision: D19053236
fbshipit-source-id: 81f7187b263e0db6a57582677088519f9b97f1d7
Summary:
D18820280 increased the log level for `globFiles()` calls from DBG3 to DBG2.
However, buck and other tools often end up huge numbers of glob calls, and
this ends up causing log spam.
This change reduces the log level of `globFiles()` back to DBG3. In a
subsequent diff I will update the code to increase the log level for slow
calls. This does mean that we will only get a log message for slow glob calls
after the call completes, but this is better than not getting a message at
all.
Reviewed By: pkaush
Differential Revision: D18924780
fbshipit-source-id: 86e83614f51424aca078e4de8a56c45018a6d286
Summary: was looking in this area and saw this TODO. moves the `setParentCommits()` call into `CheckoutConflict::finish()` so it is called while we are still holding the parents lock.
Reviewed By: chadaustin
Differential Revision: D18668329
fbshipit-source-id: 8415c792193e5b89737f15141f33c5a9799f527b
Summary:
A spike in automatic GCs usually implies something has gone wrong. Log
an event for each one, recording the cache size prior to the GC and
the cache size after.
Reviewed By: simpkins
Differential Revision: D18902580
fbshipit-source-id: 158b2635733a415a9fcc7c412b2c0f44ed04aa01
Summary: Enabling getScmStatusBetweenRevisions to work on Windows and its tests.
Reviewed By: simpkins
Differential Revision: D18431271
fbshipit-source-id: eee82538e2fc3d7e371c96fc271cd9662ea6d737
Summary: This diff contains the tests for hg status. This is not comprehensive list of tests but the tests that we have working at this time. More tests will be added as we test and fix issues.
Reviewed By: simpkins
Differential Revision: D18454851
fbshipit-source-id: 80e3ff4d24873016dc420f202fbfe53fcffc24f2
Summary: The current implementation of EdenDispatcher is projectedfs specific and was linking against its dll. This was adding the need to add Prjfs dll for running our tests. Moving it to prjfs channel should fix that.
Reviewed By: simpkins
Differential Revision: D18462171
fbshipit-source-id: ff7bfcdd14d19552d49b61ff03cdd770badfcd68
Summary: This class provides an interface to create an EdenMount for testing and use it through the APIs. It's similar to TestMount for Eden on Posix.
Reviewed By: simpkins
Differential Revision: D18454852
fbshipit-source-id: 91d64b2c535060979ef634fe4e7779f50f169fc5
Summary: WinStore interface was taking wstring for path and we were converting wchar pointers to wstring which were later converted to RelativePath. In this diff we pass the path as is and save one copy per operation.
Reviewed By: simpkins
Differential Revision: D18444773
fbshipit-source-id: fa6a3efb6a7e91cf9b0dca919e80870e1bcf0804
Summary: Moved the current FsChannel implementation in to PrjfsChannel and converted the FsChannel.h into an abstract class. Created a TestFsChannel class for testing.
Reviewed By: simpkins
Differential Revision: D18440401
fbshipit-source-id: 30d6742cd18b4b76b664e3af7ed1a528cf876046
Summary:
A successful run of hg commit ends in call to resetParents() to reset the current parent commit.
We also need resetParent to run the Status tests.
Reviewed By: genevievehelsel
Differential Revision: D18439954
fbshipit-source-id: 44f7a009e050bd5e2130d8c4fc3a85a8ec3bfad1
Summary:
GenerateStatus will parse the CurrentState and compute the status. The algorithm here is little different than of Eden with FUSE, Eden with Prjfs will loop over the CurrentState and only compare the entries present in it. CurrentState has the list of all the entries modified and a complete status could be generated from it.
This doesn't solve all the scenarios. I have few unit tests running at the top of this stack and those cases are working. I need to test the other cases and make them work. One case that I already know doesn't work is when a user resets the parent and doesn't make any change. I will work on fixing the other cases and put up diffs for that.
Reviewed By: simpkins
Differential Revision: D18195573
fbshipit-source-id: 3650f26387f1a5f8d467fd81148c266fa1307a3c
Summary:
Add an option `experimental:use-edenapi` to `EdenConfig`.
See the next diff for usage.
Reviewed By: chadaustin
Differential Revision: D18605549
fbshipit-source-id: 2786c21bb38a76229078662cc5c1ddf906d1be4a
Summary:
I think we need to make `EdenConfig` available for testing as more things are using `EdenConfig` to configure behaviors.
Right now it does not do much things other than just construct an `EdenConfig` instance without manually providing these parameters.
Reviewed By: chadaustin
Differential Revision: D18798484
fbshipit-source-id: b4a85d966a9b4f425c547bef9eb6e7570d7e2170
Summary:
Two bugs conspired to cause edenfs after a graceful restart to think
the kernel supported FUSE_NO_OPENDIR_SUPPORT when it didn't: the
connection info struct wasn't zeroed, and FUSE connection capabilities
weren't properly mirrored into the Dispatcher upon graceful
restart. Fix both and add an integration test.
Reviewed By: simpkins
Differential Revision: D18903761
fbshipit-source-id: 23f4db3e240ee7d035f707820072c606a45f1138
Summary:
Register the logging and logging_full option handlers in the open
source and macOS builds of Eden.
Reviewed By: simpkins
Differential Revision: D18870545
fbshipit-source-id: 140debe77be071a9fa2d628ad305517273e095e3
Summary: Adds a function which takes both the manifestID and the commitID to get a Tree. This will be used in `checkOutRevision()` and this allows us to skip looking up the manifestID since the caller can just pass it in themselves.
Reviewed By: wez
Differential Revision: D18719405
fbshipit-source-id: 919f0a7c84bff4a2f0bc20110c45bd272f9e9107
Summary:
It is very painful for Eden's automatic GC to delete trees from the
local store. An unfortunate number of programs scan the entire repo,
causing every tree to be loaded. If Eden's automatic GC kicks in, it
causes all trees to be downloaded again, blocking both FUSE and Thrift
requests.
Temporarily mark trees as persistent.
Reviewed By: fanzeyi
Differential Revision: D18842066
fbshipit-source-id: 4a22defcb1151912337d4167ac49852519c7789d
Summary: Saves loadFileFromPath in the DiffContext. This allows the source control differ ( at `eden/fs/store/Diff.cpp`) to call this function while it is trying to load gitIgnores and resolve symlinks.
Reviewed By: simpkins
Differential Revision: D18647088
fbshipit-source-id: bddff0158130f0c96ac263bcce81dc51fc017800
Summary: Adds a helper function that can take a path to an Inode and load a file by simply getting the Inode then calling the existing `loadFile()` function
Reviewed By: simpkins
Differential Revision: D18647090
fbshipit-source-id: 89c25bef10a7a0d0ffd660293f85479ac0e9f9a9
Summary: Move the bulk of the gitignore loading to EdenMount, this is needed later in the stack to pass this function as a callback through the DiffContext to enable gitignore loading from the source control tree differ.
Reviewed By: simpkins
Differential Revision: D18565265
fbshipit-source-id: 062349d81bbfb78235ccb34d6ec72ad445a6e2b0
Summary:
Increase the log level for messages about `globFiles()` thrift calls so that
they are enabled by default in our normal production configuration.
`globFiles()` calls can potentially cause EdenFS to fetch a lot of data, so it
is useful to be able to see them in the logs when investigating performance
issues.
Reviewed By: genevievehelsel
Differential Revision: D18820280
fbshipit-source-id: e351a9541e420f5eb47ca14d6620366437773ce4
Summary:
This message is pretty noisy when running EdenFS with an increased debug
logging level. Bump the debug level down so it doesn't spam the logs when
running at a slightly elevated debug level. Also fix a minor typo.
Reviewed By: fanzeyi
Differential Revision: D18820281
fbshipit-source-id: 668367337f4a8cf2a38003c8e0034969d335c2c0
Summary: There are a couple of functions in `store/Diff.h` which are not used elsewhere except for testing. These functions are tested by `diffCommitsForStatus()` anyways so these standalone functions are not needed.
Reviewed By: chadaustin
Differential Revision: D18690006
fbshipit-source-id: f2b24575c17403d7241896f35f4e0e16bb03b7ce
Summary:
Each entry in InodeMap::unloadedInodes_ stored the InodeNumber
twice. Remove one of them.
Reviewed By: wez
Differential Revision: D18651007
fbshipit-source-id: be85c34cb2b38fc0b2875d0874cecd1ef274aca4
Summary: Small readability improvements I noticed while reading the diff algorithm.
Reviewed By: genevievehelsel
Differential Revision: D18650987
fbshipit-source-id: 19346362711455ae87d6770812e647544a4576bf
Summary:
Add default case handling for some switch statements that do not handle the
case of an enum variable being set to something outside of one of the defined
values.
Reviewed By: fanzeyi
Differential Revision: D18652102
fbshipit-source-id: abcb3f9dc1f6b0dc761170c7115ddd9ad5145bc4
Summary:
This splits `EDEN_BUG()` into three separate version. All three crash in
debug mode builds, but in release builds they behave differently:
- `EDEN_BUG()` throws an exception
- `EDEN_BUG_FUTURE(Type)` returns a `folly::Future<Type>` that has been
fulfilled with an exception.
- `EDEN_BUG_EXCEPTION()` returns a `folly::exception_wrapper`.
The main advantage of this is that this allows the compiler to detect that
`EDEN_BUG()` can never return. Previously `EDEN_BUG()` was used for all 3 of
these different cases, and its behavior depended on whether `toException()`
was ever called. As a result we could not easily get the compiler to identify
code paths where we know at compile time that it will never return.
Reviewed By: chadaustin
Differential Revision: D18652103
fbshipit-source-id: 070107c7520f51b05696905fa243de5f8df15958
Summary:
Now that the fb-mercurial sources are available in the Eden repository, update
our CMake build files to always build them. This moves the build logic from
the centralized `FBMercurialFeatures.cmake` file into `CMakeLists.txt` files
in the appropriate subdirectories.
Reviewed By: chadaustin
Differential Revision: D18588011
fbshipit-source-id: ded9decde5c2ec766aae0bb0f4f5b021d1044a98
Summary:
Update the CMake build to always build the Rust datapack libraries, even on
Windows. This allows us to completely eliminate the `EDEN_HAVE_RUST_DATAPACK`
checks.
Note that I did leave the `EDEN_HAVE_RUST_DATAPACK` macro in place, as we
still do not build the Rust datapack code on Mac when building with Buck.
Reviewed By: fanzeyi
Differential Revision: D18588008
fbshipit-source-id: 1a4c9ceec5372d0e6a7313d2eb87edabd1e60a96
Summary:
Update Eden's top-level CMakeLists.txt file to build the Rust datapack
libraries. Previously these were built by invoking CMake separately inside th
`eden/scm` subdirectory. Now that the code has been combined into a single
location we can use a single CMake invocation to drive the build of both these
components.
The old code did not build the Rust datapack code on Windows, and this diff
does not change that behavior. I'm not aware of any reason to skip building
this on Windows, so I plan to enable building this code on Windows in a
subsequent diff.
Reviewed By: pkaush
Differential Revision: D18588006
fbshipit-source-id: 20f4f0ea9fef8595a9dd35a21115952b2808c824
Summary: Even if blobs have different hashes, they could have the same contents. For example, if between the two revisions being compared, if a file was changed and then later reverted. In that case, the contents would be the same but the blobs would have different hashes. Currently, `getScmStatusBetweenRevisions()` would report false positives in this case. This is also needed so we do not report false positives in `getScmStatus()` when hit this code path
Reviewed By: simpkins
Differential Revision: D18647086
fbshipit-source-id: 66e12648a24fd7e5612eee5e599a5b81c7c5f2d1
Summary:
Annotate some import statements so autodeps won't complain about them, since
it doesn't know where they come from in the build.
Note that `hg_import_helper.py` isn't used any more by the current code, and
should ideally be killed off soon. However, just annotating these import
statements for now in the short term is the easiest way to make autodeps
happy.
Reviewed By: fanzeyi
Differential Revision: D18652104
fbshipit-source-id: 310bc5c63447e3bf4dfd081d7b3930eed4d0498b
Summary:
The `eden/fs/sqlite` code uses `eden/fs/utils/PathFuncs.h`, so add a proper
dependency on the `eden_utils` library.
Reviewed By: pkaush
Differential Revision: D18652105
fbshipit-source-id: b8dffd93971ab5efb71d7bc3c2dcb62249b3915b
Summary:
Record the durations of each phase of checkout so it can be logged to
scribe in a future diff.
Reviewed By: simpkins
Differential Revision: D18632258
fbshipit-source-id: e447976af22cf25cffd640e79b95da2989b07448
Summary: Adding an experimental option to EdenFS so we can disable blob caching if we need.
Reviewed By: chadaustin
Differential Revision: D18441665
fbshipit-source-id: 56751c0199d6658bfbf7ec3865f746a7279324ab
Summary:
We're about to return more information from checkout, so
introduce a CheckoutResult struct.
Reviewed By: genevievehelsel
Differential Revision: D18630820
fbshipit-source-id: 78b68ef3551ca8c9e9968631de554084c333b4b9
Summary: make `future_getScmStatusV2` into `async_tm_getScmStatusV2` so we can short circut the status call if the request is no longer active.
Reviewed By: simpkins
Differential Revision: D18378212
fbshipit-source-id: b2bca014a37d99f7a92478c3e23bde0b874dd13c
Summary: This reads `enforceParents` from a config instead of always assuming true for `getScmStatusV2()`. This will allow a easy kill switch in case throwing errors from this thrift call causes issues with something that calls hg status
Reviewed By: simpkins
Differential Revision: D18258164
fbshipit-source-id: 1ae421a941c01a678d25d5453c771262b03558d0
Summary:
This diff adds the C++ conversation methods that converts the Rust struct introduced in the previous Diff into EdenFS's `Tree`.
We have to keep the RocksDB write batch in for storing proxy hashes.
Reviewed By: wez
Differential Revision: D18365504
fbshipit-source-id: 2a2cd2cfebd9f0ae4e279db608becc5890c5ccdf
Summary: Create a template functions to convert strings with a wide variety of input types like folly::StringPiece, std::string, folly::fbstring and std::string_view to wide char string.
Reviewed By: wez
Differential Revision: D18071832
fbshipit-source-id: 147af97dff929766db83c9477117ec82da42b03e
Summary: We had linking issues with the Boost while compiling with getdeps. I have moved the tests to use std::filesystem instead.
Reviewed By: wez
Differential Revision: D18195608
fbshipit-source-id: 32213a9b22105869eef78724bb3e583c3fd8e708
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.
The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:
```
void test_log() {
FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.
`nm` before:
```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```
`nm` after:
```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```
Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.
```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```
So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.
In addition to binary size improvements, switching to fmt will potentially
* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).
Reviewed By: simpkins
Differential Revision: D15485589
fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
Summary: make the error message returned in the case of out of date parents during a new status call more user friendly and provide possible remediation instructions
Reviewed By: simpkins
Differential Revision: D18328835
fbshipit-source-id: b214f45bb055d008db8b233ddd2a1843332db838
Summary:
Merge the fb-mercurial code into the Eden repository, under the
`eden/scm` subdirectory.
Reviewed By: quark-zju
Differential Revision: D18445774
fbshipit-source-id: fc3307f9937e0c7e1c8f7d03c5102c4fe5dedb10
Summary:
Allocate and hook up a StructuredLogger at startup if the EdenConfig
has the appropriate values set.
Reviewed By: simpkins
Differential Revision: D18071821
fbshipit-source-id: 3996b6644bbf0c16bb3b9950d57a79d4619a1656
Summary:
The `HgDatapackStore.cpp` file currently depends on code not available on Mac,
so do not attempt to build it there.
Reviewed By: fanzeyi
Differential Revision: D18467948
fbshipit-source-id: cce37bb480426e6d972045321f15c1f2aa4457ee
Summary:
I spent hours debugging a deadlock in my code that was caused by
accidental usage of the deprecated Synchronized::operator->. Avoid
that in the future by building Eden deprecation-clean.
Reviewed By: genevievehelsel
Differential Revision: D18354757
fbshipit-source-id: fe74250635f779bb5010f907d57951132b9edea4
Summary:
Throw errors if a read or write is called on the `Overlay` class after `close()` has been called. Adds IO reference counting to know when it is safe to release the overlay lock after calling `close()`
This takes care to catch errors that could be thrown around overlay IO and makes sure to correctly decrement the IO count in the case of errors.
Reviewed By: chadaustin
Differential Revision: D17957360
fbshipit-source-id: af5e3c3f98c77800914d03cd748fb2f1c59ae264
Summary:
If a checkout's SNAPSHOT file disappeared, the mount error given would
be that a Hash had an invalid size. Instead, throw a file not found
error on mount.
Reviewed By: genevievehelsel
Differential Revision: D18381074
fbshipit-source-id: 35282e0990189d4084a2a64330a5733561cf88c2
Summary:
Looking at a log, it wasn't immediately obvious what might have passed
an invalid hash into the Hash constructor. Improve the error message
to make the cause clearer.
Reviewed By: genevievehelsel
Differential Revision: D18380916
fbshipit-source-id: 620b8fa902a87496b87a5aa0ff304e6991585864
Summary: changes `overlay_` to be stored as a `std::shared_ptr` in order to safely create and pass `std::weak_ptr`s of the overlay to OverlayFiles that are being created.
Reviewed By: chadaustin
Differential Revision: D18302732
fbshipit-source-id: 495b613914771714ba8a8d1beb1c7a0cd72b33d4
Summary:
This moves error handling with `OverlayFile` closer to the source of the error with the use of `folly::Expected`.
This is for future IOReference counting but also lets us transform all of these generic `std::system_error`s into InodeErrors (which is a subclass of `std::system_error` that can include inode path in the error message).
Another option for error handling would be to throw folly::checkUnixError inside the `OverlayFile` instead of returning a `folly::Expected`, but I am always hesitant to assume that the user of this class wants the error to be thrown, since they may want to do some other work (cleanup etc) before throwing an error.
Reviewed By: chadaustin
Differential Revision: D18215242
fbshipit-source-id: 5e38fdc7e6efe7d646dd3895932c9b2f26674b36
Summary: This file wrapper is returned instead of a `folly::File` when talking with the `Overlay`. This is basework for supporting closing the `Overlay` class even if it is still in use so file modifications on files given from the `Overlay` can be denied if the `Overlay` is closed.
Reviewed By: chadaustin
Differential Revision: D17973664
fbshipit-source-id: e4b9778b3a830b6310fda4a9e66b459530b9fe17
Summary:
Fix the `getStatInfo()` code to avoid walking all loaded inodes. That
behavior was okay early on when we were first developing EdenFS and this
API wasn't used in many places. However, `eden rage` and `eden stats` both
exercise this code path, and it seems pretty bad to walk all loaded inodes in
order to return some stats.
I removed the counts about number of materialized inodes for now, since I
don't think we're really using this metric much.
Reviewed By: chadaustin
Differential Revision: D17578306
fbshipit-source-id: 55ab0209745869b160e91167d6cff7d95f39a95a
Summary:
We noticed that some operations were surprisingly slow and
the hypothesis was that caching in the kext wasn't working well.
I traced into this with a simple repeated `realpath` of the same file
while running the server with debug logging turned up. That showed
that we were getting two requests from the kernel for each inode in
the path resolution for each invocation of the realpath tool.
I switched to looking at the kext code and found this section to be
a potential issue:
https://github.com/osxfuse/kext/blob/support/osxfuse-3/osxfuse/fuse_internal.h#L711
```
/* XXX: truncation; user space sends us a 64-bit tv_sec */ \
VTOFUD(vp)->attr_valid.tv_sec = (time_t)struct_name ## _get_attr_valid(fuse_out); \
VTOFUD(vp)->attr_valid.tv_nsec = struct_name ## _get_attr_valid_nsec(fuse_out); \
nanouptime(&uptsp_ ## __funct__); \
```
Switching our default TTL to be the maximum possible value that fits
in a signed 32-bit time_t cuts down on the requests from the kernel to
a single lookup for the symlink:
```
V1105 15:56:28.900839 28132386 FuseChannel.cpp:1141] fuse request opcode=1 FUSE_LOOKUP unique=2 len=47 nodeid=14910747 uid=2048904527 gid=1876110778 pid=24546
V1105 15:56:28.900885 28132386 FuseChannel.cpp:1437] FUSE_LOOKUP parent=14910747 name=README
V1105 15:56:28.901111 28143988 FuseChannel.cpp:421] sendRawReply: unique=2 header->len=160 wrote=160
```
and no requests at all for a plain file in the repo.
Reviewed By: simpkins
Differential Revision: D18340767
fbshipit-source-id: caebf051a543c54f7a03852fd2e0abab68448ded
Summary:
This diff makes EdenFS to be able to import manifest from Rust code.
We are only able to import raw manifest data from Rust for now. This code still
send the imported data to the ctreemanifest for parsing. Follow up diff will
fix this.
Reviewed By: chadaustin
Differential Revision: D18097701
fbshipit-source-id: b821a1d595f43a19d9ce9cd6d5ff5174d49e34e6
Summary:
Background: https://fb.workplace.com/groups/sourcecontrolteam/permalink/2440123159442349/?comment_id=2443624539092211
This may or may not be the root cause but this is definitely gonna be causing issues.
Between D17468473 and D17866320, turning on this option will make EdenFS to use the old Rust code that does not strip copyrev data.
Reviewed By: wez
Differential Revision: D18250237
fbshipit-source-id: b3816bd08b8dc3b2a90931829c368a58731c7843
Summary: Calling a command that is not implemented will terminate the edenfs. It was ok for testing but now we have it running on Sandcastle, so will throw Eden error instead.
Reviewed By: chadaustin
Differential Revision: D18145495
fbshipit-source-id: 17e57f426d3cf80c93c2112741a027db8cbc6b8d
Summary:
Testpilot is crashing to parse the JSON with the test names. I didn't debug it any further but looked like didn't like the names and renaming them worked.
Traceback (most recent call last):
File "testinfra/testpilot/testpilot.py", line 616, in pre_main
File "testinfra/testpilot/testpilot.py", line 111, in main
File "testinfra/testpilot/testpilot.py", line 262, in main_impl
File "testinfra/testpilot/testpilot.py", line 259, in main_impl
File "testinfra/testpilot/integration/fbcode/finder.py", line 478, in get_tests
KeyError: 'eden-StateDBNodeTest.fetchDirectoryEntries-getdeps-windows-none-6 - main'
Reviewed By: wez
Differential Revision: D18208976
fbshipit-source-id: 1785aa4b1c42041578350b9c2a7e928059eeb648
Summary:
Add a new thrift API for computing the difference between the working
directory and a given source control commit.
This has the following differences from the old getScmStatus() commit:
- The parameters are accepted in a GetScmStatusParams structure now.
This makes it easier for the server-side C++ implementation to tell which
parameters have actually been specified by the caller. This will make it
easier to extend this API in the future without having to replace it with a
new function call again.
- The return value is a GetScmStatusResult, which includes both the ScmStatus
and the EdenFS version number. This will allow code like `hg status` to get
both the status results and the EdenFS version in a single call, without
needing to make multiple separate thrift calls.
- This new call will return an error if the caller requests the status against
a commit that disagrees with EdenFS's view of the current commit. Because
the individual `hg` command line processes do not perform any
synchronization of their own when reading the working directory parent,
they can often call EdenFS with stale parent information, or while a
checkout is currently in progress. This new behavior will reject the
request with an error, rather than having EdenFS perform a potentially very
expensive status computation when the results probably aren't actually
useful to the caller anyway.
Reviewed By: chadaustin
Differential Revision: D15110218
fbshipit-source-id: ebc2f74dafc090d4fd245de8e4d62e2b086500dd
Summary: Adds a non-optional EdenErrorType struct for EdenError, this can be used in case of special error case handling of errors without error message parsing. Currently this is just passed along and not consumed anywhere in the client, but later in the stack is used for specific retry of checkout if "CHECKOUT_IN_PROGRESS" on the consuming side.
Reviewed By: chadaustin
Differential Revision: D18139917
fbshipit-source-id: b3f2ec4c480fc5246ff2f46d09c436021bad8b61
Summary:
The Windows build spews a great many warnings. Address some of them by
enabling the unused-exception-parameter warning on Clang/Linux too.
Reviewed By: yfeldblum
Differential Revision: D18178930
fbshipit-source-id: efecb605b84d4f06c8c8411a23d17904bbdff746
Summary:
Add code on macOS to collect the process's resident set size and
virtual size so the memory usage counter shows up.
Reviewed By: wez
Differential Revision: D18142618
fbshipit-source-id: b42206017ace5309b7bc379f042c1cd6e455f574
Summary:
add an enforceParent flag to signal if the diff call should only be run against its parent commit. setup work for `getScmStatusV2`
The only callsite for this version of diff() is in the testcase and in `async_tm_getScmStatus`, in which both cases they set `enforceCurrentParent = false`.
Reviewed By: chadaustin
Differential Revision: D17911708
fbshipit-source-id: 361c7586e1452544172cf9e79a0cdf0888154096
Summary:
Introduce a framework that allows recording structured log events
which are encoded as JSON and piped to a configured command line
program.
Reviewed By: pkaush
Differential Revision: D18025183
fbshipit-source-id: ab6b4d510a905a30252f2cff85d107a0d32d149e
Summary: Having the header files in CMake, adds them in Visual Studio project and makes it easier to work in Visual Studio.
Reviewed By: chadaustin
Differential Revision: D18145496
fbshipit-source-id: f6607e0cbf2e6a8d9fc3ad9f8bed90458aec355e
Summary: This diff will create an instance of CurrentState in EdenMount and integrate it with the Prjfs notifications.
Reviewed By: chadaustin
Differential Revision: D18068335
fbshipit-source-id: 5c78949c8b0ad316d2b014e808297f29092ae703
Summary: This diff adds the notifications callbacks from Prjfs. These will be used in the later diffs in the stack.
Reviewed By: chadaustin
Differential Revision: D18068035
fbshipit-source-id: a8e679ef32ae39251e6c1e926feb9cd1b71258e0
Summary: This diff creates a Guid as a unique mount id. This mount id will be used to uniquely identify our repo in the Regdb.
Reviewed By: chadaustin
Differential Revision: D18055798
fbshipit-source-id: 01099d7fdf15c325498e83c8b63637dd1d85d028
Summary: The is the top level interface which will receive all the FS changes and will store them in the RegDB for later use.
Reviewed By: chadaustin
Differential Revision: D17998282
fbshipit-source-id: 7570edcfe0f6bf9164bfa0688665fb7253a21bc6
Summary: StateDirectoryEntry will contain the current state of a file or folder in the prjfs cache and will be totally in memory. In the next few diffs we will bring in the code that will load and store these in the db.
Reviewed By: chadaustin
Differential Revision: D17985967
fbshipit-source-id: 260dca369a3acce10385b04fdb5ff111cfc998be
Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary:
This saves the version number (like 20191014-154333) in the EdenServer. This will be returned to the user in a getScmStatusV2 call later up the stack. This is useful for being able to warn the user if EdenFS is running on an old or known-bad version.
I ended up splitting up `getEdenfsVersion()` and `getEdenfsBuildName()` because I don't think we need to store "edenfs " with every version number in the server / return this from the thrift call of `getScmStatusV2()`. Not sure of the use case of having "edenfs " prepended to the version number returned from the current `getEdenfsBuildName()` so didn't want to change that, but if it seems fine we could remove that prepended edenfs (or if it actually makes sense to return the string with that prepended from the ScmStatus call).
Reviewed By: chadaustin
Differential Revision: D17919528
fbshipit-source-id: 5abdf7851371c4614da862657350a8cc263eb014
Summary: We should handle errors from Rust code gracefully so it does not abort the mounting process.
Reviewed By: chadaustin
Differential Revision: D18096335
fbshipit-source-id: 17cec6f619ea2cc8c1719b0cf97287c815ff6f0a
Summary: EdenFS recently is unable to import data from API Server on devservers. This is due to the missing of service router support during the build time. This diff fixes this issue.
Reviewed By: wez
Differential Revision: D18042201
fbshipit-source-id: 15868d8e81b9403218e481b3fbf0a7ca1279b023
Summary: This diff switches EdenFS from using the `revisionstore` crate into `backingstore`
Reviewed By: chadaustin
Differential Revision: D17866320
fbshipit-source-id: f6de1fe568ae893e4c2555039df6edfb0a450885
Summary: Introduce a new ScribeLogger class that spawns and maintains a process. Log messages are newline-delimited and written to the process's stdin. If the process stops responding or responds too slowly, log messages are dropped.
Reviewed By: pkaush
Differential Revision: D17777215
fbshipit-source-id: c998d10c73fc103122d69ae19c5d84f58b7939d2
Summary:
krallin noted that we were not collecting LFS stats from the import helper.
This diff should get us to run the import helper with those stats enabled, although
we don't yet know how useful those will be in practice.
We originally defaulted to `hg.real` to bypass some startup overhead and make our tests
run faster. We've evolved more control over this in the test harness over time, so I
think we're probably fine to make the runtime non-test default just be the normal hg
entrypoint.
Reviewed By: chadaustin
Differential Revision: D18039164
fbshipit-source-id: 7593cdca9afe30fa2fc928b84ba552dbecfed388
Summary: This makes a `diff()` function definition private inside EdenMount in order to solidify use of default parameters and to consolidate callsites.
Reviewed By: chadaustin
Differential Revision: D17962075
fbshipit-source-id: 2a6d4213f98828f4ebe0ba6db43211a96d1b2bc8
Summary:
The use of futures in the checkout code was a bit confusing. Before I
make further changes, refactor that.
Reviewed By: genevievehelsel
Differential Revision: D17985158
fbshipit-source-id: 26889ee85dc5704bc61347703dae370d4529d67b
Summary: Removing this DCHECK for the searchExpression because we can recieve requests for empty string and is a valid request.
Reviewed By: chadaustin
Differential Revision: D17985604
fbshipit-source-id: 40ffbe153a7ae44e85ed7d7ad7b2e79affb1d529
Summary: Added a multi-byte version of toString() for the guid class. Also renamed the existing toString() to toWString() which returns the widechar string for the GUID.
Reviewed By: chadaustin
Differential Revision: D17985500
fbshipit-source-id: fb3d976e39c0583122e47e3e444cf0f68e6a9544
Summary:
When we implemented FUSE_NO_OPEN_SUPPORT and FUSE_NO_OPENDIR_SUPPORT,
we forgot to remove the FileHandle and FileHandleBase code.
Reviewed By: pkaush
Differential Revision: D17991710
fbshipit-source-id: dfeb26d512f017cef7710929ccff1f6940cf8641
Summary: Removing an import that is not really used.
Reviewed By: pkaush
Differential Revision: D17997050
fbshipit-source-id: 7ba6a869937d51d640c72c22417a26fcf42c6ab3
Summary: refactor current test to enable fault injector for TestMount and remove DECLARE in c++ file.
Reviewed By: chadaustin
Differential Revision: D17882339
fbshipit-source-id: 719fcf082a9a1d1ea7a1fadc561b6f368f1ccdff
Summary: Tracing was not an accurate name for what this directory had become. So rename it to telemetry.
Reviewed By: wez
Differential Revision: D17923303
fbshipit-source-id: fca07e8447d9b9b3ea5d860809a2d377e3c4f9f2
Summary: While reading the config code, I made some minor, behavior-changing improvements.
Reviewed By: genevievehelsel
Differential Revision: D17918029
fbshipit-source-id: e40bce099a9555559c028c701f6fb75a6a4fc6be
Summary: FileMetadata will also store the SCM hash to be used later.
Reviewed By: chadaustin
Differential Revision: D17883243
fbshipit-source-id: 6121ca76f78b05f074799c0f52b568aa7805d8ab
Summary: This class is to simplify working with Windows registry and would be used in the future diffs.
Reviewed By: chadaustin
Differential Revision: D17791021
fbshipit-source-id: 76de156de67be4a007a5254efe48a14bdfefb24d
Summary:
Add a configuration knob that allows environments to opt out of Eden's
custom permission checks on the unix domain socket, instead using the
standard unix domain socket permission rules for the host OS.
Reviewed By: wez
Differential Revision: D17863456
fbshipit-source-id: c60c52891e49cc8027832bd42029cfd52752547b
Summary:
Instead of having accessors for every config setting in EdenConfig,
just expose the ConfigSettings directly.
Reviewed By: fanzeyi
Differential Revision: D17847805
fbshipit-source-id: 8c6c1010c010113cf859677449797ea916f2a2a5
Summary:
I plan to simplify access to EdenConfig by making ConfigSettings
public rather than writing accessors for each setting. To avoid
exposing implementation details, switch to private inheritance.
Reviewed By: wez
Differential Revision: D17847752
fbshipit-source-id: 02ef7afe96f09cc29a54b21bfafbef9234e6f74d
Summary: Removes `TreeDiffer` class and passes `DiffContext` through standalone `TreeDiffer` functions as first argument as per comment on D17400466 for setup for processing gitignores in the `TreeDiffer` codepath. (also this allows for easy implementation of short circut of `future_getScmStatusBetweenRevisions` similar to D17531102)
Reviewed By: chadaustin
Differential Revision: D17717977
fbshipit-source-id: d480d212474bd80aeac9cd9bb901f97562b62b13
Summary:
I found the logic for whether the config should be reloaded
unnecessarily complicated, so reduce it to deciding whether to reload
or not. This removes the need to acquire a write lock in autoreload's
common case.
Reviewed By: wez
Differential Revision: D17847699
fbshipit-source-id: 50fee1aac15cc8f896333c93459fea6510646600
Summary:
To allow downstream consumers to make authentication decisions based
on effective group, store and expose the peer effective group id.
Reviewed By: yfeldblum
Differential Revision: D17844620
fbshipit-source-id: 327437d5b36213e06ca6358266109f96761ac8dc
Summary:
Give ThriftPermissionChecker flexibility by allowing it to access
UserInfo if it needs to. This also gives ThriftPermissionChecker
access to EdenConfig for use in future diffs.
Reviewed By: wez
Differential Revision: D17844453
fbshipit-source-id: 5f7badd084bfbe82bd6b3797ef68daa7b63741b5
Summary:
Some Unix applications (notably, nfsd) create regular files using vfs_create, which ends up invoking the `mknod` system call rather than `open`, which for historical reasons only supported socket creation with Eden. However, since Eden supports regular files, we can broaden the FUSE mknod handler to support regular files as well.
For context, see https://github.com/GoogleCloudPlatform/gcsfuse/issues/137#issuecomment-155273363
Reviewed By: chadaustin
Differential Revision: D17792424
fbshipit-source-id: 466fcbcb3bcb587e731bc8b2a3e0f1508ff1f4e4
Summary:
Add test cases to ensure behavior of files that exist in the source control tree but match a gitignore pattern (in that case we should report file modification/removal if such happens and should not consult the gitignore)
from `man 5 gitignore` : Note that all the gitignore files really concern only files that are not already tracked by git
Reviewed By: chadaustin
Differential Revision: D17606492
fbshipit-source-id: 1da9f0617ecf2201eb6e548561c2de170f6184fa
Summary:
We make use of the KERN_PROCARGS2 MIB data that we can
retrieve via `sysctl`.
If we can't retrieve that data then we fall back to libproc as
we were doing previously. From my testing so far it seems like
the main reason for failure is that the target process is a
protected system process.
Reviewed By: chadaustin
Differential Revision: D17724101
fbshipit-source-id: 8de1a978e6f89612bfe247e0fd540d9078f50746
Summary: This will help us identify the calling process for the FS request. This was helpful while testing watchman crawl in Eden repo.
Reviewed By: chadaustin
Differential Revision: D17719028
fbshipit-source-id: 0d646d7949a6e26268b71287b26d63473a9a87f8
Summary:
We are seeing random segment fault originating from OpenSSL on macOS when
Mononoke fetching is enabled.
The cause is that on macOS we are actually linking against libcurl shipped with
the system instead of ours. That copy of libcurl is linked with macOS's
libcrypto instead of the one we compiles during Eden's build, and it seems that
version of libcrypto does not provide concurrency safety.
The solution is to build curl on macOS and make sure it is linked to our
OpenSSL that has the concurrency callbacks registered.
Reviewed By: wez
Differential Revision: D17657822
fbshipit-source-id: 85abdf3be10b3903a5efc6b3a91624c7258de790
Summary: Moved the code from vcproj testutils folder to test folder and enabled the Cmake build on it.
Reviewed By: wez
Differential Revision: D17738227
fbshipit-source-id: bcba41cb29b7e24782c36899c0977c5592888ff5
Summary:
Make it possible for any user on the system to connect to Eden's unix
domain socket. Our own permission checking is implemented in D17459971.
Reviewed By: wez
Differential Revision: D17460113
fbshipit-source-id: 29eba62010a58837df0376c66e252f53314a25dd
Summary:
To allow other user accounts to access fb303 counters, we plan to open
our unix domain socket to the world. But that requires implementing
our own permission checks, so do that first.
Reviewed By: wez
Differential Revision: D17459971
fbshipit-source-id: 3cc512da90f725d1c93f25f2a7a5ea7c72fb2b96
Summary:
Previously, readdir-triggered prefetches would merely look up and
cache the inodes for entries. Given a lookup also needs to return stat
information, including file sizes, prefetch blob sizes too.
Reviewed By: wez
Differential Revision: D17728752
fbshipit-source-id: 1c555c123fb7405f82885b427b32cd007fdb16ef
Summary:
Converting the const ptr in EdenDispatcher because we need to call non-const function (like update Journal). Also storing the reference in EdenDispatcher and WinStore instead of ptr.
This diff should not change the behavior of the Eden.
Reviewed By: chadaustin
Differential Revision: D17719606
fbshipit-source-id: 6c371c38045c760837130bb607163c2bc469b17a
Summary: This allows us to test the datapack code easier without rebuilding Eden.
Reviewed By: wez
Differential Revision: D17468473
fbshipit-source-id: a6807b4d6e747ae8557ae51fdf798de2a54fd4f1
Summary:
When Eden is competing with hundreds of build processes using about
the same amount of RAM, it's less painful for the OOM killer to kill
the compilers, not the edenfs daemon. Bias its OOM score a little.
Reviewed By: wez
Differential Revision: D17701071
fbshipit-source-id: 5ee1078e78e5aa6251692944ca12d9a892f86a6f
Summary: Now that Eden has dropped support for flatmanifest, allow GC'ing trees in the local storage.
Reviewed By: simpkins
Differential Revision: D17236345
fbshipit-source-id: d40ec0c2e61c9971ba1d976572d002c48c7bfc20
Summary:
it turns out that buck was using this to avoid making a thrift
call to get the sha1 of files.
Let's restore the method but have it return an empty list until buck
has migrated away from that.
asp2insp is looking at fixing up that aspect of buck
Reviewed By: chadaustin
Differential Revision: D17694987
fbshipit-source-id: 9ab0771c00fc64ed3c3395948650189d50d99180
Summary: Keep track of `ResponseChannelRequest` in diff() in order to short circut function if client request is no longer active.
Reviewed By: wez, simpkins
Differential Revision: D17531102
fbshipit-source-id: 662333e5fbc2ed9b3b32b619b8d15db618ad06ec
Summary:
Force a reference to the edenfsctlPath flag, otherwise the
linker will discard it and a large number of tests will fail.
Reviewed By: simpkins
Differential Revision: D17683222
fbshipit-source-id: b7cb29e74af85b544f45a228770ad2613c8e6efc
Summary:
This diff removes the logic that consumes the legacy bind
mount list and mounts them on startup. That functionality has been
replaced with the eden redirect command.
Instead of performing the bind mounts in the server, the server will
now run `eden redirect fixup` to apply that configuration.
This diff also changes the behavior of performBindMounts: previously, if the
bind mount setup failed, we would tear down the entire repo mount. Since we're
now spawning an external process, it is much more likely that something might
fail and result in a bad experience, so we no longer bail out in that case:
we'll continue and leave the bind mounts as-is. The user can then use `eden
doctor` or `eden redirect fixup` to sort things out.
Reviewed By: simpkins
Differential Revision: D17236366
fbshipit-source-id: 8b004551a076216f0e5448942f00b5195ee18803
Summary:
The RocksDbLocalStore.cpp code updates counters, so it depends on the fb303
library.
Reviewed By: fanzeyi
Differential Revision: D17669619
fbshipit-source-id: d05bee82b05743c2ba292808ce3d77812413d2eb
Summary: `folly::ssl::init` will initialize some of the random number generator and callbacks needed for concurrency.
Reviewed By: wez
Differential Revision: D17644835
fbshipit-source-id: 8b0b4ec98b119298053d7362339ebb8bfbd30586
Summary:
Update the `Future` based version of the `ThriftLogHelper` class to log how
long the call took, even if it failed.
The code already logged on error for non-Future thrift handler functions, but
for `future_*()` calls we only logged when producing a successful result.
This switches the code from `thenValue()` to `thenTry()` so that we log on
both successful and failed results.
Reviewed By: wez
Differential Revision: D17636471
fbshipit-source-id: 859a326e5bc54183c80f16bf463ba6d451f624e5
Summary:
D17623550 deleted `ServiceCacheIf::invalidateSelection()` but did not remove
this override, which broke the build for EdenFS's tests.
Reviewed By: wez
Differential Revision: D17638351
fbshipit-source-id: 200ead0f79a2252dc7ac2318745cd98e520737a4
Summary:
Update the logging for `getFileInformation()`, `glob()`, `globFiles()`, and
`getSHA1()` to log at most 5 paths, plus the number of remaining paths.
Watchman can sometimes call `getFileInformation()` with tens of thousands of
paths. We don't enable these log messages by default, but if someone has
turned up the log levels to help debug, this can easily fill up the log files
with spam.
Reviewed By: wez
Differential Revision: D17623234
fbshipit-source-id: 3ea8ace294fb8f042f1c53382c76148ba4af95f6
Summary:
Add a CMake option to control whether or not we should build support for
fb-mercurial (aka Eden SCM). If this is disabled we avoid building anything
under eden/fs/store/hg and drop support for the "hg" backing store.
Reviewed By: wez
Differential Revision: D15980320
fbshipit-source-id: 23a49d3e5cf89199666ff4a0bf46626502c12171
Summary:
On a machine with one core, it's quite possible for the process to
freeze for a second, resulting in a flaky test.
Reviewed By: wez
Differential Revision: D17579535
fbshipit-source-id: a3e158b5895c06bdf560ebecd849a74e4efa591e
Summary: Use callback to save ScmStatus instead of storing status inside a `TreeDiffer` object. Involes a bit of restructuring of some code to avoid circular dependencies and library creation. Mostly renames and file moves, some funtion moves as well.
Reviewed By: simpkins
Differential Revision: D17400466
fbshipit-source-id: fcd194a4c20204dffd3d11cd4a083564dc0ea938
Summary:
Increase the log level if a FUSE request timeout occurs, so we can tell if
this has occurred in the logs when debugging user problems. Rate limit the
messages to at most once per second to avoid lots of log spam when a problem
occurs that causes lots of FUSE requests to time out.
Reviewed By: chadaustin
Differential Revision: D17506837
fbshipit-source-id: b84bd094034f1b31ef74038a9fd3828d4db9acf2
Summary: Use callback to save ScmStatus instead of storing status inside a `TreeDiffer` object.
Reviewed By: simpkins
Differential Revision: D17379318
fbshipit-source-id: b5f56c937509f0861f5dc06c742ebc3e1f85c695
Summary:
Add a CMake option to control whether or not we should build support for
Git. If this is disabled we avoid building anything under eden/fs/store/git
and drop support for the "git" backing store.
Reviewed By: wez
Differential Revision: D15980321
fbshipit-source-id: 434364d81b44935ce86fdf4d66697ee21ff2992f
Summary:
The folly's version of fileutil's throw's "permission denied" error on Windows while creating the config file. Updating the code to call the Win32 version from eden/fs/win/utils.
We do eventually plan to fix Folly by moving the Eden Windows's FileUtils code to it. Until then we will call it from Eden's utils.
Reviewed By: fanzeyi
Differential Revision: D17490623
fbshipit-source-id: 948c488f1dfddd7b12eb8623e056a7680fd5aa73
Summary:
The code in `EdenConfigChecks.cmake` had two separate `find_package(cpptoml)`
calls, one using its installed CMake config file, and one using a custom
`Findcpptoml.cmake` module.
This removes the custom `Findcpptoml.cmake` code, and updates everything to
only used cpptoml's installed CMake configuration.
Reviewed By: chadaustin
Differential Revision: D17401220
fbshipit-source-id: 3789703cdfc029049db3b1bd9f5751fa2a60a8d4
Summary:
I was experimenting with using direct mononoke fetches from my mac but
was hitting 400 errors. It turned out that we were resolving the VIP name and
using the resolved host in the generated URL. The VIP configuration doesn't
know how to route such a request and yields a 400 response.
The SocketAddressWithHostname type has enough information to allow for handling
this correctly, but we were discarding the hostname component!
This diff captures the entire SocketAddressWithHostname type in the `address_`
member and uses the hostname portion when generating the URL.
Reviewed By: fanzeyi
Differential Revision: D17470256
fbshipit-source-id: 14ebd517eac12e07248a64fb37c0623ecbe37e89
Summary: Remove all of the importer code that supported flatmanifest repositories.
Reviewed By: simpkins
Differential Revision: D15057249
fbshipit-source-id: 1581712c3ab9a5daeb70e61cceb5de013f16f5e1
Summary:
The Catalina beta reports `10.15` and we were throwing an
exception when we didn't match three version number components.
Allow for 2 or more components to match when scanning the string.
Reviewed By: fanzeyi
Differential Revision: D17490838
fbshipit-source-id: 240513f21c0f57e629a300a35a815e7a572e93ff
