Summary: the request will be passed with the context, so simply check the context at all three "looping" points to see if the request was cancelled. I split this into its own diff due to its unique test plan.
Reviewed By: chadaustin
Differential Revision: D18647091
fbshipit-source-id: ccb7ac765f8f3b0a60e936779cc27b658b9fd8b9
Summary:
In any case in which our Inode is not materialized and we would like to continue to diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
Also, this allows us to remove the RemoveDiffEntry since that only looked at the scmEntry, so that work can also be deferred to the source control tree differ easily.
This avoids loading inodes when we don't need to. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D19292528
fbshipit-source-id: 7b5ec161f7bd65962ae30c49fdad6dbc73954c2a
Summary:
In any case in which our Inode is not materialized and we would like to continue to diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
This reduces the number of inodes loaded by avoiding to load them if we don't need to. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D19292527
fbshipit-source-id: 2dbf6c4130214f581b6cc09e663f7567abab6ef3
Summary:
In any case in which an inode is not materialized and we are computing a diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
This avoids loading inodes when not necessary. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D18647087
fbshipit-source-id: 74632de777f04e2468d10977cc757b8993d439e9
Summary: Pass a GitIgnoreStack* and isIgnored flag through the diff operation. It is used later in the stack when we go down this code path in the TreeInode diffing code. When context->loadFileContentsFromPath() is null, the gitignore loading code will be ignored, used when we do not want to honor a gitignore file. For example, during `getScmStatusBetweenRevisions()`.
Reviewed By: chadaustin
Differential Revision: D18647089
fbshipit-source-id: 20d2abd2ef61669465e134165da5a0ac5e987cca
Summary: Produce a -Wdeprecated-declarations warning in code that uses the old SYNCHRONIZED macros.
Reviewed By: yfeldblum
Differential Revision: D19567088
fbshipit-source-id: b0637c06cbd125d09e1b39eec1e82a71ddca2324
Summary:
In the case that the overlay was closed before the initial
createOverlayFile, prevent an invalid OverlayFile from being accessed,
causing the test to fail.
Reviewed By: genevievehelsel
Differential Revision: D19591636
fbshipit-source-id: 9d961f044c3b68b5c0b2dcd108ff85db6326276d
Summary: I reproduced the KeySpace.cpp ICE on my Windows 10 VM, so bump the MSVC version check. I have not confirmed it does not ICE on MSVC 2019.
Reviewed By: pkaush
Differential Revision: D19562325
fbshipit-source-id: 03cf84e95c709bb2ebe7c3e0055a4f290eb17f8a
Summary: This code is no longer used now that `eden redirect` exists.
Reviewed By: wez
Differential Revision: D19565010
fbshipit-source-id: 4703bc39a024c4c491a83ef5a6e85711a8f7e4d8
Summary:
It's not often obvious to people where edenfs.log is located on the
filesystem, so redirect people to `eden debug log` instead.
Reviewed By: simpkins
Differential Revision: D19560709
fbshipit-source-id: 6b031c80bab2c21dfddba4d7ec37d4e70218ea08
Summary:
Now that loading too many blobs only GCs the blob cache, treat cached
trees as ephemeral again.
Reviewed By: simpkins
Differential Revision: D19358375
fbshipit-source-id: 191427f46276f0f5d024b5bc923a68f7a42044da
Summary:
Instead of clearing every single cached object when the total size
exceeds the ephemeral storage limit, keep a limit per object type and
only clear those that exceed their quota.
Reviewed By: simpkins
Differential Revision: D19358312
fbshipit-source-id: 6918d6f4cc2931aed79a9025d0e0f357ede515e0
Summary:
EdenFS no longer uses the blobsize keyspace. To avoid wasting space,
clear and campact it on startup, along with any keyspaces we
deprecated in the future.
Reviewed By: fanzeyi
Differential Revision: D19354881
fbshipit-source-id: 5285757a0e44ab1080c5f940283e06b17bec811d
Summary:
Indicate getManifestEntry is deprecated so we remember to delete it
down the line.
Reviewed By: genevievehelsel
Differential Revision: D19459793
fbshipit-source-id: 86dd173cfb12c36cf6ba19eaf4be96dfe624c05f
Summary: Simplify the definition and use of KeySpace and move it into its own header.
Reviewed By: simpkins
Differential Revision: D19353441
fbshipit-source-id: ef07677d927a48839b709711388abeb3c1ed9679
Summary: This diff addresses the unreleased mapped pack files in the old datapack code.
Reviewed By: chadaustin
Differential Revision: D19399737
fbshipit-source-id: 86a6254a2939fd69e1ce2b25b8bfcb36b0deb16b
Summary:
As reported by JT, EdenFS may hold the file descriptor of mapped pack files too long even when it is deleted by external processes, thus taking more disk spaces.
This diff fixes this problem by making EdenFS periodically rescan the pack files.
Reviewed By: chadaustin
Differential Revision: D19395439
fbshipit-source-id: 4bfd6a7ac13dceb3099d2704d62b3825433aff4b
Summary:
stderr: eden/fs/inodes/EdenMount.cpp:1122:37:
error: local variable 'unmountError' will be copied despite being thrown by name [-Werror,-Wreturn-std-move]
eden/fs/inodes/EdenMount.cpp:1122:37: note: call 'std::move' explicitly to avoid copying
Reviewed By: simpkins
Differential Revision: D19408659
fbshipit-source-id: 1575104414daaeb06892586492cdb74e2df604c5
Summary: based on comment on D18647089, changes the behavior when a directory is replaced with a file, and then the directory is ignored. The behavior should be that the file is reported as `ADDED`, previously the file would have been reported as `IGNORED`
Reviewed By: simpkins
Differential Revision: D19318796
fbshipit-source-id: 8da1b7c6df182ca591e4aea21f6573ca81790cb4
Summary:
We want to have a shared pool of `HgImporter` across different threads, but that would require `HgImporter` is safe to be *passed* between threads. (This is not making `HgImporter` entirely thread safe.)
However, `HgImporter` currently holds a pointer to the thread local fb303 counter. This diff pushes down `EdenStats` so we only access the thread local stats variable when we need to add counters. This might be a little slower but it does make it safer.
Reviewed By: chadaustin
Differential Revision: D19053250
fbshipit-source-id: 44a897acc90c6042ae22a0417eece39e099ee13f
Summary: The only reason `HgImporter` still holds a pointer to LocalStore is for looking up proxy hashes. We can directly pass in the Mercurial hash and path instead.
Reviewed By: chadaustin
Differential Revision: D19049039
fbshipit-source-id: 45d1e1f363ed73dca447b22e5891495cf9ad202b
Summary: scuba logging for mismatched parent commits, it would be nice to have a count on how often this is encountered so we can pinpoint spikes of this case. I am not sure how helpful having the parent commits would be in the event, my thought there was that it could help see how far away the commits are and if one of the commtis doesn't exist (like in the case of stripping commtis with a cancelled hg split), but I am open to having an empty event as well if storing these strings is too expensive.
Reviewed By: fanzeyi
Differential Revision: D19432378
fbshipit-source-id: fa3e7cd6aef832c4f918f339b781590b7484cfdc
Summary:
There is a bug in Projected FS on Windows 1903 and 1909, which BSODs the system on SetInformation call. Projected FS seems to be dereferencing a nullptr while attempting to check if the file is in the backing store. Providing a separate callback to check if the file exists in the backing store will avoid the buggy code (suggested by Microsoft).
This will enable Eden FS on Windows 1903 and 1909.
Reviewed By: simpkins
Differential Revision: D19377522
fbshipit-source-id: ab5e9bc8e2472f3e689d71d05a560bac86211295
Summary: This is a followup on D18666585.
Reviewed By: chadaustin
Differential Revision: D19377523
fbshipit-source-id: f9ab4011d0978a6934de77d1837a37082603cf52
Summary: We missed updating this in D18669664.
Reviewed By: simpkins
Differential Revision: D19377524
fbshipit-source-id: e23fb039047855454b4f12353bc375b1a2c8d962
Summary: `folly::cold_detail::ColdClass` was marking things (like `folly::Unexpected`) cold, but at the cost of inhibiting the inliner from doing its job. This is leading to bad codegen, which offsets any small wins we mind get for the `cold` attribute.
Reviewed By: yfeldblum
Differential Revision: D19324159
fbshipit-source-id: 7ed431b6c9d6e963c3bf438c707fa6cf6a38bf9d
Summary:
D19135344 broke EdenFS's cmake builds, and our macOS releases haven't
been going out. Rather than pass the same streaming options to the
Thrift compiler in the CMake build, remove deprecated_server_streams
from both builds.
Reviewed By: simpkins
Differential Revision: D19204397
fbshipit-source-id: c22e634ded82a683daff5c92182c8a1eb1ed4d21
Summary: This stack is deprecated in favor of the rocket stack.
Reviewed By: rhodo
Differential Revision: D18123600
fbshipit-source-id: db6e3bf4add9c49cf8f1d2f12f97b207e87c7302
Summary:
Reduce the log level for a few messages to help reduce some less important
clutter when running with an elevated debug log level.
Reviewed By: chadaustin
Differential Revision: D19188773
fbshipit-source-id: 396bb15e119fc12765ecbc707e7a1dadbdd02422
Summary:
Remove some half-baked, unnecessary logic for caching sizes separately
from SHA-1. Eden's backing stores do not support chunking large files
yet, so there's no value in caching content SHA-1 and size
separately. This fixes a scenario where fetching blob size and then
SHA-1 would result in two backing store imports.
Reviewed By: fanzeyi
Differential Revision: D19169096
fbshipit-source-id: dc32f3313e5f4230c06a5bbaa67da7bf0febaba8
Summary:
Troditionally, ObjectStore relied on HgBackingStore writing directly
to LocalStore in order to cache trees. This had the unfortunate side
effect that other backing store implementations did not benefit from
tree caching.
Move tree caching into ObjectStore so all backing stores benefit from
tree caching.
Reviewed By: simpkins, fanzeyi
Differential Revision: D19168211
fbshipit-source-id: b1019591ebb4760cc8b933b9adb82174b8e5fa1f
Summary:
If a GC ran at the same time as HgBackingStore imported a commit, the
import would fail. Rather than round-tripping the tree through the
LocalStore, just return it.
Reviewed By: genevievehelsel
Differential Revision: D19167798
fbshipit-source-id: eddb280def349483289d2e300a58eff3ad7416e1
Summary: There is one instace of `getScmStatusBetweenRevisions` in use - it is used in the eden cli in a hacky way to check if a commit hash is valid. Since this is not used anywhere else in a meaningful way, this replaces that use case with a hg call and depreciates `getScmStatusBetweenRevisions`
Reviewed By: simpkins
Differential Revision: D18690026
fbshipit-source-id: 02bd2c20a0f631ec41116f9fd4e18d14369298ef
Summary:
This diff turns the return type of `BackingStore::getBlob` from `folly::Future` into `folly::SemiFuture` to prevent executor leaks.
This also enable us to remove the need of holding `serverThreadPool` from backing stores.
----
**Changes**
* `ObjectStore` now needs to hold a `folly::Executor::KeepAlive` that is used to turn `SemiFuture`s it gets from backing stores into `Future`.
* Signature changes of the implementations of `BackingStore` class.
* For tests, I chose to use `QueuedImmediateExecutor` in place of `UnboundedQueueExecutor` as it will basically execute tasks inline. I'm concerned introducing thread pool executor in tests may turn tests flaky.
Reviewed By: wez
Differential Revision: D18669664
fbshipit-source-id: 0cae89f365dcf8b345b49d64469a530cf25d4ac5
Summary:
Add a config option to restrict the number of tree prefetches that can run in
parallel. Without this applications that recursively walk a checkout tree can
end up spawning a huge number of asynchronous prefetch operations. These
prefetch operations are quite expensive today, as we currently have to fetch
the full file contents in order to retrieve the file sizes. A recursive
directory walk can end up building up a huge backlog of in-progress tree
prefetches. This will slow down the directory walk, and it can take tens of
minutes to finish processing the prefetch backlog even after the directory
walk is aborted.
Reviewed By: chadaustin
Differential Revision: D19136685
fbshipit-source-id: cdc7a570d016fd7ca81a639cef83380b197acdfc
Summary:
EdenMount offered both getRootTree and getRootTreeFuture. Replace the
former with the latter and adjust remaining callers.
Reviewed By: genevievehelsel
Differential Revision: D18666585
fbshipit-source-id: 961361a1fb545e48db0ece970bb5e6ffbca8efef
Summary:
The last remaining user of ObjectStore::getBlobMetadata was a debug
Thrift call. Remove it and update the Thrift call to use getBlobSize
and getBlobSha1.
Reviewed By: pkaush
Differential Revision: D18663376
fbshipit-source-id: 86baefc9004a07aac4ddf5849870431be04c75f2
Summary: Fix some warnings that only show up in the macOS build.
Reviewed By: fanzeyi
Differential Revision: D19053236
fbshipit-source-id: 81f7187b263e0db6a57582677088519f9b97f1d7
Summary:
D18820280 increased the log level for `globFiles()` calls from DBG3 to DBG2.
However, buck and other tools often end up huge numbers of glob calls, and
this ends up causing log spam.
This change reduces the log level of `globFiles()` back to DBG3. In a
subsequent diff I will update the code to increase the log level for slow
calls. This does mean that we will only get a log message for slow glob calls
after the call completes, but this is better than not getting a message at
all.
Reviewed By: pkaush
Differential Revision: D18924780
fbshipit-source-id: 86e83614f51424aca078e4de8a56c45018a6d286
Summary: was looking in this area and saw this TODO. moves the `setParentCommits()` call into `CheckoutConflict::finish()` so it is called while we are still holding the parents lock.
Reviewed By: chadaustin
Differential Revision: D18668329
fbshipit-source-id: 8415c792193e5b89737f15141f33c5a9799f527b
