Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary:
s/CURLE_SSL_CONNECT_ERROR/CURLE_RECV_ERROR/.
Note CURLE_SSL_CONNECT_ERROR is not explicitly checked in
code in the eden hierarchy (or anywhere else significant in fbcode).
Reviewed By: meyering
Differential Revision: D20490844
fbshipit-source-id: be4a66f49eb4a9eabaf73785f9a203f0aa6905a4
Summary: This diff bumps the open call from FUSE to High priority (which is higher than any other blob open request atm). This has shown improvement on the user experience of EdenFS when it's importing many other things from other channels (thrift, etc.)
Reviewed By: chadaustin
Differential Revision: D20287389
fbshipit-source-id: 319bc44ef8be5c904d7cf0db7cc2f8be28b4760a
Summary: This diff implements `HgQueuedBackingStore` that uses `HgImportRequestQueue` to provide SCM data importing with priorities. This can allow us to customize how we import things and batch importing.
Reviewed By: chadaustin
Differential Revision: D19184826
fbshipit-source-id: da579b5bbff0b1449e9689e2c0159d4a3a475a83
Summary: This diff adds `HgImportRequestQueue` that is responsible for managing incoming requests by their priorities. This queue is later used in the `HgQueuedBackingStore` to prioritize works to the workers of the backing store.
Reviewed By: chadaustin
Differential Revision: D20197069
fbshipit-source-id: 246bbc086054a8021226e9ba6ab26d3bf0cfb7a3
Summary:
This class is used to represent an import request that will be used later in the queue implementation.
When the EdenFS needs to import a blob, it creates an instance of this request and send it to the worker. Then it waits for the promise associated with the request.
In the future, we should be able to change the owned `Promise` into a non-owned `SemiFuture` to a `Promise` somewhere else for merging repetitive import requests.
Reviewed By: chadaustin
Differential Revision: D19184824
fbshipit-source-id: 823aabbed1156acf6306b7aefc76580a540d310d
Summary: This diff adds `Priority` added in the previous diff to the `BackingStore` interface with the default value set to `Priority::Normal`.
Reviewed By: chadaustin
Differential Revision: D20197071
fbshipit-source-id: a92f1b49bb82e3478042e5e3b79b047d834755ea
Summary:
This diff introduces a `Priority` type for EdenFS. This type is used to pass along the priority of a request.
The priority class itself contains two parts, `kind` and `offset`. `kind` uses the first 4-bytes and the reset 12-bytes are used to store offset. The idea is that we can roughly assign a priority kind to most of the requests and offset is used to dynamically tweak the priority of some particular requests. For example, when we saw a process is generate millions of requests we can use this to express "normal priority but less important than other process's normal priority".
Reviewed By: chadaustin
Differential Revision: D20287652
fbshipit-source-id: 9a849fb6cc6ba5e443fea978d5b4dc3ab8ca906a
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary: expose the counters for number of pending imports (blobs, trees, prefetches) to allow use in tooling
Reviewed By: chadaustin
Differential Revision: D20269853
fbshipit-source-id: d2b7e2110520290751699c4a891d41ebd5b374cf
Summary:
adds a counter to track the imports queued to enable more statistics exposure.
- Add a counters to track the number of blob, tree, prefetch imports that are in the pending
- have the counters increment (increment in constructor of wrapper struct) when the import is about to be queued
- have counters decrement once the load has completed (decrement in destructor of wrapper struct)
Reviewed By: chadaustin
Differential Revision: D20256410
fbshipit-source-id: 5536b46307b30fc19dc5747414727a86961c78e1
Summary: - added logging only around the import tree call to capture non-queue related wait time
Reviewed By: chadaustin, fanzeyi
Differential Revision: D20207472
fbshipit-source-id: d88bb34ce224a26ff2be100d7789ddeff608006d
Summary:
- added logging only around the import blob call to capture non-queue related wait time
- added to `test_reading_file_gets_file_from_hg` in `integration.stats_test.HgBackingStoreStatsTest` to test import blob logging in addition to the get blob loging
(not yet done for importing trees, will do in next diff)
Reviewed By: chadaustin
Differential Revision: D20201215
fbshipit-source-id: c89281fe7d3d6e89d111ac8cce9014adff44ac40
Summary:
A bunch of files include folly/executors/GlobalExecutors.h transitively through thrift/lib/cpp2/async/Stream.h, which is going away. Explicitly include the header (and add dependency to target) in preparation for deleting Stream.h
drop-conflicts
Reviewed By: vitaut
Differential Revision: D20141838
fbshipit-source-id: 21c58cf82136287fc2d84ba5badec6b872106015
Summary:
The problem is that the datapack files are not flushed to disk when it is prefetched. By having a pair of brackets around the `HgBackingStore`, it will ensure the `HgImporter` is closed by the time when we verify the prefetch with `hg cat` since it will terminate the `debugedenimporthelper` process in its destructor, which flushes the datapack files.
The real cause of the test failure is still unclear but I believe this is the correct way of doing this test.
Reviewed By: xavierd
Differential Revision: D20090249
fbshipit-source-id: 8e3966936a402c92311919433282027846d065e8
Summary: so that it can be shared more easily with AsyncUDPSocket
Reviewed By: yangchi
Differential Revision: D19851480
fbshipit-source-id: ec8cdb852519724db6f89cf70c4a4169de5028b6
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
Summary: In addition to duration and success, log object fetch counts and checkout type to Scuba.
Reviewed By: fanzeyi
Differential Revision: D19334276
fbshipit-source-id: dabf52427f2ebda2b58df93194df39d52f4fcb4f
Summary: Log the number of object lookups and cache hit rates for a checkout operation.
Reviewed By: simpkins
Differential Revision: D19191201
fbshipit-source-id: 5e9ad501e704810f072dabcda3fce86d027c452e
Summary:
During checkout and stats, count every object fetch and which level of
cache it was served from.
Reviewed By: simpkins
Differential Revision: D19186333
fbshipit-source-id: fc0a74db297b9c723682e245996a7befd762f933
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
There were some unsafe uses of folly::collect, especially in
the diff code. Switch to collectSafe.
Reviewed By: simpkins
Differential Revision: D19529784
fbshipit-source-id: e0ac920a2d2b44767a453f2c41a9166e72246ab6
Summary: the request will be passed with the context, so simply check the context at all three "looping" points to see if the request was cancelled. I split this into its own diff due to its unique test plan.
Reviewed By: chadaustin
Differential Revision: D18647091
fbshipit-source-id: ccb7ac765f8f3b0a60e936779cc27b658b9fd8b9
Summary: Pass a GitIgnoreStack* and isIgnored flag through the diff operation. It is used later in the stack when we go down this code path in the TreeInode diffing code. When context->loadFileContentsFromPath() is null, the gitignore loading code will be ignored, used when we do not want to honor a gitignore file. For example, during `getScmStatusBetweenRevisions()`.
Reviewed By: chadaustin
Differential Revision: D18647089
fbshipit-source-id: 20d2abd2ef61669465e134165da5a0ac5e987cca
Summary: I reproduced the KeySpace.cpp ICE on my Windows 10 VM, so bump the MSVC version check. I have not confirmed it does not ICE on MSVC 2019.
Reviewed By: pkaush
Differential Revision: D19562325
fbshipit-source-id: 03cf84e95c709bb2ebe7c3e0055a4f290eb17f8a
Summary:
It's not often obvious to people where edenfs.log is located on the
filesystem, so redirect people to `eden debug log` instead.
Reviewed By: simpkins
Differential Revision: D19560709
fbshipit-source-id: 6b031c80bab2c21dfddba4d7ec37d4e70218ea08
Summary:
Now that loading too many blobs only GCs the blob cache, treat cached
trees as ephemeral again.
Reviewed By: simpkins
Differential Revision: D19358375
fbshipit-source-id: 191427f46276f0f5d024b5bc923a68f7a42044da
Summary:
Instead of clearing every single cached object when the total size
exceeds the ephemeral storage limit, keep a limit per object type and
only clear those that exceed their quota.
Reviewed By: simpkins
Differential Revision: D19358312
fbshipit-source-id: 6918d6f4cc2931aed79a9025d0e0f357ede515e0
Summary:
EdenFS no longer uses the blobsize keyspace. To avoid wasting space,
clear and campact it on startup, along with any keyspaces we
deprecated in the future.
Reviewed By: fanzeyi
Differential Revision: D19354881
fbshipit-source-id: 5285757a0e44ab1080c5f940283e06b17bec811d
Summary: Simplify the definition and use of KeySpace and move it into its own header.
Reviewed By: simpkins
Differential Revision: D19353441
fbshipit-source-id: ef07677d927a48839b709711388abeb3c1ed9679
Summary: This diff addresses the unreleased mapped pack files in the old datapack code.
Reviewed By: chadaustin
Differential Revision: D19399737
fbshipit-source-id: 86a6254a2939fd69e1ce2b25b8bfcb36b0deb16b
Summary:
As reported by JT, EdenFS may hold the file descriptor of mapped pack files too long even when it is deleted by external processes, thus taking more disk spaces.
This diff fixes this problem by making EdenFS periodically rescan the pack files.
Reviewed By: chadaustin
Differential Revision: D19395439
fbshipit-source-id: 4bfd6a7ac13dceb3099d2704d62b3825433aff4b
Summary:
We want to have a shared pool of `HgImporter` across different threads, but that would require `HgImporter` is safe to be *passed* between threads. (This is not making `HgImporter` entirely thread safe.)
However, `HgImporter` currently holds a pointer to the thread local fb303 counter. This diff pushes down `EdenStats` so we only access the thread local stats variable when we need to add counters. This might be a little slower but it does make it safer.
Reviewed By: chadaustin
Differential Revision: D19053250
fbshipit-source-id: 44a897acc90c6042ae22a0417eece39e099ee13f
Summary: The only reason `HgImporter` still holds a pointer to LocalStore is for looking up proxy hashes. We can directly pass in the Mercurial hash and path instead.
Reviewed By: chadaustin
Differential Revision: D19049039
fbshipit-source-id: 45d1e1f363ed73dca447b22e5891495cf9ad202b
Summary:
Remove some half-baked, unnecessary logic for caching sizes separately
from SHA-1. Eden's backing stores do not support chunking large files
yet, so there's no value in caching content SHA-1 and size
separately. This fixes a scenario where fetching blob size and then
SHA-1 would result in two backing store imports.
Reviewed By: fanzeyi
Differential Revision: D19169096
fbshipit-source-id: dc32f3313e5f4230c06a5bbaa67da7bf0febaba8
Summary:
Troditionally, ObjectStore relied on HgBackingStore writing directly
to LocalStore in order to cache trees. This had the unfortunate side
effect that other backing store implementations did not benefit from
tree caching.
Move tree caching into ObjectStore so all backing stores benefit from
tree caching.
Reviewed By: simpkins, fanzeyi
Differential Revision: D19168211
fbshipit-source-id: b1019591ebb4760cc8b933b9adb82174b8e5fa1f
Summary:
If a GC ran at the same time as HgBackingStore imported a commit, the
import would fail. Rather than round-tripping the tree through the
LocalStore, just return it.
Reviewed By: genevievehelsel
Differential Revision: D19167798
fbshipit-source-id: eddb280def349483289d2e300a58eff3ad7416e1
Summary:
This diff turns the return type of `BackingStore::getBlob` from `folly::Future` into `folly::SemiFuture` to prevent executor leaks.
This also enable us to remove the need of holding `serverThreadPool` from backing stores.
----
**Changes**
* `ObjectStore` now needs to hold a `folly::Executor::KeepAlive` that is used to turn `SemiFuture`s it gets from backing stores into `Future`.
* Signature changes of the implementations of `BackingStore` class.
* For tests, I chose to use `QueuedImmediateExecutor` in place of `UnboundedQueueExecutor` as it will basically execute tasks inline. I'm concerned introducing thread pool executor in tests may turn tests flaky.
Reviewed By: wez
Differential Revision: D18669664
fbshipit-source-id: 0cae89f365dcf8b345b49d64469a530cf25d4ac5
Summary:
The last remaining user of ObjectStore::getBlobMetadata was a debug
Thrift call. Remove it and update the Thrift call to use getBlobSize
and getBlobSha1.
Reviewed By: pkaush
Differential Revision: D18663376
fbshipit-source-id: 86baefc9004a07aac4ddf5849870431be04c75f2
Summary:
A spike in automatic GCs usually implies something has gone wrong. Log
an event for each one, recording the cache size prior to the GC and
the cache size after.
Reviewed By: simpkins
Differential Revision: D18902580
fbshipit-source-id: 158b2635733a415a9fcc7c412b2c0f44ed04aa01
Summary: Enabling getScmStatusBetweenRevisions to work on Windows and its tests.
Reviewed By: simpkins
Differential Revision: D18431271
fbshipit-source-id: eee82538e2fc3d7e371c96fc271cd9662ea6d737
Summary:
Add an option `experimental:use-edenapi` to `EdenConfig`.
See the next diff for usage.
Reviewed By: chadaustin
Differential Revision: D18605549
fbshipit-source-id: 2786c21bb38a76229078662cc5c1ddf906d1be4a
Summary: Adds a function which takes both the manifestID and the commitID to get a Tree. This will be used in `checkOutRevision()` and this allows us to skip looking up the manifestID since the caller can just pass it in themselves.
Reviewed By: wez
Differential Revision: D18719405
fbshipit-source-id: 919f0a7c84bff4a2f0bc20110c45bd272f9e9107
Summary:
It is very painful for Eden's automatic GC to delete trees from the
local store. An unfortunate number of programs scan the entire repo,
causing every tree to be loaded. If Eden's automatic GC kicks in, it
causes all trees to be downloaded again, blocking both FUSE and Thrift
requests.
Temporarily mark trees as persistent.
Reviewed By: fanzeyi
Differential Revision: D18842066
fbshipit-source-id: 4a22defcb1151912337d4167ac49852519c7789d
