Summary:
off_t is not suitable on Windows because it is a 32-bit signed
type. Introduce a new FileOffset type for use in EdenFS.
Reviewed By: kmancini
Differential Revision: D47406784
fbshipit-source-id: 9cc9585b4b118c54a1ca148c2ba7e7ccfb6f278c
Summary:
clang-tidy had some automated suggestions for our code. Apply the ones
that make sense.
Some of them didn't, like removal of all uses of `volatile`. I
manually reverted those changes.
Reviewed By: genevievehelsel
Differential Revision: D41051052
fbshipit-source-id: 3fe22a91e929d3bb8e6346126c2c7bf9f027eb32
Summary:
Now that ProcessId exists, we should use it instead of pid_t in
ObjectFetchContext.
Reviewed By: genevievehelsel
Differential Revision: D42037216
fbshipit-source-id: 34cd89f78be35a15d73b26edc840e917fd642723
Summary:
When implementing Mercurial filtering logic for the FilteredBackingStore, I realized that simply having a FilterCallback function would be insufficient for most filtering implementations.
There's global state that one may want to keep track of, validation logic, and possible caching that can be done by the filter handler.
Therefore, in this diff I'm introducing the Filter class. Each FilteredBackingStore user will implement their own Filter class to define how filtering is done. These custom Filter classes can add things like validation, caching, etc.
The next diff introduces the HgSparseFilter class and tests it.
Reviewed By: genevievehelsel
Differential Revision: D47161201
fbshipit-source-id: 2bf119ad8a2d01274b272f03ffb5b130e030a0ca
Summary:
We've seen a case where a large `hg update` was taking an absurdly long time in
`ObjectStore::getTree` but the telemetry was showing us that most of the time
wasn't spent fetching trees from Mercurial! The suspicion is that most of the
time was spent in the LocalStore but with no evidence to prove it.
Let's thus add some timing telemetry to various LocalStore read requests to
fill this gap.
Reviewed By: mshroyer
Differential Revision: D46154456
fbshipit-source-id: b439ac48889ed3db71db136ff6c1cc91f48c926a
Summary:
EdenFS contains several layers of caching for BlobMetadata:
- In the hgcache,
- In the in-memory BlobMetadata cache,
- In the LocalStore (configurable),
- In the FileInode,
- In the TreeEntry,
All of the above must be filled prior to reading a blob, therefore computing
the sha1+size when reading a blob is likely not needed and superfluous.
Alternatively, we could move the code to be done in the background in the
LocalStoreCachedBackingStore to avoid blocking reading the blob. Reviewers
opinions are welcome!
Reviewed By: chadaustin
Differential Revision: D45625432
fbshipit-source-id: ad7d897c32cbfd590dce5a3a9c8073349e9b70c8
Summary:
In both the ObjectStore and in the hg BackingStore, copies of the unique_ptr
were being made. For large blobs this is particularly inefficient as
potentially several MB (if not more) of data needs to be copied. Let's fix this
by changing the BackingStore API to return a shared_ptr.
In order to make the code easier to read and write, also define 3 types:
TreePtr, BlobPtr and BlobMetadataPtr and use them in the BackingStore code.
Future changes should be done at a later point to convert the whole codebase to
using these.
Reviewed By: chadaustin
Differential Revision: D45967102
fbshipit-source-id: 6086f95456232db48a5cbec47b7cf8b14e4424ed
Summary:
We already had a pretend PrivHelper on Windows. Remove the stub
methods on Windows and provide a Windows implementation of PrivHelper
where most methods are no-ops but the mounts throw a NOT_IMPLEMENTED
error.
Reviewed By: kmancini
Differential Revision: D45260187
fbshipit-source-id: 87ca8679947372e484c345acdd708a17dbb8235e
Summary:
Add a function to query the (optional) ObjectId from an arbitrary
inode.
Reviewed By: kmancini
Differential Revision: D45673053
fbshipit-source-id: 23630638ba94bbd1925a39193fa125df04abebad
Summary:
We use privhelper for NFS too, so move fs/fuse/privhelper into
fs/privhelper.
Reviewed By: kmancini
Differential Revision: D45243273
fbshipit-source-id: cfc93b324768809c87075b0085905e873c7ea88d
Summary:
It's unfortunate that changing NFS code causes most of the inode layer
to recompile. One benefit of FsChannel is we can begin to decouple
EdenMount from FUSE, NFS, and PrjFS. Remove the Nfsd3.h include from
EdenMount.h.
Reviewed By: kmancini
Differential Revision: D45167461
fbshipit-source-id: 41650390cbd053233dfe3baf051c592bc203de56
Summary:
Now that Windows is going to support NFS. We need a reliable way to check at
runtime if a file should be read from disk (Prjfs) or it should be read from
the overlay (nfs).
We read inodes during fsck which is before the mount is fully initialized, so
we need to be able to detect the mount type earlier than how we do right now.
Here I am moving the NFS detection before eden mount initialization.
While moving this I realized that the InodeMap's view if NFS is incorrect
in the takeover case, so I am also fixing that here.
Reviewed By: chadaustin
Differential Revision: D45020849
fbshipit-source-id: b0a8fd431a37174c81b0d053d92b8bac026bd0f1
Summary:
Now that the Mercurial backingstore knows how to fetch aux data, let's thread
this through the ObjectStore.
Reviewed By: kmancini
Differential Revision: D44110102
fbshipit-source-id: c57da05066d80fee199e45b4a4223168a196e3de
Summary:
DurationScope has a pair of atomic reference count operations at the
beginning and end of every recorded duration. To avoid that overhead,
reference EdenStats with RefPtr and, in production EdenFS, store a
global, unowned EdenStats object.
This gives us the benefits of explicit dependencies and no global
state in tests, while avoiding atomic RMWs in the release build.
Reviewed By: xavierd
Differential Revision: D44323723
fbshipit-source-id: 1b3384d2e6a0a2959fd79774a8ba46afc4c176ca
Summary:
ObjectStore.h includes too many headers. Replace several with forward
declarations.
Reviewed By: xavierd
Differential Revision: D44323736
fbshipit-source-id: 944cd72fd758df4363493f87a573d91245a45b6a
Summary:
Our telemetry doesn't capture any data regarding how long Overlay operations
are taking. Since there is a suspicion that checkout is bounded on the Overlay,
let's add these counters to better understand the cost of the Overlay.
Reviewed By: genevievehelsel
Differential Revision: D44118779
fbshipit-source-id: 2e8658c9b0629a0a4f9c6f4535ec876527386b5f
Summary: We may sometimes want to use EdenStats earlier in startup. For example, in the next diff we can now use EdenStats in the StructuredLoggerFactory
Reviewed By: genevievehelsel
Differential Revision: D43800710
fbshipit-source-id: 4f903bd78e901f3b4a885718b5dc58abce9f53fc
Summary:
EdenFS is often started in the background. Because of this users can not easily
see the startup status. The only place it is avaiable other than the eden start
command is the logs. Those are hard to find and not really intended to be easy
to read.
Because users don't see EdenFS making progress while it is starting, users
reachout for help in the support group (leading to the oncall having to read out
the logs to them), or they will try to self service and kill EdenFS which land
themselves in a worse state, and lead to start up taking even longer due to
unclean startup or EdenFS corruption (proxy hashes).
In this diff I am exposing a way to subscribe to the status of EdenFS's startup
process. I also introduce the `--verbose` flag on the `status` command that
displays this startup status when EdenFS is running.
Reviewed By: xavierd
Differential Revision: D38962809
fbshipit-source-id: 2a8f03218df33e4fbfd4d0be919d48d9523c85b8
Summary:
The isSafeForInodeAccess is used to know when the root inode is initialized and
valid, but this is used in racy context. For instance, the getMountPoints
collects all the mounts whose root inode is initialized, but then releases the
lock on the mountPoints_ list, subsequently, the EdenMount's root Inode is
accessed without checking. If the EdenMount is shutdown prior to the root Inode
being copied, the root Inode might be a nullptr, causing a NULL-dereference.
This pattern can be found in all of the Thrift entry points that play with
mounts and thus are all potentially sources of EdenFS crashes. To solve this,
we need to guarantee 2 things:
- isSafeForInodeAccess must only change state while the mountPoints_ lock is
held, specifically at shutdown time,
- The root inode must be copied while the mountPoints_ lock is held.
This diff does both. For the second one in particular EdenServer::getMount is
modified to also return the root Inode. This is done instead of introducing a new
function to prevent future bugs from seeping in that may be missed at review time.
Reviewed By: chadaustin
Differential Revision: D42563083
fbshipit-source-id: c0267277a54c425f330bbd58d1dc86ec3746502d
Summary:
Bring back readdir prefetching, which was disabled as part of the
edenapi aux metadata rollout, and tighten up its logic to accelerate
specific use cases.
To optimize `find .`, prefetch subtrees when a subtree is stat().
To optimize `find . -empty`, prefetch blob metadata when a file is
stat().
Add a config knob to opt out of blob prefetching, tree prefetching, or
both.
Reviewed By: xavierd
Differential Revision: D41701675
fbshipit-source-id: 29b9c28f0f09cf686e3f256bd8c993e9b466e012
Summary:
This diff adds the ability to write unit tests against TestMount with
overridden EdenConfig settings.
Reviewed By: kmancini
Differential Revision: D41883877
fbshipit-source-id: ebcaf6e92ede9c411da11679705db53d37ad052f
Summary:
In preparation for supporting arbitrary ConfigSources, replace the two
hard-coded TOML file sources in EdenConfig with a list.
Reviewed By: kmancini
Differential Revision: D41881820
fbshipit-source-id: 585ceef17e4f23dabefdde61eb92b64c8ff992ee
Summary:
EdenConfig had a bunch of hard-coded logic around stat()ing files and
parsing TOML.
Besides code clarity, this had several problems:
* Using an EdenConfig in a unit test would touch the actual filesystem.
* We didn't support reading configs from other sources. For example,
we've talked about reading EdenConfig from .hgrc.
* ReloadableConfig knew too much about EdenConfig's internal working.
* Changing configs in tests involves calling setValue(), which is an
unsafe API in the limit.
This diff introduces a new abstraction called ConfigSource, with one
implementation, TomlFileConfigSource.
ConfigSource can indicate that it's time for a reload, and can apply
its values to a non-const EdenConfig instance.
Reviewed By: kmancini
Differential Revision: D41830417
fbshipit-source-id: a0ff90c4a1adb870e4b34da5d16238a6e7b75be2
Summary:
We never copy-assign EdenConfig, and I'd argue we would never intend
to. Since copying EdenConfig is a special and expensive operation,
remove the copy assignment operator.
Reviewed By: kmancini
Differential Revision: D41830340
fbshipit-source-id: f640083f1339e5fa022e4884e3cd2e27bc382e30
Summary:
For parity with Future and SemiFuture, remove ImmediateFuture's
default constructor and replace it with `makeEmpty()` and a
`std::in_place` constructor overload.
Reviewed By: xavierd
Differential Revision: D41293865
fbshipit-source-id: 43abae81fed3d99a5b4124d8c598dcce921ef65b
Summary:
I'm about to introduce an interface named ConfigSource, which
awkwardly conflicts with this enum. Rename ConfigSource to
ConfigSourceType. I considered ConfigSourceSlot, but slot seems more
of an implementation detail.
Reviewed By: kmancini
Differential Revision: D41830066
fbshipit-source-id: 85a54f4243b14c693087c7555e27b5473e90562e
Summary:
My goal is to make EdenConfig testable. To that end, move calculation
of substitution variables into its own ConfigVariables type.
Reviewed By: kmancini
Differential Revision: D41780224
fbshipit-source-id: 3b59143b42db79c44bbea10121a7e06bb4b833ce
Summary: EdenConfig had some remnants of old cruft. Remove them.
Reviewed By: xavierd
Differential Revision: D41744767
fbshipit-source-id: d5fd50fa599c15ec8b748c323c45ea1e7eb8ddcd
Summary:
EdenFS's tests still fall back on a debugedenimporthelper process to
import trees and blobs. That's because the test repos still use the
deprecated revlog format. Migrate the test repos to use indexedlog
storage so that SaplingNativeBackingStore hits.
Note: This repo configuration is awkward and temporary, and the
Sapling team is planning to migrate to "Eager" repos to support the
`hg init` case in the future.
Reviewed By: quark-zju
Differential Revision: D41447075
fbshipit-source-id: 23c2f13dfc9f964d9386d7f4c920222db8f25144
Summary:
If an hg command fails in a C++ test, we always want more details
about why.
Reviewed By: xavierd
Differential Revision: D41393550
fbshipit-source-id: 1d0ca29919e0be8253a82938ca04a85cf149d42d
Summary:
EdenFS's C++ unit tests unnecessarily differed in how they initialized
test repositories. Move
scmstore/treemanifest/remotefilelog/etc. configuration into hgInit so
it's the same everywhere.
Reviewed By: genevievehelsel
Differential Revision: D41364290
fbshipit-source-id: 24f882766bf00c6546663e0712b6029153a4a699
Summary:
scmstore.backingstore = true has been rolled out everywhere since
January. Also enable it in EdenFS unit tests.
Reviewed By: xavierd
Differential Revision: D40968994
fbshipit-source-id: ae55fa908bdbd287177f0f485efa457c595bc4d0
Summary:
std::string_view has noexcept accessors and folly::Range doesn't, so
this allows us to make Path and PathPiece noexcept.
Reviewed By: kmancini
Differential Revision: D41145426
fbshipit-source-id: 046f6f6a532d8d0da8508ccf7896c914e19a25ec
Summary:
We no longer need a kill switch on edenapi fetches. Stop accessing the
config and always configure backing stores with edenapi on.
Reviewed By: genevievehelsel
Differential Revision: D40909096
fbshipit-source-id: fbdb01fc74add870b95d0bcc46be6b4bf9323898
Summary:
Default implementations are convenient in the short term but
getLocalBlobMetadata is core part of the BackingStore API. Therefore,
whether a BackingStore implements it should be an explicit decision.
Reviewed By: kmancini
Differential Revision: D40456066
fbshipit-source-id: 1da1ec706fe78693968d46ff672b9c9bd8ea5e8b
Summary:
Requiring that callers ensure that ObjectFetchContext outlives the
asynchronous result is a burden and a memory-safety risk. Without
Rust's borrow checker, we can't feel confident this is safe under all
error and timeout conditions. It also requires we use
folly::collectAll and collectAllSafe, which means that an error during
fanout has to wait until the fanout completes before we can report the
error to our caller.
Moreover, tying the ObjectFetchContext's lifetime to the
FUSE/NFS/Thrift/etc. request's lifetime gives us incorrect statistics
about how long the request takes. I consider this an API design
mistake (on my part).
Correct the situation by decoupling ObjectFetchContext from
RequestContext, and begin reference-counting ObjectFetchContext.
This does increase the number of atomic RMW operations, but we can
alleviate those where they matter by manually checking
*Future::isReady() before calling .then*.
Reviewed By: xavierd
Differential Revision: D40744706
fbshipit-source-id: 5baf654f6d65790ed0cafb45c0bc7f2aaa8ecec2
Summary:
At API boundaries, it's common to want to convert a Path or PathPiece
to std::string. Previously, code would call
`.stringPiece().str()`. Add an `asString()` to make this explicit.
Reviewed By: xavierd
Differential Revision: D41140757
fbshipit-source-id: 72003dc9a353cdfa4632a7d1ab433122211186e0
Summary:
On Windows, absolute paths are of multiple shape, with all of them having pros
and cons. In particular, device paths (C:/) that EdenFS is using is limited to
260 characters. To bypass this limitation, UNC paths (\\?\) can be used, but no
normalization is done automatically, specifically '/' aren't converted to '\'.
Since UNC paths allow us to support long paths, and would also bring more
safety to EdenFS by enforcing that all AbsolutePath are of the UNC form, let's
use them.
One of the other downside of UNC is that applications may not support them or
may misbehave when passed to them. For this reason, EdenFS doesn't expose UNC
paths in its Thrift API, changing the path to a device path, applications
relying on Thrift are expected to re-UNC them.
Reviewed By: chadaustin
Differential Revision: D40818724
fbshipit-source-id: 260b8732df04bbffee6b2fe52f0ceda222d3941b
Summary: This changes follows the last, changing `TreeOverlay` -> `InodeCatalog`. Its in a separate change just for readability purposes
Reviewed By: chadaustin
Differential Revision: D40518302
fbshipit-source-id: 8c1ed08e413d5e421a9dcd0b83600855573a699a
Summary: To facilitate the split of the `IOverlay` class, split the `hasOverlayData` function into `hasOverlayFile` and `hasOverlayDir`
Reviewed By: chadaustin
Differential Revision: D40526835
fbshipit-source-id: 5b075c3103e5ae3d1f1df32edb1384621bac35bd
Summary:
Eden's globber is currently hardcoded to be *mostly* case-sensitive ("mostly", because some code paths are already case-insensitive - see D39892007 (05b2300f73)). Here we augment the implementation to consistently perform case-insensitive globbing in case-insensitive mounts.
As a possible followup, we may want to expose this as an argument to `globFiles`. At the moment it is solely controlled by the checkout config.
Reviewed By: chadaustin, xavierd
Differential Revision: D39906585
fbshipit-source-id: 5b266cb75cf073651f6e2c66e7f1ebdd7026312f
Summary:
On case-insensitive but case-preserving file systems, we currently clobber the case of a glob match and synthesise a path from the *pattern*'s case - e.g. if the pattern is `foo` and the file `Foo` exists, we incorrectly return `foo` as the match. This erases information that may be important to clients.
Here we change Eden to consistently preserve the case of the path as stored in the FS.
Reviewed By: chadaustin
Differential Revision: D39892007
fbshipit-source-id: 9d857ebdee917d0045ab800ef7094b85bf129722
Summary:
I'm having some confusing conflicts between fmt and our PathPieces. To
at least make the error messages clearer, migrate them to
std::string_view.
Reviewed By: genevievehelsel
Differential Revision: D39991418
fbshipit-source-id: 34d0f1ee23618da4595ee1d072e9cdf4f51736b6
Summary: In preparation for the split between TreeOverlays and BlobOverlays, this diff makes it explicit that the OverlayType applies to tree storage
Reviewed By: chadaustin
Differential Revision: D38279396
fbshipit-source-id: c56805818e43471f5301329beb0b5a51e0b04a69
