Summary: Added `fill_from_function` which makes it much more useful
Reviewed By: fanzeyi
Differential Revision: D46342378
fbshipit-source-id: 9bc8c01dcb10806a03f1eab628f9763c4e5edc81
Summary:
Our telemetry was showing that starting mid-April the cost of
ObjectStore::getBlobmetadata went down significantly but investigating the root
cause of the improvement didn't seem to yield any results. It turns out the
root cause was that the code no longer timed the cost of the future.
Reviewed By: kmancini
Differential Revision: D46369667
fbshipit-source-id: 0c7d997c93d94498384e27b3447a9c736dcce3b2
Summary:
Both the getBlobMetadata and fetchBlobMetadata were publishing to the same counter, making them indistinguishable.
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: genevievehelsel
Differential Revision: D46343472
fbshipit-source-id: 04e09372c21dd5cb09336259ff34a5394d10d3b4
Summary:
We've seen crashes on Windows when the mount is initialized but a later step at
mount time fails. In between these 2 events, if a write to the working copy
happens the ProjectedFS code assumes that the InodeMap is initialized and thus
tries to access the rootInode. This assumption is unfortunately broken by the
call to mountFinished and the code should instead follow the regular unmount
code path to make sure that no pending notifications are inflight.
I suspect NFS and FUSE also would exhibit the same issue.
Reviewed By: kmancini
Differential Revision: D46169999
fbshipit-source-id: ed1eb17dc2f084acb7e2d88c555150b3b8c6662e
Summary: Adding a way to configure the blake3 key for blake3 hashing
Reviewed By: chadaustin
Differential Revision: D44499469
fbshipit-source-id: 344edd91cfc867c8a49d0db9369dfe2132e86c77
Summary: allow using `SqliteOverlay` on non-Windows, since OverlayChecker has now been updated to work with it.
Reviewed By: chadaustin
Differential Revision: D44105360
fbshipit-source-id: 5f78e6c9b5323faeee394da4a51d7e84f8d44a2f
Summary:
In the rare case where fetching both the BlobMetadata and the Blob failed, the
LocalStoreCachedBackingStore would still try to dereference the nullptr Blob,
leading to a crash.
Reviewed By: chadaustin
Differential Revision: D46211469
fbshipit-source-id: f98dd215dbda432507f3823439d354d3d662dc6f
Summary:
We've seen a case where a large `hg update` was taking an absurdly long time in
`ObjectStore::getTree` but the telemetry was showing us that most of the time
wasn't spent fetching trees from Mercurial! The suspicion is that most of the
time was spent in the LocalStore but with no evidence to prove it.
Let's thus add some timing telemetry to various LocalStore read requests to
fill this gap.
Reviewed By: mshroyer
Differential Revision: D46154456
fbshipit-source-id: b439ac48889ed3db71db136ff6c1cc91f48c926a
Summary:
Extends `FsckTest` to test `SqliteOverlay`. Two major changes were made to non-test files:
1) `loadInodeInfo` was moved from `OverlayChecker` into `InodeCatalog`
2) `getAllParentInodeNumbers` was added to facilitate `OverlayChecker::readInodes`
Reviewed By: chadaustin
Differential Revision: D45447172
fbshipit-source-id: 9ad291a0c4c34f1e44e1f05c1fdbf2ac7ea7d309
Summary: This will no longer be tied to the fscatalog, so it makes more sense to live in a more general location
Reviewed By: chadaustin
Differential Revision: D44980396
fbshipit-source-id: 3cabf00d87df7eb00671c2b82b04c8f12283e394
Summary:
Telemetry exist for where Blob and BlobMetadata are coming from, but not for
Tree. Let's bring the Tree telemetry up to par.
Reviewed By: genevievehelsel
Differential Revision: D46154455
fbshipit-source-id: ab3c31d55b6a91009289b3b07853fa574bbaa137
Summary:
I migrated the last caller of debugGetScmBlobMetadata to debugGetBlobMetadata
in the last diff. Let's delete the old method.
I am not really concerned about waiting for that lower diff to rollout and all
because this is a debugging command that we only sparingly ask users to run
(As far as I know like maybe 3 times).
Reviewed By: chadaustin
Differential Revision: D45972617
fbshipit-source-id: 6d4689a225fee972f8e3a802115852043f0bade8
Summary:
I created debugGetBlobMetadata a while ago and this is the last caller of
debugGetScmBlobMetadata. Let's migrate it to the new endpoint so we can delete
the old one.
Reviewed By: chadaustin
Differential Revision: D45972618
fbshipit-source-id: cbb0977260f9a1543d3bf0a31bf7b948c2f35143
Summary:
I created debugGetBlob a while ago and there are no more callers of
debugGetScmBlob from code search, so let's delete this.
Reviewed By: chadaustin
Differential Revision: D45972619
fbshipit-source-id: 97276ec2182fbe7d9022a06e9a0df34ae4e80235
Summary:
EdenFS contains several layers of caching for BlobMetadata:
- In the hgcache,
- In the in-memory BlobMetadata cache,
- In the LocalStore (configurable),
- In the FileInode,
- In the TreeEntry,
All of the above must be filled prior to reading a blob, therefore computing
the sha1+size when reading a blob is likely not needed and superfluous.
Alternatively, we could move the code to be done in the background in the
LocalStoreCachedBackingStore to avoid blocking reading the blob. Reviewers
opinions are welcome!
Reviewed By: chadaustin
Differential Revision: D45625432
fbshipit-source-id: ad7d897c32cbfd590dce5a3a9c8073349e9b70c8
Summary:
For tools that want to take advantage of the same fast-path logic when
directories don't change across updates, expose a semistable ID they
can use to cache derived data or get a rough understand when a
directory has changed its contents.
Reviewed By: kmancini, xavierd
Differential Revision: D45974142
fbshipit-source-id: 7b2b482876b07e73514a936e198de2dc31ed1597
Summary:
rustls 0.21.0 finally has support for IP addresses in certs.
Lots of other crates have (in)direct dependencies on rustls so I have to upgrade those too:
* hyper, reqwest, and auxiliary crates for those have been updated
* tonic has been updated and our local patch removed (since the relevant PR was merged upstream)
Reviewed By: zertosh
Differential Revision: D45989949
fbshipit-source-id: 658c11d29859d6d36368a412dab05e3c99313ee0
Summary:
In both the ObjectStore and in the hg BackingStore, copies of the unique_ptr
were being made. For large blobs this is particularly inefficient as
potentially several MB (if not more) of data needs to be copied. Let's fix this
by changing the BackingStore API to return a shared_ptr.
In order to make the code easier to read and write, also define 3 types:
TreePtr, BlobPtr and BlobMetadataPtr and use them in the BackingStore code.
Future changes should be done at a later point to convert the whole codebase to
using these.
Reviewed By: chadaustin
Differential Revision: D45967102
fbshipit-source-id: 6086f95456232db48a5cbec47b7cf8b14e4424ed
Summary:
getEntryAttributes did a mount lookup for every single path. That's
unnecessary: have the Thrift method do a mount lookup, ensure the
mount stays alive, and pass it down.
Reviewed By: kmancini
Differential Revision: D45973970
fbshipit-source-id: 68c5038b229b5b2e4946e6a34dd6ceaabd3fcf11
Summary:
We have three ways to look up a mount in a Thrift handler: by MountId,
with a Thrift field reference, and as a string parameter. Add
`lookupMount` overloads that handle all of them.
Reviewed By: kmancini
Differential Revision: D45973903
fbshipit-source-id: fb6d6ac1bf5001dbc111875aa4aa257d1bcccc59
Summary:
We have several legacy conditional includes that are no longer
necessary. Remove another.
Reviewed By: xavierd
Differential Revision: D45927567
fbshipit-source-id: 05bea9a94b249104c26893cfc0017fa472643ce5
Summary:
Remove a class of boilerplate by introducing a lookupMount function
that takes the two common Thrift mount identifiers and returns an
EdenMountHandle.
Reviewed By: kmancini
Differential Revision: D45924483
fbshipit-source-id: 2f919c4525ba8c82018d9b7c2ca4ee361d924791
Summary:
I want to use EdenMountHandle in EdenServiceHandler without pulling in
EdenServer.h.
Reviewed By: xavierd
Differential Revision: D45924297
fbshipit-source-id: 2043374d513c01423cd6f785b2c4e2b36d937676
Summary:
Having a strict ObjectID format is quite inconvenient. We will need to introduce a new ObjectID format for Eden x Sparse, so it's in our best interest to remove the ObjectID format restrictions before hand.
This will allow us to place the high entropy data (proxy hash in our case) in any location in the ObjectId without causing a ton of hash collisions. This will enable us to introduce FilteredObjectIDs in the form:
`<tree_or_blob_byte><filterset_id><path><ObjectId>`
where the `<ObjectId>` contains the high entropy bits we need to hash.
Reviewed By: xavierd
Differential Revision: D45793298
fbshipit-source-id: 77385e32f63d5f3d1fc37b72b9971f5717cbd872
Summary:
We are considering changing the ObjectID hash function to no-longer be constant time. Instead, it might iterate over the bytes of the ID and "mix-in" contents from each 8 byte chunk into a final hash.
In order to measure the performance hit this might have, we need to benchmark both long ObjectIds and short ObjectIDs. This diff introduces a benchmark to test hashing ObjectIDs of length 240 characters long, which is a realistic size of our current ObjectIDs.
Reviewed By: chadaustin
Differential Revision: D45798645
fbshipit-source-id: 988442555f48968edf67517aa25b55ef636de610
Summary:
Now that we have NFS support on Windows, we can enable
debugOutstandingNfsCalls.
Reviewed By: genevievehelsel
Differential Revision: D45260222
fbshipit-source-id: 38e7ec6cd38601bd940904695cc69ce2193b98c1
Summary:
Now the the ifdefs are isolated to the PrivHelper, we can remove some
from EdenServiceHandler.
Reviewed By: mshroyer
Differential Revision: D45260210
fbshipit-source-id: d0394e45320c516290a6c4323400cc054e76d612
Summary:
We already had a pretend PrivHelper on Windows. Remove the stub
methods on Windows and provide a Windows implementation of PrivHelper
where most methods are no-ops but the mounts throw a NOT_IMPLEMENTED
error.
Reviewed By: kmancini
Differential Revision: D45260187
fbshipit-source-id: 87ca8679947372e484c345acdd708a17dbb8235e
Summary:
I was hoping this would shake out some dependency issues, but they
must be triggered by a later diff in the stack.
Reviewed By: xavierd
Differential Revision: D45927428
fbshipit-source-id: ec29b331f561f9bcfe1daa6856ac8cca67e01d39
Summary:
I ended up in a crazy situation where the CMake build was failing to
link because `target_link_libraries` had a cycle. Fix that by
preventing cycles in the first place. CMake is the gift that keeps on
giving.
Reviewed By: xavierd
Differential Revision: D45927359
fbshipit-source-id: 6a378f7935195e42b3446249091898370d6e2ee5
Summary:
enums can contain arbitrary values, so gcc (and older clang?) complain
about the possibility that none of these cases match.
Reviewed By: xavierd
Differential Revision: D45927298
fbshipit-source-id: 0cd0a946d0faf7442a1a8d5de388a7d4b6dbb274
Summary: Add a function to query the (optional) ObjectId from VirtualInode.
Reviewed By: kmancini
Differential Revision: D45673165
fbshipit-source-id: 72dadaf4cbd5120c75548734aca2bb8ae715b8e2
Summary:
Add a function to query the (optional) ObjectId from an arbitrary
inode.
Reviewed By: kmancini
Differential Revision: D45673053
fbshipit-source-id: 23630638ba94bbd1925a39193fa125df04abebad
Summary:
I noticed some situations where we don't hold the root inode for the
duration of an asynchronous operation. To make that easier, I
introduced an EdenMountHandle.
Port EdenServer::getMount() to return EdenMountHandle, which
simplifies some Thrift service handlers.
Reviewed By: kmancini
Differential Revision: D45891887
fbshipit-source-id: 99dfaa4aaf609c8b237cceeaa15b91bd26981d02
Summary:
Asynchronous operations that must ensure an EdenMount (and its root)
are usable for the duration must hold a reference to the root's
InodePtr, even if they only need the EdenMount.
To make this relationship less implicit, introduce an EdenMountHandle
which is effectively the same as the previous
`pair<EdenMountPtr, TreeInodePtr>`, but more clearly documents the intent
is to preserve the usability of EdenMount and its root inode.
This also allows us to decouple that relationship in the future,
should we want to.
For now, only use EdenMountHandle in getMountPoints. Next, I'll switch
`getMount` to use it.
Reviewed By: kmancini
Differential Revision: D45830176
fbshipit-source-id: 832dd96b9a6fe04fabbb5af0ecbed0173f9d2a2b
Summary:
The EdenMount::getObjectStore gave the assumption to callers that the
ObjectStore lived longer than the EdenMount. Since this assumption is implicit
and not explicit, there is no way to guarantee it by just reading the code in a
local fashion. It forces the reader/auditor to understand the code in its
entirety. Unfortunately, EdenFS contains more of these than can be kept in
memory which leads to bugs and use-after-free.
This can be avoided by breaking the implicit lifetime assumption and simply
returning the shared_ptr. If the caller needs to keep the ObjectStore around,
they can do so explicitely which makes reading the code easier.
Reviewed By: chadaustin
Differential Revision: D45831168
fbshipit-source-id: 8bdb8e708308a8a220e3130d2fc51854c8556cb1
Summary:
This will be used in a later diff outside of EdenMount.cpp, thus let's move it
to its own file.
Reviewed By: chadaustin
Differential Revision: D45831170
fbshipit-source-id: 1cc55e1ae580ee7f02e3d9c6a717da4751c955e6
Summary:
When a BlobMetadata fails to fetch, the fetch promise is set to a `nullptr` to
force the higher level BackinStore to fetch the full blob. Unfortunately,
`HgImportRequestQueue::markImportAsFinished` attempts to deep copy that
`nullptr` leading to a crash.
Reviewed By: chadaustin
Differential Revision: D45873681
fbshipit-source-id: 3f5b777a3671cf53dd9d605dca2e119eabe3e625
Summary:
I noticed this special path in `eden rm` misspelled "still". I also
cleaned up the grammar. Sentence fragments aren't capitalized and, to
make the path clearer, I removed the trailing period and split the
message across multiple lines.
(Separately, I think this is special path is not hit as often as was
intended.)
Reviewed By: xavierd
Differential Revision: D45828487
fbshipit-source-id: 183ccde081cb022abbc9e972f00bae8e5b9f33c0
Summary:
We don't have a kext on macOS anymore, so we can remove the
configuration setting that enables it.
If we ever build our own kernel module on Linux, we can bring this back.
Reviewed By: kmancini
Differential Revision: D45260142
fbshipit-source-id: 6f99bdfb3fce236b7a17a665818327f00b5884c0
