Summary:
Both the getBlobMetadata and fetchBlobMetadata were publishing to the same counter, making them indistinguishable.
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: genevievehelsel
Differential Revision: D46343472
fbshipit-source-id: 04e09372c21dd5cb09336259ff34a5394d10d3b4
Summary:
add new ods logging for bookmark fetching from phases crate
This would help with investigation sevs like S344849, also having this info we
can setup alerts.
Reviewed By: YousefSalama
Differential Revision: D46314640
fbshipit-source-id: 66e72da19ad4eb528c78d0783c297746f57cd4f1
Summary:
We've seen crashes on Windows when the mount is initialized but a later step at
mount time fails. In between these 2 events, if a write to the working copy
happens the ProjectedFS code assumes that the InodeMap is initialized and thus
tries to access the rootInode. This assumption is unfortunately broken by the
call to mountFinished and the code should instead follow the regular unmount
code path to make sure that no pending notifications are inflight.
I suspect NFS and FUSE also would exhibit the same issue.
Reviewed By: kmancini
Differential Revision: D46169999
fbshipit-source-id: ed1eb17dc2f084acb7e2d88c555150b3b8c6662e
Summary:
This offers a new interface for Shard Manager managed repos to be able to load redacted content.
The new API entry point: `open_managed_repos_unredacted` is analogous to `open_repo_unredacted`, but supports a Shard Manager based workflow.
Reviewed By: RajivTS
Differential Revision: D46280863
fbshipit-source-id: e2e4aebcb08d01f3be97d11d6fe8da2822ba6f29
Summary: Adding a way to configure the blake3 key for blake3 hashing
Reviewed By: chadaustin
Differential Revision: D44499469
fbshipit-source-id: 344edd91cfc867c8a49d0db9369dfe2132e86c77
Summary: I am planning to enable `copytrace.fallback-to-content-similarity=True` for hg repos, and will use this metric to monitor how useful the content similarity fallback is
Reviewed By: muirdm
Differential Revision: D46163311
fbshipit-source-id: d21b0288965f8ec4da7c6c80e37d1b557b357ea6
Summary: Our cache is in the dagcopytrace instance, so let's make it a singleton.
Reviewed By: quark-zju
Differential Revision: D46105882
fbshipit-source-id: 0984015e7b052482b27f950fc05563c432aa23b1
Summary: this function will be used in the following diff
Reviewed By: quark-zju
Differential Revision: D46105885
fbshipit-source-id: 9d02fb62a44b1e5bd18eba5236522e297fae0f64
Summary:
This diff modifies the behavior of `gitimport` and `remote-gitimport` when importing `annotated-tags`. In addition to:
- storing the raw tag
- converting the tag to a bookmark
- storing the metadata of the tag as changeset
this diff maintains a mapping between the tag-name and the changeset representing the metadata of the tag. This mapping would be useful to answer queries about the tag metadata when queried through Git and also enable going back to Git from Mononoke without losing data.
Differential Revision: D46067809
fbshipit-source-id: e2463160cdf647b50fa4ecfe4674d079016b4adc
Summary: Implement an alternative to ChangesetPathHistory::history using the new commit graph. This is almost identical to the skiplist implementation except that an (incorrect) optimization was removed that was trying to minimize the number of is_ancestor queries which should no longer be needed as they are a lot faster with the new commit graph.
Differential Revision: D46109911
fbshipit-source-id: 3e8f0c36a48084116673dda3741de87e0741f1a2
Summary: Implement an alternative to ChangesetContext::history using the new commit graph by using AncestorsStreamBuilder. Also made the tests not depend on the exact output history order but instead accept any topological order.
Differential Revision: D46109910
fbshipit-source-id: 31f3e219e9addf04b49fc500ad245d541f110887
Summary:
Factors out the logic of creating an ancestors stream to a builder to allow for more customizability (required for ChangesetContext::history) and reduce code duplication.
Additionally this optimizes range_stream to only traverse changesets that are descendants of start_id. Ancestry checks are kept to a minimum (worst case O(merges)) by only checking on parents of changesets that have a skip tree parent with generation lower than the generation of start_id.
Reviewed By: liubov-dmitrieva
Differential Revision: D46276429
fbshipit-source-id: 65c4f2892c92baaab0fef4f5041f06095ec49e05
Summary: allow using `SqliteOverlay` on non-Windows, since OverlayChecker has now been updated to work with it.
Reviewed By: chadaustin
Differential Revision: D44105360
fbshipit-source-id: 5f78e6c9b5323faeee394da4a51d7e84f8d44a2f
Summary:
In the rare case where fetching both the BlobMetadata and the Blob failed, the
LocalStoreCachedBackingStore would still try to dereference the nullptr Blob,
leading to a crash.
Reviewed By: chadaustin
Differential Revision: D46211469
fbshipit-source-id: f98dd215dbda432507f3823439d354d3d662dc6f
Summary: We didn't have a way to do this before so let's add it.
Reviewed By: mitrandir77
Differential Revision: D46190481
fbshipit-source-id: 84544f0226ef3a4ef4d8ac9b4bf66551e4671793
Summary: It's not ideal that we pull mononoke_app and even instantiate MononokeApp here. This is an attempt to decouple Mononoke from Metagit.
Reviewed By: mitrandir77
Differential Revision: D45862434
fbshipit-source-id: 6eab3577743810461f22535d89cf9a00dc1cbb65
Summary: Most of the useful ways to use these methods requires providing an async closure so let's asyncify them.
Reviewed By: RajivTS
Differential Revision: D45607981
fbshipit-source-id: ee7b471cc5d60a793a79928f6c15c48c0da1d08e
Summary: Currently, the code scans the repo when revision translation is failed, which is very slow.
Reviewed By: muirdm
Differential Revision: D46200075
fbshipit-source-id: 31eaca43582b480a0b3abd7af0b27ec8d6406195
Summary:
We get both `added` and `deleted` files in the function, this will be
needed for directory move detection.
Reviewed By: muirdm
Differential Revision: D46105884
fbshipit-source-id: d0081f8afaf27ef4ab105ab7c58b3f5cc694debc
Summary:
We've seen a case where a large `hg update` was taking an absurdly long time in
`ObjectStore::getTree` but the telemetry was showing us that most of the time
wasn't spent fetching trees from Mercurial! The suspicion is that most of the
time was spent in the LocalStore but with no evidence to prove it.
Let's thus add some timing telemetry to various LocalStore read requests to
fill this gap.
Reviewed By: mshroyer
Differential Revision: D46154456
fbshipit-source-id: b439ac48889ed3db71db136ff6c1cc91f48c926a
Summary:
Extends `FsckTest` to test `SqliteOverlay`. Two major changes were made to non-test files:
1) `loadInodeInfo` was moved from `OverlayChecker` into `InodeCatalog`
2) `getAllParentInodeNumbers` was added to facilitate `OverlayChecker::readInodes`
Reviewed By: chadaustin
Differential Revision: D45447172
fbshipit-source-id: 9ad291a0c4c34f1e44e1f05c1fdbf2ac7ea7d309
Summary: In the case of a multiplexed blobstore, the put operation can exit after it succeeds in one inner blobstore before finishing in all, and leave the rest running in the background. This sleep tries to prevent exiting early before they all finish.
Reviewed By: mitrandir77
Differential Revision: D46190037
fbshipit-source-id: ac2ae0d1453ebdc923170b9da7ca98df7519bf52
Summary: This will no longer be tied to the fscatalog, so it makes more sense to live in a more general location
Reviewed By: chadaustin
Differential Revision: D44980396
fbshipit-source-id: 3cabf00d87df7eb00671c2b82b04c8f12283e394
Summary: Implements a public method changeset_children in CommitGraph that fetches the changeset ids of the children of a given changeset.
Differential Revision: D46185434
fbshipit-source-id: f0b2e2ee53c7a34fe94ee5ef1f74f6e31429d056
Summary: used cached version of API
Reviewed By: YousefSalama
Differential Revision: D46184380
fbshipit-source-id: 5deb8c455ad682afcf1f0cad822d34fdd632b88b
Summary:
Telemetry exist for where Blob and BlobMetadata are coming from, but not for
Tree. Let's bring the Tree telemetry up to par.
Reviewed By: genevievehelsel
Differential Revision: D46154455
fbshipit-source-id: ab3c31d55b6a91009289b3b07853fa574bbaa137
Summary:
I migrated the last caller of debugGetScmBlobMetadata to debugGetBlobMetadata
in the last diff. Let's delete the old method.
I am not really concerned about waiting for that lower diff to rollout and all
because this is a debugging command that we only sparingly ask users to run
(As far as I know like maybe 3 times).
Reviewed By: chadaustin
Differential Revision: D45972617
fbshipit-source-id: 6d4689a225fee972f8e3a802115852043f0bade8
Summary:
I created debugGetBlobMetadata a while ago and this is the last caller of
debugGetScmBlobMetadata. Let's migrate it to the new endpoint so we can delete
the old one.
Reviewed By: chadaustin
Differential Revision: D45972618
fbshipit-source-id: cbb0977260f9a1543d3bf0a31bf7b948c2f35143
Summary:
I created debugGetBlob a while ago and there are no more callers of
debugGetScmBlob from code search, so let's delete this.
Reviewed By: chadaustin
Differential Revision: D45972619
fbshipit-source-id: 97276ec2182fbe7d9022a06e9a0df34ae4e80235
Summary: The admin tool currently doesn't register the `fb303` extension with `MononokeApp` during its creation. This makes the tool incapable of registering for health check and responding to thrift health check requests by TW. In absence of ALIVE response from the job, TW ends up killing it. This diff adds the extension during `MononokeApp` construction. This will not impact admin tool's usage through CLI but the error messages in the integration tests will need to be updated.
Differential Revision: D46147128
fbshipit-source-id: f06b8bcdce938346726b9ff520a3d809d9fda6fc
Summary: Makes it easier to rollback
Reviewed By: mitrandir77
Differential Revision: D46156526
fbshipit-source-id: eae70801069d09ab03b3b683712d9353fb1e5a43
Summary:
EdenFS contains several layers of caching for BlobMetadata:
- In the hgcache,
- In the in-memory BlobMetadata cache,
- In the LocalStore (configurable),
- In the FileInode,
- In the TreeEntry,
All of the above must be filled prior to reading a blob, therefore computing
the sha1+size when reading a blob is likely not needed and superfluous.
Alternatively, we could move the code to be done in the background in the
LocalStoreCachedBackingStore to avoid blocking reading the blob. Reviewers
opinions are welcome!
Reviewed By: chadaustin
Differential Revision: D45625432
fbshipit-source-id: ad7d897c32cbfd590dce5a3a9c8073349e9b70c8
Summary: The bookmark `get` command simply displays the changeset that is pointed to by the bookmark. This diff modifies the command to also handle git annotated tags converted into bookmarks. If the bookmark is a tag bookmark, then the command also outputs the metadata changeset associated with the bookmark. This will be useful for validating that imported annotated tags retain their metadata through metadata changesets.
Differential Revision: D46022692
fbshipit-source-id: b3980f4799066913268ef7acc1f4956d3feb5b23
Summary: `BonsaiTagMapping` exists as a facet so let's add the appropriate builder methods in `RepoFactory` and `TestRepoFactory`. Follow-up diffs will add this new facet in the `Repo` facet container for `SCS` (for `remote-gitimport`) and in `gitimport`.
Differential Revision: D46020528
fbshipit-source-id: 88e65a498355bf21edc8c8d8aa3755a5507fe991
Summary:
The `metadata-diff` endpoint in SCS is used by Phabricator to get information about the difference between files of two commits. Currently, this endpoint works using raw file content. This diff makes the following optimizations:
- Avoids loading file content in memory if the file is `binary` or `non-utf8`
- Avoids parsing `generated` and `partially-generated` tags if the file is known to be neither
- Avoid parsing raw file bytes into UTF-8 string if the file is non-generated
These optimizations will be most helpful for the case of `binary` or `non-utf8` files but should improve performance of the endpoint in general.
Reviewed By: YousefSalama
Differential Revision: D46109336
fbshipit-source-id: b3c915005686f77defceeb53f1587049bc8b6bd8
Summary:
For tools that want to take advantage of the same fast-path logic when
directories don't change across updates, expose a semistable ID they
can use to cache derived data or get a rough understand when a
directory has changed its contents.
Reviewed By: kmancini, xavierd
Differential Revision: D45974142
fbshipit-source-id: 7b2b482876b07e73514a936e198de2dc31ed1597
Summary:
thrift_test is pretty general. For something like getFileAttributes,
getFileAttributesV2, and readdir, it's nice to have them in one file.
Reviewed By: kmancini
Differential Revision: D45974095
fbshipit-source-id: ca5ecd4795f1d278f4de5b83cb8c9af94c111902
Summary: Implement an alternative to slice_repository using the new commit graph that doesn't depend on the skiplist. Useful when doing any kind of processing that requires ancestors to be processed first (backfilling derived data, mirroring commits, ...)
Reviewed By: mitrandir77
Differential Revision: D45607347
fbshipit-source-id: b2012972472e718522e4f33ff01fc9b253b76fff
Summary: Makes slightly more sense
Reviewed By: mitrandir77
Differential Revision: D45607346
fbshipit-source-id: 6b505796da96ff24cd7d555290dcf341f93c797f
Summary: My fingers are tired from scrolling
Reviewed By: mitrandir77
Differential Revision: D45607349
fbshipit-source-id: fe366797a37fbd1682ca917d6bdd39c88195c8e9
