Summary: Slightly tweak the tsconfig to fix type errors in vscode/
Reviewed By: zzl0
Differential Revision: D59344714
fbshipit-source-id: d16519f3270e500cf17c3586611bd14ff1487beb
Summary: It seems we're not actually checking type errors in CI, so a few have slipped through over time. None of these are very serious.
Reviewed By: zzl0
Differential Revision: D59344715
fbshipit-source-id: 18d23d7baaf2f63c927d081faff30ebf4f327769
Summary:
# Context
The new AugmentedManifest format used by Sapling and Mononoke will support looking up Trees/Blobs from CAS via (Blake3,size) pairs. We already provide a way for users to get (Blake3,size) pairs for blobs, but there is currently no easy way to get this information for trees.
We want to add support to getAttributesFromFilesV2 for looking up the (Blake3,size) pairs for trees as well. To do this, we'll need to add a method for querying TreeMetadata (including size and Blake3 hashes) via the SaplingBackingStore.
This initial implementation of TreeMetadata will only include the following features:
1) Ability to query TreeMetadata from the SaplingBackingStore via a getTreeMetadata endpoint
2) The ability to request TreeMetadata from the getAttributesFromFilesV2 thrift endpoint
In the future, we will extend the implementation to support:
1) Automatically fetching TreeMetadata during BackingStore::getTree requests
2) Caching TreeMetadata in Eden's in-memory caches
# This diff
this diff adds the new TreeAuxData methods to the FFI layer so they can be used in the SaplingBackingStore
Reviewed By: jdelliot
Differential Revision: D58491605
fbshipit-source-id: c85ac039bf855e8d8695194267ff9f4cfcb8110a
Summary:
# Context
The new AugmentedManifest format used by Sapling and Mononoke will support looking up Trees/Blobs from CAS via (Blake3,size) pairs. We already provide a way for users to get (Blake3,size) pairs for blobs, but there is currently no easy way to get this information for trees.
We want to add support to getAttributesFromFilesV2 for looking up the (Blake3,size) pairs for trees as well. To do this, we'll need to add a method for querying TreeMetadata (including size and Blake3 hashes) via the SaplingBackingStore.
This initial implementation of TreeMetadata will only include the following features:
1) Ability to query TreeMetadata from the SaplingBackingStore via a getTreeMetadata endpoint
2) The ability to request TreeMetadata from the getAttributesFromFilesV2 thrift endpoint
In the future, we will extend the implementation to support:
1) Automatically fetching TreeMetadata during BackingStore::getTree requests
2) Caching TreeMetadata in Eden's in-memory caches
# This diff
This diff adds the method that will be used to fetch TreeMetadata from the SaplingNativeBackingStore.
Reviewed By: jdelliot
Differential Revision: D58427476
fbshipit-source-id: a0e95824b6839a64e8578194df8ae7de093104d6
Summary:
# Context
The new AugmentedManifest format used by Sapling and Mononoke will support looking up Trees/Blobs from CAS via (Blake3,size) pairs. We already provide a way for users to get (Blake3,size) pairs for blobs, but there is currently no easy way to get this information for trees.
We want to add support to getAttributesFromFilesV2 for looking up the (Blake3,size) pairs for trees as well. To do this, we'll need to add a method for querying TreeMetadata (including size and Blake3 hashes) via the SaplingBackingStore.
This initial implementation of TreeMetadata will only include the following features:
1) Ability to query TreeMetadata from the SaplingBackingStore via a getTreeMetadata endpoint
2) The ability to request TreeMetadata from the getAttributesFromFilesV2 thrift endpoint
In the future, we will extend the implementation to support:
1) Automatically fetching TreeMetadata during BackingStore::getTree requests
2) Caching TreeMetadata in Eden's in-memory caches
# This diff
This diff adds the strucutres that we'll use to pass around TreeMetadata information. This currently only includes the directory size (in bytes) and optional Blake3 hash, but it could be extended in the future.
Reviewed By: jdelliot
Differential Revision: D58418093
fbshipit-source-id: c1de7220053be90cc0c8f627e9c6a924a50e8263
Summary:
# Context
The new AugmentedManifest format used by Sapling and Mononoke will support looking up Trees/Blobs from CAS via (Blake3,size) pairs. We already provide a way for users to get (Blake3,size) pairs for blobs, but there is currently no easy way to get this information for trees.
We want to add support to getAttributesFromFilesV2 for looking up the (Blake3,size) pairs for trees as well. To do this, we'll need to add a method for querying TreeMetadata (including size and Blake3 hashes) via the SaplingBackingStore.
This initial implementation of TreeMetadata will only include the following features:
1) Ability to query TreeMetadata from the SaplingBackingStore via a getTreeMetadata endpoint
2) The ability to request TreeMetadata from the getAttributesFromFilesV2 thrift endpoint
In the future, we will extend the implementation to support:
1) Automatically fetching TreeMetadata during BackingStore::getTree requests
2) Caching TreeMetadata in Eden's in-memory caches
# This diff
This diff tries to be consistent with naming across different crates. FileAuxData already exists, and Metadata is already an overloaded term. Therefore, we should try to move towards calling Metadata -> Aux Data across all our products. We'll start by moving all the newly named "DirectoryMetadata" structures to be called "TreeAuxData" instead.
We'll eventually start renaming things within EdenFS as well.
Reviewed By: muirdm
Differential Revision: D58483932
fbshipit-source-id: 26b81a8b1bd0776bf7006443c793be4239d6c29f
Summary:
# Context
The new AugmentedManifest format used by Sapling and Mononoke will support looking up Trees/Blobs from CAS via (Blake3,size) pairs. We already provide a way for users to get (Blake3,size) pairs for blobs, but there is currently no easy way to get this information for trees.
We want to add support to getAttributesFromFilesV2 for looking up the (Blake3,size) pairs for trees as well. To do this, we'll need to add a method for querying TreeMetadata (including size and Blake3 hashes) via the SaplingBackingStore.
This initial implementation of TreeMetadata will only include the following features:
1) Ability to query TreeMetadata from the SaplingBackingStore via a getTreeMetadata endpoint
2) The ability to request TreeMetadata from the getAttributesFromFilesV2 thrift endpoint
In the future, we will extend the implementation to support:
1) Automatically fetching TreeMetadata during BackingStore::getTree requests
2) Caching TreeMetadata in Eden's in-memory caches
# This diff
This diff adds the ability to fetch/iterate-through DirectoryMetadata of TreeEntries
Reviewed By: muirdm
Differential Revision: D58483887
fbshipit-source-id: deaceae1f0c30c4a17aa2213433ee798d3ef4a1c
Summary: `block_in_place` wouldn't be suitable as our tests run using the `current_thread` scheduler which doesn't support it. Remove the comment to avoid future confusion.
Reviewed By: lmvasquezg
Differential Revision: D59396826
fbshipit-source-id: 1e15bae31e16f094d8c9a82a979d87a858221ef9
Summary: For large manifests, this may consume lots of CPU. Ensure we don't block other tasks by yielding periodically.
Reviewed By: lmvasquezg
Differential Revision: D59396775
fbshipit-source-id: d9c1ce616cf6f0a51373bdb7c740ff7cbf63d9b0
Summary:
Switch unsharded manifest implementations to use `SortedVectorTrieMap` as their trie map type.
Because this is done through a blanket implementation, we must do this for all types at once.
Reviewed By: andreacampi
Differential Revision: D59380492
fbshipit-source-id: 9f676836798e5b64dba60ac33d727a9c4d7a10f0
Summary:
Implement a wrapper around `SortedVectorMap` that simulates a `TrieMap` without actually constructing one.
When using unsharded manifests in the sharded manifest algorithms, they need to provide the `TrieMapOps` interface. Currently we do this by converting the unsharded manifest to a fully-expanded `TrieMap`. However, for large manifests, this is expensive, and often wasteful as large parts of the manifest are likely the same anyway.
As an alternative, we can use the original `SortedVectorMap` that the unsharded manifest is built on as a basis for the trie map traversal. We track the length of the common prefix so far traversed, and the start and end positions of all entries within the sorted vector map that start with that common prefix. The individual trie map node emulators can share the original sorted vector map, so no copying of data is required (except to copy out values when they are reached).
Note that `SortedVectorTrieMap` is read-only. It can be thought of as a trie-map "view" onto the original map.
Reviewed By: andreacampi
Differential Revision: D59380491
fbshipit-source-id: be2b52aad2673361b7151ad6492de0dcdaaa12d9
Summary: As for fsnodes, prefetch the content metadata for all files before deriving hg augmentd manifest. Do this concurrently with loading the mercurial changeset, so for most changesets this will take no extra time, and prevents content metadata fetches from stalling the derivation process.
Reviewed By: andreacampi
Differential Revision: D59380489
fbshipit-source-id: 8b9f84ab21eccf8a6dddf96e5f6b7ee66d98e781
Summary:
# Context
This check was broken because the GetScmStatusV2 call required the active filter to be passed in on FilteredFS repos.
# This diff
Fetches the active filter id from the getSnapshotInfo endpoint and then passes that filter id into the getScmStatusV2 endpoint so that it doesn't fail.
Reviewed By: kmancini
Differential Revision: D57345438
fbshipit-source-id: 5efe9426e1788770d3e5ef640e4891092cd9decc
Summary: Add tests for previous diff. This is a new test suite since OSS tests only have a single field with image uploads.
Reviewed By: zzl0
Differential Revision: D59343164
fbshipit-source-id: e5a557fcfc7c554e53b23476fe711d46932cbda1
Summary:
Previously, image uploads weren't keyed by field. So if you had two fields which could accept image uploads, uploading to one would show a spinner in both, and errors in either would appear in both.
We just need to make our data store a fieldName key which we use to filter the underlying data.
Reviewed By: zzl0
Differential Revision: D59343165
fbshipit-source-id: 7c8b2308b7c991e9ad495545c6c33db16ff77bec
Summary:
Rather than trying to make the switch in one go, I'm introducing a new constructor. This will allow me to add some callsite at a time and test thoroughly. As long as the content of the DB and Configerator match, this should not cause any observable behavior, regardless of which code path is taken.
When the DB is empty (which is the case now) this is a giant noop anyway.
Differential Revision: D59367428
fbshipit-source-id: 08e9ac0734ad280381379ce8089bc214489e808c
Summary:
# Context
We'd like to add log rotation for EdenFS (and eventually Watchman, Myles, etc). There are services that already exists on Linux and macOS (logrotate and newsyslog respectively). However, these services require our daemon's to implement log-reopening logic in order to work correctly (on Linux) or at all (on macOS).
We could deal with slightly correctness issues on Linux, but since we need log-reopening logic for log rotation to work at all on macOS, we should implement log-reopening so that we can use log rotation on at least two of our environments.
# Implementation
The basic way to implement log-reopening in Eden is to add a signal handler for SIGHUP. SIGHUP is used to indicate "hanging up" (in other words, the file we're writing to is now closed). When we receive SIGHUP, we can reopen the log file that we're writing to so that we can continue to persist our logs.
# This diff
This diff adds SIGHUP signal handling that reopens the log file. It also moves the log file path to be a global variable since signal handlers must be static void functions.
Reviewed By: chadaustin
Differential Revision: D59303686
fbshipit-source-id: c8ff0c54f9720388b2516c8074716a4dc3f2fbc5
Summary:
# Context
We'd like to add log rotation for EdenFS (and eventually Watchman, Myles, etc). There are services that already exists on Linux and macOS (logrotate and newsyslog respectively). However, these services require our daemon's to implement log-reopening logic in order to work correctly (on Linux) or at all (on macOS).
We could deal with slightly correctness issues on Linux, but since we need log-reopening logic for log rotation to work at all on macOS, we should implement log-reopening so that we can use log rotation on at least two of our environments.
# Implementation
The basic way to implement log-reopening in Eden is to add a signal handler for SIGHUP. SIGHUP is used to indicate "hanging up" (in other words, the file we're writing to is now closed). When we receive SIGHUP, we can reopen the log file that we're writing to so that we can continue to persist our logs.
# This diff
This diff adds an integration test to show that SIGHUP handling doesn't work right now. The next diff should fix the test and prove that we handle SIGHUP correctly (and it fixes writing to a closed log file).
Reviewed By: chadaustin
Differential Revision: D59303688
fbshipit-source-id: dbbd83ba7078995918fd592d78bcae95f574e183
Summary:
# Context
We'd like to add log rotation for EdenFS (and eventually Watchman, Myles, etc). There are services that already exists on Linux and macOS (logrotate and newsyslog respectively). However, these services require our daemon's to implement log-reopening logic in order to work correctly (on Linux) or at all (on macOS).
We could deal with slightly correctness issues on Linux, but since we need log-reopening logic for log rotation to work at all on macOS, we should implement log-reopening so that we can use log rotation on at least two of our environments.
# Implementation
The basic way to implement log-reopening in Eden is to add a signal handler for SIGHUP. SIGHUP is used to indicate "hanging up" (in other words, the file we're writing to is now closed). When we receive SIGHUP, we can reopen the log file that we're writing to so that we can continue to persist our logs.
# This diff
This diff adds a stub for SIGHUP handling so that I can write an integration test in the next diff that avoids crashing the Eden daemon (sending SIGHUP to a process that doesn't handle SIGHUP will cause that process to exit).
Reviewed By: chadaustin
Differential Revision: D59303687
fbshipit-source-id: dd924957beac71868aea3adea37992733de64df3
Summary: I seem to have missed some endpoints when I did D56042552.
Reviewed By: zzl0
Differential Revision: D59305679
fbshipit-source-id: 158797cb561c779b5999dfe3812e9d4e42c009b8
Summary: `PushRedirection` was very confusing, especially when used next to `unbundle::PushRedirector` and `wireproto_handler::PushRedirectorBase`.
Reviewed By: gustavoavena
Differential Revision: D59369155
fbshipit-source-id: f4d626be157de35b22f49c2e55d05d8a75cf7a66
Summary:
A key part of the `git push` payload is the `packfile` which contains the all the Git objects that the client wants to send over to the server. This packfile is encoded, compressed and deltified.
This diff adds the capability to parse the packfile and verify its validity. It produces a stream of objects that need to be persisted in Mononoke blobstore (which will be handled in a follow-up diff)
Reviewed By: andreacampi
Differential Revision: D59332462
fbshipit-source-id: 5c7b24649c17293c4bb69194dcfae295c9de621d
Summary:
Before this change, there were two ways to interact with data derivation:
* `derive_exactly_batch`: a function that can be used efficiently but requires a lot of care (for instance, knowing for sure that the parents of each commit in the batch are already derived and ensuring the ordering in the batch is topological)
* `derive`: a function that's easy to use, and "does the right thing", but does so very inefficiently, with no batching and "O(n_commits * derived_data_size)" memory usage
The rule of thumb was: practically never call `derive` except if you're sure it's delegating to the derived data service that does the right thing of calling the batched interface or you will cause a SEV eventually.
Reconcile both of these implementations and sprinkle some `CommitGraph` magic on top so we can have the best of both worlds.
NOTE: We sacrifice a tiny surface of functionality, namely, logging the time spent finding underived commits, but we also make it so much more efficient that it really shouldn't matter anymore)
Reviewed By: YousefSalama
Differential Revision: D58202861
fbshipit-source-id: f30406ef9a8f0042e0f30ff9ea6f73184de775f8
Summary:
In the child diff, we replace the implementation of `derive` with a new version that sections the work into batches and derives in batches.
It will be useful to configure the behaviour so that for each individual derived data type, we can specify a reasonable batch size.
Add the config mechanics now and use them in the child diff.
Reviewed By: YousefSalama
Differential Revision: D59329112
fbshipit-source-id: fc58e2671163e8657b295f684135bf100c980955
Summary: Small API change to simplify / split the rest of the stack.
Reviewed By: RajivTS
Differential Revision: D59328461
fbshipit-source-id: 60b195b5f92e3f82082261345083a679a27c4578
Summary: Small API change to simplify / split the rest of the stack.
Reviewed By: RajivTS
Differential Revision: D59328417
fbshipit-source-id: e44c26ae671ac85250cb23fb77eaf509468d5a71
Summary:
The current plan is
{F1737830691}
Util now mononoke supported only MTLS, this diff allows using TLS.
Reviewed By: andreacampi
Differential Revision: D59278026
fbshipit-source-id: 3bc81e4bb4fdb9812e23663590fa7e5fd496f705
Summary: Adds a "linear" flag to `scsc log` to allow querying the linear history of a commit. This flag is not supported with any of the "path", "before" or "after" flags.
Reviewed By: markbt
Differential Revision: D59327662
fbshipit-source-id: caf7067c32a264594538f3d08670ca920f86f453
Summary: Uses the commit graph `LinearAncestorsStreamBuilder` to implement SCS method `commit_linear_history`. Mostly a copy of `commit_history` without the timestamp parameters, and with the skip parameter handled in `LinearAncestorsStreamBuilder` not applied on top of the commit graph stream.
Reviewed By: markbt
Differential Revision: D59324409
fbshipit-source-id: 8e6f5f479371da44b8be1b4b01f87e02aaaa92f0
Summary:
Implements a builder for streams of linear ancestors. The builder allows constraining the stream to exclude the linear ancestors of a changeset, or to include only the linear descendants of a changeset. It additionally allows efficiently skipping the first N linear ancestors of the stream.
This will be used to power commit_linear_history SCS method.
Reviewed By: markbt
Differential Revision: D59324016
fbshipit-source-id: 8d5f3ac9f3f13d516457449cf28c916b02e20a8d
Summary: It was accessing the skip tree instead of the p1 linear tree.
Reviewed By: andreacampi
Differential Revision: D59323964
fbshipit-source-id: 4ad30964a38b9b97f3cd75e88b9ef655cb6ecc93
Summary:
The method doesn't calculate the p1 linear lowest common ancestor, but the skip tree one instead. Modify unit test `test_p1_linear_tree` graph to showcase this bug.
I changed to a vertical drawdag because the horizontal drawdag was giving confusing parent order. Now the order of all parents is from left to right.
Reviewed By: andreacampi
Differential Revision: D59323881
fbshipit-source-id: eccdf70042c369c565c2ae4f949d0bbccd85eaa8
Summary: Skipping is done in O(log) so we don't have to constrain the skip parameter to i32. None of our repos is that large but there's no reason not to use i64. This is not used so the thrift error is safe to bypass.
Reviewed By: andreacampi
Differential Revision: D59324153
fbshipit-source-id: 2ce8e7257297a895c4886077f6510312c9495444
Summary:
The `test_writing_untracked_file_bumps_write_counter` test was in the skip list for windows. The reason was the difference between the counter names in windows.
This diff fix the counter names for Windows platform in this test and remove this test from the skip list
Reviewed By: kmancini
Differential Revision: D58888655
fbshipit-source-id: c4027cf2b3e67b635c216a8e31050be8c7034fdc
Summary: Explanation of how stats are working at EdenFS
Reviewed By: genevievehelsel, kmancini
Differential Revision: D59252923
fbshipit-source-id: 126085c2f78abd15e00ebc39ab0a3734af141857
Summary:
The cross-repo sync libraries construct really large futures that are on the border of causing stack overflows.
With D58202861, some shuffling of how derived data gets evaluated brings it over the limit and causes stack overflows in unit tests and integration tests, which manifests in SEGFAULTS.
This diff breaks down the stack in all the API functions that the cross-repo sync exposes (forward and backward sync), by boxing large futures.
Reviewed By: YousefSalama
Differential Revision: D59329111
fbshipit-source-id: 73037da9a06b69d47d1fe6ffbaffc4d1e7e5c775
Summary: This makes tests hang for some reason when running from Emacs. Didn't investigate, but --chg isn't important now that most tests supports debugruntest, so just remove it.
Reviewed By: quark-zju
Differential Revision: D59232877
fbshipit-source-id: c1fc0f2a4099e8255bef62073b5f9259c0a33c56