Summary:
Switch unsharded manifest implementations to use `SortedVectorTrieMap` as their trie map type.
Because this is done through a blanket implementation, we must do this for all types at once.
Reviewed By: andreacampi
Differential Revision: D59380492
fbshipit-source-id: 9f676836798e5b64dba60ac33d727a9c4d7a10f0
Summary:
Implement a wrapper around `SortedVectorMap` that simulates a `TrieMap` without actually constructing one.
When using unsharded manifests in the sharded manifest algorithms, they need to provide the `TrieMapOps` interface. Currently we do this by converting the unsharded manifest to a fully-expanded `TrieMap`. However, for large manifests, this is expensive, and often wasteful as large parts of the manifest are likely the same anyway.
As an alternative, we can use the original `SortedVectorMap` that the unsharded manifest is built on as a basis for the trie map traversal. We track the length of the common prefix so far traversed, and the start and end positions of all entries within the sorted vector map that start with that common prefix. The individual trie map node emulators can share the original sorted vector map, so no copying of data is required (except to copy out values when they are reached).
Note that `SortedVectorTrieMap` is read-only. It can be thought of as a trie-map "view" onto the original map.
Reviewed By: andreacampi
Differential Revision: D59380491
fbshipit-source-id: be2b52aad2673361b7151ad6492de0dcdaaa12d9
Summary: As for fsnodes, prefetch the content metadata for all files before deriving hg augmentd manifest. Do this concurrently with loading the mercurial changeset, so for most changesets this will take no extra time, and prevents content metadata fetches from stalling the derivation process.
Reviewed By: andreacampi
Differential Revision: D59380489
fbshipit-source-id: 8b9f84ab21eccf8a6dddf96e5f6b7ee66d98e781
Summary:
# Context
This check was broken because the GetScmStatusV2 call required the active filter to be passed in on FilteredFS repos.
# This diff
Fetches the active filter id from the getSnapshotInfo endpoint and then passes that filter id into the getScmStatusV2 endpoint so that it doesn't fail.
Reviewed By: kmancini
Differential Revision: D57345438
fbshipit-source-id: 5efe9426e1788770d3e5ef640e4891092cd9decc
Summary: Add tests for previous diff. This is a new test suite since OSS tests only have a single field with image uploads.
Reviewed By: zzl0
Differential Revision: D59343164
fbshipit-source-id: e5a557fcfc7c554e53b23476fe711d46932cbda1
Summary:
Previously, image uploads weren't keyed by field. So if you had two fields which could accept image uploads, uploading to one would show a spinner in both, and errors in either would appear in both.
We just need to make our data store a fieldName key which we use to filter the underlying data.
Reviewed By: zzl0
Differential Revision: D59343165
fbshipit-source-id: 7c8b2308b7c991e9ad495545c6c33db16ff77bec
Summary:
Rather than trying to make the switch in one go, I'm introducing a new constructor. This will allow me to add some callsite at a time and test thoroughly. As long as the content of the DB and Configerator match, this should not cause any observable behavior, regardless of which code path is taken.
When the DB is empty (which is the case now) this is a giant noop anyway.
Differential Revision: D59367428
fbshipit-source-id: 08e9ac0734ad280381379ce8089bc214489e808c
Summary:
# Context
We'd like to add log rotation for EdenFS (and eventually Watchman, Myles, etc). There are services that already exists on Linux and macOS (logrotate and newsyslog respectively). However, these services require our daemon's to implement log-reopening logic in order to work correctly (on Linux) or at all (on macOS).
We could deal with slightly correctness issues on Linux, but since we need log-reopening logic for log rotation to work at all on macOS, we should implement log-reopening so that we can use log rotation on at least two of our environments.
# Implementation
The basic way to implement log-reopening in Eden is to add a signal handler for SIGHUP. SIGHUP is used to indicate "hanging up" (in other words, the file we're writing to is now closed). When we receive SIGHUP, we can reopen the log file that we're writing to so that we can continue to persist our logs.
# This diff
This diff adds SIGHUP signal handling that reopens the log file. It also moves the log file path to be a global variable since signal handlers must be static void functions.
Reviewed By: chadaustin
Differential Revision: D59303686
fbshipit-source-id: c8ff0c54f9720388b2516c8074716a4dc3f2fbc5
Summary:
# Context
We'd like to add log rotation for EdenFS (and eventually Watchman, Myles, etc). There are services that already exists on Linux and macOS (logrotate and newsyslog respectively). However, these services require our daemon's to implement log-reopening logic in order to work correctly (on Linux) or at all (on macOS).
We could deal with slightly correctness issues on Linux, but since we need log-reopening logic for log rotation to work at all on macOS, we should implement log-reopening so that we can use log rotation on at least two of our environments.
# Implementation
The basic way to implement log-reopening in Eden is to add a signal handler for SIGHUP. SIGHUP is used to indicate "hanging up" (in other words, the file we're writing to is now closed). When we receive SIGHUP, we can reopen the log file that we're writing to so that we can continue to persist our logs.
# This diff
This diff adds an integration test to show that SIGHUP handling doesn't work right now. The next diff should fix the test and prove that we handle SIGHUP correctly (and it fixes writing to a closed log file).
Reviewed By: chadaustin
Differential Revision: D59303688
fbshipit-source-id: dbbd83ba7078995918fd592d78bcae95f574e183
Summary:
# Context
We'd like to add log rotation for EdenFS (and eventually Watchman, Myles, etc). There are services that already exists on Linux and macOS (logrotate and newsyslog respectively). However, these services require our daemon's to implement log-reopening logic in order to work correctly (on Linux) or at all (on macOS).
We could deal with slightly correctness issues on Linux, but since we need log-reopening logic for log rotation to work at all on macOS, we should implement log-reopening so that we can use log rotation on at least two of our environments.
# Implementation
The basic way to implement log-reopening in Eden is to add a signal handler for SIGHUP. SIGHUP is used to indicate "hanging up" (in other words, the file we're writing to is now closed). When we receive SIGHUP, we can reopen the log file that we're writing to so that we can continue to persist our logs.
# This diff
This diff adds a stub for SIGHUP handling so that I can write an integration test in the next diff that avoids crashing the Eden daemon (sending SIGHUP to a process that doesn't handle SIGHUP will cause that process to exit).
Reviewed By: chadaustin
Differential Revision: D59303687
fbshipit-source-id: dd924957beac71868aea3adea37992733de64df3
Summary: I seem to have missed some endpoints when I did D56042552.
Reviewed By: zzl0
Differential Revision: D59305679
fbshipit-source-id: 158797cb561c779b5999dfe3812e9d4e42c009b8
Summary: `PushRedirection` was very confusing, especially when used next to `unbundle::PushRedirector` and `wireproto_handler::PushRedirectorBase`.
Reviewed By: gustavoavena
Differential Revision: D59369155
fbshipit-source-id: f4d626be157de35b22f49c2e55d05d8a75cf7a66
Summary:
A key part of the `git push` payload is the `packfile` which contains the all the Git objects that the client wants to send over to the server. This packfile is encoded, compressed and deltified.
This diff adds the capability to parse the packfile and verify its validity. It produces a stream of objects that need to be persisted in Mononoke blobstore (which will be handled in a follow-up diff)
Reviewed By: andreacampi
Differential Revision: D59332462
fbshipit-source-id: 5c7b24649c17293c4bb69194dcfae295c9de621d
Summary:
Before this change, there were two ways to interact with data derivation:
* `derive_exactly_batch`: a function that can be used efficiently but requires a lot of care (for instance, knowing for sure that the parents of each commit in the batch are already derived and ensuring the ordering in the batch is topological)
* `derive`: a function that's easy to use, and "does the right thing", but does so very inefficiently, with no batching and "O(n_commits * derived_data_size)" memory usage
The rule of thumb was: practically never call `derive` except if you're sure it's delegating to the derived data service that does the right thing of calling the batched interface or you will cause a SEV eventually.
Reconcile both of these implementations and sprinkle some `CommitGraph` magic on top so we can have the best of both worlds.
NOTE: We sacrifice a tiny surface of functionality, namely, logging the time spent finding underived commits, but we also make it so much more efficient that it really shouldn't matter anymore)
Reviewed By: YousefSalama
Differential Revision: D58202861
fbshipit-source-id: f30406ef9a8f0042e0f30ff9ea6f73184de775f8
Summary:
In the child diff, we replace the implementation of `derive` with a new version that sections the work into batches and derives in batches.
It will be useful to configure the behaviour so that for each individual derived data type, we can specify a reasonable batch size.
Add the config mechanics now and use them in the child diff.
Reviewed By: YousefSalama
Differential Revision: D59329112
fbshipit-source-id: fc58e2671163e8657b295f684135bf100c980955
Summary: Small API change to simplify / split the rest of the stack.
Reviewed By: RajivTS
Differential Revision: D59328461
fbshipit-source-id: 60b195b5f92e3f82082261345083a679a27c4578
Summary: Small API change to simplify / split the rest of the stack.
Reviewed By: RajivTS
Differential Revision: D59328417
fbshipit-source-id: e44c26ae671ac85250cb23fb77eaf509468d5a71
Summary:
The current plan is
{F1737830691}
Util now mononoke supported only MTLS, this diff allows using TLS.
Reviewed By: andreacampi
Differential Revision: D59278026
fbshipit-source-id: 3bc81e4bb4fdb9812e23663590fa7e5fd496f705
Summary: Adds a "linear" flag to `scsc log` to allow querying the linear history of a commit. This flag is not supported with any of the "path", "before" or "after" flags.
Reviewed By: markbt
Differential Revision: D59327662
fbshipit-source-id: caf7067c32a264594538f3d08670ca920f86f453
Summary: Uses the commit graph `LinearAncestorsStreamBuilder` to implement SCS method `commit_linear_history`. Mostly a copy of `commit_history` without the timestamp parameters, and with the skip parameter handled in `LinearAncestorsStreamBuilder` not applied on top of the commit graph stream.
Reviewed By: markbt
Differential Revision: D59324409
fbshipit-source-id: 8e6f5f479371da44b8be1b4b01f87e02aaaa92f0
Summary:
Implements a builder for streams of linear ancestors. The builder allows constraining the stream to exclude the linear ancestors of a changeset, or to include only the linear descendants of a changeset. It additionally allows efficiently skipping the first N linear ancestors of the stream.
This will be used to power commit_linear_history SCS method.
Reviewed By: markbt
Differential Revision: D59324016
fbshipit-source-id: 8d5f3ac9f3f13d516457449cf28c916b02e20a8d
Summary: It was accessing the skip tree instead of the p1 linear tree.
Reviewed By: andreacampi
Differential Revision: D59323964
fbshipit-source-id: 4ad30964a38b9b97f3cd75e88b9ef655cb6ecc93
Summary:
The method doesn't calculate the p1 linear lowest common ancestor, but the skip tree one instead. Modify unit test `test_p1_linear_tree` graph to showcase this bug.
I changed to a vertical drawdag because the horizontal drawdag was giving confusing parent order. Now the order of all parents is from left to right.
Reviewed By: andreacampi
Differential Revision: D59323881
fbshipit-source-id: eccdf70042c369c565c2ae4f949d0bbccd85eaa8
Summary: Skipping is done in O(log) so we don't have to constrain the skip parameter to i32. None of our repos is that large but there's no reason not to use i64. This is not used so the thrift error is safe to bypass.
Reviewed By: andreacampi
Differential Revision: D59324153
fbshipit-source-id: 2ce8e7257297a895c4886077f6510312c9495444
Summary:
The `test_writing_untracked_file_bumps_write_counter` test was in the skip list for windows. The reason was the difference between the counter names in windows.
This diff fix the counter names for Windows platform in this test and remove this test from the skip list
Reviewed By: kmancini
Differential Revision: D58888655
fbshipit-source-id: c4027cf2b3e67b635c216a8e31050be8c7034fdc
Summary: Explanation of how stats are working at EdenFS
Reviewed By: genevievehelsel, kmancini
Differential Revision: D59252923
fbshipit-source-id: 126085c2f78abd15e00ebc39ab0a3734af141857
Summary:
The cross-repo sync libraries construct really large futures that are on the border of causing stack overflows.
With D58202861, some shuffling of how derived data gets evaluated brings it over the limit and causes stack overflows in unit tests and integration tests, which manifests in SEGFAULTS.
This diff breaks down the stack in all the API functions that the cross-repo sync exposes (forward and backward sync), by boxing large futures.
Reviewed By: YousefSalama
Differential Revision: D59329111
fbshipit-source-id: 73037da9a06b69d47d1fe6ffbaffc4d1e7e5c775
Summary: This makes tests hang for some reason when running from Emacs. Didn't investigate, but --chg isn't important now that most tests supports debugruntest, so just remove it.
Reviewed By: quark-zju
Differential Revision: D59232877
fbshipit-source-id: c1fc0f2a4099e8255bef62073b5f9259c0a33c56
Summary:
fix broken test
15 to 16 is expected to change since we added a new dd type but this test was
missed in D59230541 since CI for that diff failed to show the failing test
Reviewed By: YousefSalama
Differential Revision: D59324139
fbshipit-source-id: 4334edb0fd556510ca77dd0fd53962cfabd0de9b
Summary:
Minor improvements to the submodule expansion validation code.
- Provide more specific context about which submodule expansion failed validation.
- Print and log what were the unexpected files that led to validation failing.
Reviewed By: RajivTS
Differential Revision: D59272753
fbshipit-source-id: b0573d296810c7a763cd0faa164b818c61b310de
Summary: The parent diff had a few review comments + I wanted to do some extra cleanup so I combined those together in this diff.
Reviewed By: gustavoavena
Differential Revision: D59320456
fbshipit-source-id: ab739d00eba519dcccf8a9928accc484a3fe8022
Summary:
The core part of `git push` is handled by the `git-receive-pack` endpoint. This diff adds all the necessary scaffolding to make the communication between the client and the server possible. However, it does not actually move the bookmark just yet. Given that the surrounding code itself is complex enough, I have decided to split that into two different diffs.
Follow up diffs will implement the actual logic of invoking hooks and moving bookmarks.
Reviewed By: gustavoavena
Differential Revision: D59285259
fbshipit-source-id: 6f580f8bf7f993389e48d22e4879ebfaa9e3ea43
Summary:
add a compatibility test mononoke with sapling!
it is very important to have a compatibility test between the types
Reviewed By: RajivTS
Differential Revision: D59274564
fbshipit-source-id: bd3178712f91c811d75142461bc8d6f9fd48d4e0
Summary:
Context: For querying commit history from SCS we have the commit_history method. However, depending on the parameters passed to this method, it can be very inefficient. Most of the inefficiency comes from having to handle merge commits in the history, but most of our clients do not actually care about merges. Therefore I'm going to introduce another method that ignores merge commits, which allows it to be implemented in a way that's efficient for all possible parameters (efficient as in O((output size) * log))
Adds a new method for querying the linear ancestors of a commit. Linear ancestors of a commit are commits that can be reached by repeatedly following the first parent, including the commit itself. I'm explicilty ignoring the `after_timestamp` and `before_timestamp` parameters because they are well known to cause users to write inefficient queries.
Reviewed By: markbt
Differential Revision: D59273639
fbshipit-source-id: bd5c92b7b46b56637e91600f0a3c73aa084f4a58
Summary: I thought `valuable` was enabled; it wasn't. Let's *actually* enable it.
Reviewed By: Wilfred
Differential Revision: D59301882
fbshipit-source-id: 60337d3b69f8babb4cd6f12d044fd2f856539056
