Summary: Add support for deriving all types of derived data for a given repository
Reviewed By: StanislavGlebik
Differential Revision: D23842909
fbshipit-source-id: 2fa5c4a9444169b26c5cf70d91a6cc707cca8022
Summary: This change make it possible to find underived changesets for multiple derived data types at once, and then batch derive everything taking into account dependencies between derived data types.
Reviewed By: StanislavGlebik
Differential Revision: D23762152
fbshipit-source-id: 8112fc5deb86f3c55f2fa5079cf917a0506f045c
Summary:
When the server returns a 429, the intention is that the client will wait for a
little bit then try again later (there is no harm in that, as we haven't really
used many server resources for this). However, it turned out that right now we
just abort. Let's fix it!
Note that this changes the behavior a bit for the error cases, in the sense
that we no longer return `Ok(None)` but instead return an `Err`. Xavier noted
this should make sense here.
I've also had the client send its retry attempt via a header, because who
knows, that might be useful.
Reviewed By: kulshrax
Differential Revision: D24308127
fbshipit-source-id: 35639956f36342dfb0056b0d348dc4ad56bd576c
Summary:
This is Ruby's `iter.each_slice(n)` or Rust's `slice.chunks(n)`.
It will be used by phabstatus to batch requests.
Reviewed By: singhsrb
Differential Revision: D24289050
fbshipit-source-id: 528b35d43874222f1b6783738e965d9a3b159deb
Summary: Make it easier to iterate contexes instead of revs.
Reviewed By: simpkins
Differential Revision: D24289053
fbshipit-source-id: deffdc4a5d4e7e3644a98f8112f5c8b8bdbd154d
Summary:
Previously `--master` falls back to `null` if `-r` is provided for
compatibility. That could be surprising as smartlog's sorting does not work
well without a master branch.
Change `--master` to a consistent value regardless of `-r` is provided or not.
Reviewed By: simpkins
Differential Revision: D24297586
fbshipit-source-id: c6e695333ae17db11dcb98d9cdb57cc2f298ff64
Summary: Introduces fetching of child entry IDs, and child file metadata for a specified tree manifest ID. The aux data lookup will only be performed if `with_file_metadata` is set, which is actually kind of wrong. Instead `with_children` from the wire type should be exposed in the API request type, and `with_*_metadata` should be hidden or used for data other than the child entry `Key`s.
Reviewed By: kulshrax
Differential Revision: D23886678
fbshipit-source-id: 0cba72cea7be47ae3348a406d407a19b60976c0c
Summary: This code predates the use of `pin_project`, which simplifies the implementation of pinned types using proc macros. We now use `pin_project` in most other places in this crate where we define custom `Future`s and `Stream`s, so let's use it in `SignalStream` as well. (This makes the code easier to modify and removes the need for `unsafe` code.)
Reviewed By: farnz
Differential Revision: D24285792
fbshipit-source-id: da018b9ed62c33d9d06ecf6caa4748969a057e66
Summary:
Add a stream combinator that causes a `TryStream` to end upon encountering the error. The resulting stream remains a `TryStream`, but will always terminate after yielding an error item.
Later in this stack, this will be used in the LFS server to cause mid-stream errors to always terminate the response.
Reviewed By: krallin
Differential Revision: D24301254
fbshipit-source-id: e71ddc16e79caeb195e23be7d6a93ee63a96a713
Summary:
We had a small issue with our logging when pushredirection was enabled. See for
example https://fburl.com/scuba/mononoke_test_perf/orsld6yh. If you count the
number of "Start processing" and "Command processed" entries you'll notice that
there's one more instance of the former. That's wrong, and turned out this
additional "Start processing" line is coming from backsyncing.
Turned out that the problem was two fold:
1) start_command() logs "Start processing" row and also it sets "Start
processing" as default message for every next scuba calls. That doesn't seem
intentional, so this diff clones scuba sample before logging "Start
processing" to avoid "poisoning" the "tag" field
2) Backsyncing weren't setting the log_tag at all. So this diff fixes it
Reviewed By: ikostia
Differential Revision: D24305167
fbshipit-source-id: 3448b88693e0c5c8f5a9b90f602bc2dfe99db15a
Summary: Some of these tests were already passing. Some required minor fixes to pass.
Reviewed By: quark-zju
Differential Revision: D24290635
fbshipit-source-id: 789d3033142466c868057da8f9f23d27aede8ea4
Summary:
They now pass. Didn't test the p4 tests since I don't have p4
available, but I don't think we use it anymore.
Reviewed By: quark-zju
Differential Revision: D24284643
fbshipit-source-id: 184697052c0b7000ad9765b4033964b4d382b811
Summary:
Template filters are expecting the output of the template to be a unicode
string, not a byte string. Thus we need to ensure that diff return a unicode
string.
Reviewed By: DurhamG
Differential Revision: D24290908
fbshipit-source-id: 70899249c3e26edd1714fd901fa6af405b61e544
Summary:
This reverts D24066918 (a684e53a31). That diff changed the phabstatus code to use the last
created revset iterator rather than the first for peekahead purposes.
The first revset iterator does appear to be the correct one to wrap. After
D24066918 (a684e53a31) I consistently see `hg log -r 'draft()' -T'{phabstatus}\n'` make
Phabricator queries for only one diff at a time. Reverting the changes from
that diff fixes the issue.
Note that the revset argument passed to `hg log` appears to make a difference
in the behavior here. When using simple revsets including just specific
commit ranges then the peekahead behavior works as expected both before and
after D24066918 (a684e53a31). However, when using `draft()` in the revset specifier then
the behavior works before D24066918 (a684e53a31), but not after it.
Reviewed By: singhsrb
Differential Revision: D24269910
fbshipit-source-id: c2c2abf6360fe3129a1fc73b5b433b763bce1fd2
Summary:
In D19023924 (e9df42c192) ikostia allowed non-prefix free movers, however he left the
safeguard in just in case. Well, now we finally need to use non-prefix-free
mapping, so let's remove the safeguard.
Reviewed By: ikostia
Differential Revision: D24277040
fbshipit-source-id: 4d658ad813171ab0dcb23656e95e3e443ec9961a
Summary:
This is a tool that's handy to use right before the binding. It accepts a list
of large repo changesets, and for each commit does not yet have a mapping it
inserts a NotSyncCandidate. This is exactly what we want to do right before the
bind - we'd like large repo to have most of its commits marked as
NotSyncCandidate.
Reviewed By: ikostia
Differential Revision: D24276642
fbshipit-source-id: b040e5dee981c06da8a4c58dacbb8c2660f0917d
Summary:
The infinitepush sqlindex was returning bytes instead of strings, which
caused various formatters to serialize as "b'sadf'" instead of "sadf". Let's fix
the type converter to produce strings on python 3. Note, this is the opposite to
what we do in the hgsql converter. hgsql works with binary data, so we default
to bytes there. The sqlindex doesn't deal with any binary data, so defaulting to
strings is better.
Also updates a comment on the hgsql converter to be accurate. I confirmed there
are no other customer converters in the code base that need updating.
Reviewed By: krallin
Differential Revision: D24282899
fbshipit-source-id: 708732c17626f6ebc2c3cbe7f00afa5771f31d7b
Summary:
Likely `MPath` constructor got more clever since this was written. In any case
I am just happy that it builds.
Reviewed By: farnz
Differential Revision: D24278551
fbshipit-source-id: 2378a43bd01b1826b7f81d294a035053fba5322f
Summary:
Bookmark filler doesn't make much sense outside of FB. In fact the commit
filler is already in the `facebook/` dir.
D24253307 contains the fbpkg change that has to be landed in-sync with this one.
Reviewed By: lukaspiatkowski
Differential Revision: D24253070
fbshipit-source-id: 52734ae34779801b4cae4882a6d0880586ef505f
Summary:
Include the client correlator string from the `clienttelemetry` extension in each EdenAPI HTTP request via the `X-Client-Correlator` header.
The `ClientIdentityMiddleware` in `gotham_ext` already understands this header (as it is already used by the LFS server), and `gotham_ext`'s `ScubaMiddleware` will automatically include the provided correlator in the server's Scuba samples.
Reviewed By: farnz
Differential Revision: D24282244
fbshipit-source-id: 13d04e706eda38893cff6e740bd1d7bf104e43dd
Summary: Add option to redirect the eden rage output to stderr
Reviewed By: chadaustin
Differential Revision: D24210031
fbshipit-source-id: 736af81de6a41e49c2459a1cd8124123d6928323
Summary: For the purposes of megarepo, we need to treat certain paths differently depending on whether the push is native to a large repo, or is push-redirected from a small repo. Let's add a separate patterns list to the `deny_files` hook. Of course, we can also add a list, specific for push-redirected pushes, but it seems unneeded atm.
Reviewed By: StanislavGlebik
Differential Revision: D24278409
fbshipit-source-id: 9fd815940bb656ceac6ab234f3a0647a5c57db06
Summary:
If we fail to read from MySQL, we immediately go back to attempting to read.
That's the exact opposite of what we should be doing. This fixes that by not
changing our poll schedule if we hit an error.
Reviewed By: farnz
Differential Revision: D24279086
fbshipit-source-id: 2c05ec9c33a1e0cbdfb63dda63a68f2a93615512
Summary:
This diff adds all third party dependencies that are required by getdeps to be able to build and runn Mononoke's integration tests.
Also add a stub Makefile with no-op steps that will be filled in next diff.
Reviewed By: ahornby
Differential Revision: D24251894
fbshipit-source-id: 67384ecfd0ced6762dddc3c6e61feb1240b1162d
Summary: Reorder scrub walk expansion to reduce queue depth. The bounded_traversal_stream inserts to its unscheduled queue at the front, therefore reversing the order of the children, so we want to add the easiest to complete Nodes to the list of edges last.
Reviewed By: farnz
Differential Revision: D24247625
fbshipit-source-id: 59caa5898e7f38f41cc04a15723370de38f8474f
Summary:
Delegate chaosblob put to put_impl to remove some duplication.
This is possible as all chaosblob construction has an inner BlobstorePutOps.
Reviewed By: StanislavGlebik
Differential Revision: D24258401
fbshipit-source-id: dee9fce888e5ef9c2f34865c97921b9cc87ac3bf
Summary:
This change adds some wiring to allow the hooks to treat native vs push-redirected pushes differently. This is needed because `deny-files` needs to block native pushes to `.ovrsource-rest` and `arvr-legacy`, while it needs to allow push-redirection into these directories.
The plan further is to change the actual hook body for `deny_files` to have different handling of the two cases.
Reviewed By: StanislavGlebik
Differential Revision: D24257454
fbshipit-source-id: 2f5931149115210aeeeebb3294a6512effd36350
Summary:
The logic inside `eden/scm/tests/features.py` script enables certain
features of HG on tests based on the name of the test. Mononoke's integration
tests suit reuses eden/scm's tests suit and as a consequence it triggers the
same `features.py` logic. It was fine until D24201934 (f5988c415c) introduced a feature for
test that is named in the same way as one of the Mononoke tests - the
`test-clienttelemetry.t`.
In order to fix this problem Mononoke will pass a `--nofeatures` flag to the
`eden/scm/run-tests.py` invocations that will turn off the usage of
`features.py`.
Reviewed By: farnz
Differential Revision: D24276294
fbshipit-source-id: eb28ed55a05de3b012e37407603c2370adaaad16
Summary:
mononoke_api crate which provides high-level abstraction over most of mononoke
has been used in externally facing scs_server so far. That's why all of it's entry
points are validating user identity first.
In D24136239 (42a783999d) I've started using it for iternal sync job. In this world all
operations are authorized and there's no need for extra ACL checks.
I could just add additional arg to the new method but I don't want to make it
easy to accidentally skip ACL checks - that's why I went for separate method
with descriptive namea.
Reviewed By: krallin
Differential Revision: D24261117
fbshipit-source-id: 32f601d970d286cd383ba5c488596d64d09a535e
Summary: Factor out common async limit code in throttledblob to throttled_access function.
Reviewed By: StanislavGlebik
Differential Revision: D24254357
fbshipit-source-id: 8f3277ccc065e3eb93f089edb2f3bd12e80ef352
