Summary:
The `support_bundle2_listkeys` flag controls at runtime whether we support
`listkeys` in bundle2. Since this was added before tunables were available,
it uses a value in the mutable counters SQL store.
We could migrate this to tunables, but in practice we have never disabled it,
so let's just make it the default.
Reviewed By: krallin
Differential Revision: D21546246
fbshipit-source-id: 066a375693757ea841ecf0fddb0cc91dc144fd6f
Summary:
When the client pulls draft commits, include mutation information in the bundle
response.
Reviewed By: farnz
Differential Revision: D20871339
fbshipit-source-id: a89a50426fbd8f9ec08bbe43f16fd0e4e3424e0b
Summary:
Advertise support for `b2x:infinitepushmutation`. When the client sends us
mutation information, store it in the mutation store.
Reviewed By: mitrandir77
Differential Revision: D20871340
fbshipit-source-id: ab0b3a20f43a7d97b3c51dcc10035bf7115579af
Summary: Add the mutation store to blobrepo.
Reviewed By: krallin
Differential Revision: D20871336
fbshipit-source-id: 777cba6c2bdcfb16b711dbad61fc6d0d2f337117
Summary: What it says in the title
Reviewed By: StanislavGlebik
Differential Revision: D21549635
fbshipit-source-id: 75939ebbfb317a9beaa9acd1fc1a7c6f41b0f88f
Summary:
subcommand_single calls `derived_data_utils.regenerate(vec![cs_id])` with the
intention that derived data for this commit will be regenerated. However
previously it didn't work because DerivedDataUtils::derive() was ignoring
regenerate parameter. This diff fixes it.
Reviewed By: krallin
Differential Revision: D21527344
fbshipit-source-id: 56d93135071a7f3789262b7a9d9ad84a0896c895
Summary: Add a `/history` endpoint that serves EdenAPI history data. Like the other endpoints, this one currently buffers the response in memory, and will be modified to return a streaming response in a later diff.
Reviewed By: krallin
Differential Revision: D21489463
fbshipit-source-id: 259d2d1b7d700251fe902f1ac741545e5895404a
Summary: Factor out logic that will be common to many handlers into new functions in the `util` module.
Reviewed By: krallin
Differential Revision: D21489469
fbshipit-source-id: 9aff4e5182748ab0a0bedd6038852692b8e721a7
Summary: Break up the EdenAPI server integration tests to prevent the test from getting too long.
Reviewed By: krallin
Differential Revision: D21464056
fbshipit-source-id: 076aaf8717547fe9188f40c078d577961c02325d
Summary: Add an endpoint that serves trees. Uses the same underlying logic as the files endpoint, and returns the requested nodes in a CBOR DataResponse.
Reviewed By: krallin
Differential Revision: D21412987
fbshipit-source-id: a9bcc169644a5889c3118a3207130228a5246b2f
Summary: Change `make_req` to take a JSON array as input when constructing `DataRequest`s instead of a JSON object. This is more correct because DataRequests can include multiple `Key`s with the same path; this cannot be represented as an object since an object is effectively a hash map wherein we would have duplicate keys.
Reviewed By: quark-zju
Differential Revision: D21412989
fbshipit-source-id: 07a092a15372d86f3198bea2aa07b973b1a8449d
Summary: EdenAPI data responses can contain data for either files or trees. As such, the implementation of both the files and trees endpoints is almost identical. To allow the logic to be shared between both, this diff makes the handler code generic.
Reviewed By: krallin
Differential Revision: D21412986
fbshipit-source-id: 89501915b0401214b373ed1db2eb09e59de2e5b7
Summary: In order to allow writing code that is generic over files and trees, move the functionality common between the two to a separate trait. This will allow for a significant amount of code sharing in the EdenAPI server. (This diff does not introduce any new functionality; it's mostly just moving existing code into the new traits.)
Reviewed By: krallin
Differential Revision: D21412988
fbshipit-source-id: 31b55904f62ccb965b0f9425de875fc069e10b5a
Summary:
Add an endpoint that serves Mercurial file data.
The data for all files involved is fetched concurrently from Mononoke's backend but in this initial version the results are first buffered in memory before the response is returned; I plan to change this to stream the results in a later diff.
For now this version demonstrates the basic functionality as well as things like ACL enforcement (a valid client identity header with appropriate access permissions must be present for requests to succeed).
Reviewed By: krallin
Differential Revision: D21330777
fbshipit-source-id: c02a70dff1f646d02d75b9fc50c19e79ad2823e6
Summary:
Right now, we debug-print the root cause and pretty-print everything else. This
is pretty bad because the root cause is usually the one thing we would want to
pretty print so we can add instructions there (such as "your hooks failed, fix
it").
This fixes this so we stop pretty-printing the root cause, but also debug print
the whole error, which gives us more developer-friendly context and is easier
for automation to match on.
This is actually in common/rust ... but we're the only people using it AFAICT.
Reviewed By: StanislavGlebik
Differential Revision: D21522518
fbshipit-source-id: 10158811574b56024e14852229e4541da19d5609
Summary:
Limits on concurrent calls are a bit hard to reason about, and it's not super
obvious what a good limit when all our underlying limits are expressed in QPS
(and when our data sets don't have peak concurrency - instead they have
completion time + # blob accesses).
Considering our past experience with ThrottledBlob has been quite positive
overall, I'd like to just use the same approach in ContextConcurrencyBlobstore.
To be safe, I've also updated this to be driven by tunables, which make it
easier to rollout and rollback.
Note that I removed `Debug` on `CoreContext` as part of this because it wasn't
used anywhere. We can bring back a meaningful implementation of `Debug` there
in the future if we want to. That triggered some warnings about unused fields,
which for now I just silenced.
Reviewed By: farnz
Differential Revision: D21449405
fbshipit-source-id: 5ca843694607888653a75067a4396b36e572f070
Summary:
The motivation for making this function async is that it needs to spawn things,
so it should only ever execute while polled by an executor. If we don't do
this, then it can panic if there is no executor, which is annoying.
I've been wanting to do this for a while but hadn't done it because it required
refactoring a lot of things (see the rest of this stack). But, now, it's done.
Reviewed By: mitrandir77
Differential Revision: D21427348
fbshipit-source-id: bad077b90bcf893f38b90e5c470538d2781c51e9
Summary:
This updates our blobrepo factory code to async / await. The underlying
motivation is to make this easier to modify. I've ran into this a few times
now, and I'm sure others have to, so I think it's time.
In doing so, I've simplified the code a little bit to stop passing futures
around when values will do. This makes the code a bit more sequential, but
considering none of those futures were eager in any way, it shouldn't really
make any difference.
Reviewed By: markbt
Differential Revision: D21427290
fbshipit-source-id: e70500b6421a95895247109cec75ca7fde317169
Summary:
I'd like to add some async fns to session creation. The motivation is that I
want to instantiate an AsyncRateLimiter there, and that requires an async
context because it needs to spawn on the Tokio runtime, and the best way to
enforce this is to just make the function async.
Reviewed By: StanislavGlebik
Differential Revision: D21427291
fbshipit-source-id: 75b0d41b62a77ade3d624e24adc57a938b722d9c
Summary:
At least let's tell the use what to do about the problem and, where we can,
what the conflicting file was (see the attached task).
Reviewed By: farnz
Differential Revision: D21459412
fbshipit-source-id: 52b90cf7d41ebe6550083c6673b4e93b10edf5e2
Summary:
I initially wanted to modify this and it'll be easier to do so if it's
async-await. While in there, add tests and update the code to bail early if any
conflict is hit.
In writing the tests, I noted that the code that we need is already there and
his does work as expected, so I'm not actually going to modify this more, but
it's probably stil worth it to land the tests.
Reviewed By: StanislavGlebik
Differential Revision: D21457899
fbshipit-source-id: 91350962fa2d96a88e4595d1ae47ef7678dad8cb
Summary: I'm going to asyncify some things here. Let's start with this.
Reviewed By: farnz
Differential Revision: D21451761
fbshipit-source-id: 64c78de4ab640b826a3ec1d6d84149d46f225024
Summary:
Right now we're only logging hooks that outright fail, which isn't great. Let's
log rejections as well.
Reviewed By: johansglock
Differential Revision: D21522804
fbshipit-source-id: 6bfc6b12394099b04faa9d23f164b436935f9fb3
Summary:
Make the repo path in Option<WrappedPath> available in stream output in preparation for using it in the corpus dumper to write to disk
The path is Option as not all nodes can have an associated file system path (e.g. BonsaiChangeset)
The headlines changes are in sampling.rs and sizing.rs. The progress.rs change slightly generalises to allow any type convertible to NodeType as the main walk identifier in the output stream.
Some refactors done as part of this
* NodeSamplingHandler is renamed to WalkSampleMapping to reflect this is what it stores.
* WalkSampleMapping generic parameters are extended to take both a key and a sample type
* NodeSamplingHandler::start_node() is moved to a new SampleTrigger::map_keys() type. This is so that SamplingWalkVisitor doesn't need the full WalkSampleMapping generic parameters.
Reviewed By: krallin
Differential Revision: D20835662
fbshipit-source-id: 58db622dc63d7f869a092739d1187a34b77219f6
Summary: Make sampling blobstore handlers fallible in preparation for corpus dumper so we can know if writes to disk/directory creations failed.
Reviewed By: farnz
Differential Revision: D21168632
fbshipit-source-id: d25123435e8f54c75aaabfc72f5fa653e5cf573d
Summary:
Not all node types can have a path associated
Reset the tracked path to None if the route is taking us through a node type that can't have a repo path.
Reviewed By: krallin
Differential Revision: D21228372
fbshipit-source-id: 2b1e291f09232500adce79c630d428f09cd2d2cc
Summary:
Add new --sample-offset argument so that in combination with the existing --sample-rate the whole repo can be sampled in slices
For --sample-rate=N, this allows us to scrub or corpus dump 1/Nth of the repo a time, which is particularly useful for corpus dumping on machines with limited disk.
Also factored out the sampling args construction as 3 of the 4 walk variants use them (only validate does not)
Reviewed By: krallin
Differential Revision: D21158486
fbshipit-source-id: 94f98ceb71c22e0e9d368a563cdb04225b6fc459
Summary: use ArcIntern for WrappedPath to reduced walker memory usage for paths
Reviewed By: farnz
Differential Revision: D21230828
fbshipit-source-id: 525bac5a14b205659e177e03bd83bf06d1444617
Summary:
How is this Dag structure going to be used? This is probably the interesting
question for this diff.
On one side the structure could maintain a view of all the repositories and
manage the DAGs for all repositories in a central place. On the other side the
`Dag` is just an instance of a Changelog and Mononoke manages repositories that
each have a `Dag`. I went with the former pattern as it seems to me to be more
in line with the general architecture of Mononoke.
We can see the Dag being another part of the BlobRepo in the future. We will
want to avoid depending on the BlobRepo for actual functionality to avoid
cyclic dependencies. Currently the BlobRepo is used in construction for
convenience but that will have to change in the future.
Reviewed By: StanislavGlebik
Differential Revision: D21418367
fbshipit-source-id: 7c133eac0f38084615c2b9ba1466de626d2ffcbe
Summary: This removes .compat() from edenapi_server/main.rs. The actual removal probably could be done with less code, but in addition to removing compat(), I made most of the blocking code async.
Reviewed By: kulshrax, farnz
Differential Revision: D21426641
fbshipit-source-id: 1b3de4dc0b24d06faeb73de2e8658f0629d9491d
Summary:
Add a simple `/repos` endpoint that returns the list of repos available in a JSON response.
While the handler itself is quite simple, this diff establishes the general pattern by which new handlers will be added to the server.
Reviewed By: krallin
Differential Revision: D21330778
fbshipit-source-id: 77f57c969c34c8c1f7c94979fac383ec442a1e14
Summary:
When I refactored MPath to limit path length to 255 throughout, I had to change
the logic in this hook because it couldn't represent problematic paths anymore.
Unfortunately, I didn't realize that this would break in cases where the file
fits in 254 or 255 characters with one of the less compact encodings (but that
doesn't fit in 255 if you add `.i`), but also fits in 255 characters once you
add `.i` using one of the more compact encodings.
This results in the hook rejecting things that could have been represented in a
more compact encoding un-necessarily.
This fixes that, but to do so it also requires no requiring MPath in fsencode
(and instead allowing any slice of bytes) which is basically the bulk of
changes here.
Reviewed By: StanislavGlebik
Differential Revision: D21462205
fbshipit-source-id: d4fe6129b379675e842bff5b20bd776cb39157b2
Summary:
This diff logs the delay in deriving data. In particular it logs how much time
has left since an underived commit was created.
Note that this code makes an assumption about monotonic dates - for repos with pushrebase
repos that should be the case.
Reviewed By: krallin
Differential Revision: D21427265
fbshipit-source-id: bfddf594467dfd2424f711f895275fb54a4e1c60
Summary:
Two things will be simplified:
1) Do not pass sqlbookmarks, we can always get them from blobrep
2) Instead of passing repo per derived data type let's just always pass
unredacted repo
Add a very simple unittest
Differential Revision: D21426885
fbshipit-source-id: 712ef23340466438bf34a086517f7ba33d4eabed
Summary: Small refactoring that will make the next diffs easier
Differential Revision: D21426166
fbshipit-source-id: f3c3ae00794046828eaf3c0912dbabc233c97e77
Summary:
The transformation is pretty direct. I didn't add additional functionality
to the IdMap and I did not update the construction algorithm yet. The querying
method on IdMap were updated to async and then there are the SQL interaction
details.
In follow up changes I want to update the construction algorithm and add support
for multiple repositories.
I am not happy with the names of the columns or naming in general in this code.
Open to suggestions. One idea could be matching the client nomenclature as much
as possible.
Reviewed By: StanislavGlebik
Differential Revision: D20929576
fbshipit-source-id: 12104892faa69f37c141e8baf54d5fb24fc5df6b
Summary: This also unblocks the MacOS Mononoke builds, so enabling them back
Reviewed By: farnz
Differential Revision: D21455422
fbshipit-source-id: 4eae10785db5b93b1167f580a1c887ee4c8a96a2
Summary: What it says in the title. I'd like to set up alarms on this.
Reviewed By: farnz
Differential Revision: D21450584
fbshipit-source-id: 539299407cea84c67ff14b30184e8df4282415f8
Summary:
If a bundle comes from the commit cloud forward filler, we need to ignore
and not record it.
To do so, we need to start paying attention to stream-level params for the
first time.
Reviewed By: krallin
Differential Revision: D21427620
fbshipit-source-id: 9ee417cd2a5f2f5bb6ec342cd63071c6ca822475
Summary:
We want to be able to record all the bundles Mononoke processes to be later
replayed by Mercurail.
Reviewed By: krallin
Differential Revision: D21427622
fbshipit-source-id: b88e10e03d07dae35369286fe31022f36a1ee5cf
Summary: To make it easier to navigate the codebase the oss-only code will be from now on stored in a separate module, similarly to how the fbcode-only code is stored.
Reviewed By: markbt
Differential Revision: D21429060
fbshipit-source-id: aa7e80961de2897dae31bd0ec83488c683633b7a
Summary:
Something was up yesterday with the warm bookmarks cache. It started failing on
some hosts, and went out of sync for > 1 hour on some hosts. The logs reported
a lot of failures, but without any context they weren't super useful:
P130438422.
This adds a bit more logging. If this happens again, we'll be able to better
understand what happened.
Reviewed By: StanislavGlebik
Differential Revision: D21447043
fbshipit-source-id: 67a3924c4486991df5e4d38a995ff8054c145cf9
Summary: This is an sqlite equivalent of what exists in xdb now.
Reviewed By: krallin
Differential Revision: D21427621
fbshipit-source-id: 7024fbf7a8773c4465d2e6ee327aadeaf87cb213
Summary:
There was a bug in scrub blobstore that caused failures while traversing the
fsnodes.
If all blobstores returned None, then we need to return None and not fail as we
do now. So the situation we ran into was:
1) fsnodes is not derived, all blobstore return None
2) Previously it returned the error, which later checked in
https://fburl.com/diffusion/mhhhnkxv - this check makes sure there's no entry
with the same key on the queue. However by that time fsnodes might already be
derived and someone else might insert a new entry in the blobstore and in the
queue. This would return an error to the client.
The fix here is to not fail if all blobstores returned None
Reviewed By: ahornby
Differential Revision: D21405418
fbshipit-source-id: 21fe130ce65a0087c408a5014e5b108c7ce8fe6c
Summary: Cover as much as remining code with `Cargo.toml`s, for the rest create an exlusion list in the autocargo config.
Reviewed By: krallin
Differential Revision: D21383620
fbshipit-source-id: 64cc78a38ce0ec482966f32a2963ab4939e20eba
