Summary: Change `HistoryResponse` so that instead of repeatedly sending the file path alongside every history entry, we group the history entries by file and send the path once per group. This will prevent the server from transmitting redundant information.
Reviewed By: quark-zju
Differential Revision: D21982558
fbshipit-source-id: f1c5d2573c97940c7bf1645ed7fef6e1887c0d42
Summary: Last usage was removed in D20342689.
Reviewed By: xavierd
Differential Revision: D22011195
fbshipit-source-id: 58455ad3dc77438704e80c69eb728a319ee71a39
Summary: D21880220 renamed the `depth` parameter in Mononoke's history fetching functions to be `length`. This diff makes the same change for EdenAPI's `HistoryRequest` struct.
Reviewed By: StanislavGlebik
Differential Revision: D21948599
fbshipit-source-id: fe8649a5789f07d8262ad3d5e2f477a8b50f2c6f
Summary:
Add a new `subtree` endpoint to the EdenAPI server, which fetches trees using the underlying server-side logic for the `gettreepack` wire protocol command. This is intended for use in situations where compatibility with `gettreepack` is desired when using HTTP for tree fetching.
The name of the endpoint is up for bikeshedding. It seemed weird to name the endpoint `gettreepack` since that name is a verb and refers to a "pack" which is not relevant in this context (there are no wirepacks or packfiles involved). I chose the name `subtree` since the endpoint logically returns all of the nodes underneath a given node in the tree (though in the most frequent case, the node will be the root node and therefore the subtree will be the entire tree).
In practice, this initial implementation is not ideal because it buffers the trees in memory, which is problematic because `gettreepack` requests are likely to produce a very large number of trees. Later in this stack, the endpoint will be updated to produce a streaming response instead.
Reviewed By: StanislavGlebik
Differential Revision: D21782764
fbshipit-source-id: 726925858352c33c923da1979da9d20fbcf930f6
Summary:
Several of the structs used by EdenAPI were previously defined in Mercurial's
`types` crate. This is not ideal since these structs are used for data interchange
between the client and server, and changes to them can break Mononoke, Mercurial,
or both. As such, they are more fragile than other types and their use should be
limited to the EdenAPI server and client to avoid the need for extensive breaking
changes or costly refactors down the line.
I'm about to make a series of breaking changes to these types as part of the
migration to the new EdenAPI server, so this seemed like an ideal time to split
these types into their own crate.
Reviewed By: krallin
Differential Revision: D21857425
fbshipit-source-id: 82dedc4a2d597532e58072267d8a3368c3d5c9e7
Summary: The replacement will make lfs_server, mononoke_api, edenapi_server and walker OSS buildable.
Reviewed By: krallin
Differential Revision: D21884324
fbshipit-source-id: e6cdf8356b13056a9944ab9da18dc15977b6a2ec
Summary:
The max_depth parameter was confusing - one might have thought that it returns
all entries that are `max_depth` hops away from `startnode`. This is not the
case - in reality it just limits what's the total number of entries can be
returned.
In fact, I think it can be replaced with Stream::take() method, however that's
a larger refactoring, and I'm not going to do it now.
It doesn't matter much because usually we have linear histories, but I think
it's still worth to make it clearer.
Reviewed By: krallin
Differential Revision: D21880220
fbshipit-source-id: d209e1e129383f9181ae11b489992dc4c3ce2d5c
Summary:
Replace the use of `RepoConfigs::read*` associated functions with free
functions. These didn't really need to be associated functions (and in the
case of the common and storage configs, really didn't belong there either).
Reviewed By: krallin
Differential Revision: D21837270
fbshipit-source-id: 2dc73a880ed66e11ea484b88b749582ebdf8a73f
Summary: Add a `/history` endpoint that serves EdenAPI history data. Like the other endpoints, this one currently buffers the response in memory, and will be modified to return a streaming response in a later diff.
Reviewed By: krallin
Differential Revision: D21489463
fbshipit-source-id: 259d2d1b7d700251fe902f1ac741545e5895404a
Summary: Factor out logic that will be common to many handlers into new functions in the `util` module.
Reviewed By: krallin
Differential Revision: D21489469
fbshipit-source-id: 9aff4e5182748ab0a0bedd6038852692b8e721a7
Summary: Add an endpoint that serves trees. Uses the same underlying logic as the files endpoint, and returns the requested nodes in a CBOR DataResponse.
Reviewed By: krallin
Differential Revision: D21412987
fbshipit-source-id: a9bcc169644a5889c3118a3207130228a5246b2f
Summary: EdenAPI data responses can contain data for either files or trees. As such, the implementation of both the files and trees endpoints is almost identical. To allow the logic to be shared between both, this diff makes the handler code generic.
Reviewed By: krallin
Differential Revision: D21412986
fbshipit-source-id: 89501915b0401214b373ed1db2eb09e59de2e5b7
Summary: In order to allow writing code that is generic over files and trees, move the functionality common between the two to a separate trait. This will allow for a significant amount of code sharing in the EdenAPI server. (This diff does not introduce any new functionality; it's mostly just moving existing code into the new traits.)
Reviewed By: krallin
Differential Revision: D21412988
fbshipit-source-id: 31b55904f62ccb965b0f9425de875fc069e10b5a
Summary:
Add an endpoint that serves Mercurial file data.
The data for all files involved is fetched concurrently from Mononoke's backend but in this initial version the results are first buffered in memory before the response is returned; I plan to change this to stream the results in a later diff.
For now this version demonstrates the basic functionality as well as things like ACL enforcement (a valid client identity header with appropriate access permissions must be present for requests to succeed).
Reviewed By: krallin
Differential Revision: D21330777
fbshipit-source-id: c02a70dff1f646d02d75b9fc50c19e79ad2823e6
Summary: This removes .compat() from edenapi_server/main.rs. The actual removal probably could be done with less code, but in addition to removing compat(), I made most of the blocking code async.
Reviewed By: kulshrax, farnz
Differential Revision: D21426641
fbshipit-source-id: 1b3de4dc0b24d06faeb73de2e8658f0629d9491d
Summary:
Add a simple `/repos` endpoint that returns the list of repos available in a JSON response.
While the handler itself is quite simple, this diff establishes the general pattern by which new handlers will be added to the server.
Reviewed By: krallin
Differential Revision: D21330778
fbshipit-source-id: 77f57c969c34c8c1f7c94979fac383ec442a1e14
Summary: The changes to server/context, gotham_ext and the code that depends on them are the only reminding places where aclchecker is used directly and it is not easy to split this diff to convert them separately.
Reviewed By: krallin
Differential Revision: D21067809
fbshipit-source-id: a041ab141caa6fe6871e1fda6013e33f1f09bc56
Summary:
Add a new function that initialises common parts of Mononoke:
- cachelib
- logging
- runtime
In the next diff, I will also update `init_mononoke` to initialise tunables
too.
I left some users of `init_runtime` unchanged (such as `mononoke_admin`) as
they conditionally enable cachelib.
Reviewed By: krallin
Differential Revision: D21177253
fbshipit-source-id: 40b534d1b244fd8dcd05e1cff1c9f3edfb32a4b9
Summary:
Previously our jobs would have exit with error code 0 even if there was an
actual error. The reason for this was because error was just ignored (or rather
just printed to stderr).
This is not a huge problem but it makes tw output confusing - it shows that
the task was "Completed" while in reality it "Failed"
Reviewed By: ahornby
Differential Revision: D20693297
fbshipit-source-id: 4f615e2ef11f2edbb9bdbcf49cb1635929fdae89
Summary: Use constants for flag names instead of duplicating them as hardcoded strings.
Reviewed By: xavierd
Differential Revision: D21072045
fbshipit-source-id: 4617d169d034e05dcf11eb138ad0b6eaf915edec
Summary:
Add TLS session data middleware to the EdenAPI server to allow inspecting HTTPS traffic to the server for debugging purposes.
An additional side effect of setting this up is that we use `bind_server_with_socket_data` to bind to the listening socket, wherein we construct a `TlsSocketData`, which constructs a `TlsCertificateIdentities` containing the client certificates, which are later added to the server's State by Gotham. Having the client certificate information in the State is necessary in order for the ClientIdentityMiddleware to work, and will allow enforcement of ACLs based on the provided client identities.
Reviewed By: krallin
Differential Revision: D21054213
fbshipit-source-id: 7002c73b7458f21e3c4a51a3029d27d1dea7a927
Summary: Add a few debug-level log lines during server startup so we can see which part of startup is slow.
Reviewed By: quark-zju
Differential Revision: D21054216
fbshipit-source-id: 5dfb7b58fffb360506f34e3f2bb9e8b51fcc5e6b
Summary: `#![deny(warnings)]` does nothing outside of the crate root file, so this was a no-op.
Reviewed By: singhsrb
Differential Revision: D21054214
fbshipit-source-id: dc1931c0a186eb42aae7700dd006550616f29a70
Summary: Use the client identity middleware from gotham_ext in the EdenAPI server. This middleware parses validated client identities from HTTP headers inserted by Proxygen; these identities can then be used to enforce repo ACLs.
Reviewed By: HarveyHunt
Differential Revision: D20744887
fbshipit-source-id: 651e171d1b20448b3e99bfc938d118fb6dddea91
Summary:
This updates the lfs server and eden api server to use a newer version of
Gotham, which comes along with an updated version of Bytes and Hyper.
A few things had to change for this:
- New bytes don't support concatenation, so we need to fold them ourselves,
except...
- ... new Hyper bodies don't tell you how big they are (either in requests or
responses), so we need to inspect headers to find the size instead (I added
this in `gotham_ext::body_ext::BodyExt`, although it arguably belongs more in
a `hyper_ext` crate, but creating a new crate for just this seems overkill).
- New Hyper requires its data stream to be `Sync` for reasons that have more to
do with developer experience than runtime
(https://github.com/hyperium/hyper/pull/1857). Unfortunately, our Filestore
streams aren't `Sync`, because our `BoxFuture` contains a `dyn Future` that
isn't explicitly `Sync` (which is how we pull things out of blobstores). Even
if `BoxFuture` contained a `Sync` future, that still wouldn't be enough
anyway, because `compat()` explicitly implements `!Sync` on the stream it
returns. I'll ask upstream in Hyper if this can possibly change in the
future, but for now we can work around it by wrapping the stream in a
channel. I'll keep an eye out for performance here.
- When I updated our "pre state data" tweaks on top of Gotham, I renamed those
to "socket data", since that's a better name or what they are (hence the
changes here).
- I updated the lfs_protocol to stop depending on Hyper and instead depend on
http, since that's all we need here.
As you review this, please pay close attention to the updated implementation of
`SignalStream`. Since this is a custom `Stream` in new futures, it requires a
bit of `unsafe { ... }`.
Note that, unfortunately, the diff includes both the Gotham update and the
server updates, since they have to happen together.
Reviewed By: kulshrax, dtolnay
Differential Revision: D20342689
fbshipit-source-id: a490db96ca7c4da8ff761cb80c1e7e3c836bad87
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.
rs changes performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs,
rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
intersect
rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
)" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```
Reviewed By: jsgf
Differential Revision: D20168958
fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0
Summary:
This updates our middleware stack and introduces two new pieces of functinality:
- Middleware can now be async.
- Middleware can now preempt requests and dispatch a response.
The underlying motivation for this is to allow implementing Mononoke LFS's rate
limiting middleware in our existing middleware stack.
Reviewed By: kulshrax
Differential Revision: D20191213
fbshipit-source-id: fc1df7a14eb0bbefd965e32c1fca5557124076b5
Summary: We're about to start adding more handlers to the server. Rather than putting them all in the same file, let's create a submodule for them.
Reviewed By: krallin
Differential Revision: D19957012
fbshipit-source-id: 38192664371f0b0ef5eadb4969739f7cb6e5c54c
Summary: Add a `RequestContext` type that stores per-request state, along with a `Middleware` implementation that injects a `RequestContext` into Gotham's `State` object for each request. This is essentially a stripped-down version of the `RequestContextMiddleware` used in the LFS server. Given that the RequestContext contains application-specific functionality, this Middleware lives alongside the rest of the EdenAPI server code rather than in the `gotham_ext` crate (where all of the generic Middleware lives).
Reviewed By: krallin
Differential Revision: D19957013
fbshipit-source-id: 6fad2b92aea0b3662403a69e6a6598e4cd26f083
Summary: common/sql_ext will now be buildable in OSS
Reviewed By: krallin
Differential Revision: D19876764
fbshipit-source-id: 0f51abd1169f6b8108e7e4cab85b5f193c28e2cd
Summary: Later in this stack, we're going to have to introduce a few more context types, including Mononoke's per-session `CoreContext` as well as an LFS-server inspired per-request `RequestContext`. To make the scope of the `EdenApiContext` more clear (namely, that unlike the other contexts, this one persists for the entirety of the server's execution), let's rename this to `EdenApiServerContext`. This mirrors the naming convention for these contexts used in the LFS server.
Reviewed By: xavierd
Differential Revision: D19863296
fbshipit-source-id: 8ad785070328523d0beaf824c86c7350ff6a2697
Summary: Use `ServerIdentityMiddleware` from `gotham_ext` in the EdenAPI server to provide server information in the HTTP response headers returned by the server. This is useful for debugging.
Reviewed By: xavierd
Differential Revision: D19863297
fbshipit-source-id: 6ed8bac35e05af580e062423e6f6389a18d17c2a
Summary: The exact format of the GPL license header changed when the Mononoke directory moved from `/scm/mononoke` to `/eden/mononoke`. This file ended up with the old header somehow (was created prior to the rename but landed after), so let's fix it to make the linter happy.
Reviewed By: farnz
Differential Revision: D19848640
fbshipit-source-id: 39c5b49850e5a3cba1951bf4e5b813cd08940f01
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
Summary: Make the EdenAPI server report that it is exiting when asked to shut down. This ensures that Proxygen will stop sending traffic to servers that are about to be shut down by Tupperware. This diff is basically the same as krallin's diff D17626009 for the LFS server.
Reviewed By: quark-zju
Differential Revision: D19782432
fbshipit-source-id: 41b9e6761145402e7dcf18c53a2b33799588594c
Summary: This diff sets up the Mononoke API (from the `mononoke_api` crate) in the EdenAPI server, and makes it available to route handlers by adding it to a new `EdenApiContext` struct that is maintained as part of the server's global state. The server's route handlers should use the Mononoke API for accessing source control data, and any new source control business logic should be incorporated into that crate rather than being part of the EdenAPI server itself.
Reviewed By: xavierd
Differential Revision: D19778441
fbshipit-source-id: bc2efb82e0276d75c49980c52fa0d3017a4ce2f1
