Summary: Move client hostname reverse DNS lookup from inside of the LFS server's `RequestContext` to an async method on `ClientIdentity`, allowing it to be used elsewhere. The behavior of `RequestContext::dispatch_post_request` should remain unchanged.
Reviewed By: krallin
Differential Revision: D22835610
fbshipit-source-id: 15c1183f64324f216bd639630396c9c6f19bcaaa
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.
Reviewed By: dtolnay
Differential Revision: D22403809
fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
Summary:
This diff is probably going to sound weird ... but xavierd and I both think
this is the best approach for where we are right now. Here is why this is
necessary.
Consider the following scenario
- A client creates a LFS object. They upload it to Mononoke LFS, but not
upstream.
- The client shares this (e.g. with Sandcastle), and includes a LFS pointer.
- The client tries to push this commit
When this happens, the client might not actually have the object locally.
Indeed, the only pieces of data the client is guaranteed to have is
locally-authored data.
Even if the client does have the blob, that's going to be in the hgcache, and
uploading from the hgcache is a bit sketchy (because, well, it's a cache, so
it's not like it's normally guaranteed to just hold data there for us to push
it to the server).
The problem boils down to a mismatch of assumptions between client and server:
- The client assumes that if the data wasn't locally authored, then the server
must have it, and will never request this piece of data again.
- The server assumes that if the client offers a blob for upload, it can
request this blob from the client (and the client will send it).
Those assumptions are obviously not compatible, since we can serve
not-locally-authored data from LFS and yet want the client to upload it, either
because it is missing in upstream or locally.
This leaves us with a few options:
- Upload from the hg cache. As noted above, this isn't desirable, because the
data might not be there to begin with! Populating the cache on demand (from
the server) just to push data back to the server would be quite messy.
- Skip the upload entirely, either by having the server not request the upload
if the data is missing, by having the server report that the upload is
optional, or by having the client not offer LFS blobs it doens't have to the
server, or finally by having the client simply disobey the server if it
doesn't have the data the server is asking for.
So, why can we not just skip the upload? The answer is: for the same reason we
upload to upstream to begin with. Consider the following scenario:
- Misconfigured client produces a commit, and upload it to upstream.
- Misconfigured client shares the commit with Sandcastle, and includes a LFS
pointer.
- Sandcastle wants to push to master, so it goes to check if the blob is
present in LFS. It isn't (Mononoke LFS checks both upstream and internal, and
only finds the blob in upstream, so it requests that the client submit the
blob), but it's also not not locally authored, so we skip the push.
- The client tries to push to Mononoke
This push will fail, because it'll reference LFS data that is not present in
Mononoke (it's only in upstream).
As for how we fix this: the key guarantee made by our proxying mechanism is
that if you write to either LFS server, your data is readable in both (the way
we do this is that if you write to Mononoke LFS, we write it to upstream too,
and if you write to upstream, we can read it from Mononoke LFS too).
What does not matter there is where the data came from. So, when the client
uploads, we simply let it submit a zero-length blob, and if so, we take that to
mean that the client doesn't think it authored data (and thinks we have it), so
we try to figure out where the blob is on the server side.
Reviewed By: xavierd
Differential Revision: D22192005
fbshipit-source-id: bf67e33e2b7114dfa26d356f373b407f2d00dc70
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary: Add an `errors` module containing an `ErrorKind` implemented via `thiserror`. All errors in the server's HTTP handles been annotated with an appropriate `ErrorKind` via `anyhow::Context`. Notably, there is specialized handling of `MononokeError`s from the `mononoke_api` crate via a new `MononokeErrorExt` trait.
Reviewed By: krallin
Differential Revision: D22012237
fbshipit-source-id: 5f6948696e4741816636f1c6fee12069ee1a4b32
Summary:
Move the LFS server's `StreamBody` into `gotham_ext`, along with some changes to decouple it from the LFS server.
In particular, the `Content-Length` header and post-request `Sender` have been made optional fields that can be set via a builder-style interface. The LFS server's `StreamBody` has been renamed to `LfsStreamBody` and is now a thin wrapper around `gotham_ext`'s `StreamBody` that preserves the old behavior.
Reviewed By: krallin
Differential Revision: D21988855
fbshipit-source-id: a9bf9c04bb791388d761fc705ebc38472a713b65
Summary:
There are people that are hurt by usage of these terms, this should be more
then enough reason to replace these. Newly chosen terms are more
self-explanatory as well.
This doesn't yet touch the actualy config files, as that requires a bit more
effort than 1 diff and will require more coordination.
Reviewed By: krallin
Differential Revision: D21924440
fbshipit-source-id: e24fc638dc8c9d6d20b6f3fa5f0d0bbc91bbf77b
Summary: This methods will replace the fbcode_build provided functions. Also add Cargo files for crates that are now thanks to that buildable in OSS.
Reviewed By: farnz
Differential Revision: D21549860
fbshipit-source-id: 69f4179aa7a9081d33fac1f2f88aa3b32cd31e17/
Summary: The `secure_utils` crate from common/rust/secure_utils was moved to rust-shed, the remaining crates in that folder are being refactored here into a single crate `identity_ext` for clarity.
Reviewed By: StanislavGlebik
Differential Revision: D21549861
fbshipit-source-id: 4da6566a09ba7a772e8062632f9d7520af2e09e6
Summary:
Add a simple `/repos` endpoint that returns the list of repos available in a JSON response.
While the handler itself is quite simple, this diff establishes the general pattern by which new handlers will be added to the server.
Reviewed By: krallin
Differential Revision: D21330778
fbshipit-source-id: 77f57c969c34c8c1f7c94979fac383ec442a1e14
Summary: The changes to server/context, gotham_ext and the code that depends on them are the only reminding places where aclchecker is used directly and it is not easy to split this diff to convert them separately.
Reviewed By: krallin
Differential Revision: D21067809
fbshipit-source-id: a041ab141caa6fe6871e1fda6013e33f1f09bc56
Summary: Add a convenience method to HttpError to return a JSON-formatted representation of the error to the client.
Reviewed By: krallin
Differential Revision: D21193939
fbshipit-source-id: e1ff1555b0016f46dbcd1847239d96daf8b45685
Summary: Move the various body types out of the LFS server into `gotham_ext`. `StreamBody` was intentionally left behind for now since it contains some LFS-specific logic that would need to be factored out before it can be moved.
Reviewed By: krallin
Differential Revision: D21193941
fbshipit-source-id: 638b9e93e9dc7385f7fde9dbb3a2392ad0e18385
Summary: Move `SignalStream` out of the LFS server into `gotham_ext`. This is a step towards extracting all of the functionality needed to support streaming bodies in `gotham_ext`.
Reviewed By: krallin
Differential Revision: D21193940
fbshipit-source-id: 832a5254c80e4ee085ece371b45b38a4519403f3
Summary: Move the `HttpError` type out of the LFS server into `gotham_ext` so it can be used by the EdenAPI server too.
Reviewed By: krallin
Differential Revision: D21193937
fbshipit-source-id: dff59e3ae995fe5771db47174a96e31b2c9f4c73
Summary:
This updates the lfs server and eden api server to use a newer version of
Gotham, which comes along with an updated version of Bytes and Hyper.
A few things had to change for this:
- New bytes don't support concatenation, so we need to fold them ourselves,
except...
- ... new Hyper bodies don't tell you how big they are (either in requests or
responses), so we need to inspect headers to find the size instead (I added
this in `gotham_ext::body_ext::BodyExt`, although it arguably belongs more in
a `hyper_ext` crate, but creating a new crate for just this seems overkill).
- New Hyper requires its data stream to be `Sync` for reasons that have more to
do with developer experience than runtime
(https://github.com/hyperium/hyper/pull/1857). Unfortunately, our Filestore
streams aren't `Sync`, because our `BoxFuture` contains a `dyn Future` that
isn't explicitly `Sync` (which is how we pull things out of blobstores). Even
if `BoxFuture` contained a `Sync` future, that still wouldn't be enough
anyway, because `compat()` explicitly implements `!Sync` on the stream it
returns. I'll ask upstream in Hyper if this can possibly change in the
future, but for now we can work around it by wrapping the stream in a
channel. I'll keep an eye out for performance here.
- When I updated our "pre state data" tweaks on top of Gotham, I renamed those
to "socket data", since that's a better name or what they are (hence the
changes here).
- I updated the lfs_protocol to stop depending on Hyper and instead depend on
http, since that's all we need here.
As you review this, please pay close attention to the updated implementation of
`SignalStream`. Since this is a custom `Stream` in new futures, it requires a
bit of `unsafe { ... }`.
Note that, unfortunately, the diff includes both the Gotham update and the
server updates, since they have to happen together.
Reviewed By: kulshrax, dtolnay
Differential Revision: D20342689
fbshipit-source-id: a490db96ca7c4da8ff761cb80c1e7e3c836bad87
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
This updates our middleware stack and introduces two new pieces of functinality:
- Middleware can now be async.
- Middleware can now preempt requests and dispatch a response.
The underlying motivation for this is to allow implementing Mononoke LFS's rate
limiting middleware in our existing middleware stack.
Reviewed By: kulshrax
Differential Revision: D20191213
fbshipit-source-id: fc1df7a14eb0bbefd965e32c1fca5557124076b5
Summary: Move several pieces of Middleware related to TLS from the LFS server to `gotham_ext`, for reuse in the EdenAPI server. These modules are already well-abstracted and consequently require no modification to be moved out of the LFS server.
Reviewed By: xavierd
Differential Revision: D19890537
fbshipit-source-id: 8de420183e7bd5dd0de10a75c8cfe83825f0c23c
Summary: Move the generally-useful ServerIdentityMiddleware into gotham_ext so we can use it in the EdenAPI server.
Reviewed By: xavierd
Differential Revision: D19845282
fbshipit-source-id: 3a01b15dc64cee99cefafcdac229c0b70a4db683
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
