Summary: I'd like to get this in Scuba to do a bit more reporting there.
Reviewed By: farnz
Differential Revision: D17600786
fbshipit-source-id: 90d34362b5212abf3ff6642f55e57a997f86006f
Summary:
The method name was incorrectly added to the ODS key, rather than the repo
name. Fix this. :-)
Reviewed By: krallin
Differential Revision: D17600416
fbshipit-source-id: 203f3be428b15a0b7d1cce90dd85a3932311c89c
Summary:
This is nice when running locally to know when we're done with setting up
cachelib and such.
Reviewed By: farnz
Differential Revision: D17569829
fbshipit-source-id: adfe5944991c8842a459df8606d9d81c2dcd02de
Summary:
This updates the lfs_server to publish client_hostname to Scuba. It also
unifies the various mechanisms we had to get data about the client into a
single piece of middleware, which ensures we get a consistent view of e.g.
client IPs in Scuba and stderr lgoging.
Reviewed By: HarveyHunt
Differential Revision: D17570749
fbshipit-source-id: e5b62abf440d5d09e78c1f51632444200768126c
Summary:
When I made the LFS server not wait for upstream if it can reply immediately, I
also made the tests for its proxy functionality a bit racy, since it no longer
always connects to upstream immediately. This fixes that.
While I'm in there, I also:
- Added an integration test for the "skip upstream" functionality.
- Made our stderr logging log empty responses.
- Made our invalid URI error reporting a little more comprehensive.
Reviewed By: HarveyHunt
Differential Revision: D17547889
fbshipit-source-id: 47f150136ef91a7f6334bb09f95782357b72f01a
Summary:
Previously, the RequestContext stored a string representing the method
type. Replace this with an enum, so that matching on it is easier - allowing
for different histogram configs for each method type.
For the batch endpoint, use a histogram of durations from 0 -> 500ms.
Reviewed By: farnz
Differential Revision: D17530495
fbshipit-source-id: 23ff424fc0aca5c1022f9d6521bbd70d0af3ccf6
Summary:
If we have all the objects a client is asking for, we might as well answer
immediately with them, rather than wait for upstream.
This effectively means that this LFS server can be faster than its upstream on
batch downloads (for uploads, we still have to check in with upstream), as it's
no longer limited by upstream's performance (or errors!) when responding to
batch requests.
Reviewed By: HarveyHunt
Differential Revision: D17526491
fbshipit-source-id: 5d008011fd5ac444c84569f8235ecc9bde6cef9c
Summary: This adds the request_id to our request slog logger. This is nicer because it lets us tie logs for a given request together.
Reviewed By: HarveyHunt
Differential Revision: D17526490
fbshipit-source-id: c6a57828a097f96b0e752373b98a1ea834009f95
Summary: We no longer have any middleware that requires to be able to capture variables from the "inbound" phase (i.e. prior to handling the request) to the "outbound" phase (i.e. once the response is ready). Instead, we're passing everything through the State. So, let's get rid of the dynamism we don't need.
Reviewed By: HarveyHunt
Differential Revision: D17503373
fbshipit-source-id: 569d180250821aa3707245133a223b1f4efba3b6
Summary:
Middleware executes when a response is ready, but since responses can contain a stream, that might not be the full story if we're e.g. downloading a big file.
This diff updates our middleware to introduce a post-send context that lets us conditionally dispatch those actions after we've finished sending the response to the client.
Reviewed By: HarveyHunt
Differential Revision: D17503374
fbshipit-source-id: 4dc97c0057d6e1705d116cbc1d283fc73de213ef
Summary:
This reworks our middleware to not use the Gotham router for middleware, and
instead creates our own service to wrap the one provided by the Gotham Router.
The upshot of this is that we can then run our own middleware even if the
Gotham Router finds a 404 or 405 (in which case it doesn't normally run any
middleware).
This also simplifies our middleware a little bit, since our middleware isn't
modifying responses at all, but Gotham middleware allows that (we can still run
response transformation in Gotham).
As part of this change, I've also cleaned up our LoggingContext (which didn't
really belong in `lfs_server_context`), and moved our router implementation in
its own module.
Reviewed By: HarveyHunt
Differential Revision: D17500363
fbshipit-source-id: 9b0d7449f707f158d9f5433e2953d270b3446c8f
Summary:
This adds a little more headers to make it easier for clients to identify which
server they are talking to.
Reviewed By: HarveyHunt
Differential Revision: D17498981
fbshipit-source-id: 758562f0ed631dc6f54eb567200e3bf1af04b078
Summary:
Add a string representing the method to LoggingContext,
which can then be used by the ODS and Scuba middleware.
NOTE: This changes the batch endpoint key from:
mononoke.lfs.request.<repo>.objects
to:
mononoke.lfs.request.<repo>.batch
Reviewed By: krallin
Differential Revision: D17475551
fbshipit-source-id: 8692f165719c9f69bf0d783845ed9d87b8baf86f
Summary:
Store a wrapped instance of ScubaSampleBuilder in gotham's State, so that it can be
accessed from outside of the middleware. Further, refactor the Scuba middleware
to use a separate function and replace the and_then() combinator with then() so that all internal
errors are handled by the scuba logger.
Additionally, use the scuba logger to log the number of objects in a batch as well
as the size of uploads.
Reviewed By: krallin
Differential Revision: D17425371
fbshipit-source-id: bb86995c2e561062c1b1951fcea98fa25300103c
Summary:
Add statistics to the LFS server that report useful information, such as
the number of requests to an endpoint or a histogram of file sizes.
Reviewed By: krallin
Differential Revision: D17367739
fbshipit-source-id: bca99c059c61f11e7f78319ebccd22ebb31c4ae0
Summary:
This diff moves initFacebook calls that used to happen just before FFI calls to instead happen at the beginning of main.
The basic assumption of initFacebook is that it happens at the beginning of main before there are additional threads. It must be allowed to modify process-global state like env vars or gflags without the possibility of a data race from other code concurrently reading those things. As such, the previous approach of calling initFacebook through `*fbinit::FACEBOOK` near FFI calls was prone to race conditions.
The new approach is based on attribute macros added in D17245802.
---
The primary remaining situations that still require `*fbinit::FACEBOOK` are when we don't directly control the function arguments surrounding the call to C++, such as in lazy_static:
lazy_static! {
static ref S: Ty = {
let _ = *fbinit::FACEBOOK;
/* call C++ */
};
}
and quickcheck:
quickcheck! {
fn f(/* args that impl Arbitrary */) {
let _ = *fbinit::FACEBOOK;
/* call C++ */
}
}
I will revisit these in a separate diff. They are a small fraction of total uses of fbinit.
Reviewed By: Imxset21
Differential Revision: D17328504
fbshipit-source-id: f80edb763e7f42b3216552dd32f1ea0e6cc8fd12
Summary:
Timing a request is going to be important for multiple logging
middlewares, so let's move it into its own middleware and place it
in the pipeline.
Further, create a new middleware/ directory and move middlewares into there.
Reviewed By: krallin
Differential Revision: D17346276
fbshipit-source-id: f84c6c06d76e95c11aab18c3a24200a67429bebf
Summary:
As the filestore can cheaply calculate file size, update the Scuba logging to
also log that. Further, set the content length header when responding to download requests.
Reviewed By: krallin
Differential Revision: D17319666
fbshipit-source-id: 858372316930c384f19b89e2b69b08faaf656237
Summary:
This wires up the stdlog crate with our slog output. The upshot is that we can
now run binaries with `RUST_LOG` set and expect it to work.
This is nice because many crates use stdlog (e.g. Tokio, Hyper), so this is
convenient to get access to their logging. For example, if you run with
`RUST_LOG=gotham=info,hyper=debug`, then you get debug logs from Hyper and info
logs from Gotham.
The way this works is by registering a stdlog logger that uses the env_logger's
filter (the one that "invented" `RUST_LOG`) to filter logs, and routes them to
slog if they pass. Note that the slog Logger used there doesn't do any
filtering, since we already do it before sending logs there.
One thing to keep in mind is that we should only register the stdlog global
logger once. I've renamed `get_logger` to `init_logging` to make this clearer.
This behavior is similar to what we do with `init_cachelib`. I've updated
callsites accordingly.
Note that we explicitly tell the stdlog framework to ignore anything that we
won't consider for logging. If you don't set `RUST_LOG`, then the default
logging level is `Error`, which means that anything below error that is sent to
stdlog won't even get to out filtering logic (the stdlog macros for logging
check for the global level before actually logging), so this is cheap unless
you do set `RUST_LOG`.
As part of this, I've also updated all our binaries (and therefore, tests) to
use glog for logging. We had been meaning to do this, and it was convenient to
do it here because the other logger factory we were using didn't make it easy
to get a Drain without putting it a Logger.
Reviewed By: ahornby
Differential Revision: D17314200
fbshipit-source-id: 19b5e8edc3bbe7ba02ccec4f1852dc3587373fff
Summary:
This updates the LFS server to route hg client correlators to Scuba. This will
help in troubleshooting issues should any arise.
Reviewed By: HarveyHunt
Differential Revision: D17319280
fbshipit-source-id: d4323925a425203f53aba184d5854dd674462da6
Summary:
This adds support for logging client identities to Scuba. This is useful to
know who's connecting to the LFS Server.
Reviewed By: StanislavGlebik
Differential Revision: D17318696
fbshipit-source-id: adba75e4133e54af7eef5183a245a3934527db05
Summary:
This updates our error reporting to actually log the chained causes we bothered
to put on the errors. It also adds a few more chained causes.
Reviewed By: StanislavGlebik
Differential Revision: D17315689
fbshipit-source-id: d3b83f73fa06b56b863e23f2f76e78f699af8e36
Summary:
Add per request logging to the LFS server, so that it's easier to debug LFS server issues. In
a future diff, I will also log response size.
Reviewed By: krallin
Differential Revision: D17314566
fbshipit-source-id: d40dc23a55bc56f6f768c9c0119553d03ea568c5
Summary: Add header so we can tell response is from mononoke LFS.
Reviewed By: krallin, StanislavGlebik
Differential Revision: D17288640
fbshipit-source-id: 73dc4b29c2b865f8f5407636de4235819ca8ffdb
Summary: I think these are left over from pre-2018 code where they may have been necessary. In 2018 edition, import paths in `use` always begin with a crate name or `crate`/`super`/`self`, so `use $ident;` always refers to a crate. Since extern crates are always in scope in every module, `use $ident` does nothing.
Reviewed By: Imxset21
Differential Revision: D17290473
fbshipit-source-id: 23d86e5d0dcd5c2d4e53c7a36b4267101dd4b45c
Summary:
Previously, the LFS server's internal functions returned HandlerResponses
which included HTTP status codes. The bail_http macros were used to implement this,
but made it difficult pass state to them for logging, as well as not allowing the use
of the ? operator, which is a nice benefit of async / await syntax.
Refactor these functions to that they don't return such Gotham specific types
and remove the relevant http macros.
Reviewed By: krallin
Differential Revision: D17286238
fbshipit-source-id: 05ff791d4761b0f742d22a2966d5ecc5968728ba
Summary: This removes the LFS implementation from the apiserver. We haven't used it, and we weren't going to.
Reviewed By: StanislavGlebik
Differential Revision: D17263036
fbshipit-source-id: cad3e8a7d54abef06ff1bc1cb5e73c55342743b6
Summary:
This adds support in the LFS Server for running without an upstream. This is
useful to run tests, because it lets chain LFS Servers and have one acting as a
proxy and one acting as a backend.
This also highlighted a bug in my implementation (we were expecting a download
action from upstream if the data didn't need to be uploaded there), which I
fixed here.
For the time being, we won't use this in production.
Reviewed By: HarveyHunt
Differential Revision: D17263039
fbshipit-source-id: 7cba550054e5f052a4b8953ebe0195907919aade
Summary:
This adds a logging middleware that logs requests to stdlog (and therefore to slog through slog-stdlog).
It's convenient for tests, and it's also a pretty standard thing for HTTP servers.
Reviewed By: StanislavGlebik
Differential Revision: D17263040
fbshipit-source-id: 992c5e46ba9ae5b829001a26536b827130aa813c
Summary: It looks like I forgot a few prior to landing the LFS server. This removes a few extra `Fallible`s I still had laying around.
Reviewed By: StanislavGlebik
Differential Revision: D17263037
fbshipit-source-id: 0e39e04daf6fc1336a7a35aaafd243dc88c8836e
Summary: This is helpful for Tupperware.
Reviewed By: HarveyHunt
Differential Revision: D17227429
fbshipit-source-id: 5b30833eb5b7f4618ee3360daca486d13b8142b3
Summary: Not much to see here. This is just a POC of some logging being setup.
Reviewed By: StanislavGlebik
Differential Revision: D17203576
fbshipit-source-id: 14e0d8d3d91b9142029e4198fa7be688e666952f
Summary: This updates the lfs_server to support TLS.
Reviewed By: StanislavGlebik
Differential Revision: D17203298
fbshipit-source-id: aa91a1b6304c7203018cd5f9feb90645e085cb31
Summary:
This introduces a LFS server for Mononoke, which is designed to also act as a
reverse proxy to a fallback LFS server. The goal of this LFS server is to
uphold the following guarantees:
- If a client uploads successfully, then the LFS content will be present in
Mononoke as well as the upstream server.
- If a piece of content is present in either Mononoke or the upstream server,
then downloading this content will succeed. If the content is available in
Mononoke, it'll be served there. Otherwise, the client will be redirected to
the upstream server.
Note that while the LFS server for Mononoke exposes a route per repo, that
isn't the case for the upstream server.
Also, note that this only does proxying for the batch endpoint and uploads.
Clients must have access to the upstream to download from there blobs that
don't exist in Mononoke.
Implementation-wise, this uses async Rust and the Gotham web framework (Gotham
is based on Hyper, which we also use to talk to upstream LFS). Note that
neither Gotham nor Mononoke's filestore doesn't use Tokio 0.2 yet, so we still
have to run on Tokio 0.1 as the executor.
Reviewed By: StanislavGlebik
Differential Revision: D17182753
fbshipit-source-id: 24c9c9c7183506c38d7534665a19d42acf4b3442
