Summary:
This diff moves initFacebook calls that used to happen just before FFI calls to instead happen at the beginning of main.
The basic assumption of initFacebook is that it happens at the beginning of main before there are additional threads. It must be allowed to modify process-global state like env vars or gflags without the possibility of a data race from other code concurrently reading those things. As such, the previous approach of calling initFacebook through `*fbinit::FACEBOOK` near FFI calls was prone to race conditions.
The new approach is based on attribute macros added in D17245802.
---
The primary remaining situations that still require `*fbinit::FACEBOOK` are when we don't directly control the function arguments surrounding the call to C++, such as in lazy_static:
lazy_static! {
static ref S: Ty = {
let _ = *fbinit::FACEBOOK;
/* call C++ */
};
}
and quickcheck:
quickcheck! {
fn f(/* args that impl Arbitrary */) {
let _ = *fbinit::FACEBOOK;
/* call C++ */
}
}
I will revisit these in a separate diff. They are a small fraction of total uses of fbinit.
Reviewed By: Imxset21
Differential Revision: D17328504
fbshipit-source-id: f80edb763e7f42b3216552dd32f1ea0e6cc8fd12
Summary:
This wires up the stdlog crate with our slog output. The upshot is that we can
now run binaries with `RUST_LOG` set and expect it to work.
This is nice because many crates use stdlog (e.g. Tokio, Hyper), so this is
convenient to get access to their logging. For example, if you run with
`RUST_LOG=gotham=info,hyper=debug`, then you get debug logs from Hyper and info
logs from Gotham.
The way this works is by registering a stdlog logger that uses the env_logger's
filter (the one that "invented" `RUST_LOG`) to filter logs, and routes them to
slog if they pass. Note that the slog Logger used there doesn't do any
filtering, since we already do it before sending logs there.
One thing to keep in mind is that we should only register the stdlog global
logger once. I've renamed `get_logger` to `init_logging` to make this clearer.
This behavior is similar to what we do with `init_cachelib`. I've updated
callsites accordingly.
Note that we explicitly tell the stdlog framework to ignore anything that we
won't consider for logging. If you don't set `RUST_LOG`, then the default
logging level is `Error`, which means that anything below error that is sent to
stdlog won't even get to out filtering logic (the stdlog macros for logging
check for the global level before actually logging), so this is cheap unless
you do set `RUST_LOG`.
As part of this, I've also updated all our binaries (and therefore, tests) to
use glog for logging. We had been meaning to do this, and it was convenient to
do it here because the other logger factory we were using didn't make it easy
to get a Drain without putting it a Logger.
Reviewed By: ahornby
Differential Revision: D17314200
fbshipit-source-id: 19b5e8edc3bbe7ba02ccec4f1852dc3587373fff
Summary: I think these are left over from pre-2018 code where they may have been necessary. In 2018 edition, import paths in `use` always begin with a crate name or `crate`/`super`/`self`, so `use $ident;` always refers to a crate. Since extern crates are always in scope in every module, `use $ident` does nothing.
Reviewed By: Imxset21
Differential Revision: D17290473
fbshipit-source-id: 23d86e5d0dcd5c2d4e53c7a36b4267101dd4b45c
Summary: None of these do anything.
Reviewed By: StanislavGlebik
Differential Revision: D16963565
fbshipit-source-id: bb0dd344daf151dd1d5c45392c43b014d2b17a07
Summary: Update populate_healer to act directly on a blobstore config rather than indirectly via a repo config.
Reviewed By: StanislavGlebik
Differential Revision: D15065424
fbshipit-source-id: 638778a61283dc9ed991c49936a21d02b8d2e3f3
Summary:
The healer is a blobstore-level operation, which is orthogonal to the concept of a repo; therefore, there should be no mention of repoid in any of the healer's structures or tables.
For now this leaves the schema unmodified, and fills the repoid with a dummy value (0). We can clean that up later.
Reviewed By: lukaspiatkowski, HarveyHunt
Differential Revision: D15051896
fbshipit-source-id: 438b4c6885f18934228f43d85cdb8bf2f0e542f1
Summary:
It's not cool to include the `flat/` prefix into the queue. Not cool at all.
It's not part of the actual blobstore key.
Let's make the populator cool again.
Reviewed By: ahornby
Differential Revision: D16494119
fbshipit-source-id: 8d3a34271d19ab0531cbb59e56b7a1ac04f66b07
Summary:
My previous diff (D16327788), despite claiming a 1000x improvement, was merely a ~220x
improvement. This diff is some tweaking with numbers, whihc brings about
another 1.3x improvement, bringing the total one to be 290x.
Reviewed By: krallin
Differential Revision: D16356932
fbshipit-source-id: 3d3f0c844eec9866217cf5a57f285fe8a56152de
Summary:
In earlier diffs in this stack, I updated the callsites that reference XDB tiers to use concrete &str types (which is what they were receiving until now ... but it wasn't spelled out as-is).
In this diff, I'm updating them to use owned `String` instead, which lets us hoist up `to_string()` and `clone()` calls in the stack, rather than pass down reference only to copy them later on.
This allows us to skip some unnecessary copies. Tt turns out we were doing quite a few "turn this String into a reference, pass it down the stack, then turn it back into a String".
Reviewed By: farnz
Differential Revision: D16260372
fbshipit-source-id: faec402a575833f6555130cccdc04e79ddb8cfef
Summary:
This implements and uses the `add_many` method of the blob healer queue. This method allows us to do batched adds, which in turn allows us to use `chunks` on Manifold iteration.
NB 1: I deliberately removed control symbols form progress print message. If we only print it on the same line, we loose it when the job crashes.
NB 2: I deliberately use range of `entries[0]`, as I want to pessimistically restart from the earliest in case of a failure.
Reviewed By: krallin
Differential Revision: D16327788
fbshipit-source-id: 8d9f3cf85ee7cbca657a8003a787b5ea84a1b9b0
Summary: With these fixes running the healer for `repo_id` 502 seems to work.
Reviewed By: aslpavel
Differential Revision: D16140872
fbshipit-source-id: bf2caf7428a74cf14cd61ea8b52b7e5822432646
Summary:
`local_instances` option was used to create fileblobstore or sqlite blobstore.
Now we use mononoke config for this purpose. Since this option is no longer
useful let's delete it
Reviewed By: krallin
Differential Revision: D16120065
fbshipit-source-id: 375a168b27e7f2cf1a6a77f487c5e013f9004546
Summary:
This migrates the internal structures representing the repo and storage config,
while retaining the existing config file format.
The `RepoType` type has been replaced by `BlobConfig`, an enum containing all
the config information for all the supported blobstores. In addition there's
the `StorageConfig` type which includes `BlobConfig`, and also
`MetadataDBConfig` for the local or remote SQL database for metadata.
Reviewed By: StanislavGlebik
Differential Revision: D15065421
fbshipit-source-id: 47636074fceb6a7e35524f667376a5bb05bd8612
Summary:
In the later diff we'll add batching of BlobstoreSyncQueue writes. It would be
much harder to add the batching if we also have to return this boolean.
And since noboby uses it, let's just remove it
Reviewed By: farnz
Differential Revision: D15248290
fbshipit-source-id: 72c64770c1b023e9de23a5dfccd8b4482302fe96
Summary: Tool that is used to populate blobstore synchronization queue from manifold bucket
Reviewed By: StanislavGlebik
Differential Revision: D13880216
fbshipit-source-id: 6527e4f7027dc8ce8e73f9949e080e53eda1c541
