Summary:
The `MononokeApp` interface is almost like a builder. Make it a proper builder
with a fluent API. This will allow us to add new options in the future without
changing all the places it is called.
As an example of why this is useful, we also clean up the options for selecting
a repo. The default behaviour is to add `--repo-id` and `--repo-name` options
that are optional. This is not usually what programs want, so they can now
specify `.with_repo_required()` to make one of these options mandatory, or
`.with_all_repos()` to remove the options entirely for commands that operate
on all repos.
The LFS server is updates to specify `.with_all_repos()`. We will update the
remaining existing commands separately as needed.
Reviewed By: krallin
Differential Revision: D18009138
fbshipit-source-id: 5ce46ee2418c92c3697138cccb2b3e2807e46faa
Summary:
Follow up from D17765160. Let's unify logview initialization in one place.
This will let us log to logview from all binaries we have in mononoke
Note - the diff touches a lot of files, but main changes are in
cmdlib/src/args.rs, apiserver/src/main.rs and server/src/main.rs.
Reviewed By: krallin
Differential Revision: D17895480
fbshipit-source-id: c922adfb385461ff168bd788e42ea1b88891f7cf
Summary:
This diff updates all license headers to use the new text and style.
Also, a few internal files were missing the header, but now they have it.
`fbcode/common/rust/netstring/` had the internal header, but now it has
GPLV2PLUS - since that goes to Mononoke's Github too.
Differential Revision: D17881539
fbshipit-source-id: b70d2ee41d2019fc7c2fe458627f0f7c01978186
Summary:
This is better from the usability perspective, as it can resolve hg changeset, bonsai changeset and a
bookmark.
However, this also means that when the user does provide an Hg commit hash and we need an hg commit hash, we are doing one more database query (`get_hg_from_bonsai_changeset`). This is a little bad, but all of the use-cases I changed here are from the admin tool, so we should be fine.
Reviewed By: StanislavGlebik
Differential Revision: D17740448
fbshipit-source-id: 7c1620979e25631f1a4e44d6310668fe634b2075
Summary: This is the bare minimum to ask the remapping table for any values it has. Combined with other admin tool commands, you can use this to track down a synced commit.
Reviewed By: StanislavGlebik
Differential Revision: D17554261
fbshipit-source-id: f5f967a55182b614b68b5a8c1421401921c9a268
Summary:
This diff moves initFacebook calls that used to happen just before FFI calls to instead happen at the beginning of main.
The basic assumption of initFacebook is that it happens at the beginning of main before there are additional threads. It must be allowed to modify process-global state like env vars or gflags without the possibility of a data race from other code concurrently reading those things. As such, the previous approach of calling initFacebook through `*fbinit::FACEBOOK` near FFI calls was prone to race conditions.
The new approach is based on attribute macros added in D17245802.
---
The primary remaining situations that still require `*fbinit::FACEBOOK` are when we don't directly control the function arguments surrounding the call to C++, such as in lazy_static:
lazy_static! {
static ref S: Ty = {
let _ = *fbinit::FACEBOOK;
/* call C++ */
};
}
and quickcheck:
quickcheck! {
fn f(/* args that impl Arbitrary */) {
let _ = *fbinit::FACEBOOK;
/* call C++ */
}
}
I will revisit these in a separate diff. They are a small fraction of total uses of fbinit.
Reviewed By: Imxset21
Differential Revision: D17328504
fbshipit-source-id: f80edb763e7f42b3216552dd32f1ea0e6cc8fd12
Summary:
This wires up the stdlog crate with our slog output. The upshot is that we can
now run binaries with `RUST_LOG` set and expect it to work.
This is nice because many crates use stdlog (e.g. Tokio, Hyper), so this is
convenient to get access to their logging. For example, if you run with
`RUST_LOG=gotham=info,hyper=debug`, then you get debug logs from Hyper and info
logs from Gotham.
The way this works is by registering a stdlog logger that uses the env_logger's
filter (the one that "invented" `RUST_LOG`) to filter logs, and routes them to
slog if they pass. Note that the slog Logger used there doesn't do any
filtering, since we already do it before sending logs there.
One thing to keep in mind is that we should only register the stdlog global
logger once. I've renamed `get_logger` to `init_logging` to make this clearer.
This behavior is similar to what we do with `init_cachelib`. I've updated
callsites accordingly.
Note that we explicitly tell the stdlog framework to ignore anything that we
won't consider for logging. If you don't set `RUST_LOG`, then the default
logging level is `Error`, which means that anything below error that is sent to
stdlog won't even get to out filtering logic (the stdlog macros for logging
check for the global level before actually logging), so this is cheap unless
you do set `RUST_LOG`.
As part of this, I've also updated all our binaries (and therefore, tests) to
use glog for logging. We had been meaning to do this, and it was convenient to
do it here because the other logger factory we were using didn't make it easy
to get a Drain without putting it a Logger.
Reviewed By: ahornby
Differential Revision: D17314200
fbshipit-source-id: 19b5e8edc3bbe7ba02ccec4f1852dc3587373fff
Summary: None of these do anything.
Reviewed By: StanislavGlebik
Differential Revision: D16963565
fbshipit-source-id: bb0dd344daf151dd1d5c45392c43b014d2b17a07
Summary:
Add two helpful commands:
1) Fetch the phase of a commit
2) List all public commits
Reviewed By: ikostia
Differential Revision: D16830576
fbshipit-source-id: 03f503cb30a7f150ea383d62fb71913dd2b93e6e
Summary:
This updates the `mononoke_admin` filenodes command to allow showing filenodes
by path + ID in addition to revision + path. This makes it possible to locate
the linknode given a filenode ID but no revision.
This updates the filenodes CLI command a little bit (so if you want the old
command you need to add `by-revision`), but this should be fine since this is
a command that a human will normally be calling.
Reviewed By: farnz
Differential Revision: D16936628
fbshipit-source-id: 1079ad580b235076739c92d0c1f3c9bfc80db4c1
Summary:
This is a mechanical part of rename, does not change any commit messages in
tests, does not change the scuba table name/config setting. Those are more
complex.
Reviewed By: krallin
Differential Revision: D16890120
fbshipit-source-id: 966c0066f5e959631995a1abcc7123549f7495b6
Summary: Clean up non-test usage of CoreContext::test_mock left over from T37478150
Reviewed By: farnz
Differential Revision: D16804838
fbshipit-source-id: f420b8186557a42e9b6c78437c0fb76c9a343b31
Summary: In the following diffs, I will implement the actual logic.
Reviewed By: krallin
Differential Revision: D16639720
fbshipit-source-id: f42b91476035343a6f00b11e4921e26836b1a574
Summary:
Sometimes, our Mononoke CLI tests report errors in mode/opt where they don't
actually output error messages at all when returning an error. Those error
messages actually get buffered by our logger, and are normally emitted when
said logger is dropped.
However, I have a suspicion that by calling process::exit(), we're not giving
Rust a chance to drop the logger (because that eventually just calls the exit
syscall), and as a result our buffer isn't getting flushed.
This patch attempts to fix this by removing all calls to process::exit in the
admin CLI, and instead returning an exit code from main, which I hope will
allow main to drop everything before it returns.
More broadly speaking, I don't think it's very good practice to exit from all
over the place in the admin CLI, so this fixes that :)
Reviewed By: HarveyHunt
Differential Revision: D16687747
fbshipit-source-id: 38e987463363e239d4f9166050a8dd26a4bef985
Summary: This adds debug subcommands metadata and verify in the Filestore. Those respectively output the metadata for a file verify that the file is reachable through all aliases.
Reviewed By: ahornby
Differential Revision: D16621789
fbshipit-source-id: 4a2156bfffb9d9641ce58f6d5f691364ba9dc145
Summary:
This is useful to inspect the Mercurial filenodes in Mononoke, like in S183272.
For example, I intend to use this subcommand to verify how well the future linknode healing works.
Reviewed By: krallin
Differential Revision: D16621516
fbshipit-source-id: 4266f85bce29b59072bf9c4f3e63777dae09a4f1
Summary: This helps us investigate the workings of the (upcoming) xdb blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D16208770
fbshipit-source-id: 33542f3d34a5c8b4287bb14b0aa97d3802b0e0d6
Summary:
`local_instances` option was used to create fileblobstore or sqlite blobstore.
Now we use mononoke config for this purpose. Since this option is no longer
useful let's delete it
Reviewed By: krallin
Differential Revision: D16120065
fbshipit-source-id: 375a168b27e7f2cf1a6a77f487c5e013f9004546
Summary:
Subcommand `blacklist` to censor multiple files. It stores the censored keys of the blacklisted files in table censored_contents.
The subcommand takes as arguments
- list of files
- commit hash
- string, representing the task ID for censoring the files (the task should contain the reason why the files is censored)
Reviewed By: ikostia
Differential Revision: D15939450
fbshipit-source-id: 7261ab2358cc4905d61a14f354de2949a2a94e7c
Summary: This help text was wrong. This fixes it.
Reviewed By: HarveyHunt
Differential Revision: D15555767
fbshipit-source-id: 5202d46a8527c9e44c5588d52f7425d3efb38898
Summary: Let's make it a bit more manageable.
Reviewed By: StanislavGlebik
Differential Revision: D15407978
fbshipit-source-id: 490e4d48c36349cf65171f38df0ef0372883281d
Summary: Practice commit. The behaviour will be implemented in the future
Reviewed By: ikostia
Differential Revision: D15372554
fbshipit-source-id: c9b3de81fcd2d79d34b8f8c898e28de2564cd887
Summary: I'm going to be doing some work on this file, but it's not up-to-date with rustfmt. To minimize merge conflicts and simplify diff reviews, I ran that earlier.
Reviewed By: StanislavGlebik
Differential Revision: D15364673
fbshipit-source-id: 32658629cc7ee29cdfc21be8dd526766d2dcbb0e
Summary:
Printint id will make it easier to debug sync job problems (for example,
rewiding the latest replayed id counter)
Reviewed By: krallin
Differential Revision: D15322624
fbshipit-source-id: 5c94be9cc0dcced9df51162adb598b6498f1c749
Summary:
Seems redundant to also require callers to open_ssl to also pass a
(mostly) identical string.
Also make open_ssl special-case filenodes with sharding (though filenodes
aren't currently opened through it).
Reviewed By: StanislavGlebik
Differential Revision: D15157834
fbshipit-source-id: 0df45307f17bdb2c021673b3153606031008bee2
Summary:
This migrates the internal structures representing the repo and storage config,
while retaining the existing config file format.
The `RepoType` type has been replaced by `BlobConfig`, an enum containing all
the config information for all the supported blobstores. In addition there's
the `StorageConfig` type which includes `BlobConfig`, and also
`MetadataDBConfig` for the local or remote SQL database for metadata.
Reviewed By: StanislavGlebik
Differential Revision: D15065421
fbshipit-source-id: 47636074fceb6a7e35524f667376a5bb05bd8612
Summary:
Reads current replay counter, and where a bookmark would point to after this
bundle is replayed. That can be useful for debugging
Reviewed By: aslpavel
Differential Revision: D15216378
fbshipit-source-id: fd250e27c2a6d7ee407510561a36b820cc5a1d2b
Summary:
The mononoke admin integration tests can be flaky when there is logging and an error, because those are respectively sent to stdout and stderr, which means they're not ordered relative to one another.
I attempted to fix this with minimal changes in D15146392, but that didn't solve the issue: StanislavGlebik reported that he still ran into a flaky test.
The reason for this is presumably that even though we write to stderr first then to stdout, there's no guarantee that the `.t` test runner will read whetever we output to stderr before it reads what we output to stdout.
I noted in that earlier diff that a more proper fix would be to write errors to stderr so they are indeed ordered relative to logging. That is what this diff does.
For consistency, I updated other fatal outcomes (bad arguments) to also log to stderr.
Reviewed By: StanislavGlebik
Differential Revision: D15181944
fbshipit-source-id: 3ca48870c39f11a7dcc57f1341f25ce61ccae360
Summary:
One of the tests for mononoke admin (which I introduced recently) appears to be flaky. Sometimes, output from stdout (the error that terminates the program) and stderr (logs emitted while the program runs) is flipped.
I suspect this is caused by buffering, so this patch flushes all output before writing the error (if there was one).
An alternative approach here might be to write the final error to stderr instead of stdosut (so everything goes to stderr). That feels cleaner, but it does change the interface a little bit, so I didn't take that approach just yet. That said, if nobody objects, I'm happy to pick that approach instead.
Reviewed By: farnz
Differential Revision: D15146392
fbshipit-source-id: 67481afd4802cb48d24d19052988be4a83433efd
Summary: This adds the ability to exclude blobimport entries when querying the count of remaining entries in the HG sync replay log.
Reviewed By: ikostia
Differential Revision: D15097549
fbshipit-source-id: ae1a9a31f51a044924fdebbdd219fff1e2b3d46a
Summary:
This introduces a new `--skip` flag in Mononoke admin under `hg-sync-bundle last-processed`. This will update the last-synced counter to the last blobimport that preceeds a log entry that is no a blobimport.
In other words, it makes it so that the last change to be processed is *not* a blobimport.
It will fail if:
- There is no valid log entry to jump ahead to (e.g. all the further log entries are blobimports).
- The current change to be processed is not a blobimport.
- The mutable counter was changed by someone else in the meantime.
Reviewed By: ikostia
Differential Revision: D15081759
fbshipit-source-id: 8465321b08d9c7b5bc97526400518bcf3ac77f13
Summary: This adds a command in mononoke admin to verify the consistency of remaining bundles to sync -- i.e. whether all bundles are blobimports or all of them are not blobimports.
Reviewed By: ikostia
Differential Revision: D15097935
fbshipit-source-id: a0df221c38e84897213edf232972ba420977e9d3
Summary:
The code in admin/main.rs passes `bookmarks` as the name of the SQLBookmarks database, but other pieces of code (the server which writes this data, and the hg sync job that normally consumes it) use `books`.
This hasn't been a problem until now since no integration tests exist for the admin tool. This path mismatch was never exposed since in production we don't actually care about this name.
Reviewed By: ikostia, StanislavGlebik
Differential Revision: D15081757
fbshipit-source-id: c4fcd328568160023f3c15fa2ab7d77accf2ad68
Summary: We'll do batching to save time on the sync job. We need to sync faster
Reviewed By: ikostia, farnz
Differential Revision: D14929027
fbshipit-source-id: 3139d0ece07f344cdafa5e39b698bc3b02625f0a
Summary: Slim down the blobstore trait crate as much as possible.
Reviewed By: aslpavel
Differential Revision: D14542675
fbshipit-source-id: faf09255f7fe2236a491742cd836226474f5967c
Summary:
Pointing the configs to a new package allows us to control the rollout more easily than overriding the existing skiplist in manifold.
This skiplist features a single skip entry for each node with a depth of up to 512 commits. Previously depth was 32k. This means on average we will have to do 16k iterations to find the commit we want after doing the last skiplist jump.
If we were to jump to the very beginning of the repo that would mean before we had to do ~130 skips and then on average 16k steps. With this skiplist we would do ~8k skips and 256 steps. As skips and steps are equally expensive this should be faster in almost every scenario. The total size of the skiplist is not affected by this change (as each node still contains one skip entry) but the new skiplist also covers the complete history, not only the 2kk most recent commits. This leads to size increase of ~20%.
Reviewed By: StanislavGlebik, HarveyHunt
Differential Revision: D14540318
fbshipit-source-id: 86fda5b6c57a06bc4a77e30625014ec119e7a155
Summary: To learn how far behind are we in the absolute bundle numbers.
Reviewed By: StanislavGlebik
Differential Revision: D14491672
fbshipit-source-id: 31d16f115b2b6fe4b88c25a847ce229e123b048b
Summary: We want to be able to manipulate hg-sync counters from the Mononoke admin.
Reviewed By: StanislavGlebik
Differential Revision: D14477676
fbshipit-source-id: 11218390bf469d4f297f7f13e9daee2d5f9bb35b
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary: Move complex things out of `blobstore` to thin out the dep graph. This didn't work as well as I'd hoped because `blobstore`->`mononoke-types`->`thrift`/`sql`.
Reviewed By: StanislavGlebik
Differential Revision: D13915511
fbshipit-source-id: c210dda23fa7102168c0ca14a035ed5c03a6993c
Summary:
Let's split reachability index crate. The main goal is to reduce compilation
time. Now crates like revsets will only depend on traits definition but not on
the actual implementation (skiplist of genbfs).
Reviewed By: lukaspiatkowski
Differential Revision: D13878403
fbshipit-source-id: 022eca50ac4bc7416e9fe5f3104f0a9a65195b26
Summary:
Some crates, namely revsets and reachabilityindex, currently depend on
blobrepo, while all they need is the ability to fetch commits.
By moving changeset_fetcher outside this dependency will be removed. That may
make builds faster
Reviewed By: lukaspiatkowski
Differential Revision: D13878369
fbshipit-source-id: 9ee8973a9170557a4dede5404dd374aa4a000405
Summary:
Currently if a crate depends even on a single type from metaconfig then in
order to compile this trait buck first compiles metaconfig crate with all the
logic of parsing the configs.
This diff split metaconfig into two crates. The first one just holds the types for
"external consumption" by other crates. The second holds the parsing logic.
That makes builds faster
Reviewed By: jsgf, lukaspiatkowski
Differential Revision: D13877592
fbshipit-source-id: f353fb2d1737845bf1fa0de515ff8ef131020063
