Summary:
This new method syncs all log entries until the the certain log entry.
This function will first read from replica, however if it can't find new entries in the replica it will try to fetch from master.
Reviewed By: krallin
Differential Revision: D17929152
fbshipit-source-id: 77fa0ef28ae3c530d427e079079d2d4d6b6c6b62
Summary: Print calculated statistics in csv format, so we can generate csv file by redirecting output to some file. As a TODO we may want to add new fields to RepoStatistics struct, refactor code and create csv file using e.g. serde deserialization.
Reviewed By: krallin
Differential Revision: D17907650
fbshipit-source-id: 0e7f0af522cc72c067d59431039e44998d5dd354
Summary:
Some backfill types, e.g. fsnodes, require access to the unredacted repo in
order to correctly compute their values. For example, fsnodes require the
content hashes of the redacted blobs to correctly form the fnsode graph.
This kind of access is ok, as we are only deriving summary information
(hashes), and won't reveal the content of the redacted blob.
Add the ability for `backfill_derive_data` to access the unredacted repo,
limited only to whitelisted derived data types.
Reviewed By: ikostia
Differential Revision: D17877191
fbshipit-source-id: 6c9b1dfdfb2e6f431815ddf3de60029fbb180454
Summary:
Follow up from D17765160. Let's unify logview initialization in one place.
This will let us log to logview from all binaries we have in mononoke
Note - the diff touches a lot of files, but main changes are in
cmdlib/src/args.rs, apiserver/src/main.rs and server/src/main.rs.
Reviewed By: krallin
Differential Revision: D17895480
fbshipit-source-id: c922adfb385461ff168bd788e42ea1b88891f7cf
Summary:
This diff updates all license headers to use the new text and style.
Also, a few internal files were missing the header, but now they have it.
`fbcode/common/rust/netstring/` had the internal header, but now it has
GPLV2PLUS - since that goes to Mononoke's Github too.
Differential Revision: D17881539
fbshipit-source-id: b70d2ee41d2019fc7c2fe458627f0f7c01978186
Summary: Added option to calculate statistics for old changesets saved in file. Currently it prints calculated statistics.
Reviewed By: StanislavGlebik
Differential Revision: D17811305
fbshipit-source-id: 162946941b9e153ffedc1fc28539eebacab77132
Summary:
This updates the blobstore healer to avoid using a hardcoded list of regions.
Facebook
I'll remove this from the TW spec once this makes it through Conveyor.
This Configerator file is referenced in the DBA region turnup instructions: https://our.intern.facebook.com/intern/wiki/MySQL/DBaaS/Internal/Regional_Turnup_Steps/#configs (they update `configerator/raw_configs/myrouter/routing_v3.jsonc`, which includes this file)
Reviewed By: StanislavGlebik
Differential Revision: D17787579
fbshipit-source-id: ec9128202679ce0dbe18d77fb84c3fbf6186cd19
Summary:
There's a few things broken with common/rust/sql and the blobstore healer's handling of replication lag right now:
- If Seconds_Behind_Master isn't an int (it'll be NULL if replications is paused), it just panics.
- If it's talking to a server that it expected to be a replica but is a master, it returns None for the replication lag, but 0 would be more appropriate.
- If a region no longer has a replica, it errors out.
This diff fixes that:
- If replication is paused, we return None for lag.
- If we're talking to a master, we return 0.
- If a region has no replica, we ignore it.
Reviewed By: StanislavGlebik
Differential Revision: D17787580
fbshipit-source-id: 9e5e7682456870b88910afec12e1c409fd8c5ba6
Summary:
This is better from the usability perspective, as it can resolve hg changeset, bonsai changeset and a
bookmark.
However, this also means that when the user does provide an Hg commit hash and we need an hg commit hash, we are doing one more database query (`get_hg_from_bonsai_changeset`). This is a little bad, but all of the use-cases I changed here are from the admin tool, so we should be fine.
Reviewed By: StanislavGlebik
Differential Revision: D17740448
fbshipit-source-id: 7c1620979e25631f1a4e44d6310668fe634b2075
Summary: Skip calculating number of lines for files with size greater than 10MB
Reviewed By: krallin
Differential Revision: D17710287
fbshipit-source-id: 025e94b66539b471e5e48aee0ec7e61808cb0bd6
Summary:
Attached tasks have more details, but in short - if a commit has two parents
then the order of them in mercurial can change but the commit hash won't
change. In Mononoke it's different - change of parent order changes the hash.
This option allows us to fix parent order if needed
Reviewed By: krallin
Differential Revision: D17683807
fbshipit-source-id: 23549e55369b625ea5d597bc336b465373cbbc74
Summary: print_statistics and log_statistics were used in the same places in code.
Reviewed By: StanislavGlebik
Differential Revision: D17665790
fbshipit-source-id: 5d41f1b9d77edbe12db25f109a8bb171c8072fb4
Summary: Set time when changeset was created instead of current time (default time) in samples that are saved in Scuba
Reviewed By: StanislavGlebik
Differential Revision: D17629069
fbshipit-source-id: a4c34f38eff520b96f7e8a85445ee2255c220941
Summary:
As suggested by yfeldblum in https://our.intern.facebook.com/intern/diff/D17544447/?transaction_id=2964022370277861.
> the `rust_` bit seems superfluous; the `_thrift` bit opens the door for confusion with apache thrift. Any issue using a module name like `fbthrift`?
---
```
$ fastmod '\brust_thrift\b' fbthrift -d ~/fbcode -e rs,toml
$ fastmod '\brust_thrift\b' fbthrift -d ~/fbcode -g '**/TARGETS'
$ hg revert experimental
$ arc lint -a
```
Reviewed By: bolinfest
Differential Revision: D17611123
fbshipit-source-id: b621a422480b00eb2e339ff7542cc66c3ca5b8ec
Summary: This is the bare minimum to ask the remapping table for any values it has. Combined with other admin tool commands, you can use this to track down a synced commit.
Reviewed By: StanislavGlebik
Differential Revision: D17554261
fbshipit-source-id: f5f967a55182b614b68b5a8c1421401921c9a268
Summary: Modified statistics_collector tool that now it saves statistics to Scuba
Reviewed By: HarveyHunt
Differential Revision: D17570918
fbshipit-source-id: b22094dcc915bc9ba9cb76c09f73e9590c65e641
Summary: Add `fsnodes` as a kind of derived data that can be backfilled.
Reviewed By: krallin
Differential Revision: D17549835
fbshipit-source-id: 14d7540567c0bd0eae4f53944a5c4c19e5902105
Summary:
This is currently a bit hacky, as bonsai->hg conversion does not work like other derived data, but it does the job.
We want to be able to dump Bonsai changesets from synced commits onto a bookmark, and have a tailer fill in Mercurial changesets behind us. This code does that via the existing mechanism for creating changesets
Reviewed By: StanislavGlebik
Differential Revision: D17552077
fbshipit-source-id: d5b13f884332e57e7d8c55256ccb4ea918fe4921
Summary: I modified tool that now it calculates statistics starting from current master (or other provided by us bookmark), then it waits for new commits and updates statistics.
Reviewed By: StanislavGlebik
Differential Revision: D17502571
fbshipit-source-id: c98b06529cc7fc6ab9ee358b87dc9fa161546da5
Summary: This is convenient to get the content ID for a file at a given path.
Reviewed By: HarveyHunt
Differential Revision: D17498393
fbshipit-source-id: f40706b9289ca77e99cb3d7070b396ee134f79c3
Summary: Modified code to calculate next statistic: total number of lines in source files.
Reviewed By: krallin
Differential Revision: D17395757
fbshipit-source-id: f19ee888b9836b42e755acee2d4c76e1fef6265e
Summary:
Collect together all derived data in the `derived_data` directory. This moves
the `unodes` derivation crate.
Reviewed By: StanislavGlebik
Differential Revision: D17314208
fbshipit-source-id: f1b575446192cb9799e443efb1cb863b80ef72d6
Summary:
This diff moves initFacebook calls that used to happen just before FFI calls to instead happen at the beginning of main.
The basic assumption of initFacebook is that it happens at the beginning of main before there are additional threads. It must be allowed to modify process-global state like env vars or gflags without the possibility of a data race from other code concurrently reading those things. As such, the previous approach of calling initFacebook through `*fbinit::FACEBOOK` near FFI calls was prone to race conditions.
The new approach is based on attribute macros added in D17245802.
---
The primary remaining situations that still require `*fbinit::FACEBOOK` are when we don't directly control the function arguments surrounding the call to C++, such as in lazy_static:
lazy_static! {
static ref S: Ty = {
let _ = *fbinit::FACEBOOK;
/* call C++ */
};
}
and quickcheck:
quickcheck! {
fn f(/* args that impl Arbitrary */) {
let _ = *fbinit::FACEBOOK;
/* call C++ */
}
}
I will revisit these in a separate diff. They are a small fraction of total uses of fbinit.
Reviewed By: Imxset21
Differential Revision: D17328504
fbshipit-source-id: f80edb763e7f42b3216552dd32f1ea0e6cc8fd12
Summary: `single` subcommand which regenerates specified derived data for provided changeset. This is useful for debugging purposes
Reviewed By: StanislavGlebik
Differential Revision: D17318968
fbshipit-source-id: 3d3a551991b0628a05335addedd7d5b315fd45d2
Summary: I modified statistics_collector tool, so now it calculates both number of files, and total file size in repo.
Reviewed By: krallin
Differential Revision: D17342512
fbshipit-source-id: 94217d8b61c2a7350f1793a2ef33f84d600bbb54
Summary:
Previously rebuilding skiplist required fetching all the commits from mysql,
and then rebuilding the index from scratch. That was quite slow (> 16 mins to
finish). Instead let's try to read the key to check if it has the data already
and prepopulate skiplist with it.
Reviewed By: krallin
Differential Revision: D17343950
fbshipit-source-id: e8a446b94af61dbbd224d853f7dd8dd41510549d
Summary:
This wires up the stdlog crate with our slog output. The upshot is that we can
now run binaries with `RUST_LOG` set and expect it to work.
This is nice because many crates use stdlog (e.g. Tokio, Hyper), so this is
convenient to get access to their logging. For example, if you run with
`RUST_LOG=gotham=info,hyper=debug`, then you get debug logs from Hyper and info
logs from Gotham.
The way this works is by registering a stdlog logger that uses the env_logger's
filter (the one that "invented" `RUST_LOG`) to filter logs, and routes them to
slog if they pass. Note that the slog Logger used there doesn't do any
filtering, since we already do it before sending logs there.
One thing to keep in mind is that we should only register the stdlog global
logger once. I've renamed `get_logger` to `init_logging` to make this clearer.
This behavior is similar to what we do with `init_cachelib`. I've updated
callsites accordingly.
Note that we explicitly tell the stdlog framework to ignore anything that we
won't consider for logging. If you don't set `RUST_LOG`, then the default
logging level is `Error`, which means that anything below error that is sent to
stdlog won't even get to out filtering logic (the stdlog macros for logging
check for the global level before actually logging), so this is cheap unless
you do set `RUST_LOG`.
As part of this, I've also updated all our binaries (and therefore, tests) to
use glog for logging. We had been meaning to do this, and it was convenient to
do it here because the other logger factory we were using didn't make it easy
to get a Drain without putting it a Logger.
Reviewed By: ahornby
Differential Revision: D17314200
fbshipit-source-id: 19b5e8edc3bbe7ba02ccec4f1852dc3587373fff
Summary:
Fetching commits takes a long time. I've added a subcommand that saves them to
a file, and "backfill" command can later read it. Note that another option
would be to add indices to the db to make commit fetching faster, however this diff
was simpler to do.
I also fixed a panic which was because of division by zero.
Reviewed By: aslpavel
Differential Revision: D17341640
fbshipit-source-id: b0335ebf8799cd48884c19fa8a0ee8023eb751af
Summary:
`extern crate` is usually no longer needed in 2018 edition of Rust. This diff removes `extern crate` lines from fbcode where possible, replacing #[macro_use] with individual import of macros.
Before:
```
#[macro_use]
extern crate futures_ext;
extern crate serde_json;
```
After:
```
use futures_ext::try_boxfuture;
```
Reviewed By: Imxset21
Differential Revision: D17313537
fbshipit-source-id: 70462a2c161375017b77fa44aba166884ad2fdc3
Summary: First version of binary that counts number of files in specific repo and print it.
Reviewed By: StanislavGlebik
Differential Revision: D17285812
fbshipit-source-id: 30bc9e2c11ee75fcfb8d94610bd4e320a56dafc7
Summary:
We need this for `megarepo_test` blobimport, which will prefix `fbsource` and
`ovrsource` bookmarks.
Reviewed By: krallin
Differential Revision: D17286275
fbshipit-source-id: 40bb3f97b0a06dcd636f891d9b32c7ef9b55a0fc
Summary: I think these are left over from pre-2018 code where they may have been necessary. In 2018 edition, import paths in `use` always begin with a crate name or `crate`/`super`/`self`, so `use $ident;` always refers to a crate. Since extern crates are always in scope in every module, `use $ident` does nothing.
Reviewed By: Imxset21
Differential Revision: D17290473
fbshipit-source-id: 23d86e5d0dcd5c2d4e53c7a36b4267101dd4b45c
Summary:
Commit sync will operate based on the following idea:
- there's one "large" and potentially many "small" repos
- there are two possible sync directions: large-to-small and small-to-large
- when syncing a small repo into a large repo, it is allowed to change paths
of each individual file, but not to drop files
- large repo prepends a predefined prefix to every bookmark name from the small repo, except for the bookmarks, specified in `common_pushrebase_bookmarks` list, which refers to the bookmarks that can be advanced by any small repo
Reviewed By: krallin
Differential Revision: D17258451
fbshipit-source-id: 6cdaccd0374250f6bbdcbc9a280da89ccd7dff97
Summary: Admin subcommand to force unode regeneration for specified commit. This is useful for debugging unodes generation perforamance
Reviewed By: farnz
Differential Revision: D17225913
fbshipit-source-id: 0ec700f670edd01e4c7659fe437e91c2f2c43497
Summary: None of these do anything.
Reviewed By: StanislavGlebik
Differential Revision: D16963565
fbshipit-source-id: bb0dd344daf151dd1d5c45392c43b014d2b17a07
Summary:
This make it easier to benchmark things, in particular Manifold CDN
performance.
Reviewed By: StanislavGlebik
Differential Revision: D16963422
fbshipit-source-id: c110bd620fec6cfb25aa45cffdd059024495ff32
Summary:
This avoids typos when accessing arguments, and thus avoids panics when we
unwrap arguments that don't exist.
Reviewed By: StanislavGlebik
Differential Revision: D16963424
fbshipit-source-id: cf35dfaf026be8842902af3c817301672a8929d2
Summary: Update bounded_traversal_stream to take IntoIterator of initial values. This allows simultaneous navigation of a graph from multiple roots.
Reviewed By: farnz
Differential Revision: D17163609
fbshipit-source-id: c999e7653cb620c215331ecc46f5a800ced8ef37
