Commit Graph

442 Commits

Author SHA1 Message Date
Daniel Grzegorzewski
aef2f4d6d1 Add info logging in statistics_collector tool
Summary: Logging basic info about statistics_collector tool.

Reviewed By: StanislavGlebik

Differential Revision: D17605520

fbshipit-source-id: 1b44bcaa0fb5dd2ce4f54a5a02d49979f0e91e2f
2019-09-27 02:22:45 -07:00
David Tolnay
1cd333be79 rust/thrift: Rename rust_thrift runtime library to fbthrift
Summary:
As suggested by yfeldblum in https://our.intern.facebook.com/intern/diff/D17544447/?transaction_id=2964022370277861.

> the `rust_` bit seems superfluous; the `_thrift` bit opens the door for confusion with apache thrift. Any issue using a module name like `fbthrift`?

 ---

```
$ fastmod '\brust_thrift\b' fbthrift -d ~/fbcode -e rs,toml
$ fastmod '\brust_thrift\b' fbthrift -d ~/fbcode -g '**/TARGETS'
$ hg revert experimental
$ arc lint -a
```

Reviewed By: bolinfest

Differential Revision: D17611123

fbshipit-source-id: b621a422480b00eb2e339ff7542cc66c3ca5b8ec
2019-09-26 20:23:47 -07:00
Simon Farnsworth
e65a44a8c7 Add a quick query tool for the commit sync remapping database
Summary: This is the bare minimum to ask the remapping table for any values it has. Combined with other admin tool commands, you can use this to track down a synced commit.

Reviewed By: StanislavGlebik

Differential Revision: D17554261

fbshipit-source-id: f5f967a55182b614b68b5a8c1421401921c9a268
2019-09-25 09:25:46 -07:00
Daniel Grzegorzewski
2e6c46e7c9 Save statistics from statistics_collector to Scuba
Summary: Modified statistics_collector tool that now it saves statistics to Scuba

Reviewed By: HarveyHunt

Differential Revision: D17570918

fbshipit-source-id: b22094dcc915bc9ba9cb76c09f73e9590c65e641
2019-09-25 07:58:37 -07:00
Mark Thomas
3995bf9927 backfill_derived_data: add fsnodes
Summary: Add `fsnodes` as a kind of derived data that can be backfilled.

Reviewed By: krallin

Differential Revision: D17549835

fbshipit-source-id: 14d7540567c0bd0eae4f53944a5c4c19e5902105
2019-09-25 05:19:55 -07:00
Simon Farnsworth
2e4fe33290 Add hgchangesets to backfill_derived_data
Summary:
This is currently a bit hacky, as bonsai->hg conversion does not work like other derived data, but it does the job.

We want to be able to dump Bonsai changesets from synced commits onto a bookmark, and have a tailer fill in Mercurial changesets behind us. This code does that via the existing mechanism for creating changesets

Reviewed By: StanislavGlebik

Differential Revision: D17552077

fbshipit-source-id: d5b13f884332e57e7d8c55256ccb4ea918fe4921
2019-09-25 01:00:24 -07:00
Andres Suarez
239e008c73 mononoke: fix broken toml fixture
Reviewed By: krallin

Differential Revision: D17557847

fbshipit-source-id: 7c003d8075f394d07333f8294addc5235ab95305
2019-09-24 14:56:23 -07:00
Daniel Grzegorzewski
191194f60f Modified statistics_collector tool to calculate statistics in new commits
Summary: I modified tool that now it calculates statistics starting from current master (or other provided by us bookmark), then it waits for new commits and updates statistics.

Reviewed By: StanislavGlebik

Differential Revision: D17502571

fbshipit-source-id: c98b06529cc7fc6ab9ee358b87dc9fa161546da5
2019-09-24 10:35:52 -07:00
Thomas Orozco
2ac2e95d08 mononoke/admin: allow showing envelopes for filenodes
Summary: This is convenient to get the content ID for a file at a given path.

Reviewed By: HarveyHunt

Differential Revision: D17498393

fbshipit-source-id: f40706b9289ca77e99cb3d7070b396ee134f79c3
2019-09-20 04:08:53 -07:00
Kostia Balytskyi
a3a2b0200d mononoke: extract non-arg-related things into cmdlib::helpers from cmdlib::args
Summary: What the title says.

Reviewed By: farnz

Differential Revision: D17475432

fbshipit-source-id: b117ab37df6c1965c20da6154cd32d6c68dd9882
2019-09-19 06:56:39 -07:00
Daniel Grzegorzewski
8515e6f741 Modified statistics_collector tool to calculate total number of lines in source files
Summary: Modified code to calculate next statistic: total number of lines in source files.

Reviewed By: krallin

Differential Revision: D17395757

fbshipit-source-id: f19ee888b9836b42e755acee2d4c76e1fef6265e
2019-09-17 04:56:21 -07:00
Mark Thomas
aa6ad17e93 unodes: move crate to derived_data
Summary:
Collect together all derived data in the `derived_data` directory.  This moves
the `unodes` derivation crate.

Reviewed By: StanislavGlebik

Differential Revision: D17314208

fbshipit-source-id: f1b575446192cb9799e443efb1cb863b80ef72d6
2019-09-17 02:49:41 -07:00
David Tolnay
713973975d Replace *fbinit::FACEBOOK with #[fbinit::main]
Summary:
This diff moves initFacebook calls that used to happen just before FFI calls to instead happen at the beginning of main.

The basic assumption of initFacebook is that it happens at the beginning of main before there are additional threads. It must be allowed to modify process-global state like env vars or gflags without the possibility of a data race from other code concurrently reading those things. As such, the previous approach of calling initFacebook through `*fbinit::FACEBOOK` near FFI calls was prone to race conditions.

The new approach is based on attribute macros added in D17245802.

 ---

The primary remaining situations that still require `*fbinit::FACEBOOK` are when we don't directly control the function arguments surrounding the call to C++, such as in lazy_static:

    lazy_static! {
        static ref S: Ty = {
            let _ = *fbinit::FACEBOOK;
            /* call C++ */
        };
    }

and quickcheck:

    quickcheck! {
        fn f(/* args that impl Arbitrary */) {
            let _ = *fbinit::FACEBOOK;
            /* call C++ */
        }
    }

I will revisit these in a separate diff. They are a small fraction of total uses of fbinit.

Reviewed By: Imxset21

Differential Revision: D17328504

fbshipit-source-id: f80edb763e7f42b3216552dd32f1ea0e6cc8fd12
2019-09-13 20:17:29 -07:00
Pavel Aslanov
0ca9f6e068 single subcommand for debugging purposes
Summary: `single` subcommand which regenerates specified derived data for provided changeset. This is useful for debugging purposes

Reviewed By: StanislavGlebik

Differential Revision: D17318968

fbshipit-source-id: 3d3a551991b0628a05335addedd7d5b315fd45d2
2019-09-13 05:13:34 -07:00
Daniel Grzegorzewski
fe39645958 Modified statistics_collector tool to calculate total file size.
Summary: I modified statistics_collector tool, so now it calculates both number of files, and total file size in repo.

Reviewed By: krallin

Differential Revision: D17342512

fbshipit-source-id: 94217d8b61c2a7350f1793a2ef33f84d600bbb54
2019-09-13 05:08:23 -07:00
Stanislau Hlebik
60d9fffda0 mononoke: make skiplist creation faster
Summary:
Previously rebuilding skiplist required fetching all the commits from mysql,
and then rebuilding the index from scratch. That was quite slow (> 16 mins to
finish). Instead let's try to read the key to check if it has the data already
and prepopulate skiplist with it.

Reviewed By: krallin

Differential Revision: D17343950

fbshipit-source-id: e8a446b94af61dbbd224d853f7dd8dd41510549d
2019-09-12 07:54:53 -07:00
Thomas Orozco
53bf2886cc mononoke: connect stdlog and slog
Summary:
This wires up the stdlog crate with our slog output. The upshot is that we can
now run binaries with `RUST_LOG` set and expect it to work.

This is nice because many crates use stdlog (e.g.  Tokio, Hyper), so this is
convenient to get access to their logging.  For example, if you run with
`RUST_LOG=gotham=info,hyper=debug`, then you get debug logs from Hyper and info
logs from Gotham.

The way this works is by registering a stdlog logger that uses the env_logger's
filter (the one that "invented" `RUST_LOG`) to filter logs, and routes them to
slog if they pass. Note that the slog Logger used there doesn't do any
filtering, since we already do it before sending logs there.

One thing to keep in mind is that we should only register the stdlog global
logger once. I've renamed `get_logger` to `init_logging` to make this clearer.
This behavior is similar to what we do with `init_cachelib`. I've updated
callsites accordingly.

Note that we explicitly tell the stdlog framework to ignore anything that we
won't consider for logging. If you don't set `RUST_LOG`, then the default
logging level is `Error`, which means that anything below error that is sent to
stdlog won't even get to out filtering logic (the stdlog macros for logging
check for the global level before actually logging), so this is cheap unless
you do set `RUST_LOG`.

As part of this, I've also updated all our binaries (and therefore, tests) to
use glog for logging. We had been meaning to do this, and it was convenient to
do it here because the other logger factory we were using didn't make it easy
to get a Drain without putting it a Logger.

Reviewed By: ahornby

Differential Revision: D17314200

fbshipit-source-id: 19b5e8edc3bbe7ba02ccec4f1852dc3587373fff
2019-09-12 04:13:11 -07:00
Stanislau Hlebik
6cf20fc88f mononoke: prefetch commits in backfill derive data
Summary:
Fetching commits takes a long time. I've added a subcommand that saves them to
a file, and "backfill" command can later read it. Note that another option
would be to add indices to the db to make commit fetching faster, however this diff
was simpler to do.

I also fixed a panic which was because of division by zero.

Reviewed By: aslpavel

Differential Revision: D17341640

fbshipit-source-id: b0335ebf8799cd48884c19fa8a0ee8023eb751af
2019-09-12 02:25:20 -07:00
David Tolnay
577a31b2a1 Delete extern crate lines
Summary:
`extern crate` is usually no longer needed in 2018 edition of Rust. This diff removes `extern crate` lines from fbcode where possible, replacing #[macro_use] with individual import of macros.

Before:

```
#[macro_use]
extern crate futures_ext;
extern crate serde_json;
```

After:

```
use futures_ext::try_boxfuture;
```

Reviewed By: Imxset21

Differential Revision: D17313537

fbshipit-source-id: 70462a2c161375017b77fa44aba166884ad2fdc3
2019-09-11 22:01:58 -07:00
Daniel Grzegorzewski
ab56c2051e Binary that counts number of files in the repo
Summary: First version of binary that counts number of files in specific repo and print it.

Reviewed By: StanislavGlebik

Differential Revision: D17285812

fbshipit-source-id: 30bc9e2c11ee75fcfb8d94610bd4e320a56dafc7
2019-09-11 03:21:53 -07:00
Kostia Balytskyi
a878f7ed13 mononoke: make blobimport capable of prefixing bookmarks
Summary:
We need this for `megarepo_test` blobimport, which will prefix `fbsource` and
`ovrsource` bookmarks.

Reviewed By: krallin

Differential Revision: D17286275

fbshipit-source-id: 40bb3f97b0a06dcd636f891d9b32c7ef9b55a0fc
2019-09-11 03:19:38 -07:00
David Tolnay
ecc30ffe8a Delete lines of the form "use [a-z_]+;"
Summary: I think these are left over from pre-2018 code where they may have been necessary. In 2018 edition, import paths in `use` always begin with a crate name or `crate`/`super`/`self`, so `use $ident;` always refers to a crate. Since extern crates are always in scope in every module, `use $ident` does nothing.

Reviewed By: Imxset21

Differential Revision: D17290473

fbshipit-source-id: 23d86e5d0dcd5c2d4e53c7a36b4267101dd4b45c
2019-09-10 15:06:19 -07:00
Pavel Aslanov
4d22d9f260 add skip and regenerate logic to backfill subcommand
Summary:
Add support for:
- regeneration of already backfilled derived data.
- skipping changesets (useful when we want to resume interrupted backfill)

Reviewed By: StanislavGlebik

Differential Revision: D17282548

fbshipit-source-id: 1e6e9ac5e3ac834b7d942431a007a52e0ddd75b8
2019-09-10 08:22:18 -07:00
Kostia Balytskyi
800824a2bd mononoke: add config reading logic for commit sync configs
Summary:
Commit sync will operate based on the following idea:
- there's one "large" and potentially many "small" repos
- there are two possible sync directions: large-to-small and small-to-large
- when syncing a small repo into a large repo, it is allowed to change paths
  of each individual file, but not to drop files
- large repo prepends a predefined prefix to every bookmark name from the small repo, except for the bookmarks, specified in `common_pushrebase_bookmarks` list, which refers to the bookmarks that can be advanced by any small repo

Reviewed By: krallin

Differential Revision: D17258451

fbshipit-source-id: 6cdaccd0374250f6bbdcbc9a280da89ccd7dff97
2019-09-10 07:23:15 -07:00
Pavel Aslanov
2f8314e204 actually allow to used fastlog derived data type
Summary: I've forgotten to amend before lending changes

Reviewed By: StanislavGlebik

Differential Revision: D17261853

fbshipit-source-id: 99a050a06348dd4fcc2b45e6243c1e56369bb0e9
2019-09-10 02:44:05 -07:00
Pavel Aslanov
740d4276aa add fastlog support
Summary: Extend `backfill_derived_data` utility to support `RootFastlog` derived data type.

Reviewed By: krallin

Differential Revision: D17258729

fbshipit-source-id: e2e1fc2ad067fb5f7b597f54727ade89a4721cb4
2019-09-09 07:48:08 -07:00
Pavel Aslanov
3284fb896c added ability to regenerate selected unode
Summary: Admin subcommand to force unode regeneration for specified commit. This is useful for debugging unodes generation perforamance

Reviewed By: farnz

Differential Revision: D17225913

fbshipit-source-id: 0ec700f670edd01e4c7659fe437e91c2f2c43497
2019-09-09 03:59:50 -07:00
Thomas Orozco
73f713ca70 mononoke: remove manifold / MySQL flags + safe_writes
Summary: None of these do anything.

Reviewed By: StanislavGlebik

Differential Revision: D16963565

fbshipit-source-id: bb0dd344daf151dd1d5c45392c43b014d2b17a07
2019-09-06 07:17:45 -07:00
Thomas Orozco
a85c5a80b4 mononoke/benchmark_filestore: support delay, debug, randomize
Summary:
This make it easier to benchmark things, in particular Manifold CDN
performance.

Reviewed By: StanislavGlebik

Differential Revision: D16963422

fbshipit-source-id: c110bd620fec6cfb25aa45cffdd059024495ff32
2019-09-06 07:17:45 -07:00
Thomas Orozco
a0e7d5d1ac mononoke/benchmark_filestore: use constants for arg names
Summary:
This avoids typos when accessing arguments, and thus avoids panics when we
unwrap arguments that don't exist.

Reviewed By: StanislavGlebik

Differential Revision: D16963424

fbshipit-source-id: cf35dfaf026be8842902af3c817301672a8929d2
2019-09-06 07:17:45 -07:00
Alex Hornby
8c46a4cc03 rust/futures-ext: add multiple initial value support to bounded_traversal_stream
Summary: Update bounded_traversal_stream to take IntoIterator of initial values. This allows simultaneous navigation of a graph from multiple roots.

Reviewed By: farnz

Differential Revision: D17163609

fbshipit-source-id: c999e7653cb620c215331ecc46f5a800ced8ef37
2019-09-04 08:31:16 -07:00
Daniel Grzegorzewski
7354e0a60c Change Arc<BlobRepo> to BlobRepo
Reviewed By: StanislavGlebik

Differential Revision: D17162141

fbshipit-source-id: fb7466e0aaed4c2494ebeaeb49b38c15f521745d
2019-09-03 06:51:28 -07:00
Pavel Aslanov
53c956751f add fb303 to tail subcommand
Summary: Adds `fb303` back_fille_derived data when it runs in tailing mode

Reviewed By: krallin

Differential Revision: D17130955

fbshipit-source-id: d16fd2a729227ae3450c1c152a21a8ae8a01d7fb
2019-08-30 08:23:06 -07:00
Pavel Aslanov
3dd41ecbda added tail subcommand
Summary:
Tail subcommand:
- requests public bookmarks
- derives requested data
- repeat above in a loop

Reviewed By: StanislavGlebik

Differential Revision: D17091133

fbshipit-source-id: 936d27563a89e5d079113bb2b185dde33c54c951
2019-08-30 03:25:47 -07:00
David Tolnay
45c906ed80 Update to Rust 1.38.0-beta.3
Reviewed By: bolinfest

Differential Revision: D17128329

fbshipit-source-id: caa2699bf0ae94b33bebd42fb4bbf09d22405056
2019-08-30 02:01:47 -07:00
Alex Hornby
f144298321 mononoke: add --drain-only option to blobstore_healer
Summary: Add --drain-only option to blobstore_healer so we can drain a queue without healing when required.

Reviewed By: ikostia

Differential Revision: D17111436

fbshipit-source-id: e2fba4914e9699a4836933d0f35def015cd3dadc
2019-08-29 07:59:53 -07:00
Alex Hornby
5152659169 mononoke: blobstore_healer log more information on replication lag
Summary: Log the region with max replication lag as well as the lag size

Reviewed By: StanislavGlebik

Differential Revision: D17097266

fbshipit-source-id: c449167b09dbd2cdce7d65bfc956e7ccf59b2415
2019-08-29 07:59:53 -07:00
Alex Hornby
0dcf6e695a mononoke: blobstore_healer performance improvements
Summary:
blobstore_healer performance improvements

* speed up delete by using bulk delete
* speed up rep lag check by only doing it once if lag within bounds
* log some stats from each healer chunk run to help diagnose issues
* dont log the full storage config on each log line ( this was making it hard to read the logs )

Reviewed By: StanislavGlebik

Differential Revision: D17093809

fbshipit-source-id: 34299ad1835c5548b9502ad9d5a3e18eff1ab95b
2019-08-29 07:59:53 -07:00
Pavel Aslanov
7b88a9f880 add tracing for derive_impl and derive_manifest
Summary:
- added tracing for `derive_impl` and `derive_manifest`
- added `--trace` argument for `admin unodes` subcommand

Reviewed By: krallin

Differential Revision: D17092726

fbshipit-source-id: 8717ab6c29cadf83ca35319cf6f31e24f2d5dc45
2019-08-28 08:35:18 -07:00
Pavel Aslanov
767aa146e0 add unodes admin subcommand
Summary:
add `unodes` admin subcommands
- `verify`- compare with hg manifest
- `tree`  - recursively list all entries starting from provided prefix

Reviewed By: farnz

Differential Revision: D16940067

fbshipit-source-id: a9949da7720b384418500e59b65092caa01d4838
2019-08-27 07:19:37 -07:00
Pavel Aslanov
242741ab89 added backfill_derived_data utility
Summary:
Utility to backfill derived data:
 - currently only support `unodes`

Reviewed By: krallin

Differential Revision: D16827897

fbshipit-source-id: aa0bb8ae4fe895233e3f11775b3432184afb37f9
2019-08-27 07:19:37 -07:00
Pavel Aslanov
7f331ea1b0 move blob changeset to mercurial crate
Summary:
Move blob changeset to mercurial crate
- Allows for `Loadable` instance for `HgChangesetId`
- Moves mercurial logic to mercurial crate

Reviewed By: StanislavGlebik

Differential Revision: D17004609

fbshipit-source-id: b49ca5e9297be1664c585019867aa0de501cf25a
2019-08-27 04:25:54 -07:00
Pavel Aslanov
7ef6c45b66 move blobs loading logic to merucrial/types/blobs
Summary:
This is a part of an effort to move mercurial related stuff into mercurial crate:
- `BlobManifest|HgBlobEntry` -> `mercurial/types/blobs`
- helper `Id` type is no longer required and `BlobManifeset` can be loaded with `HgManifesetId::load`
- `BlobManifest` implements `HgManifest` and typed `Manifest` traits

Reviewed By: StanislavGlebik

Differential Revision: D16984745

fbshipit-source-id: 33d4eb9c33137258956917328efa1f5ec6737ee9
2019-08-27 04:25:53 -07:00
Stanislau Hlebik
2fcd6c0d82 mononoke: add phases helpers in mononoke_admin
Summary:
Add two helpful commands:
1) Fetch the phase of a commit
2) List all public commits

Reviewed By: ikostia

Differential Revision: D16830576

fbshipit-source-id: 03f503cb30a7f150ea383d62fb71913dd2b93e6e
2019-08-24 04:13:31 -07:00
Thomas Orozco
5d4306f87d mononoke/admin: allow showing filenodes by path + ID
Summary:
This updates the `mononoke_admin` filenodes command to allow showing filenodes
by path + ID in addition to revision + path. This makes it possible to locate
the linknode given a filenode ID but no revision.

This updates the filenodes CLI command a little bit (so if you want the old
command you need to add `by-revision`), but this should be fine since this is
a command that a human will normally be calling.

Reviewed By: farnz

Differential Revision: D16936628

fbshipit-source-id: 1079ad580b235076739c92d0c1f3c9bfc80db4c1
2019-08-22 01:37:02 -07:00
Thomas Orozco
ce005af688 mononoke: add lfs_import command
Summary: This adds a command that allows importing a set of LFS blobs into Mononoke by streaming them out of another storage location (e.g. Dewey).

Reviewed By: HarveyHunt

Differential Revision: D16917806

fbshipit-source-id: 4917d56e11a187c89e00c23a32c6e791b351f8ef
2019-08-21 02:33:31 -07:00
Kostia Balytskyi
496749e2a7 mononoke: add create_repo_unredatected to the args helper
Summary:
We need this to accomodate `blobimport`'s needs of using (opening(D16914875)
and creating(this)) unredacted repos.

Note: this is nearly identical to D16914875.

Reviewed By: StanislavGlebik

Differential Revision: D16915557

fbshipit-source-id: 6b522e6fcee4582870df27cf66e8ad55e91235fe
2019-08-20 06:19:50 -07:00
Kostia Balytskyi
83065085fd mononoke: make blobimport use unredacted repo
Summary:
Currently blobimport fails if it tries to import a redacted blob. We should
only enforce redaction in the scenarios where end users could see the data.
Internal imports should be fine to proceed.

As a separate note, problems like this would probably go away when redaction
would be moved to the `FileStore` level.

Reviewed By: krallin

Differential Revision: D16914875

fbshipit-source-id: e531a6248f66b47de87c7aedcbc928bb7ad4ebc9
2019-08-20 04:01:00 -07:00
Kostia Balytskyi
cff091b4c6 mononoke: rename censorship/blacklisting into a redaction
Summary:
This is a mechanical part of rename, does not change any commit messages in
tests, does not change the scuba table name/config setting. Those are more
complex.

Reviewed By: krallin

Differential Revision: D16890120

fbshipit-source-id: 966c0066f5e959631995a1abcc7123549f7495b6
2019-08-20 04:01:00 -07:00
Stanislau Hlebik
b0bb654cb2 mononoke: move Loadable/Storable to blobstore crate
Summary:
mononoke_types doesn't look like a good crate for it. Let's move to blobstore
crate instead

Reviewed By: krallin

Differential Revision: D16914142

fbshipit-source-id: 57602c5a644adc87beea0cbc92beeb0d901d5873
2019-08-20 02:12:39 -07:00