Commit Graph

133 Commits

Author SHA1 Message Date
Kostia Balytskyi
e972a82545 mononoke: migrate megarepotool move subcommand to core mononoke movers
Summary:
Rather than using hardcoded path maps and ad-hoc moveres, lets use the
logic which creates moves from config.

Reviewed By: farnz

Differential Revision: D17424678

fbshipit-source-id: 64a0a0b1c7332661408444a6d81f5931ed680c3c
2019-09-19 03:33:53 -07:00
Thomas Orozco
53bf2886cc mononoke: connect stdlog and slog
Summary:
This wires up the stdlog crate with our slog output. The upshot is that we can
now run binaries with `RUST_LOG` set and expect it to work.

This is nice because many crates use stdlog (e.g.  Tokio, Hyper), so this is
convenient to get access to their logging.  For example, if you run with
`RUST_LOG=gotham=info,hyper=debug`, then you get debug logs from Hyper and info
logs from Gotham.

The way this works is by registering a stdlog logger that uses the env_logger's
filter (the one that "invented" `RUST_LOG`) to filter logs, and routes them to
slog if they pass. Note that the slog Logger used there doesn't do any
filtering, since we already do it before sending logs there.

One thing to keep in mind is that we should only register the stdlog global
logger once. I've renamed `get_logger` to `init_logging` to make this clearer.
This behavior is similar to what we do with `init_cachelib`. I've updated
callsites accordingly.

Note that we explicitly tell the stdlog framework to ignore anything that we
won't consider for logging. If you don't set `RUST_LOG`, then the default
logging level is `Error`, which means that anything below error that is sent to
stdlog won't even get to out filtering logic (the stdlog macros for logging
check for the global level before actually logging), so this is cheap unless
you do set `RUST_LOG`.

As part of this, I've also updated all our binaries (and therefore, tests) to
use glog for logging. We had been meaning to do this, and it was convenient to
do it here because the other logger factory we were using didn't make it easy
to get a Drain without putting it a Logger.

Reviewed By: ahornby

Differential Revision: D17314200

fbshipit-source-id: 19b5e8edc3bbe7ba02ccec4f1852dc3587373fff
2019-09-12 04:13:11 -07:00
Kostia Balytskyi
800824a2bd mononoke: add config reading logic for commit sync configs
Summary:
Commit sync will operate based on the following idea:
- there's one "large" and potentially many "small" repos
- there are two possible sync directions: large-to-small and small-to-large
- when syncing a small repo into a large repo, it is allowed to change paths
  of each individual file, but not to drop files
- large repo prepends a predefined prefix to every bookmark name from the small repo, except for the bookmarks, specified in `common_pushrebase_bookmarks` list, which refers to the bookmarks that can be advanced by any small repo

Reviewed By: krallin

Differential Revision: D17258451

fbshipit-source-id: 6cdaccd0374250f6bbdcbc9a280da89ccd7dff97
2019-09-10 07:23:15 -07:00
Thomas Orozco
cff0b359b3 mononoke/lfs_server: run without upstream + integration tests
Summary:
This adds support in the LFS Server for running without an upstream. This is
useful to run tests, because it lets chain LFS Servers and have one acting as a
proxy and one acting as a backend.

This also highlighted a bug in my implementation (we were expecting a download
action from upstream if the data didn't need to be uploaded there), which I
fixed here.

For the time being, we won't use this in production.

Reviewed By: HarveyHunt

Differential Revision: D17263039

fbshipit-source-id: 7cba550054e5f052a4b8953ebe0195907919aade
2019-09-10 06:39:53 -07:00
Kostia Balytskyi
1ece1d8688 mononoke: add a tool for megarepo manipulations
Reviewed By: StanislavGlebik

Differential Revision: D17181600

fbshipit-source-id: 87565ac13f8bca36a65a09c0a6169c3dcf692ff9
2019-09-06 05:28:18 -07:00
Daniel Grzegorzewski
94f41ba5aa Added option to use reponame insted of repo-id
Summary: Currently in mononoke_admin tool we have to use repo-id to identify repository. Sometimes it can be inconvenient. Changed it, so we can use either repo-id or reponame.

Reviewed By: StanislavGlebik

Differential Revision: D17202962

fbshipit-source-id: d33ad55f53c839afc70e42c26501ecd4421e32c0
2019-09-05 09:36:27 -07:00
Thomas Orozco
ce005af688 mononoke: add lfs_import command
Summary: This adds a command that allows importing a set of LFS blobs into Mononoke by streaming them out of another storage location (e.g. Dewey).

Reviewed By: HarveyHunt

Differential Revision: D16917806

fbshipit-source-id: 4917d56e11a187c89e00c23a32c6e791b351f8ef
2019-08-21 02:33:31 -07:00
Kostia Balytskyi
cff091b4c6 mononoke: rename censorship/blacklisting into a redaction
Summary:
This is a mechanical part of rename, does not change any commit messages in
tests, does not change the scuba table name/config setting. Those are more
complex.

Reviewed By: krallin

Differential Revision: D16890120

fbshipit-source-id: 966c0066f5e959631995a1abcc7123549f7495b6
2019-08-20 04:01:00 -07:00
Harvey Hunt
b778fd2482 mononoke: Add a rechunker binary
Summary:
Create a new binary that can be used to rechunk files content using the filestore.
The binary accepts multiple filenodes, that it will then go and rechunk using the filestore
config provided to it.

Reviewed By: krallin

Differential Revision: D16802701

fbshipit-source-id: d7c05729f5072ff2925bbc90cdd89fcfed56bba2
2019-08-16 09:08:45 -07:00
Thomas Orozco
cfa4c8332f mononoke/integration: disable blackhole for apiserver tests
Summary:
The network blackhole is causing the API server to occasionally hang while serving requests, which has broken some LFS tests. This appears to be have happened in the last month or so, but unfortunately, I haven't been able to root cause why this is happening.

From what I can tell, we have an hg client that tries an upload to the API Server, and uploads everything... and then the API server just hangs. If I kill the hg client, then the API server responds with a 400 (so it's not completely stuck), but otherwise it seems like the API server is waiting for something to happen on the client-side, but the client isn't sending that.

As far as I can tell, the API Server isn't actualy trying to make outbound requests (strace does report that it has a Scribe client that's trying to connect, but Scuba logging isn't enabled, and this is just trying to connect but not send anything), but something with the blackhole is causing this hg - API server interaciton to fail.

In the meantime, this diff disables the blackhole for those tests that definitely don't work when it's enabled ...

Reviewed By: HarveyHunt

Differential Revision: D16599929

fbshipit-source-id: c6d77c5428e206cd41d5466e20405264622158ab
2019-08-01 07:36:02 -07:00
Stanislau Hlebik
807a7054e9 mononoke: add test_instance config argument
Summary:
This config option should be enabled in tests. We can use it to not run prod
services in non-test enviroment.

It will be used in the next diff to not load configerator in tests

Reviewed By: krallin

Differential Revision: D16221512

fbshipit-source-id: 7e0ba9c1d46b652a06e0e767de5df78d1671951a
2019-07-12 07:21:59 -07:00
Alex Hornby
576f060f04 mononoke: integration tests, remove unnecessary usages of cat
Summary: Remove unnecessary usages of cat subshell commands as suggested on comments of D16107759

Reviewed By: krallin

Differential Revision: D16162973

fbshipit-source-id: dd9edc60cb4c9bcc26d7ee39bee87b0708772d3c
2019-07-10 02:38:11 -07:00
Alex Hornby
7a5f96ed69 mononoke: lock repo from sync job itself, not from failure handler
Summary:
Sync job is used to replay pushes from Mononoke onto mercurial. It's code is in https://fburl.com/dt1hkf4g. For more about the sync job see https://fburl.com/wiki/sd2i6rch.

The goal of the sync job is to not only replicate pushes from Mononoke onto Mercurial, but also to verify that the push was correct i.e. push on Mononoke produced the same hash as on mercurial.

The problem is that we lock and unlock in two completely differen codepaths - locking is done by an external script called "failure handler" (https://fburl.com/7txm3jxf), while unlocking is a part of the sync job.

Ideally we'd like to lock and unlock in the sync job itself and remove "failure handler" entirely.

Reviewed By: StanislavGlebik

Differential Revision: D16107759

fbshipit-source-id: a418f8d0f48fa6db82476be72a91adbc03b66168
2019-07-10 02:38:10 -07:00
Thomas Orozco
db49ed1679 mononoke: expose option to disable infinitepush server write support
Summary:
This adds the ability to provide an infinitepush namespace configuration without actually allowing infinite pushes server side. This is useful while Mercurial is the write master for Infinite Push commits, for two reasons:

- It lets us enable the infinitepush namespace, which will allow the sync to proceed between Mercurial and Mononoke, and also prevents users from making regular pushes into the infinitepush namespace.
- It lets us prevent users from sending commit cloud backups to Mononoke (we had an instance of this reported in the Source Control @ FB group).

Note that since we are routing backfills through the shadow tier, I've left infinitepush enabled there.

Reviewed By: StanislavGlebik

Differential Revision: D16071684

fbshipit-source-id: 21e26f892214e40d94358074a9166a8541b43e88
2019-07-02 10:39:52 -07:00
George-Catalin Tintareanu
f71269ddbd enable/disable censoring
Summary:
Added an option to control for which repositories should censoring be
enabled or disabled. The option is added in `server.toml` as `censoring` and it
is set to true or false. If `censoring` is not specified, then the default
option is set to true ( censoring is enabled).

By disabling `censoring` the verification if the key is blacklisted or not is
omitted, therefor all the files are fetchable.

Reviewed By: ikostia

Differential Revision: D16029509

fbshipit-source-id: e9822c917fbcec3b3683d0e3619d0ef340a44926
2019-07-02 05:04:34 -07:00
Thomas Orozco
cbb5bf0c80 mononoke: bookmarks filler: add backfill filtering
Summary: This will make it easier to use the same bookmarks filling binary to process a backfill scratch bookmarks from Mercurial before we start actually processing new scratch bookmarks as they come in (see D16028731 for the plan I'm going to follow).

Reviewed By: farnz

Differential Revision: D16028817

fbshipit-source-id: 195e5bf746284e34c70ae2cbd2b9270fbc0c02c7
2019-06-28 02:44:44 -07:00
Thomas Orozco
196a363236 mononoke: rebuild test framework
Summary:
Our test framework as it stands right now is a light passthrough to the hg `run-tests.py` test framework, which attempts to place all the files it needs to run (including tests) into a `python_binary`, then runs the hg test runner from that directory.

It heavily relies on how Buck works to offer functionality:

- It expects that all the sources it registers for its master binary will all be in the same directory when it builds
- It expects that the sources will be symlinks to the real files so that `--interactive` can work.

This has a few problems:

- It doesn't work in `mode/opt`. The archive that gets built in `mode/opt` doesn't actually have all the sources we registered, so it's impossible to run tests.
- To add a new test, you must rebuild everything. We don't do that very often, but it'd be nice if we didn't have to.
- Iterating on the runner itself is painful, because as far as Buck is concerned, it depends on the entire world. This means that every change to the runner has to scan a lot more stuff than necessary. There's some functionality I'd like to get into the runner (like reporting test timings) that hasn't been easy to add as a result.

This diff attempts to solve these problems by separating concerns a little more:

- The runner is now just a simple `python_binary`, so it's easier to make changes to it.
- The runner now provides the logic of working from local files when needed (this means you can add a new test and it'll work immediately),
- All the binaries we need are dependencies of the integration test target, not the runner's. However, to make it possible to run the runner incrementally while iterating on something, there's a manifest target that points at all the various paths the runner needs to work. This will also help integrate the test runner with other build frameworks if necessary (e.g. for open-sourcing).
- We have separate targets for various assets we need to run the tests (e.g. the hg test framework).
- The runner now controls whether to use the network blackhole. This was necessary because the network blackhole breaks PAR archives (because tmp is no longer owned by the right owner, because we use a user namespace). We should be able to bring this back at some point if we want to by using a proper chroot for opt tests.

I included a README to explain this new design as well.

There are some things that could yet stand to be improved here (notably, I think we should put assets and tests in different directories for the sake of clarity), but so far I've been aiming at providing a 1-1 translation of the old system into the new one. I am planning to make further improvements in followup diffs.

Reviewed By: farnz

Differential Revision: D15921732

fbshipit-source-id: 09052591c419acf97f7e360b1e88ef1f412da6e5
2019-06-25 08:41:31 -07:00
Thomas Orozco
43fbc24427 mononoke: scratch bookmarks sync: wire up bookmarks replay
Summary:
This wires up the bookmarks replay logic by connecting the `process_replay_stream` function to an implementation that actually makes bookmark moves through the blobrepo.

This also slightly refactors `process_replay_stream`:

- Use specific errors instead of a generic `failure_ext` `Error`.
- Add logging + Scuba reporting (and report all moves we attempted to make for a given bookmark, as well as their outcome).
- Support passing in a closure instead of a static function, since we need a closure to get a `BlobRepo` reference into the bookmark replay function.

Reviewed By: StanislavGlebik

Differential Revision: D15902312

fbshipit-source-id: d6d57ba880066677748137d684e10796fb8633d5
2019-06-24 09:09:53 -07:00
Thomas Orozco
2ee46eb252 mononoke: don't require cachelib to talk to a remote DB
Summary:
Currently, we implicitly expect that caching is enabled if we're dealing with a remote repository, but that means cachelib must be enabled when running with a remote repository, and that is ... slow.

This can be problematic in two cases:

In tests. It makes MySQL tests unbearably slow, and a little more flaky because we end up using so much CPU. With this patch, MySQL tests remain slower than SQLite tests, but by a factor of < 2, which is a pretty substantial improvement.

Running trivial administrative commands (e.g. a `mononoke_admin`), notably using a dev build (which right now unbearably slow). With this patch, such a trivial command is about 6x faster:

```
[torozco@devvm4998.lla1 ~/fbcode] time buck-out/gen/scm/mononoke/admin#binary/admin --repo-id 2102 --mononoke-config-path /home/torozco/local/.mononoke_exec/config/PROD --skip-caching bookmarks list --kind publishing
Jun 21 08:57:36.658 INFO using repo "instagram-server" repoid RepositoryId(2102)
master	c96ac4654e4d2da45a9597af859adeac9dba3d7ca964cb42e5c96bc153f185e3	2c5713ad27262b91bf1dfaf21b3cf34fe3926c8d

real	0m5.299s
user	0m5.097s
sys	0m0.699s
[torozco@devvm4998.lla1 ~/fbcode] time buck-out/gen/scm/mononoke/admin#binary/admin --repo-id 2102 --mononoke-config-path /home/torozco/local/.mononoke_exec/config/PROD bookmarks list --kind publishing
I0621 08:57:59.299988 1181997 CacheAllocator-inl.h:3123] Started worker 'PoolRebalancer'
Jun 21 08:57:59.328 INFO using repo "instagram-server" repoid RepositoryId(2102)
master	c96ac4654e4d2da45a9597af859adeac9dba3d7ca964cb42e5c96bc153f185e3	2c5713ad27262b91bf1dfaf21b3cf34fe3926c8d

real	0m28.620s
user	0m27.680s
sys	0m2.466s
```

This is also nice because it means the MySQL tests won't talk to Memcache anymore.

 ---

Note: in this refactor, I made `Caching` an enum so it can't accidentally be swapped with some other boolean.

 ---

Finally, it also uses up quite a bit less RAM (we no longer need 2GB of RAM to output one line of bookmarks — although we're still using quite a bit!):

```
[torozco@devvm4998.lla1 ~/fbcode] env time buck-out/gen/scm/mononoke/admin#binary/admin --skip-caching --repo-id 2102 --mononoke-config-path /home/torozco/local/.mononoke_exec/config/PROD bookmarks list --kind publishing
Jun 21 09:18:36.074 INFO using repo "instagram-server" repoid RepositoryId(2102)
master	abdd2f78dafeaa8d4b96897955a63844b31324f9d89176b3a62088d0e2ae2b22	1702392d14bf7a332bf081518cb1ea3c83a13c39
5.08user 0.68system 0:05.28elapsed 109%CPU (0avgtext+0avgdata 728024maxresident)k
6776inputs+0outputs (8major+115477minor)pagefaults 0swaps

[torozco@devvm4998.lla1 ~/fbcode] env time buck-out/gen/scm/mononoke/admin#binary/admin --repo-id 2102 --mononoke-config-path /home/torozco/local/.mononoke_exec/config/PROD bookmarks list --kind publishing
I0621 09:19:01.385933 1244489 CacheAllocator-inl.h:3123] Started worker 'PoolRebalancer'
Jun 21 09:19:01.412 INFO using repo "instagram-server" repoid RepositoryId(2102)
master	abdd2f78dafeaa8d4b96897955a63844b31324f9d89176b3a62088d0e2ae2b22	1702392d14bf7a332bf081518cb1ea3c83a13c39
26.96user 2.27system 0:27.93elapsed 104%CPU (0avgtext+0avgdata 2317716maxresident)k
11416inputs+5384outputs (17major+605118minor)pagefaults 0swaps
```

Reviewed By: farnz

Differential Revision: D15941644

fbshipit-source-id: 0df4a74ccd0220a786ebf0e883e1a9b8aab0a647
2019-06-24 06:07:18 -07:00
Thomas Orozco
5e4c97bb17 mononoke: also run (some) integration tests using MySQL
Summary:
This adds suppport for running Mononoke integration tests using MySQL.

This is useful to ensure at least a nominal amount of testing happens on MySQL code. After all, MySQL is what we actually use in production. Notably, this feels like something we should do before deploying automatically to prod.

Currently, this requires whitelisting tests for MySQL support. I whitelisted two tests to act as a proof of concept.

Reviewed By: farnz

Differential Revision: D15855430

fbshipit-source-id: c554e2dbc6a5ae21df670a60edb713e2945de10a
2019-06-18 07:52:22 -07:00
Stanislau Hlebik
f52f6cda7f mononoke: fix filenode parents generation to match hg
Summary:
Mercurial has complicatied logic of finding filenode parents. And this diff
implements one of those tricky cases.

If a file existed and it was copied over (with `hg cp --force`) then Mononoke
set both p1 and copy information. Mercurial though just sets copy information.
This diff implements the same logic in Mononoke (note that for now it's only
for non-merge cases).

Reviewed By: krallin

Differential Revision: D15737250

fbshipit-source-id: 0b2117097801a7137ccc7382477452a0b7bf4504
2019-06-10 03:02:09 -07:00
Stanislau Hlebik
e6175b99c3 mononoke: fix pushrebase replay test
Summary:
It was failing and disabled, this diff fixes it.
Note that the test is quite heavy and we had problem with it failing on
sandcastle before. If this attempt of fixing it won't work, then it might be
worth removing the test completely because it doesn't give a lot of benefit to
us.

Reviewed By: farnz

Differential Revision: D15657387

fbshipit-source-id: 3e0ed588861cbadbaab2178826b13e1ba201fbc7
2019-06-05 06:19:36 -07:00
Kostia Balytskyi
2d9a45ca45 mononoke: reject pure pushes if the appropriate config is not set
Summary:
This is the final step in making sure we have control over whether
non-pushrebase pushes are supported by a given repo.

Reviewed By: krallin

Differential Revision: D15522276

fbshipit-source-id: 7e3228f7f0836f3dcd0b1a3b2500545342af1c5e
2019-05-31 10:50:10 -07:00
Thomas Orozco
16131b624f mononoke: limit match count in list_all_bookmarks_with_prefix
Summary:
This adds a sanity check that limits the count of matches in `list_all_bookmarks_with_prefix`.

If we find more matches than the limit, then an error will be returned (right now, we don't have support for e.g. offsets in this functionality, so the only alternative approach is for the caller to retry with a more specific pattern).

The underlying goal is to ensure that we don't trivially expose Mononoke to accidental denial of service when a list lists `*` and we end up querying literally all bookmarks.

I picked a fairly conservative limit here (500,000), which is > 5 times the number of bookmarks we currently have (we can load what we have right now successfully... but it's pretty slow);

Note that listing pull default bookmarks is not affected by this limit: this limit is only used when our query includes scratch bookmarks.

Reviewed By: StanislavGlebik

Differential Revision: D15413620

fbshipit-source-id: 1030204010d78a53372049ff282470cdc8187820
2019-05-30 07:14:33 -07:00
Thomas Orozco
7f0e3eb64b mononoke: create scratch bookmarks from B2xInfinitepush
Summary:
This updates our receive path for B2xInfinitepush to create new scratch bookmarks.

Those scratch bookmarks will:

- Be non-publishing.
- Be non-pull-default.
- Not be replicated to Mercurial (there is no entry in the update log).

I added a sanity check on infinite pushes to validate that bookmarks fall within a given namespace (which is represented as a Regexp in configuration). We'll want to determine whether this is a good mechanism and what the regexp for this should be prior to landing (I'm also considering adding a soft-block mode that would just ignore the push instead of blocking it).

This ensures that someone cannot accidentally perform an infinitepush onto master by tweaking their client-side configuration.

 ---

Note that, as of this diff, we do not support the B2xInfinitepushBookmarks part (i.e. backup bookmarks). We might do that separately later, but if we do, it won't be through scratch Bookmarks (we have too many backup bookmarks for this to work)

Reviewed By: StanislavGlebik

Differential Revision: D15364677

fbshipit-source-id: 23e67d4c3138716c791bb8050459698f8b721277
2019-05-30 07:14:32 -07:00
Durham Goode
3ac603d29d tests: add Mononoke integration test target
Summary:
This adds a new test target that will use Mononoke as the server.
Initially we only support one test, but we can add to this list in the future.

Reviewed By: quark-zju

Differential Revision: D15382714

fbshipit-source-id: ad9b3bd35ffefc01239ff05f9f65829fb7a94555
2019-05-29 01:30:24 -07:00
Durham Goode
3dd18cce48 Allow customizing the cert directory
Summary:
Previously the cert directory was assumed to be inside $TESTDIR. This
doesn't work with the Mercurial tests, since they run inside the actual test
directory instead of in a buck-out/gen created directory.

To allow the Mercurial tests to use these same certs, let's allow customizing
the location via HGTEST_CERTDIR. In a future diff the Mercurial tests will set
this when running with Mononoke integration.

Reviewed By: StanislavGlebik

Differential Revision: D15381789

fbshipit-source-id: e1be3d1cc17f5622d1cc277a377d628d5eb45160
2019-05-28 03:16:33 -07:00
Durham Goode
0495f1e94e Refactor setup repo logic to be reusable
Summary:
Previously it always set up a repo named 'repo'. For Mercurial
integration tests we'll want the ability to name the repository other things, so
let's refactor the configuration logic into a more generic function.

Reviewed By: quark-zju

Differential Revision: D15354044

fbshipit-source-id: e222e1389ca700af712af79062aae4977e125dc4
2019-05-28 03:16:33 -07:00
Kostia Balytskyi
364fff371c mononoke: make sync job able to skip entries which were successfully applied
Summary:
In some cases, the sync binary gets killed after it had synced the bundle,
but before it had updated the counter in the database. This causes any
sunsequent sync to fail with the "conflicting changes" error. One of the ways
to mitigate this problem is to check on failure whether the bookmark on the server
is already pointing to the expected location, and pretend that the sync
succeeded, if it is.

Reviewed By: StanislavGlebik

Differential Revision: D15460956

fbshipit-source-id: 55d2df3545f8a34ebc61eb6da493b8d244629089
2019-05-24 16:23:21 -07:00
Durham Goode
69dbba936d Add flag for disabling session output
Summary:
We're trying to make it possible to run the Mercurial tests with the
Mononoke server. The session id output makes it difficult to have the same
output of the Mercurial server, so let's add a flag that allows us to disable
it.

Reviewed By: krallin

Differential Revision: D15462727

fbshipit-source-id: 63e4a7f31ddc41cd5ae767712f7ee5fbcdffda56
2019-05-23 13:22:34 -07:00
Thomas Orozco
98466df0e9 mononoke: ignore urllib warning in tests
Summary:
The verify_integrity hook is throwing a warning, which broke our verify_integrity hook tests:

https://our.intern.facebook.com/intern/test/281474986840689

Right now, we're seeing this error:

```
+  remote:     rust:verify_integrity for ee90164f902b3261fb0a8d830747fc17e0241902: hook failed /mnt/btrfs/trunk-hg-fbcode-fbsource-53494257-1558362645/fbcode/buck-out/dev/gen/security/source_control/verify_integrity#link-tree/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported version!
+  remote:       RequestsDependencyWarning)
```

Reviewed By: StanislavGlebik

Differential Revision: D15414119

fbshipit-source-id: 7004deb8db35f148abb25f77a465a73e63c1659c
2019-05-21 12:26:03 -07:00
Durham Goode
aa8712a082 treemanifest: add flatcompat mode for test migration
Summary:
We want to migrate the tests to run using treemanifest. As part of
that, we want to first transition to using treemanifest without actually
changing the hash, so we can check that the tests still work first, then update
the hashes second.

This diff adds the flatcompat mode and enables it by default. A future diff will
start enabling treemanifest for existing tests.

Reviewed By: quark-zju

Differential Revision: D15030252

fbshipit-source-id: 06c82be749282d62f1d9cfb43246695c427f8165
2019-05-21 12:25:50 -07:00
Stanislau Hlebik
058d2a4eed mononoke: integration test for multiplexed blobstore
Summary: Can be used to verify multiplexer

Reviewed By: aslpavel

Differential Revision: D15264133

fbshipit-source-id: cb9044b6b51e099b61e751925367c71fd506332e
2019-05-21 12:25:48 -07:00
Jeremy Fitzhardinge
9b54d0f85b mononoke: move storage configuration to a common file
Summary:
This change has two goals:
- Put storage configuration that's common to multiple repos in a common place,
  rather than replicating it in each server.toml
- Allow tools that only operate on the blobstore level - like blobstore healing
  - to be configured directly in terms of the blobstore, rather than indirectly
    by using a representative repo config.

This change makes several changes to repo configuration:
1. There's a separate common/storage.toml which defines named storage
configurations (ie, a combination of a blobstore and metadata DB)
2. server.toml files can also define local storage configurations (mostly
useful for testing)
3. server.toml files now reference which storage they're using with
`storage_config = "name"`.
4. Configuration of multiplex blobstores is now explicit. Previously if a
server.toml defined multiple blobstores, it was assumed that it was a
multiplex. Now storage configuration only accepts a single blobstore config,
but that config can be explicitly a multiplexed blobstore, which has the
sub-blobstores defined within it, in the `components` field. (This is
recursive, so it could be nested, but I'm not sure if this has much value in
practice.)
5. Makes configuration parsing more strict - unknown fields will be treated as
an error rather than ignored. This helps flag problems in refactoring/updating
configs.

I've updated all the configs to the new format, both production and in
integration tests. Please review to make sure I haven't broken anything.

Reviewed By: StanislavGlebik

Differential Revision: D15065423

fbshipit-source-id: b7ce58e46e91877f4e15518c014496fb826fe03c
2019-05-21 12:25:44 -07:00
Jeremy Fitzhardinge
e56d695cec mononoke: use SqlConstructors label for local db names
Summary:
Seems redundant to also require callers to open_ssl to also pass a
(mostly) identical string.

Also make open_ssl special-case filenodes with sharding (though filenodes
aren't currently opened through it).

Reviewed By: StanislavGlebik

Differential Revision: D15157834

fbshipit-source-id: 0df45307f17bdb2c021673b3153606031008bee2
2019-05-21 12:25:44 -07:00
Kostia Balytskyi
2b647034f3 mononoke: make sync job print hg server names into logs
Summary:
This will help us understand which servers behave poorly and whether there's
any clustering.

Reviewed By: StanislavGlebik

Differential Revision: D15229101

fbshipit-source-id: 1aae9196702ed6fb791b5265c3bdfe90e7e24ae4
2019-05-21 12:25:36 -07:00
Stanislau Hlebik
aac6a6c835 mononoke: add security config
Summary:
The config will be used to whitelist connections with certain identities and
blacklist everything else.

Differential Revision: D15150921

fbshipit-source-id: e4090072ea6ba9714575fb8104d9f45e92c6fefb
2019-05-21 12:25:31 -07:00
Stanislau Hlebik
b42c5a2b66 mononoke: set bookmark for hooks and use it in verify_integrity
Summary:
At the moment verify integrity script blocks master commits that do not pass
the check. Non-master commits are not blocked even if they don't pass the check
(for example, if they don't have a valid reviewer), but verify_integrity hook
still logs them to scuba.

Mononoke's verify_integrity was enabled only on master, meaning that the hook
won't run on non-master commits at all, and we won't log non-master commits to scuba.
This diff fixes it.

Reviewed By: farnz

Differential Revision: D15146725

fbshipit-source-id: ab432fb2eccae0fbcc10755f5c8447964c490285
2019-05-21 12:25:29 -07:00
Thomas Orozco
72fbf818e7 mononoke: hg sync job: Prefetch Bundle Data
Summary:
This introduces prefetching in the hg sync job, which allows us to separate two logically separate steps, and buffer between them:

- Fetching bundle data from Manifold so it can be replayed in Mercurial (we call this "preparing"). This can happen out of order.
- Submitting the changes to Mercurial. This must happen sequentially.

The upshot is that we can fetch new data from Manifold while we wait for Mercurial to apply it, which should increase our throughput.

 ---

This required a bit of refactoring for the main loop here. The reason for that is that, currently, we heavily rely on closures to capture the `log_entry` so we can pass it to Scuba, etc.

That doesn't work if we want to have buffering, because there isn't a single scope that represents

Instead, every step of the stream has its own scope, so we need to pass the `log_entry` through the stream itself. We also need to track the log entry in errors as well (e.g. to report them to Scuba).

Reviewed By: ikostia

Differential Revision: D15146682

fbshipit-source-id: f83284571f6ca90cb621c621c5165f7fbafd81b5
2019-05-21 12:25:28 -07:00
Kostia Balytskyi
5756e29d5b mononoke: stop using fastmanifest on servers in tests
Summary: Enabling it was a mistake as we can't enable it on our real hg servers.

Reviewed By: farnz

Differential Revision: D15122167

fbshipit-source-id: 5e28968354b7487bd9c965d8b3f23245f6bd265a
2019-05-21 12:25:26 -07:00
Thomas Orozco
1a05aaa1b2 Support hg-sync-bundle verify
Summary: This adds a command in mononoke admin to verify the consistency of remaining bundles to sync -- i.e. whether all bundles are blobimports or all of them are not blobimports.

Reviewed By: ikostia

Differential Revision: D15097935

fbshipit-source-id: a0df221c38e84897213edf232972ba420977e9d3
2019-05-21 12:25:24 -07:00
Thomas Orozco
37c34af244 Introduce a test tool to write stub log entries
Summary:
Stanislau and Kostia mentioned that adding a few more commands for managing the HG sync job in the Mononoke Admin tool would be useful — in particular those commands should interact with the list of log entries and the last sync.

Notably, they were:

- Add support for skipping ahead any blobimport log entries.
- Add a verify command to check that all entries looking forward are consistent — i.e. they're either all blobimport or all non-blobimport.

However, as it stands, it's hard to test those in integration tests, because one can't easily generate stub data to exercise on the admin tool.

Notably, inserting data through the `sqlite3` database command line results in inconsistent values for the `reason` column that don't compare equal to those we query from the Rust side of things.

We could work around this by updating the Rust code to e.g. use LIKE instead of equality checks, but then we're updating the production code exclusively for the purpose of serving tests.

So, instead, this patch proposes introduces a lightweight command line  tool to use in tests that basically just inserts data into the bookmarks database, but from the Rust side of things.

Reviewed By: StanislavGlebik

Differential Revision: D15081758

fbshipit-source-id: 574739d2e19e6ce59d729f1aee73c368612fb92b
2019-05-21 12:25:23 -07:00
Pavel Aslanov
3b6081a6f5 make it possible to enable bookmark cache from configs
Summary: Now it is possible to configure and enable/disable bookmark cache from configs

Reviewed By: StanislavGlebik

Differential Revision: D14952840

fbshipit-source-id: 3080f7ca4639da00d413a949547705ad480772f7
2019-05-21 12:25:22 -07:00
Johan Schuijt-Li
5e4a97cecc tests: provision fb identity in to testing certificates
Summary:
In order to do more complete tests around authentication we need to provide
fb identity in to our test certificates.

Reviewed By: StanislavGlebik

Differential Revision: D15046017

fbshipit-source-id: 3f3cd450425944a2970c6f02e7eb92a878076a05
2019-05-21 12:25:19 -07:00
Thomas Orozco
178931cc02 mononoke: add obsmarkers to pushrebase output
Summary: Added obsmarkers to pushrebase output. This allows the client to hide commits that were rebased server-side, and check out the rebased commit.

Reviewed By: StanislavGlebik

Differential Revision: D14932842

fbshipit-source-id: f215791e86e32e6420e8bd6cd2bc25c251a7dba0
2019-05-21 12:25:17 -07:00
Kostia Balytskyi
629bc92983 mononoke: enable fastmanifest for mononoke tests
Reviewed By: StanislavGlebik

Differential Revision: D15034167

fbshipit-source-id: 5c276c4eed4a61381b51a13ad126e749c8892d1c
2019-05-21 12:25:16 -07:00
Stanislau Hlebik
4038fd1266 mononoke: retry sync in case of a failure
Summary:
Sometimes we have one-off failures that retry can fix. Let's retry 3 times
before giving up.

Note that it'll "retry" 3 times even in the case of UnexpectedBookmarkMove,
and retrying in that case is useless. However it's not a big problem

Reviewed By: ikostia

Differential Revision: D14796293

fbshipit-source-id: a145213cf74e9d436212690042f187cee56e26cd
2019-05-21 12:25:00 -07:00
Pavel Aslanov
905ff80ee6 forbid rebases when root is not a p1 of the rebase set
Summary: We want to forbid pushrebases when root node is a p2 of its parents for now, since mercurial swaps them after pushrebase which causes inconsistency

Reviewed By: StanislavGlebik

Differential Revision: D14642177

fbshipit-source-id: f8f6e9565c53958e8cff5df6f4d006ddfe5a69c0
2019-05-21 12:24:59 -07:00
David Budischek
2098c4a304 Prevent merge commits
Summary: In Mononoke we want to be able to block merge commits on a repo per repo basis.

Reviewed By: aslpavel

Differential Revision: D14455502

fbshipit-source-id: 400e85834c20df811674405bc0c391860cf677dd
2019-03-18 11:20:10 -07:00
David Budischek
2a93fe345c Block non fastforward bookmark moves
Summary:
This is a hook in mercurial, in Mononoke it will be part of the implementation. By default all non fastforward pushes are blocked, except when using the NON_FAST_FORWARD pushvar (--non-forward-move is also needed to circumvent client side restrictions). Additionally certain bookmarks (e.g. master) shouldn't be able to be moved in a non fastforward manner at all. This can be done by setting block_non_fast_forward field in config.

Pushrebase can only move the bookmark that is actually being pushrebased so we do not need to check whether it is a fastforward move (it always is)

Reviewed By: StanislavGlebik

Differential Revision: D14405696

fbshipit-source-id: 782b49c26a753918418e02c06dcfab76e3394dc1
2019-03-18 04:12:09 -07:00