Commit Graph

41 Commits

Author SHA1 Message Date
Stanislau Hlebik
a80dece152 mononoke: add a way to limit s3 blobstore
Summary:
S3 blobstore is sensitive to the number of open connections and it causes perf
problems. Let's add a way to limit the number of simultaneous open connections.

Also see D26911520

Reviewed By: krallin

Differential Revision: D26911679

fbshipit-source-id: 1e3e4922b26d0a29c94c89b4aa76b9b5928effe6
2021-03-11 00:55:14 -08:00
Ran Yaniv
e5d163411a run autocargo
Summary: Run autocargo.

Reviewed By: c-ryan747

Differential Revision: D26878408

fbshipit-source-id: 14e2efb280c906fa7a79253da74d7518e1a89079
2021-03-08 12:40:50 -08:00
Stanislau Hlebik
cc0c8cd0ff mononoke: allow changing xrepo mapping extra for if config is set
Summary:
Normally we prohibit landing commits that might accidentally
change the x-repo mapping. However we do want to allow landing commits like
that to backup repos, because backup repos should have all commits as their
counterpart repositories.

This change also has another side-effect - we don't call `load_additional_changeset()`
which can be very expensive for backup repos because of the issues in configuration -
in particular, we don't have `hooks_ancestors_of` option set, and that caused all ancestors to be considered
as "additional changesets". It would make since more properly later.

Differential Revision: D26883910

fbshipit-source-id: 07ceb7b96bc6cae851ac6ff57071eff5cef387e4
2021-03-08 08:39:30 -08:00
Thomas Orozco
2a803fc10d third-party/rust: update futures
Summary:
Those newer versions of Futures have compatibility improvements with Tokio,
notably:

- https://github.com/rust-lang/futures-rs/pull/2333
- https://github.com/rust-lang/futures-rs/pull/2358

Reviewed By: farnz

Differential Revision: D26778794

fbshipit-source-id: 5a9dc002083e5edfa5c614d8d2242e586a93fcf6
2021-03-04 06:42:55 -08:00
Lukas Piatkowski
f317302b0f autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.

Reviewed By: ikostia

Differential Revision: D26618363

fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-25 15:10:56 -08:00
Stefan Filip
d45198d79d config: sync repo.thrift changes to fbcode
Summary:
```
configerator-thrift-updater scm/mononoke/repos/repos.thrift
```

(Note: this ignores all push blocking failures!)

Reviewed By: quark-zju

Differential Revision: D26628122

fbshipit-source-id: 69124458996393f71098f707b375e184b929e466
2021-02-24 16:51:50 -08:00
Lukas Piatkowski
279eb2b538 autocargo v1: changes to how thrift-related generation is done to match v2
Summary: Done some reordering of fields in Cargo.toml, added test and doctest = false, name of the target that generated the Cargo.toml file and sorted the cratemap.

Reviewed By: ahornby

Differential Revision: D26581275

fbshipit-source-id: 4c363369438c72d43d8ccf4799f103ff092457cc
2021-02-23 11:38:45 -08:00
Lukas Piatkowski
cd0b6d50e2 autocargo v1: changes to match autocargo v2 generation results.
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.

Reviewed By: zertosh

Differential Revision: D26544150

fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 11:03:55 -08:00
Alex Hornby
246937b6f3 mononoke: sync configerator repos thrift
Summary: Sync across the thrift schema

Differential Revision: D26276002

fbshipit-source-id: 24f4a8a459f2fcdb18a14627dc7bb5e8d835b829
2021-02-05 08:22:46 -08:00
Thomas Orozco
90e4bbca0a mononoke/server: add a trivial control API and use it to fix some tests
Summary:
We have some tests that are a bit racy because they write bookmarks from
one process then look at them from another process, but that can fail because
we have a cache on bookmarks that holds them for 2 seconds.

This is normally not a huge issue because we don't access said bookmarks, but
now that as of my earlier diff we run a warm bookmarks cache, it *is* a
problem. This fixes that. We can expand it later to do things like reload
tunables, but for now this satisfies one basic use case.

Reviewed By: ahornby

Differential Revision: D26149371

fbshipit-source-id: 11c7f64b1ae45f6a0de142be25ab367cb25df567
2021-02-01 07:53:17 -08:00
Thomas Orozco
d863816e0e mononoke: sync Configerator repos schema
Summary: See D26019054 for what this does.

Reviewed By: StanislavGlebik

Differential Revision: D26019125

fbshipit-source-id: 071f0115d5ee8435bbc7ff5208ace8ef6e948272
2021-01-27 08:32:38 -08:00
Daniel Xu
5715e58fce Add version specificiation to internal dependencies
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.

Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.

See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .

Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.

Reviewed By: lukaspiatkowski

Differential Revision: D26028982

fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
2021-01-25 22:10:24 -08:00
Mark Juggurnauth-Thomas
0afac29010 sync configerator thrift
Reviewed By: StanislavGlebik

Differential Revision: D25952897

fbshipit-source-id: 0fbd43fa36c2cb749a6eed103b88840681a9f706
2021-01-19 08:11:11 -08:00
Daniel Xu
f83fb64f3e Do full regen
Summary:
D25850262 (772680f2f7) updated a dep but only did a partial regen. This broke
other diffs.

This diff does a full regen to sync up again.

Differential Revision: D25872444

fbshipit-source-id: b5749b3d634ffe355a5a3cf6e94066ca5158001f
2021-01-11 16:14:36 -08:00
Egor Tkachenko
ccc9e4e39a Verification of backup repos
Summary:
This package will be used by chronos job on regular basis. It will spawn legocastle jobs for each of backup repos, pull them in together with origin repo from production tier and compare hashes of commits.

`fbpkg.builder` allows to create package and add it to contbuild together with creating a binary.
`https://www.internalfb.com/intern/wiki/Fbpkg/fbpkg.builder/#overview`

Reviewed By: HarveyHunt

Differential Revision: D25573914

fbshipit-source-id: 3f7ff3a6b2d7acefed46683122b72c5bc862e1aa
2020-12-17 14:36:11 -08:00
Egor Tkachenko
7ba932a332 Add field backup_source_name to repo_config
Summary:
Remove logic of striping `_backup` suffix and get source repo from configerator.
Has dependency on D25571595. And should be landed after it.

Reviewed By: krallin

Differential Revision: D25420829

fbshipit-source-id: 50e2fe2bede849ba622dcbde2a360b08e640901a
2020-12-17 04:30:03 -08:00
Egor Tkachenko
0109baf930 Remove optional backup source name string field
Summary: I'm changing this field to the Struct type in later diffs, so need to delete it first. It wasn't used anywhere yet, next diff in the stack was supposed to use it, so it's safe to delete it

Reviewed By: HarveyHunt

Differential Revision: D25584263

fbshipit-source-id: 1d0aab01b52dc575cd6d466857ca850cbf8cd973
2020-12-16 05:30:37 -08:00
Mark Juggurnauth-Thomas
01251de9bb sync configerator thrift
Reviewed By: StanislavGlebik

Differential Revision: D25246318

fbshipit-source-id: fd97296e3540b43c72af00ab045c81c496e06728
2020-12-14 09:24:57 -08:00
Mateusz Kwapich
10bc4843d3 new mode to reverse-fill bookmarks
Summary:
Adds `--reversefill` mode to bookmarks filler that fetches bookmark updates
from the queue and syncs them to infinitepush database.

Reviewed By: farnz

Differential Revision: D24538317

fbshipit-source-id: 5ac7ef601f2ff120c4efd8df08a416e00df0ceb9
2020-11-10 02:58:00 -08:00
Mateusz Kwapich
89fd335b1e populate bookmark filler queue on scratch bookmark updates
Summary:
Summarry:
This is the first part of syncing new scratch bookmark pushes from Mononoke to
Mercurial: on each bookmark movement we log this bookmark movement to filler's
queue.

Reviewed By: liubov-dmitrieva

Differential Revision: D24480546

fbshipit-source-id: 27103b4b4f8c4600aaf485826db2936eaffcc4a9
2020-11-10 02:58:00 -08:00
Jan Mazur
8152776a35 bypassing hooks for members of scm
Summary: Members of `scm` hipster group will be able to push to mononoke bypassing hooks when `BYPASS_ALL_HOOKS` pushvar is passed.

Reviewed By: krallin

Differential Revision: D24477468

fbshipit-source-id: ac910bf27e5510e1975c4a7cd0bfeff5216da70e
2020-10-29 10:14:25 -07:00
Thomas Orozco
24560ded2e thrift: call hooks from Rust codegen
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.

There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.

Reviewed By: jsgf

Differential Revision: D24445298

fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
2020-10-22 17:01:47 -07:00
Mateusz Kwapich
42a783999d enable single process to work on multiple repos.
Summary:
This change enables the filler job to work on all repos available instead of a
single one. We are still going to be able to dedicate the job to a certain repo
(by crafting a config with a single repo enabled) but we can put the entire
long tail for low-traffic repos under a single job.

This requires D24110335 to land in configerator to work.

Reviewed By: krallin

Differential Revision: D24136239

fbshipit-source-id: 4b77d1667c37cc55f11c3087b02a09dbae29db0f
2020-10-08 05:38:01 -07:00
Egor Tkachenko
4d0ae8ae41 Added S3 blobstore
Summary:
Implemented S3 blobstore
Isilon implements S3 as 1:1 mapping into filesystem, and it limits the maximum number of blobs in the single directory. To overcome it lets shard the keys using base64 encoding and making 2 level dir structure with 2 chars dir names.

Reviewed By: krallin

Differential Revision: D23562541

fbshipit-source-id: c87aca2410381a07babb191cbd8cf28233556e03
2020-09-22 04:15:34 -07:00
Stanislau Hlebik
0740f99f13 mononoke: allow logging censored scuba accesses to file
Summary:
In the next diff I'm going to add log-only mode to redaction, and it would be
good to have a way of testing it (i.e. testing that it actually logs accesses
to bad keys).

In this diff let's use a config option that allows logging censored scuba
accesses to file, and let's update redaction integration test to use it

Reviewed By: ikostia

Differential Revision: D23537797

fbshipit-source-id: 69af2f05b86bdc0ff6145979f211ddd4f43142d2
2020-09-04 07:37:14 -07:00
Lukas Piatkowski
eebdc0b896 mononoke/metaconfig: sync thrift changes from configerator for HookConfig
Summary: Use the new fields from RawHookConfig in HookConfig

Reviewed By: StanislavGlebik

Differential Revision: D23499766

fbshipit-source-id: 43e9d2dfdcfb0fa0dd4de6310ea0013db1b69474
2020-09-04 02:02:06 -07:00
David Tolnay
75c2118e01 Remove crate_root from Rust dependency info
Reviewed By: danobi

Differential Revision: D23430948

fbshipit-source-id: c4b374021325fc247121ceecd0e82a0291aa75d6
2020-08-31 14:43:24 -07:00
Stefan Filip
a86409361c thrift: sync repos.thrift to fbcode
Summary:
`fbcode` sync after adding the `update_algorithm` option to
`RawSegmentedChangelogConfig`.

Reviewed By: quark-zju

Differential Revision: D23376694

fbshipit-source-id: df8a378303ec00d5d7d7ac3fff7a4a8cffc99328
2020-08-28 16:37:57 -07:00
Mark Thomas
f15976637e bookmarks_movement: restrict bookmarks that are marked allow_only_external_sync
Reviewed By: StanislavGlebik

Differential Revision: D23294907

fbshipit-source-id: ed89e5fd841e7d516b5d259c1f5de4e9f8f40ee3
2020-08-25 09:14:09 -07:00
Mark Thomas
02e382b203 sync repos.thrift
Reviewed By: StanislavGlebik

Differential Revision: D23222561

fbshipit-source-id: 09c25e11b5e022547c1a1363a4cbe1926714be86
2020-08-25 09:14:08 -07:00
Viet Hung Nguyen
dcb565409d mononoke: Sync repo.thrift changes to fbcode
Summary:
Related commit: D23213192

Currently repo configs don't store their phabricator callsigns and because of this,
it's hard to generate them with a given repo name (e.g. fbsource) in Mononoke.
This diff synced the changes between configerator (check the other commit) and fbsource.

Reviewed By: farnz

Differential Revision: D23214677

fbshipit-source-id: cf102d69958c7dc32f8b72cecbfad017d0a31329
2020-08-19 10:48:41 -07:00
Stanislau Hlebik
f68531f81c mononoke: add a flag to disable short history fetching
Summary: Let's use new flag to enable/disable short history for getpack request

Reviewed By: krallin

Differential Revision: D23080200

fbshipit-source-id: 7aa0be6ded0601fa4d31d4b9ff8792a4f8d91b19
2020-08-13 10:37:40 -07:00
Stanislau Hlebik
97041ca4dc sync configerator thrift update
Reviewed By: farnz

Differential Revision: D23057328

fbshipit-source-id: b57fced1a91e47006ca5d44c5bbcebd1a516e7d0
2020-08-12 12:03:18 -07:00
Simon Farnsworth
a9b8793d2d Add a write-mostly blobstore mode for populating blobstores
Summary:
We're going to add an SQL blobstore to our existing multiplex, which won't have all the blobs initially.

In order to populate it safely, we want to have normal operations filling it with the latest data, and then backfill from Manifold; once we're confident all the data is in here, we can switch to normal mode, and never have an excessive number of reads of blobs that we know aren't in the new blobstore.

Reviewed By: krallin

Differential Revision: D22820501

fbshipit-source-id: 5f1c78ad94136b97ae3ac273a83792ab9ac591a9
2020-08-03 04:36:19 -07:00
Stanislau Hlebik
6557e2e364 sync D22817643 to fbsource
Reviewed By: farnz

Differential Revision: D22817699

fbshipit-source-id: 015eb428fad6ae7726e5072d302e66cdb3551b5e
2020-07-30 08:41:15 -07:00
Harvey Hunt
0e7738768f mononoke: Sync repo.thrift changes to fbcode
Summary: Sync the changes from configerator.

Reviewed By: krallin

Differential Revision: D22762452

fbshipit-source-id: 56f3e3bf70c58be38947b7d0fd4737c5ef895d3c
2020-07-27 06:21:20 -07:00
Arun Kulshreshtha
5f0181f48c Regenerate all Cargo.tomls after upgrade to futures 0.3.5
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.

Reviewed By: dtolnay

Differential Revision: D22403809

fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
2020-07-06 20:49:43 -07:00
Stanislau Hlebik
783e1fcc3b mononoke: sync configerator update
Reviewed By: krallin

Differential Revision: D22373947

fbshipit-source-id: b4aa9a4767ff135cd07ffccbbbf8dcaa5de65703
2020-07-03 09:58:46 -07:00
Simon Farnsworth
aa00466319 Sync hook_type change from D22282802
Summary: Also fix up the parser test that now fails with this change

Reviewed By: StanislavGlebik

Differential Revision: D22306340

fbshipit-source-id: 820aad48068471b03cbc1c42107c443bfa680607
2020-06-30 11:20:54 -07:00
Stanislau Hlebik
c3276b4c5e mononoke: sync configerator update
Reviewed By: krallin

Differential Revision: D22211347

fbshipit-source-id: 2bb2d02277d06462e4cda9347bfd8a2ae3fe7222
2020-06-25 05:51:29 -07:00
Lukas Piatkowski
14f7dd70e4 Re-sync with internal repository 2020-06-24 21:35:50 +02:00