Summary:
S3 blobstore is sensitive to the number of open connections and it causes perf
problems. Let's add a way to limit the number of simultaneous open connections.
Also see D26911520
Reviewed By: krallin
Differential Revision: D26911679
fbshipit-source-id: 1e3e4922b26d0a29c94c89b4aa76b9b5928effe6
Summary:
Normally we prohibit landing commits that might accidentally
change the x-repo mapping. However we do want to allow landing commits like
that to backup repos, because backup repos should have all commits as their
counterpart repositories.
This change also has another side-effect - we don't call `load_additional_changeset()`
which can be very expensive for backup repos because of the issues in configuration -
in particular, we don't have `hooks_ancestors_of` option set, and that caused all ancestors to be considered
as "additional changesets". It would make since more properly later.
Differential Revision: D26883910
fbshipit-source-id: 07ceb7b96bc6cae851ac6ff57071eff5cef387e4
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary: Done some reordering of fields in Cargo.toml, added test and doctest = false, name of the target that generated the Cargo.toml file and sorted the cratemap.
Reviewed By: ahornby
Differential Revision: D26581275
fbshipit-source-id: 4c363369438c72d43d8ccf4799f103ff092457cc
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary:
We have some tests that are a bit racy because they write bookmarks from
one process then look at them from another process, but that can fail because
we have a cache on bookmarks that holds them for 2 seconds.
This is normally not a huge issue because we don't access said bookmarks, but
now that as of my earlier diff we run a warm bookmarks cache, it *is* a
problem. This fixes that. We can expand it later to do things like reload
tunables, but for now this satisfies one basic use case.
Reviewed By: ahornby
Differential Revision: D26149371
fbshipit-source-id: 11c7f64b1ae45f6a0de142be25ab367cb25df567
Summary: See D26019054 for what this does.
Reviewed By: StanislavGlebik
Differential Revision: D26019125
fbshipit-source-id: 071f0115d5ee8435bbc7ff5208ace8ef6e948272
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
D25850262 (772680f2f7) updated a dep but only did a partial regen. This broke
other diffs.
This diff does a full regen to sync up again.
Differential Revision: D25872444
fbshipit-source-id: b5749b3d634ffe355a5a3cf6e94066ca5158001f
Summary:
This package will be used by chronos job on regular basis. It will spawn legocastle jobs for each of backup repos, pull them in together with origin repo from production tier and compare hashes of commits.
`fbpkg.builder` allows to create package and add it to contbuild together with creating a binary.
`https://www.internalfb.com/intern/wiki/Fbpkg/fbpkg.builder/#overview`
Reviewed By: HarveyHunt
Differential Revision: D25573914
fbshipit-source-id: 3f7ff3a6b2d7acefed46683122b72c5bc862e1aa
Summary:
Remove logic of striping `_backup` suffix and get source repo from configerator.
Has dependency on D25571595. And should be landed after it.
Reviewed By: krallin
Differential Revision: D25420829
fbshipit-source-id: 50e2fe2bede849ba622dcbde2a360b08e640901a
Summary: I'm changing this field to the Struct type in later diffs, so need to delete it first. It wasn't used anywhere yet, next diff in the stack was supposed to use it, so it's safe to delete it
Reviewed By: HarveyHunt
Differential Revision: D25584263
fbshipit-source-id: 1d0aab01b52dc575cd6d466857ca850cbf8cd973
Summary:
Adds `--reversefill` mode to bookmarks filler that fetches bookmark updates
from the queue and syncs them to infinitepush database.
Reviewed By: farnz
Differential Revision: D24538317
fbshipit-source-id: 5ac7ef601f2ff120c4efd8df08a416e00df0ceb9
Summary:
Summarry:
This is the first part of syncing new scratch bookmark pushes from Mononoke to
Mercurial: on each bookmark movement we log this bookmark movement to filler's
queue.
Reviewed By: liubov-dmitrieva
Differential Revision: D24480546
fbshipit-source-id: 27103b4b4f8c4600aaf485826db2936eaffcc4a9
Summary: Members of `scm` hipster group will be able to push to mononoke bypassing hooks when `BYPASS_ALL_HOOKS` pushvar is passed.
Reviewed By: krallin
Differential Revision: D24477468
fbshipit-source-id: ac910bf27e5510e1975c4a7cd0bfeff5216da70e
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.
There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.
Reviewed By: jsgf
Differential Revision: D24445298
fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
Summary:
This change enables the filler job to work on all repos available instead of a
single one. We are still going to be able to dedicate the job to a certain repo
(by crafting a config with a single repo enabled) but we can put the entire
long tail for low-traffic repos under a single job.
This requires D24110335 to land in configerator to work.
Reviewed By: krallin
Differential Revision: D24136239
fbshipit-source-id: 4b77d1667c37cc55f11c3087b02a09dbae29db0f
Summary:
Implemented S3 blobstore
Isilon implements S3 as 1:1 mapping into filesystem, and it limits the maximum number of blobs in the single directory. To overcome it lets shard the keys using base64 encoding and making 2 level dir structure with 2 chars dir names.
Reviewed By: krallin
Differential Revision: D23562541
fbshipit-source-id: c87aca2410381a07babb191cbd8cf28233556e03
Summary:
In the next diff I'm going to add log-only mode to redaction, and it would be
good to have a way of testing it (i.e. testing that it actually logs accesses
to bad keys).
In this diff let's use a config option that allows logging censored scuba
accesses to file, and let's update redaction integration test to use it
Reviewed By: ikostia
Differential Revision: D23537797
fbshipit-source-id: 69af2f05b86bdc0ff6145979f211ddd4f43142d2
Summary: Use the new fields from RawHookConfig in HookConfig
Reviewed By: StanislavGlebik
Differential Revision: D23499766
fbshipit-source-id: 43e9d2dfdcfb0fa0dd4de6310ea0013db1b69474
Summary:
Related commit: D23213192
Currently repo configs don't store their phabricator callsigns and because of this,
it's hard to generate them with a given repo name (e.g. fbsource) in Mononoke.
This diff synced the changes between configerator (check the other commit) and fbsource.
Reviewed By: farnz
Differential Revision: D23214677
fbshipit-source-id: cf102d69958c7dc32f8b72cecbfad017d0a31329
Summary: Let's use new flag to enable/disable short history for getpack request
Reviewed By: krallin
Differential Revision: D23080200
fbshipit-source-id: 7aa0be6ded0601fa4d31d4b9ff8792a4f8d91b19
Summary:
We're going to add an SQL blobstore to our existing multiplex, which won't have all the blobs initially.
In order to populate it safely, we want to have normal operations filling it with the latest data, and then backfill from Manifold; once we're confident all the data is in here, we can switch to normal mode, and never have an excessive number of reads of blobs that we know aren't in the new blobstore.
Reviewed By: krallin
Differential Revision: D22820501
fbshipit-source-id: 5f1c78ad94136b97ae3ac273a83792ab9ac591a9
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.
Reviewed By: dtolnay
Differential Revision: D22403809
fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
Summary: Also fix up the parser test that now fails with this change
Reviewed By: StanislavGlebik
Differential Revision: D22306340
fbshipit-source-id: 820aad48068471b03cbc1c42107c443bfa680607