Commit Graph

15 Commits

Author SHA1 Message Date
Mark Juggurnauth-Thomas
fd388aac1a derived_data_manager: implement data derivation
Summary:
Implement a new version of data derivation in the derived data manager.  This is different from the old version in a few ways:

* `derived_data::BonsaiDerivable` is replaced by `derived_data_manager::BonsaiDerivable`.  This trait defines both how to perform derivation and how to store and retrieve mapping values.  Derivation is performed with reference to the derived data manager, rather than `BlobRepo`.

* The old `Mapping` structs and traits are replaced with a direct implementation in the derived data manager, using the `BonsaiDerivable` trait to handle the derived-data-type-specific parts.

* The new implementation assumes we will stick with parallel derivation, and doesn't implement serial derivation.

Code is copied from the `derived_data` crate, as it is intended to be a replacement once all the derived data types are migrated, and re-using code would create a circular dependency during migration.

This only covers the basic derivation implementation used during production.  The derived data manager will also take over backfilling, but that will happen in a later diff.

Reviewed By: yancouto

Differential Revision: D30805046

fbshipit-source-id: b9660dd957fdf762f621b2cb37fc2eea7bf03074
2021-09-20 03:23:17 -07:00
Mark Juggurnauth-Thomas
4776c94ae7 derived_data_manager: remove builder
Summary:
The builder pattern turned out to be unnecessary, as mappings don't need to be
stored in the manager after all.

Reviewed By: StanislavGlebik

Differential Revision: D30944565

fbshipit-source-id: 4300cdcc871c89f98e42d5b47600ac640b4b94eb
2021-09-20 03:23:17 -07:00
Gus Wynn
c797db63a1 vendor tokio-console and console-subscriber
Summary:
this relies on local changes to make it so cargo metadata ACTUALLY finds this
binary: https://github.com/tokio-rs/console/pull/146 is where I try to upstream
it

Reviewed By: jsgf

Differential Revision: D30944630

fbshipit-source-id: 5d34a32a042f83eff7e7ae7445e23badf10fffe3
2021-09-17 16:23:43 -07:00
Jeremy Fitzhardinge
113060beb1 third-party/rust: bump all the tracing packages
Summary:
This includes recent changes:
- `skip_all` in #[instrument] https://github.com/tokio-rs/tracing/pull/1548
- NoSubscriber no-op Subscriber https://github.com/tokio-rs/tracing/pull/1549
- Box/Arc Layer (https://github.com/tokio-rs/tracing/pull/1536 +
  https://github.com/tokio-rs/tracing/pull/1547)
- log filtering fixes https://github.com/tokio-rs/tracing/pull/1543

Local: force `log` and `log-always` features off. This removes the
`enable-trace-logging` flag from smartplatform/reconciler.

Reviewed By: guswynn

Differential Revision: D30906032

fbshipit-source-id: fdc3e4e29a39e9e181d9f4ff017e6b3e16acdcd8
2021-09-15 16:52:25 -07:00
Jeremy Fitzhardinge
cbceb08640 third-party/rust: local patch to tracing-subscriber
Summary:
Add impls for Layer for Box/Arc<L: Layer> and <dyn Layer>. Also a pile of other
updates in git which haven't been published to crates.io yet, including proper
level filtering of trace events being fed into log.

Reviewed By: dtolnay

Differential Revision: D30829927

fbshipit-source-id: c01c9369222df2af663e8f8bf59ea78ee12f7866
2021-09-09 22:38:25 -07:00
Mark Juggurnauth-Thomas
81364c16e1 dangerous_override: override blobstore in derived data manager
Summary:
The derived data manager also has a reference to the repo blobstore.  This must
also be overridden when we override the blobstore.

Reviewed By: yancouto

Differential Revision: D30738354

fbshipit-source-id: b0e16ef810c5244cd056a3c9e5b9ceaaddb5ecea
2021-09-07 07:44:27 -07:00
Thomas Orozco
35e3466031 third-party/rust: update daemonize to 0.5
Summary:
Like it says in the title, this updates us to use Daemonize 0.5, though from
Github and not Crates.io, because it hasn't been released to the latter yet.

The main motivation here is to pull in
https://github.com/knsd/daemonize/pull/39 to avoid leaking PID files to
children of the daemon.

This required some changes in `hphp/hack/src/facebook/hh_decl`  and `xplat/rust/mobium` since the way to
run code after daemonization has changed (and became more flexible).

Reviewed By: ndmitchell

Differential Revision: D30694946

fbshipit-source-id: d99768febe449d7a079feec78ab8826d0e29f1ef
2021-09-02 06:27:03 -07:00
Thomas Orozco
0d2bfbeccd Update autocargo component on FBS:master
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------

build-break (bot commits are not reviewed by a human)

Reviewed By: farnz

Differential Revision: D30717040

fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
2021-09-02 02:33:56 -07:00
David Tolnay
ba87c55127 third-party/rust: Patch mockall_derive to fix nondeterminism failures in Conveyor
Summary:
The mockall crate's `automock` attribute previously created nondeterministic output, which leads to frequent random "Found possibly newer version of crate" failures in Buck builds that involve cache.

The affected trait in Conveyor is:

https://www.internalfb.com/code/fbsource/[4753807291f7275a061d67cead04ea12e7b38ae2]/fbcode/conveyor/common/just_knobs/src/lib.rs?lines=13-23

which has a method with two lifetime parameters. Mockall's generated code shuffled them in random order due to emitting the lifetimes in HashSet order. The generated code would randomly contain one of these two types:

`Box<dyn for<'b, 'a> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`

`Box<dyn for<'a, 'b> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`

Reviewed By: jsgf

Differential Revision: D30656936

fbshipit-source-id: c1a251774333d7a4001a7492c1995efd84ff22e5
2021-08-30 21:12:18 -07:00
Gus Wynn
87a09132dc tokio -> 1.10
Reviewed By: dtolnay

Differential Revision: D30647831

fbshipit-source-id: 7094873ec5cfbf80cd7c3564fdd011268053b0d3
2021-08-30 15:55:16 -07:00
CodemodService Bot
0a375b8e5d Daily common/rust/cargo_from_buck/bin/autocargo
Reviewed By: StanislavGlebik

Differential Revision: D30535840

fbshipit-source-id: a941161547246c1e9aac0735a1994f20389ce1ae
2021-08-25 03:07:04 -07:00
David Tolnay
cf16f0b157 Add nested-values feature to slog
Reviewed By: zertosh

Differential Revision: D30387633

fbshipit-source-id: 27b1d601a73abf522d835c2f857d5a621c2b693b
2021-08-18 10:47:58 -07:00
Robbin Xu
0af6c0031f Revert D30351289: Add nested-values feature to slog
Differential Revision:
D30351289 (637bd00002)

Original commit changeset: b6c1c896b06c

fbshipit-source-id: c226f283a744170bb6bc2ed0b00e59249f9392c3
2021-08-17 16:33:26 -07:00
Matt Smith
637bd00002 Add nested-values feature to slog
Summary:
The diff is giant, but it's just a one-line change to add the
nested-values feature to slog, we just have a whole bunch of projects dependent
on slog.

Reviewed By: dtolnay

Differential Revision: D30351289

fbshipit-source-id: b6c1c896b06cbdf23b1f92c0aac9a97aa116085d
2021-08-17 15:28:16 -07:00
Yan Soares Couto
1d639f5a93 derived_data: introduce DerivedDataManager
Summary:
The `DerivedDataManager` will manage the ordering of derivation for derived
data, taking into account dependencies between types as well as the topological
ordering of the repository.  It will replace the free functions in
`derived_data` as well as much of the `utils` crate.

This is the first step: it introduces the manager, although currently it only takes
over management of the derived data lease.

Reviewed By: mitrandir77

Differential Revision: D30281634

fbshipit-source-id: 04c3a34d97ea02cc8c26d34096cca341e800da9b
2021-08-17 10:30:07 -07:00