3b948257fc
* spelling: accurate Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: administrator Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: analyze Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: andtwenty Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: ascii Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: association Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: at least Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: attestor Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: barbaric Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: bucket Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: by Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: can Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: case-insensitive Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: case-sensitive Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: checking Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: command-line Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: commit Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: committed Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: conclusion Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: corresponding Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: created Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: dataset Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: default Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: defines Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: dependabot Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: dependency Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: depending Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: desired Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: different Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: disclose Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: download Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: each Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: enforce Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: every time Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: exist Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: existing Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: fields Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: files Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: for Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: force-push Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: github Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: gitlab Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: ignoreed Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: implementation Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: implements Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: increase Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: indicates Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: initialized Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: instructions Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: invalid Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: marshal Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: match Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: name Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: nonexistent Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: organization Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: package Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: provenance Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: query Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: readers Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: receive Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: registered Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: remediate Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: representation Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: requests Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: requires Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: return Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: scorecard Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: separator Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: serialization Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: sign up Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: specifications Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: specified Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: success Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: successfully Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: the Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: their Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: twenty Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: unexpected Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: unused Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: unverified Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: validate Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: vendor Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: vulnerabilities Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: vulns Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: will Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: without Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: workflow Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> * spelling: workflows Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> --------- Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> |
||
---|---|---|
.. | ||
fakegcs | ||
config.yaml | ||
projects.csv | ||
README.md |
Configuring a local environment to test the Scorecard Cron Job
This emulator focuses on being able to test the worker
, which pulls messages from a pubsub, processes them, and writes the results to a Google Cloud Storage (GCS) bucket.
It's necessary to support pubsub, gcs, and the controller
to get the worker
working.
In general, you'll need 4-5 terminals (or tmux) to run everything needed.
GCS emulator
fake-gcs-server meets our needs and is written in Go. We may be able to use it as a library for unit tests in the future.
For now, the binary is good enough, so install it from source (or Releases):
go install github.com/fsouza/fake-gcs-server@latest
Now you can run the fake from the root of the Scorecard repo in your first window:
fake-gcs-server -scheme http -public-host 0.0.0.0:4443 \
-backend filesystem -filesystem-root cron/internal/emulator/fakegcs
pubsub emulator:
Google Cloud has a pubsub emulator with complete install instructions. I've summarized some of them below.
One time setup
gcloud components install pubsub-emulator
gcloud components update
Anywhere outside your scorecard repo:
git clone https://github.com/googleapis/python-pubsub
cd python-pubsub/samples/snippet
pip install -r requirements.txt
Running the pubsub emulator (needed to do every time)
In a second window from any directory, run the emulator itself:
export PUBSUB_PROJECT_ID=test
gcloud beta emulators pubsub start --project=$PUBSUB_PROJECT_ID
In a third window (from the samples/snippet
directory wherever you cloned python-pubsub
) create the topic and subscription:
export PUBSUB_PROJECT_ID=test
export TOPIC_ID=scorecard-batch-requests
export SUBSCRIPTION_ID=scorecard-batch-worker
$(gcloud beta emulators pubsub env-init)
python3 publisher.py $PUBSUB_PROJECT_ID create $TOPIC_ID
python3 subscriber.py $PUBSUB_PROJECT_ID create $TOPIC_ID $SUBSCRIPTION_ID
alias drain-pubsub="python3 subscriber.py $PUBSUB_PROJECT_ID receive $SUBSCRIPTION_ID"
At any point you can drain the queue by running the following in the same window. Make sure to stop the command when testing the worker
:
drain-pubsub
run Scorecard cron components
Commands intended to be run from the base of the Scorecard repo. Since this is intended to be used during development, go run
is used but there's no reason you can't use go build
.
The repos in cron/internal/emulator/projects.csv
and the cron/internal/emulator/config.yaml
file can be changed as needed.
controller
$(gcloud beta emulators pubsub env-init)
export STORAGE_EMULATOR_HOST=0.0.0.0:4443
go run $(ls cron/internal/controller/*.go | grep -v _test.go) \
--config cron/internal/emulator/config.yaml \
cron/internal/emulator/projects.csv
worker
$(gcloud beta emulators pubsub env-init)
export STORAGE_EMULATOR_HOST=0.0.0.0:4443
go run $(ls cron/internal/worker/*.go | grep -v _test.go) \
--ignoreRuntimeErrors=true \
--config cron/internal/emulator/config.yaml