scorecard/probes
Raghav Kaul 256d5a3b50
🌱 Add script to set up probe boilerplate (#3948)
* Add script

Signed-off-by: Raghav Kaul <raghavkaul+github@google.com>

* script -> go

Signed-off-by: Raghav Kaul <raghavkaul+github@google.com>

* v4 -> v5

Signed-off-by: Raghav Kaul <raghavkaul+github@google.com>

* update

Signed-off-by: Raghav Kaul <raghavkaul+github@google.com>

* update

Signed-off-by: Raghav Kaul <raghavkaul+github@google.com>

---------

Signed-off-by: Raghav Kaul <raghavkaul+github@google.com>
2024-05-08 17:58:02 +00:00
..
archived ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
blocksDeleteOnBranches ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
blocksForcePushOnBranches ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
branchesAreProtected ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
branchProtectionAppliesToAdmins ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
codeApproved ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
codeReviewOneReviewers ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
contributorsFromOrgOrCompany ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
createdRecently ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
dependencyUpdateToolConfigured ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
dismissesStaleReviews ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
fuzzed ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasBinaryArtifacts ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasDangerousWorkflowScriptInjection ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasDangerousWorkflowUntrustedCheckout ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasFSFOrOSIApprovedLicense ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasLicenseFile ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasNoGitHubWorkflowPermissionUnknown ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasOpenSSFBadge ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasOSVVulnerabilities ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasPermissiveLicense ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasRecentCommits ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
hasUnverifiedBinaryArtifacts ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
internal 🌱 Add script to set up probe boilerplate (#3948) 2024-05-08 17:58:02 +00:00
issueActivityByProjectMember ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
jobLevelPermissions ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
packagedWithAutomatedWorkflow ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
pinsDependencies ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
releasesAreSigned 🐛 fix signed-releases lookback limit precedence (#4060) 2024-05-02 10:43:13 -07:00
releasesHaveProvenance 🐛 fix signed-releases lookback limit precedence (#4060) 2024-05-02 10:43:13 -07:00
requiresApproversForPullRequests ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
requiresCodeOwnersReview ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
requiresLastPushApproval ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
requiresPRsToChangeCode ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
requiresUpToDateBranches ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
runsStatusChecksBeforeMerging ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
sastToolConfigured ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
sastToolRunsOnAllCommits ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
securityPolicyContainsLinks ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
securityPolicyContainsText ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
securityPolicyContainsVulnerabilityDisclosure ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
securityPolicyPresent ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
testsRunInCI ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
topLevelPermissions ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
utils New probes: code-review (#3302) 2024-01-26 19:24:56 +00:00
webhooksUseSecrets ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
zrunner ⚠️ Replace v4 module references with v5 (#4027) 2024-04-12 14:51:50 -07:00
entries.go allow probes to collect their own data from repo clients (#4052) 2024-04-25 18:23:54 +00:00
README.md 📖 Review and update some probe documentation (#4023) 2024-04-11 22:08:55 -07:00

OpenSSF Scorecard Probes

This directory contains all the Scorecard probes. Each probe has its own subdirectory.

A probe is an individual heuristic, which provides information about a distinct behavior a project under analysis may or may not be doing. The probes follow a camelcase naming convention that describe the exact heuristic a particular probe assesses. The name should be phrased in a way that can be answered by "true" or "false".

Probes can return one or more findings, where a finding is a piece of data with an outcome, message, and optionally a location where the behavior was observed. The primary outcomes are finding.OutcomeTrue and finding.OutcomeFalse, but other outcomes are available as well. For example, the finding.OutcomeNotAvailable is often used for scenarios, where Scorecard cannot assess a behavior because there is no data to analyze.

A probe consists of three files:

  • def.yml: The documentation of the probe.
  • impl.go: The actual implementation of the probe.
  • impl_test.go: The probe's test.

Reusing code in probes

When multiple probes use the same code, the reused code can be placed in a package under probes/internal/

How do I know which probes to add?

In general, browsing through the Scorecard GitHub issues is the best way to find new probes to add. Requests for support for new tools, fuzzing engines or other heuristics can often be converted into specific probes.

Probe definition formatting

Probe definitions can display links following standard markdown format.

Probe definitions can display dynamic content. This requires modifications in def.yml and impl.go and in the evaluation steps.

The following snippet in def.yml will display dynamic data provided by impl.go:

${{ metadata.dataToDisplay }}

And then in impl.go add the following metadata:

f, err := finding.NewWith(fs, Probe,
	"Message", nil,
	finding.OutcomeTrue)
f = f.WithRemediationMetadata(map[string]string{
	"dataToDisplay": "this is the text we will display",
})

Example

Consider a probe with following line in its def.yml:

The project ${{ metadata.oss-fuzz-integration-status }} integrated into OSS-Fuzz.

and the probe sets the following metadata:

f, err := finding.NewWith(fs, Probe,
	"Message", nil,
	finding.OutcomeTrue)
f = f.WithRemediationMetadata(map[string]string{
	"oss-fuzz-integration-status": "is",
})

The probe will then output the following text:

The project is integrated into OSS-Fuzz.

Should the changes be in the probe or the evaluation?

The remediation data must be set in the probe.