9440b761df
* 🌱 Bump github.com/goreleaser/goreleaser in /tools (#3238) Bumps [github.com/goreleaser/goreleaser](https://github.com/goreleaser/goreleaser) from 1.18.2 to 1.19.1. - [Release notes](https://github.com/goreleaser/goreleaser/releases) - [Changelog](https://github.com/goreleaser/goreleaser/blob/main/.goreleaser.yaml) - [Commits](https://github.com/goreleaser/goreleaser/compare/v1.18.2...v1.19.1) --- updated-dependencies: - dependency-name: github.com/goreleaser/goreleaser dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: André Backman <andre.backman@nokia.com> * begin implementing probe: minTwoCodeReviewers Signed-off-by: André Backman <andre.backman@nokia.com> * print raw results Signed-off-by: André Backman <andre.backman@nokia.com> * print raw results Signed-off-by: André Backman <andre.backman@nokia.com> * print raw results Signed-off-by: André Backman <andre.backman@nokia.com> * rename probe directory: minimumCodeReviewers Signed-off-by: André Backman <andre.backman@nokia.com> * rename probe CodeReviewers Signed-off-by: André Backman <andre.backman@nokia.com> * rename import for CodeReviewers probe Signed-off-by: André Backman <andre.backman@nokia.com> * update code reviewers definition Signed-off-by: André Backman <andre.backman@nokia.com> * update code reviewers implementation; fixed embed FS usage Signed-off-by: André Backman <andre.backman@nokia.com> * printing all findings, work out where to concatenate them Signed-off-by: André Backman <andre.backman@nokia.com> * concatenated findings to one single finding, outcome is based on the least found unique reviewers Signed-off-by: André Backman <andre.backman@nokia.com> * refactored uniqueCodeReviewers probe, needs more error checks Signed-off-by: André Backman <andre.backman@nokia.com> * add error handling for cases of non-existant author and/or reviewer logins Signed-off-by: André Backman <andre.backman@nokia.com> * add error handling for cases of non-existant author and/or reviewer logins Signed-off-by: André Backman <andre.backman@nokia.com> * rename probe Signed-off-by: André Backman <andre.backman@nokia.com> * update codeReviewTwoReviewers definition Signed-off-by: André Backman <andre.backman@nokia.com> * rename unique code reviewers probe Signed-off-by: André Backman <andre.backman@nokia.com> * implement codeApproved probe, validation of reviews needs fixing Signed-off-by: André Backman <andre.backman@nokia.com> * update codeApproved probe, validation of reviews needs fixing Signed-off-by: André Backman <andre.backman@nokia.com> * working version of codeApproved probe Signed-off-by: André Backman <andre.backman@nokia.com> * codeReviewed probe implemented Signed-off-by: André Backman <andre.backman@nokia.com> * clean up comments, add imports, run all probes Signed-off-by: André Backman <andre.backman@nokia.com> * update license comments Signed-off-by: André Backman <andre.backman@nokia.com> * Update def.yml license Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update def.yml license Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update def.yml license Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update impl.go license Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update impl.go license to Apache 2 Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update impl.go license to Apache 2 Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update code_review.go license Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update entries.go; CodeReviewChecks now called CodeReview Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Update impl.go, refactor codeReviewTwoReviewers; moved utility functions into impl.go Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * Delete code_review.go utilities moved utility functions to the impl.go they are used in Signed-off-by: André Backman <88145164+andrelmbackman@users.noreply.github.com> * rename probe Signed-off-by: André Backman <andre.backman@nokia.com> * update codeReviewTwoReviewers definition Signed-off-by: André Backman <andre.backman@nokia.com> * implement codeApproved probe, validation of reviews needs fixing Signed-off-by: André Backman <andre.backman@nokia.com> * update codeApproved probe, validation of reviews needs fixing Signed-off-by: André Backman <andre.backman@nokia.com> * working version of codeApproved probe Signed-off-by: André Backman <andre.backman@nokia.com> * codeReviewed probe implemented Signed-off-by: André Backman <andre.backman@nokia.com> * clean up comments, add imports, run all probes Signed-off-by: André Backman <andre.backman@nokia.com> * update license comments Signed-off-by: André Backman <andre.backman@nokia.com> * update license comments Signed-off-by: André Backman <andre.backman@nokia.com> * 🌱 Included unit tests (#3242) - Included unit tests Signed-off-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com> Signed-off-by: André Backman <andre.backman@nokia.com> * 🌱 Bump golang.org/x/text from 0.10.0 to 0.11.0 (#3243) Bumps [golang.org/x/text](https://github.com/golang/text) from 0.10.0 to 0.11.0. - [Release notes](https://github.com/golang/text/releases) - [Commits](https://github.com/golang/text/compare/v0.10.0...v0.11.0) --- updated-dependencies: - dependency-name: golang.org/x/text dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: André Backman <andre.backman@nokia.com> * 🌱 Bump golang.org/x/oauth2 from 0.9.0 to 0.10.0 (#3244) Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.9.0 to 0.10.0. - [Commits](https://github.com/golang/oauth2/compare/v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: André Backman <andre.backman@nokia.com> * 📖 Update Branch-Protection admin and non-admin requirements (#2772) * docs: Branch protection admin-only requirements Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Branch protection requirements by tier Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: How get a perfect score in branch protection Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Fix local images ref in doc Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Fix typo Co-authored-by: Pedro Nacht <pedro.k.night@gmail.com> Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Fix check specific table of contents Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * fix: Code owners setting is non admin Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Fix branch protection applied not only to main branch Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Add alt text for images Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: You can get a perfect score with non admin access Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: update max tier scores Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: update tier 1 max points explanation Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Move changes to internal checks doc Move changes done in docs/checks.md to docs/checks/internal/checks.yaml. Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Revert changes on checks doc Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Fix admin settings evaluated on branch protection Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Change branch protection model status checks Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Change tiers score to expected score The expected score for the code to output is 3/10 for Tier 1 case and 7/10 for Tier 3 case. The scoring issue will be reported as bug. Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Fix Tier 3 score Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> --------- Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> Co-authored-by: Pedro Nacht <pedro.k.night@gmail.com> Signed-off-by: André Backman <andre.backman@nokia.com> * 🌱 Linter workflow cleanup (#3247) * Fix linter timeout by renaming deprecated deadline. Signed-off-by: Spencer Schrock <sschrock@google.com> * Disable depguard linter. As of golangci-lint v3.5.0, the depguard linter is complaining. We don't use a .depguard.yml file, so just disabling the linter. Signed-off-by: Spencer Schrock <sschrock@google.com> * Move linter into own workflow. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix bash command substitution. Signed-off-by: Spencer Schrock <sschrock@google.com> * Add harden runner. Signed-off-by: Spencer Schrock <sschrock@google.com> * switch names to existing linter job Signed-off-by: Spencer Schrock <sschrock@google.com> * Update golangci-lint to v1.53.3 Signed-off-by: Spencer Schrock <sschrock@google.com> --------- Signed-off-by: Spencer Schrock <sschrock@google.com> Signed-off-by: André Backman <andre.backman@nokia.com> * 🌱 Bump tj-actions/changed-files from 37.0.5 to 37.1.0 (#3253) Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 37.0.5 to 37.1.0. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits]( |
||
---|---|---|
.. | ||
blocksDeleteOnBranches | ||
blocksForcePushOnBranches | ||
branchProtectionAppliesToAdmins | ||
codeApproved | ||
codeReviewOneReviewers | ||
contributorsFromOrgOrCompany | ||
dismissesStaleReviews | ||
freeOfAnyBinaryArtifacts | ||
freeOfUnverifiedBinaryArtifacts | ||
fuzzedWithCLibFuzzer | ||
fuzzedWithClusterFuzzLite | ||
fuzzedWithCppLibFuzzer | ||
fuzzedWithGoNative | ||
fuzzedWithJavaJazzerFuzzer | ||
fuzzedWithOSSFuzz | ||
fuzzedWithPropertyBasedHaskell | ||
fuzzedWithPropertyBasedJavascript | ||
fuzzedWithPropertyBasedTypescript | ||
fuzzedWithPythonAtheris | ||
fuzzedWithRustCargofuzz | ||
fuzzedWithSwiftLibFuzzer | ||
hasDangerousWorkflowScriptInjection | ||
hasDangerousWorkflowUntrustedCheckout | ||
hasFSFOrOSIApprovedLicense | ||
hasLicenseFile | ||
hasLicenseFileAtTopDir | ||
hasOpenSSFBadge | ||
hasOSVVulnerabilities | ||
hasRecentCommits | ||
internal/utils | ||
issueActivityByProjectMember | ||
notArchived | ||
notCreatedRecently | ||
packagedWithAutomatedWorkflow | ||
releasesAreSigned | ||
releasesHaveProvenance | ||
requiresApproversForPullRequests | ||
requiresCodeOwnersReview | ||
requiresLastPushApproval | ||
requiresUpToDateBranches | ||
runsStatusChecksBeforeMerging | ||
sastToolCodeQLInstalled | ||
sastToolPysaInstalled | ||
sastToolQodanaInstalled | ||
sastToolRunsOnAllCommits | ||
sastToolSnykInstalled | ||
sastToolSonarInstalled | ||
securityPolicyContainsLinks | ||
securityPolicyContainsText | ||
securityPolicyContainsVulnerabilityDisclosure | ||
securityPolicyPresent | ||
testsRunInCI | ||
toolDependabotInstalled | ||
toolPyUpInstalled | ||
toolRenovateInstalled | ||
utils | ||
webhooksUseSecrets | ||
zrunner | ||
entries.go | ||
README.md |
Scorecard probes
This directory contains all the Scorecard probes.
A probe is an assessment of a focused, specific heuristic typically isolated to a particular ecosystem. For example, Scorecards fuzzing check consists of many different probes that assess particular ecosystems or aspects of fuzzing.
Each probe has its own directory in scorecard/probes
. The probes follow a camelcase naming convention that describe the exact heuristic a particular probe assesses.
Probes can return multiple or a single finding, where a finding is a piece of data with an outcome, message, and optionally a location. Probes should be designed in such a way that a finding.OutcomePositive
reflects a positive result, and finding.OutcomeNegative
reflects a negative result. Scorecard has other finding.Outcome
types available for other results; For example, the finding.OutcomeNotAvailable
is often used for scenarios, where Scorecard cannot assess a project with a given probe. In addition, probes should also be named in such a way that they answer "yes" or "no", and where "yes" answers positively to the heuristic, and "no" answers negatively. For example, probes that check for SAST tools in the CI are called toolXXXInstalled
so that finding.OutcomePositive
reflects that it is positive to use the given tool, and that "yes" reflects what Scorecard considers the positive outcome. For some probes, this can be a bit trickier to do; The notArchived
probe checks whether a project is archived, however, Scorecard considers archived projects to be negative, and the probe cannot be called isArchived
. These naming conventions are not hard rules but merely guidelines. Note that probes do not do any formal evaluation such a scoring; This is left to the evaluation part once the outcomes have been produced by the probes.
A probe consists of three files:
def.yml
: The documentation of the probe.impl.go
: The actual implementation of the probe.impl_test.go
: The probes test.
Reusing code in probes
When multiple probes use the same code, the reused code can be placed on scorecard/probes/internal/utils
How do I know which probes to add?
In general, browsing through the Scorecard GitHub issues is the best way to find new probes to add. Requests for support for new tools, fuzzing engines or other heuristics can often be converted into specific probes.
Probe definition formatting
Probe definitions can display links following standard markdown format.
Probe definitions can display dynamic content. This requires modifications in def.yml
and impl.go
and in the evaluation steps.
The following snippet in def.yml
will display dynamic data provided by impl.go
:
${{ metadata.dataToDisplay }}
And then in impl.go
add the following metadata:
f, err := finding.NewWith(fs, Probe,
"Message", nil,
finding.OutcomePositive)
f = f.WithRemediationMetadata(map[string]string{
"dataToDisplay": "this is the text we will display",
})
Example
Consider a probe with following line in its def.yml
:
The project ${{ metadata.oss-fuzz-integration-status }} integrated into OSS-Fuzz.
and the probe sets the following metadata:
f, err := finding.NewWith(fs, Probe,
"Message", nil,
finding.OutcomePositive)
f = f.WithRemediationMetadata(map[string]string{
"oss-fuzz-integration-status": "is",
})
The probe will then output the following text:
The project is integrated into OSS-Fuzz.
Should the changes be in the probe or the evaluation?
The remediation data must be set in the probe.