scorecard/docs/checks.md

Check Documentation

This page contains information on how each check works and provide remediation steps to fix the failure. All of these checks are basically "best-guesses" currently, and operate on a set of heuristics.

They are all subject to change, and have room for improvement! If you have ideas for things to add, or new ways to detect things, please contribute!

Binary-Artifacts

This check tries to determine if the project has binary artifacts in the source repository. Binaries are a threat to auditability and vulnerability management. In addition, a binary could be compromised or malicious. A low score is therefore considered High risk.

Remediation steps

• Remove the binary artifacts from the repository.
• Build from source.

Branch-Protection

Branch protection allows defining rules to enforce certain workflows for branches, such as requiring a review or passing certain status checks. Branch protection ensures compromised contributors cannot intentionally inject malicious code. A low score is therefore considered High risk. This check determines if the default and release branches are protected with GitHub's branch protection settings. The check only works when the token has Admin access to the repository. This check determines if the default and release branches are protected.

Remediation steps

• Enable branch protection settings in your source hosting provider to avoid force pushes or deletion of your important branches.
• For GitHub, check out the steps here.

CI-Tests

This check tries to determine if the project runs tests before pull requests are merged. Running tests helps developers catch mistakes early on. A low score is considered 'Low' risk. The check works by looking for a set of well-known CI-system names in GitHub CheckRuns and Statuses among the recent commits (~30). A CI-system is considered well-known if its name contains any of the following: appveyor, buildkite, circleci, e2e, github-actions, jenkins, mergeable, test, travis-ci. The check succeeds if at least 75% of successful pull requests have at least one successful check associated with them.

Remediation steps

• Check-in scripts that run all the tests in your repository.
• Integrate those scripts with a CI/CD platform that runs it on every pull request (e.g. GitHub Actions, Prow, etc).

CII-Best-Practices

This check tries to determine if the project has a CII Best Practices Badge. This badge tells us the repo maintainers are aware of best development practices. A low score is considered 'Low' risk. The check uses the URL for the Git repo and the CII API.

Remediation steps

• Sign up for the CII Best Practices program.

Code-Review

This check tries to determine if the project requires code review before pull requests are merged. Reviewing code improves the quality of code in general. In addition, it ensures compromised contributors cannot intentionally inject malicious code. A low score is therefore considered High risk. The check first tries to detect if Branch-Protection is enabled on the default branch ,and the number of reviewers is at least 1. If this fails, it checks if the recent (~30) commits have a Github-approved review or if the merger is different from the committer (implicit review). It also performs similar check for reviews using Prow (labels "lgtm" or "approved") and Gerrit ("Reviewed-on" and "Reviewed-by").

Remediation steps

• Follow security best practices by performing strict code reviews for every new pull request.
• Make "code reviews" mandatory in your repository configuration. E.g. GitHub.
• Enforce the rule for administrators / code owners as well. E.g. GitHub

Contributors

This check tries to determine if the project has a set of contributors from multiple companies. Low score has 'Low' risk. The check works by looking at the authors of recent commits and checking the Company field on the GitHub user profile. A contributor must have at least 5 commits in the last 30 commits. The check succeeds if all contributors span at least 2 different companies.

Remediation steps

• There is NO remediation work needed here. This is to provide some insights on which organization(s) have contributed to the project and making trust decisions based on that. But you can ask your contributors to join their respective organizations.

Dependency-Update-Tool

This check tries to determine if the project uses a dependency update tool. Not updating dependencies makes a project vulnerable to known flaws and prone to attacks. A low score is therefore considered High risk. The checks looks for dependabot or renovatebot. This check only looks if it is enabled and does not ensure that it is run and pull requests are merged.

Remediation steps

• Signup for automatic dependency updates with dependabot or renovatebot and place the config file in the locations that are recommended by these tools.

Fuzzing

This check tries to determine if the project uses fuzzing. Fuzzing is important to reduce the number of vulnerabilities in code. A low score is considered 'Medium' risk. The check currently works by checking if the repo name is in the OSS-Fuzz project list.

Remediation steps

• Integrate the project with OSS-Fuzz by following the instructions here.

Maintained

This check tries to determine if the project is "actively maintained". A project which is not active may not be patched, may not have its dependencies patched, or may not be actively tested and used. A low score is therefore considered High risk. The check currently works by looking whether the repo is archived or not. If it is archived, it returns the minimum score. If it is not, the check looks for commits within the last 90 days, and outputs the highest score if there are at least 1 commit/week during this period.

Remediation steps

• There is NO remediation work needed here. This is just to indicate your project activity and maintenance commitment.

Packaging

This check tries to determine if the project is published as a package that other developers can install/download. Packaging your project is essential for users to receive updates and security patches automatically. A low score is considered Medium risk. The check currently looks for GitHub packaging workflows and language-specific GitHub Actions that upload the package to a corresponding hub, e.g., Npm. There is a plan to add better support to query package manager hubs directly in the future, e.g., for Npm, PyPi.

Remediation steps

• Publish your project as a downloadable package.
• Use a GitHub action to release your package to language-specific hubs.

Pinned-Dependencies

This check tries to determine if the project has declared and pinned its dependencies. Pinning dependencies is important to mitigate compromised dependencies from undermining the security of the project. Low score is therefore considered Medium risk. The checks works by (1) looking for the following files in the root directory: go.mod, go.sum (Golang), package-lock.json, npm-shrinkwrap.json (Javascript), requirements.txt, pipfile.lock (Python), gemfile.lock (Ruby), cargo.lock (Rust), yarn.lock (package manager), composer.lock (PHP), vendor/, third_party/, third-party/; (2) looks for unpinned dependencies in Dockerfiles, shell scripts and GitHub workflows.

Remediation steps

• Declare all your dependencies with specific versions in your package format file (e.g. package.json for npm, requirements.txt for python). For C/C++, check in the code from a trusted source and add a README on the specific version used (and the archive SHA hashes).
• If the package manager supports lock files (e.g. package-lock.json for npm), make sure to check these in the source code as well. These files maintain signatures for the entire dependency tree and saves from future exploitation in case the package is compromised.
• For Dockerfiles and GitHub workflows, pin dependencies by hash. See example main.yaml and Dockerfile examples.
• To help update your dependencies after pinning them, use tools such as Github's dependabot or renovate bot.

SAST

This check tries to determine if the project uses static code analysis. SAST tool may prevent known classes of bugs to be inadvertently introduced in the codebase. A low score is considered Medium risk. The checks currently looks for known Github apps such as github-code-scanning (codeql) and sonarcloud in the recent (~30) merged PRs. The check also looks for the use of "github/codeql-action" in a GitHub workflow.

Remediation steps

• Run CodeQL checks in your CI/CD by following the instructions here.

Security-Policy

This check tries to determine if the project has published a security policy. It works by looking for a file named SECURITY.md (case-insensitive) in a few well-known directories.

Remediation steps

• Place a security policy file SECURITY.md in the root directory of your repository. This makes it easily discoverable by a vulnerability reporter.
• The file should contain information on what constitutes a vulnerability and a way to report it securely (e.g. issue tracker with private issue support, encrypted email with a published public key).

Signed-Releases

This check tries to determine if the project cryptographically signs release artifacts. Signed releases attest to the provenance of the artifact. A low score is considered 'High' risk. It works by looking for filenames: *.minisig (https://github.com/jedisct1/minisign), *.asc (pgp), *.sign. for the last 5 GitHub releases. The check does not verify the signatures.

Remediation steps

• Publish the release.
• Generate a signing key.
• Download the release as an archive locally.
• Sign the release archive with this key (should output a signature file).
• Attach the signature file next to the release archive.
• For GitHub, check out the steps here.

Token-Permissions

This check tries to determine if the project's GitHub workflows follow the principle of least privilege, i.e. if the GitHub tokens are set read-only by default. Attackers may use a compromised token with write access to push malicious code into the project. A low score is therefore considered High risk. For each workflow yaml file, the check looks for the permission definitions. To obtain the highest score, the permissions should be set as read-only at the top level and the required write permissions should be declared at the run-level. The check cannot detect if the "read-only" GitHub permission settings is enabled, as there is no API available.

Remediation steps

• Set permissions as read-all or contents: read as described in GitHub's documentation.

Vulnerabilities

This check determines if the project has open, unfixed vulnerabilities using the OSV service. An existing vulnerability is can readily be used by attackers, so a low score is considered High risk.

Remediation steps

• Fix the vulnerabilities. The details of each vulnerability can be found on https://osv.dev.