ossf/scorecard
1
1
Fork 0
mirror of https://github.com/ossf/scorecard.git synced 2024-09-17 11:57:12 +03:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

📖 Edit and rework checks.md (via checks.yaml and main.go) (#1114)

Browse Source 
* Update checks.yaml to generate new checks.md docs

* Update main.go

Update overview text and add link

* -m "update TODO for Contrib and Signed-Releases"

* -m "Add admin setting  info to Branch-Protection"

* -m "generate docs, fix typos"

* -m "generate docs"

* -m "add links, small edits to checks.yml"

* -m "generate docs.md"

* Clarify Pinned-Dependencies remediation

* "Generate docs"

* "Add admin-only Branch-Protection checks"

* "Regenerate docs"

Co-authored-by: Azeem Shaikh <azeemshaikh38@gmail.com>
This commit is contained in:
olivekl 2021-10-06 18:07:49 -04:00 committed by GitHub
parent 6935be8110
commit aaff0e530c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 836 additions and 473 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

495
docs/checks.md
View File

@ -2,20 +2,53 @@
<!-- Do not edit this file manually! Edit checks.yaml instead. -->
# Check Documentation
This page contains information on how each check works and provide remediation
steps to fix the failure. All of these checks are basically "best-guesses"
currently, and operate on a set of heuristics.
They are all subject to change, and have room for improvement!
If you have ideas for things to add, or new ways to detect things,
please contribute!
This page describes each Scorecard check in detail, including scoring criteria,
remediation steps to improve the score, and an explanation of the risks
associated with a low score. The checks are continually changing and we welcome
community feedback. If you have ideas for additions or new detection techniques,
please [contribute](CONTRIBUTING.md)!
## Binary-Artifacts
This check tries to determine if the project has generated executable (binary) artifacts in the source repository.
Including generated executables in the source repository increases user risks. Many programming language systems can generate executables from source code (e.g., C/C++ generated machine code, Java `.class` files, Python `.pyc` files, and minified JavaScript). Users will often directly use executables if they are included in the source repository, leading to many dangerous behaviors. One problem is that reviews become ineffective and thus enable the use of obsolete or maliciously-subverted executables. Reviews generally review source code, not executables, since it's difficult to audit executables to ensure that they correspond. Since it's hard to audit this, over time the included executables might not correspond to the source code. Another problem is that it allows the executable generation process to atrophy, which can lead to an inability to create working executables. These problems can be countered with verified reproducible builds, but it's easier to implement verified reproducible builds when executables are not included in the source repository (since the executable generation process is less likely to have atrophied).
It's fine to include, in the source repository, files that are simultaneously reviewable source code and executables, since these are reviewable. (Some interpretive systems, such as many operating system shells, don't have a mechanism for storing generated executables that are different from the source file.) Similarly, we allow including in the source repository source code generated by other tools (e.g., by bison, yacc, flex, and lex). There are potential downsides to generated source code, but generated source code tends to be much easier to review and thus presents a lower risk. Generated source code is also often difficult for external tools to detect.
We allow including generated documentation in source repositories. Generated documentation is intended for use by humans (not computers), and humans can take context into account. Thus, generated documentation doesn't pose the same level of risk.
A low score is therefore considered `High` risk.
Risk: `High` (non-reviewable code)
This check determines whether the project has generated executable (binary)
artifacts in the source repository.
Including generated executables in the source repository increases user risk.
Many programming language systems can generate executables from source code
(e.g., C/C++ generated machine code, Java `.class` files, Python `.pyc` files,
and minified JavaScript). Users will often directly use executables if they are
included in the source repository, leading to many dangerous behaviors.
Problems with generated executable (binary) artifacts:
- Binary artifacts cannot be reviewed, allowing possible obsolete or
maliciously subverted executables. Reviews generally review source code, not
executables, since it's difficult to audit executables to ensure that they
correspond to the source code. Over time the included executables might not
correspond to the source code.
- Generated executables allow the executable generation process to atrophy,
which can lead to an inability to create working executables. These problems
can be countered with verified reproducible builds, but it's easier to
implement verified reproducible builds when executables are not included in
the source repository (since the executable generation process is less
likely to have atrophied).
Allowed by Scorecards:
- Files in the source repository that are simultaneously reviewable source
code and executables, since these are reviewable. (Some interpretive
systems, such as many operating system shells, don't have a mechanism for
storing generated executables that are different from the source file.)
- Source code in the source repository generated by other tools (e.g., by
bison, yacc, flex, and lex). There are potential downsides to generated
source code, but generated source code tends to be much easier to review and
thus presents a lower risk. Generated source code is also often difficult
for external tools to detect.
- Generated documentation in source repositories. Generated documentation is
intended for use by humans (not computers) who can evaluate the context.
Thus, generated documentation doesn't pose the same level of risk.
**Remediation steps**
- Remove the generated executable artifacts from the repository.
@ -23,11 +56,48 @@ A low score is therefore considered `High` risk.
## Branch-Protection
[Branch protection](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches) can enable various rules to enforce certain workflows for branches, such as preventing rewriting of public history (e.g., a *force push*), requiring review before acceptance into a main branch, or passing certain status checks before acceptance into a main branch.
Branch protection can reduce the risk of unintentional or malicious code from entering the "main" branch. Branch protection rules that prevent rewriting of public history (e.g., preventing *force push* of public branches) prevent history from changing without external notice. Branch protection rules that require status checks ensure that at least those checks are met before a change is accepted. Branch protection rules that require at least one other reviewer reviews greatly reduces the risk that a compromise of a contributor's account will lead to injection of malicious code. Review also increases the likelihood that an unintentional vulnerability in a contribution will be detected and fixed before the change is accepted. A low score is therefore considered `High` risk.
Note, however, that requiring reviews for every proposed change is impractical for many projects, since many projects simply don't have that many active participants. For more discussion, see [Code Reviews](#code-reviews).
In some cases these rules will need to be suspended. For example, if a past commit includes illegal content (such as child pornography), it may be impractical to hide the commit and in such cases the history may need to be rewritten.
This check determines if the default and release branches are protected with GitHub's branch protection settings. The check only works when the token has [Admin access](https://github.community/t/enable-branch-protection-get-api-without-admin/14197) to the repository. This check determines if the default and release branches are protected.
Risk: `High` (vulnerable to intentional malicious code injection)
This check determines whether a project's default and release branches are
protected with GitHub's
[branch protection](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches)
settings. Branch protection allows maintainers to define rules that enforce
certain workflows for branches, such as requiring review or passing certain
status checks before acceptance into a main branch, or preventing rewriting of
public history.
Note: The following settings queried by the Branch-Protection check require an admin token: `DismissStaleReviews`, `EnforceAdmin`, and `StrictStatusCheck`. If
the provided token does not have admin access, the check will query the branch
settings accessible to non-admins and provide results based only on these settings.
Even so, we recommend using a non-admin token, which provides a thorough enough
result to meet most user needs.
Different types of branch protection protect against different risks:
- Require code review: requires at least one reviewer, which greatly
reduces the risk that a compromised contributor can inject malicious code.
Review also increases the likelihood that an unintentional vulnerability in
a contribution will be detected and fixed before the change is accepted.
- Prevent force push: prevents use of the `--force` command on public
branches, which overwrites code irrevocably. This protection prevents the
rewriting of public history without external notice.
- Require
[status checks](https://docs.github.com/en/github/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks):
ensures that all required CI tests are met before a change is accepted.
Although requiring code review can greatly reduce the chance that
unintentional or malicious code enters the "main" branch, it is not feasible for
all projects, such as those that don't have many active participants. For more
discussion, see
[Code Reviews](https://github.com/ossf/scorecard/blob/main/docs/checks.md#code-reviews).
Additionally, in some cases these rules will need to be suspended. For example,
if a past commit includes illegal content such as child pornography, it may be
necessary to use a force push to rewrite the history rather than simply hide the
commit.
**Remediation steps**
- Enable branch protection settings in your source hosting provider to avoid force pushes or deletion of your important branches.
@ -35,9 +105,28 @@ This check determines if the default and release branches are protected with Git
## CI-Tests
This check tries to determine if the project runs tests before pull requests are merged.
Running tests helps developers catch mistakes early on. A low score is considered 'Low' risk.
The check works by looking for a set of CI-system names in GitHub `CheckRuns` and `Statuses` among the recent commits (~30). A CI-system is considered well-known if its name contains any of the following: appveyor, buildkite, circleci, e2e, github-actions, jenkins, mergeable, test, travis-ci. It does not currently support other source hosting repositories (forges). A project may meet this criterion yet have a failing scorecard report; there are many ways to implement this criterion, and it's challenging for an automated tool (like scorecard) to detect them all. If a project's system was not detected and you think it should be, please [open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
Risk: `Low` (possible unknown vulnerabilities)
This check tries to determine if the project runs tests before pull requests are
merged. It is currently limited to repositories hosted on GitHub, and does not
support other source hosting repositories (i.e., Forges).
Running tests helps developers catch mistakes early on, which can reduce the
number of vulnerabilities that find their way into a project.
The check works by looking for a set of CI-system names in GitHub `CheckRuns`
and `Statuses` among the recent commits (~30). A CI-system is considered
well-known if its name contains any of the following: appveyor, buildkite,
circleci, e2e, github-actions, jenkins, mergeable, test, travis-ci.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement CI testing, and it is
challenging for an automated tool like Scorecard to detect them all. A low score
is therefore not a definitive indication that the project is at risk.
If a project's system was not detected and you think it should be, please
[open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
**Remediation steps**
- Check-in scripts that run all the tests in your repository.
@ -45,116 +134,347 @@ The check works by looking for a set of CI-system names in GitHub `CheckRuns` an
## CII-Best-Practices
This check tries to determine if the project has earned a [CII Best Practices Badge](https://bestpractices.coreinfrastructure.org/).
This badge tells us if the project is applying a particular set of security-focused best development practices for open source software. The CII Best Practices badge has 3 tiers (passing, silver, and gold); we give full credit if the [passing criteria are met](https://bestpractices.coreinfrastructure.org/criteria/0), as achieving a passing badge is a significant achievement for many projects. We give a little credit if the project is at least working to achieve a badge, and increasingly more as more criteria are met.
For example, to earn the passing badge, the project MUST publish the process for reporting vulnerabilities on the project site, it MUST provide a working build system that can automatically rebuild the software from source code (where applicable), it MUST have a general policy that tests will be added to an automated test suite when major new functionality is added, it MUST meet various cryptography criteria where applicable, it MUST have at least one primary developer who knows how to design secure software, at least one of the project's primary developers MUST know of common kinds of errors that lead to vulnerabilities in this kind of software (as well as at least one method to counter or mitigate each of them), and at least one static code analysis tool (beyond compiler warnings and "safe" language modes) MUST be applied to any proposed major production release.
A low score is considered 'Low' risk. The check uses the URL for the Git repo and the CII API.
Risk: `Low` (possibly not following security best practices)
This check determines whether the project has earned a [CII Best Practices
Badge](https://bestpractices.coreinfrastructure.org/), which indicates that the
project uses a set of security-focused best development practices for open
source software. The check uses the URL for the Git repo and the CII API.
The CII Best Practices badge has 3 tiers: passing, silver, and gold. We give
full credit to projects that meet the [passing
criteria](https://bestpractices.coreinfrastructure.org/criteria/0), which is a
significant achievement for many projects. Lower scores represent a project that
is at least working to achieve a badge, with increasingly more points awarded as
more criteria are met.
To earn the passing badge, the project MUST:
- publish the process for reporting vulnerabilities on the project site
- provide a working build system that can automatically rebuild the software
from source code (where applicable)
- have a general policy that tests will be added to an automated test suite
when major new functionality is added
- meet various cryptography criteria where applicable
- have at least one primary developer who knows how to design secure software
- have at least one primary developer who knows of common kinds of errors
that lead to vulnerabilities in this kind of software (and at least one
method to counter or mitigate each of them)
- apply at least one static code analysis tool (beyond compiler warnings and
"safe" language modes) to any proposed major production release.
Some of these criteria overlap with other Scorecards checks.
**Remediation steps**
- Sign up for the [CII Best Practices program](https://bestpractices.coreinfrastructure.org/en).
## Code-Review
This check tries to determine if the project requires code review before pull requests (aka merge requests) are merged.
Reviewing code improves the quality of code in general, because reviews may detect various unintentional problems that can be fixed immediately before they are merged. Such problems include unintentional vulnerabilities, so unintentional vulnerabilities can be reduced through review. Reviews also make it more difficult for an attacker to insert malicious code (either as a malicious contributor or as an attacker who has subverted a contributor's account), because a reviewer might detect the subversion. This increased difficulty can even deter attackers. A low score is therefore considered `High` risk.
However, requiring that all proposed changes be reviewed by someone else before merging them is impractical for some projects. A project with only one active participant cannot practically enforce multi-person review, and even a project with multiple active participants may not have enough active participation to be able to require review of all proposed changes. Such projects will not be able to practically enforce rules such as requiring a minimum number of reviewers, and administrators in practice will be exempt from reviews. Projects with a small number of active participants, but more than one, instead sometimes aim for a review of a percentage of proposals (e.g., "at least half of all proposed changes are reviewed").
In addition, note that requiring review does not eliminate all risks. The other reviewer(s) might fail to notice unintentional vulnerabilities or malicious code. They might be colluding with a malicious developer. The "other" reviewers might even be the same person (aka a "sock puppet").
In short, requiring others' review before accepting a change reduces risks to users, as those additional reviewers may detect problems early, but it is not a practical requirement for some projects.
The check first tries to detect if Branch-Protection is enabled on the default branch ,and the number of reviewers is at least 1. If this fails, it checks if the recent (~30) commits have a Github-approved review or if the merger is different from the committer (implicit review). It also performs similar check for reviews using [Prow](https://github.com/kubernetes/test-infra/tree/master/prow#readme) (labels "lgtm" or "approved") and Gerrit ("Reviewed-on" and "Reviewed-by").
Risk: `High` (unintentional vulnerabilities or possible injection of malicious
code)
This check determines whether the project requires code review before pull
requests (merge requests) are merged.
Reviews detect various unintentional problems, including vulnerabilities that
can be fixed immediately before they are merged, which improves the quality of
the code. Reviews may also detect or deter an attacker trying to insert
malicious code (either as a malicious contributor or as an attacker who has
subverted a contributor's account), because a reviewer might detect the
subversion.
The check first tries to detect whether [Branch-Protection](checks.md#branch-protection) is enabled on the
default branch with at least one required reviewer. If this fails, the check
determines whether the most recent (~30) commits have a Github-approved review
or if the merger is different from the committer (implicit review). It also
performs a similar check for reviews using
[Prow](https://github.com/kubernetes/test-infra/tree/master/prow#readme) (labels
"lgtm" or "approved") and [Gerrit](https://www.gerritcodereview.com/) ("Reviewed-on" and "Reviewed-by").
Note: Requiring reviews for all changes is infeasible for some projects, such as
those with only one active participant. Even a project with multiple active
contributors may not have enough active participation to be able to require
review of all proposed changes. Projects with a small number of active
participants instead sometimes aim for a review of a
percentage of proposals (e.g., "at least half of all proposed changes are
reviewed").
Requiring review does not eliminate all risks. The other reviewers might fail to
notice unintentional vulnerabilities or malicious code, be colluding with a
malicious developer, or even be the same person (using a "[sock
puppet](https://en.wikipedia.org/wiki/Sock_puppet_account)" account).
**Remediation steps**
- If the project has only one contributor, or does not have enough reviewers to practically require that all contributions be reviewed, try to recruit more maintainers to the project who will be willing to review others' work. Ideally at least some of these people will be from different organizations. If the project has very limited utility, consider expanding its intended utility so more people will be interested in improving it, and make that larger scope clear to potential contributors.
- If the project has only one contributor, or does not have enough reviewers to practically require that all contributions be reviewed, try to recruit more maintainers to the project who will be willing to review others' work. Ideally at least some of these people will be from different organizations (see [Contributors](checks.md#contributors)). If the project has very limited utility, consider expanding its intended utility so more people will be interested in improving it, and make that larger scope clear to potential contributors.
- Follow security best practices by performing strict code reviews for every new pull request / merge request.
- Make "code reviews" mandatory in your repository configuration. E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging).
- Enforce the rule for administrators / code owners as well. E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
- Make "code reviews" mandatory in your repository configuration. ([Instructions for GitHub.](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging))
- Enforce the rule for administrators / code owners as well. ([Instructions for GitHub.](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators))
## Contributors
This check tries to determine if the project has a set of contributors from multiple organizations (e.g., companies).
Low score has 'Low' risk.
Some projects cannot meet this requirement. Obviously projects with only one active participant cannot meet it. Many projects have a narrow scope and may not be able to attract the interest of multiple organizations. See [Code Reviews](#code-reviews) for added notes about projects with a small number of participants.
The check works by looking at the authors of recent commits and checking the `Company` field on the GitHub user profile. A contributor must have at least 5 commits in the last 30 commits. The highest score is achieved when there are contributors from at least 3 different companies in the last 30 commits. It does not currently support other source hosting repositories (forges).
Risk: `Low` (lower number of trusted code reviewers)
This check tries to determine if the project has recent contributors from
multiple organizations (e.g., companies). It is currently limited to
repositories hosted on GitHub, and does not support other source hosting
repositories (i.e., Forges).
The check looks at the `Company` field on the GitHub user profile for authors of
recent commits. To receive the highest score, the project must have had
contributors from at least 3 different companies in the last 30 commits; each of
those contributors must have had at least 5 commits in the last 30 commits.
Note: Some projects cannot meet this requirement, such as small projects with
only one active participant, or projects with a narrow scope that cannot attract
the interest of multiple organizations. See
[Code Reviews](https://github.com/ossf/scorecard/blob/main/docs/checks.md#code-reviews)
for more information about evaluating projects with a small number of
participants.
**Remediation steps**
- There is *NO* remediation work needed here. This is to provide some insights on which organization(s) have contributed to the project and making trust decisions based on that. But you can ask your contributors to join their respective organizations.
- Ask contributors to [join their respective organizations](https://docs.github.com/en/organizations/managing-membership-in-your-organization/inviting-users-to-join-your-organization), if they have not already. Otherwise, there is no remediation for this check; it simply provides insight into which organizations have contributed so that you can make a trust-based decision based on that information.
## Dependency-Update-Tool
This check tries to determine if the project uses a dependency update tool.
Not updating dependencies makes a project vulnerable to known flaws and prone to attacks. A low score is therefore considered `High` risk.
The checks looks for [dependabot](https://dependabot.com/docs/config-file/) or [renovatebot](https://docs.renovatebot.com/configuration-options/). This check only looks if it is enabled and does not ensure that it is run and pull requests are merged. A project may meet this criterion yet have a failing scorecard report; there are many ways to implement this criterion, and it's challenging for an automated tool (like scorecard) to detect them all.
Risk: `High` (possibly vulnerable to attacks on known flaws)
This check tries to determine if the project uses a dependency update tool,
specifically [dependabot](https://dependabot.com/docs/config-file/) or
[renovatebot](https://docs.renovatebot.com/configuration-options/). Out-of-date
dependencies make a project vulnerable to known flaws and prone to attacks.
These tools automate the process of updating dependencies by scanning for
outdated or insecure requirements, and opening a pull request to update them if
found.
This check can determine only whether the dependency update tool is enabled; it
does not ensure that the tool is run or that the tool's pull requests are
merged.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement dependency updates,
and it is challenging for an automated tool like Scorecard to detect them all. A
low score is therefore not a definitive indication that the project is at risk.
**Remediation steps**
- Signup for automatic dependency updates with dependabot or renovatebot and place the config file in the locations that are recommended by these tools.
- Signup for automatic dependency updates with [dependabot](https://dependabot.com/docs/config-file/) or [renovatebot](https://docs.renovatebot.com/configuration-options/) and place the config file in the locations that are recommended by these tools.
## Fuzzing
This check tries to determine if the project uses fuzzing.
Fuzzing is important to reduce the number of vulnerabilities in code. A low score is considered 'Medium' risk.
The check currently works by checking if the repo name is in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project list. A project may meet this criterion yet have a failing scorecard report; there are many ways to implement this criterion, and it's challenging for an automated tool (like scorecard) to detect them all.
Risk: `Medium` (possible vulnerabilities in code)
This check tries to determine if the project uses
[fuzzing](https://owasp.org/www-community/Fuzzing) by checking if the repository
name is included in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project
list.
Fuzzing, or fuzz testing, is the practice of feeding unexpected or random data
into a program to expose bugs. Regular fuzzing is important to detect
vulnerabilities that may be exploited by others, especially since attackers can
also use fuzzing to find the same flaws.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement fuzzing, and it is
challenging for an automated tool like Scorecard to detect them all. A low score
is therefore not a definitive indication that the project is at risk.
**Remediation steps**
- Integrate the project with OSS-Fuzz by following the instructions [here](https://google.github.io/oss-fuzz/).
## Maintained
This check tries to determine if the project is "actively maintained".
A project which is not active might not be patched, might not have its dependencies patched, or might not be actively tested and used. A low score is therefore considered `High` risk.
A lack of active maintenance is not necessarily a problem. Some software, especially smaller utility functions, does not normally need to be maintained. For example, a library that determines if an integer is even would not normally need maintenance (it might only need to be modified if its underlying implementation language definition changed, e.g., by adding a new type). A lack of active maintenance simply suggests that potential users should investigate further.
The check currently works by looking whether the repo is archived or not. If it is archived, it returns the minimum score. If it is not, the check looks for commits within the last 90 days, and outputs the highest score if there are at least 1 commit/week during this period.
Risk: `High` (possibly unpatched vulnerabilities)
This check determines whether the project is actively maintained. If the project
is archived, it receives the lowest score. If there is at least one commit per
week during the previous 90 days, the project receives the highest score.
A project which is not active might not be patched, have its
dependencies patched, or be actively tested and used. However, a lack
of active maintenance is not necessarily always a problem. Some software,
especially smaller utility functions, does not normally need to be maintained.
For example, a library that determines if an integer is even would not normally
need maintenance unless an underlying implementation language definition
changed. A lack of active maintenance should signal that potential users should
investigate further to judge the situation.
**Remediation steps**
- There is *NO* remediation work necessarily needed here from projects; this just indicates the project activity and maintenance commitment. External users should determine if this is the kind of software that would not normally need active maintenance.
- There is no remediation work needed from projects with a low score; this check simply provides insight into the project activity and maintenance commitment. External users should determine whether the software is the type that would not normally need active maintenance.
## Packaging
This check tries to determine if the project is published as a package that others can easily download, install, easily update, and uninstall.
It's important that the project provide an easy way to download, install, update, and uninstall the software. It's particularly important to make it easy for users to receive security patches as updates.
This is often done by creating a "package" that is easy to install and uninstall by a package manager. Many program language ecosystems have a generally-used packaging format supported by a language-level package manager tool and public package repository. Many operating system platforms also have at least one package format, tool, and public repository (in some cases the source repository generates system-independent source packages, which are then used by others to generate system executable packages). Container images are yet another way to package software. In some situations packaging is not sensible, but it's wise to package software in so many circumstances that it's worth checking for it.
A low score is considered `Medium` risk.
The check currently looks for [GitHub packaging workflows]( https://docs.github.com/en/packages/learn-github-packages/publishing-a-package) and language-specific GitHub Actions that upload the package to a corresponding hub, e.g., [Npm](https://www.npmjs.com/). It does not currently support other source hosting repositories (forges). There is a plan to add better support to query package manager hubs directly in the future, e.g., for [Npm](https://www.npmjs.com/), [PyPi](https://pypi.org/). A project may meet this criterion yet have a failing scorecard report; some widely-used tools are relatively easy to detect, but it's challenging for an automated tool (like scorecard) to detect them all. If scorecard fails to detect the way you publish a package and you think scorecard should support your use case, please [open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
Risk: `Medium` (users possibly missing security updates)
This check tries to determine if the project is published as a package. It is
currently limited to repositories hosted on GitHub, and does not support other
source hosting repositories (i.e., Forges).
Packages give users of a project an easy way to download, install, update, and
uninstall the software by a package manager. In particular, they make it easy
for users to receive security patches as updates.
The check currently looks for
[GitHub packaging workflows](https://docs.github.com/en/packages/learn-github-packages/publishing-a-package)
and language-specific GitHub Actions that upload the package to a corresponding
hub, e.g., [Npm](https://www.npmjs.com/). We plan to add better support to query
package manager hubs directly in the future, e.g., for
[Npm](https://www.npmjs.com/), [PyPi](https://pypi.org/).
You can create a package in several ways:
- Many program language ecosystems have a generally-used packaging format
supported by a language-level package manager tool and public package
repository.
- Many operating system platforms also have at least one package format,
tool, and public repository (in some cases the source repository generates
system-independent source packages, which are then used by others to
generate system executable packages).
- Using container images.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to package software, and it is
challenging for an automated tool like Scorecards to detect them all. A low
score is therefore not a definitive indication that the project is at risk. If
Scorecards fails to detect the way you publish a package and you think we should
support your use case, please let us know by [opening an
issue](https://github.com/ossf/scorecard/issues/new/choose).
**Remediation steps**
- Publish your project as a downloadable package, e.g., if hosted on GitHub, use [GitHub's mechanisms for publishing a package]](https://docs.github.com/en/packages/learn-github-packages/publishing-a-package).
- Publish your project as a downloadable package, e.g., if hosted on GitHub, use [GitHub's mechanisms for publishing a package](https://docs.github.com/en/packages/learn-github-packages/publishing-a-package).
- If hosted on GitHub, use a GitHub action to release your package to language-specific hubs.
## Pinned-Dependencies
This check tries to determine if the project is an application that has declared and pinned its dependencies. A "pinned dependency" is a dependency that is explicitly set to a specific version instead of allowing a range of versions. If this project is a library (not an application), this check should automatically pass (but see limitations below).
It's important for applications to pin dependencies to ensure that checking and deployment are all done with the same software, reducing deployment risks, simplifying debugging, and enabling reproducibility. In some cases pinning dependencies can help mitigate compromised dependencies from undermining the security of the project (in the case where you've evaluated the pinned dependency, you are confident it's not compromised, and a later version is released that is compromised). In particular, pinning dependencies is one way to [counter dependency confusion (aka substitution) attacks](https://azure.microsoft.com/en-us/resources/3-ways-to-mitigate-risk-using-private-package-feeds/). In a dependency confusion attack, an application uses multiple feeds to acquire software packages (a "hybrid configuration"), and attackers can fool the user into using a malicious package via a feed that was not expected for that package. A risk of pinning dependencies is that it can inhibit software updates (e.g., because of a security vulnerability or because the pinned version is compromised); this can be mitigated by [having applications and *not* libraries pin to specific versions](https://jbeckwith.com/2019/12/18/package-lock/), using automated tools to notify applications when their dependencies are outdated, and by applications that *do* pin dependencies update quickly. Low score is therefore considered `Medium` risk.
The checks works by (1) looking for the following files in the root directory: go.mod, go.sum (Golang), package-lock.json, npm-shrinkwrap.json (Javascript), requirements.txt, pipfile.lock (Python), gemfile.lock (Ruby), cargo.lock (Rust), yarn.lock (package manager), composer.lock (PHP), vendor/, third_party/, third-party/; (2) looks for unpinned dependencies in Dockerfiles, shell scripts and GitHub workflows. It does not currently support other source hosting repositories (forges) other than GitHub.
*Limitations:* This check should only apply to applications, as libraries shouldn't normally have enforced pinned dependencies. Unfortunately, Scorecard currently can't detect if a project is a library or application. Even when [application detection is added](https://github.com/ossf/scorecard/issues/689), it's always possible for an automated tool like Scorecard to incorrectly categorize software (especially in projects that include both libraries and applications).
If the project is hosted on GitHub, You can learn more about dependencies for projects using [GitHub dependency graph](https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph).
Risk: `Medium` (possible compromised dependencies)
This check tries to determine if the project is an application that has declared
and pinned its dependencies. A "pinned dependency" is a dependency that is
explicitly set to a specific version instead of allowing a range of versions. It
is currently limited to repositories hosted on GitHub, and does not support
other source hosting repositories (i.e., Forges). If this project is a library
(not an application), this check should automatically pass (but see limitations
below).
The check works by looking for:
- the following files in the root directory: go.mod, go.sum (Golang),
package-lock.json, npm-shrinkwrap.json (Javascript), requirements.txt,
pipfile.lock (Python), gemfile.lock (Ruby), cargo.lock (Rust), yarn.lock
(package manager), composer.lock (PHP), vendor/, third_party/, third-party/;
- unpinned dependencies in Dockerfiles, shell scripts and GitHub workflows.
Pinned dependencies reduce several security risks:
- They ensure that checking and deployment are all done with the same
software, reducing deployment risks, simplifying debugging, and enabling
reproducibility.
- They can help mitigate compromised dependencies from undermining the
security of the project (in the case where you've evaluated the pinned
dependency, you are confident it's not compromised, and a later version is
released that is compromised).
- They are one way to
[counter dependency confusion (aka substitution) attacks](https://azure.microsoft.com/en-us/resources/3-ways-to-mitigate-risk-using-private-package-feeds/),
in which an application uses multiple feeds to acquire software packages (a
"hybrid configuration"), and attackers fool the user into using a malicious
package via a feed that was not expected for that package.
However, pinning dependencies can inhibit software updates, either because of a
security vulnerability or because the pinned version is compromised. Mitigate
this risk by:
- [having applications and _not_ libraries pin to specific versions](https://jbeckwith.com/2019/12/18/package-lock/);
- using automated tools to notify applications when their dependencies are
outdated;
- quickly updating applications that do pin dependencies.
_Limitations:_ This check should apply only to applications, as libraries
shouldn't normally have enforced pinned dependencies. Unfortunately, Scorecards
currently can't detect if a project is a library or application. Even when
[application detection is added](https://github.com/ossf/scorecard/issues/689),
it's always possible for an automated tool like Scorecards to incorrectly
categorize software (especially in projects that include both libraries and
applications). For projects hosted on GitHub, you can learn more about
dependencies using the
[GitHub dependency graph](https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph).
**Remediation steps**
- First, determine if your project is producing a library or application. If it is a library, then generally you don't want to pin dependencies of library users. If your project is producing an application, consider the remediations below.
- Declare all your dependencies with specific versions in your package format file (e.g. `package.json` for npm, `requirements.txt` for python). For C/C++, check in the code from a trusted source and add a `README` on the specific version used (and the archive SHA hashes).
- First determine if your project is producing a library or application. If it is a library, you generally don't want to pin dependencies of library users, and should not follow any remediation steps.
- If your project is producing an application, declare all your dependencies with specific versions in your package format file (e.g. `package.json` for npm, `requirements.txt` for python). For C/C++, check in the code from a trusted source and add a `README` on the specific version used (and the archive SHA hashes).
- If the package manager supports lock files (e.g. `package-lock.json` for npm), make sure to check these in the source code as well. These files maintain signatures for the entire dependency tree and saves from future exploitation in case the package is compromised.
- For Dockerfiles and GitHub workflows, pin dependencies by hash. See example [main.yaml](https://github.com/ossf/scorecard/blob/f55b86d6627cc3717e3a0395e03305e81b9a09be/.github/workflows/main.yml#L27) and [Dockerfile](https://github.com/ossf/scorecard/blob/main/cron/worker/Dockerfile) examples.
- To help update your dependencies after pinning them, use tools such as Github's [dependabot](https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/) or [renovate bot](https://github.com/renovatebot/renovate).
- For Dockerfiles and GitHub workflows, pin dependencies by hash. See [main.yaml](https://github.com/ossf/scorecard/blob/f55b86d6627cc3717e3a0395e03305e81b9a09be/.github/workflows/main.yml#L27) and [Dockerfile](https://github.com/ossf/scorecard/blob/main/cron/worker/Dockerfile) for examples.
- To help update your dependencies after pinning them, use tools such as
Github's
[dependabot](https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/) or [renovate bot](https://github.com/renovatebot/renovate).
## SAST
This check tries to determine if the project uses static code analysis.
SAST tool may prevent known classes of bugs to be inadvertently introduced in the codebase. A low score is considered `Medium` risk.
The checks currently looks for known Github apps such as [github-code-scanning](https://securitylab.github.com/tools/codeql) (codeql) and sonarcloud in the recent (~30) merged PRs. The check also looks for the use of "github/codeql-action" in a GitHub workflow. It does not currently support other source hosting repositories (forges) other than GitHub. A project may meet this criterion yet have a failing scorecard report; there are many ways to implement this criterion and it's especially difficult for an automated tool (like scorecard) to detect them all.
Risk: `Medium` (possible unknown bugs)
This check tries to determine if the project uses Static Application Security
Testing (SAST), also known as
[static code analysis](https://owasp.org/www-community/controls/Static_Code_Analysis).
It is currently limited to repositories hosted on GitHub, and does not support
other source hosting repositories (i.e., Forges).
SAST is testing run on source code before the application is run. Using SAST
tools can prevent known classes of bugs from being inadvertently introduced in the
codebase.
The checks currently looks for known Github apps such as
[CodeQL](https://codeql.github.com/) (github-code-scanning) and
[SonarCloud](https://sonarcloud.io/) in the recent (~30) merged PRs, or the use
of "github/codeql-action" in a GitHub workflow.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement SAST, and it is
challenging for an automated tool like Scorecard to detect them all. A low score
is therefore not a definitive indication that the project is at risk.
**Remediation steps**
- Run CodeQL checks in your CI/CD by following the instructions [here](https://github.com/github/codeql-action#usage).
## Security-Policy
This check tries to determine if the project has published a security policy. It works by looking for a file named `SECURITY.md` (case-insensitive) in a few well-known directories.
Risk: `Medium` (possible insecure reporting of vulnerabilities)
This check tries to determine if the project has published a security policy. It
works by looking for a file named `SECURITY.md` (case-insensitive) in a few
well-known directories.
A security policy (typically a `SECURITY.md` file) can give users information
about what constitutes a vulnerability and how to report one securely so that
information about a bug is not publicly visible.
**Remediation steps**
- Place a security policy file `SECURITY.md` in the root directory of your repository. This makes it easily discoverable by a vulnerability reporter.
- The file should contain information on what constitutes a vulnerability and a way to report it securely (e.g. issue tracker with private issue support, encrypted email with a published public key).
- For GitHub, see more information [here](https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.).
## Signed-Releases
This check tries to determine if the project cryptographically signs release artifacts.
Signed releases attest to the provenance of the artifact. A low score is considered 'High' risk.
It works by looking for filenames: *.minisig (https://github.com/jedisct1/minisign), *.asc (pgp), *.sign. for the last 5 releases if it's hosted on GitHub. The check does not verify the signatures. It does not currently support other source hosting repositories (forges) other than GitHub.
Risk: `High` (possibility of installing malicious releases)
This check tries to determine if the project cryptographically signs release
artifacts. It is currently limited to repositories hosted on GitHub, and does
not support other source hosting repositories (i.e., Forges).
Signed releases attest to the provenance of the artifact.
This check looks for the following filenames in the project's last five
releases: [*.minisig ](https://github.com/jedisct1/minisign), *.asc (pgp),
*.sign.
Note: The check does not verify the signatures.
**Remediation steps**
- Publish the release.
@ -166,17 +486,38 @@ It works by looking for filenames: *.minisig (https://github.com/jedisct1/minisi
## Token-Permissions
This check tries to determine if the project's automated workflows follow the principle of least privilege, i.e. if the tokens are set read-only by default.
Attackers may use a compromised token with write access to push malicious code into the project. A low score is therefore considered `High` risk.
The check looks at the GitHub workflows. For each workflow yaml file, for the permission definitions. To obtain the highest score, the permissions should be set as read-only at the [top level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#permissions) and the required write permissions should be declared at the [run-level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idpermissions). The check cannot detect if the "read-only" GitHub permission settings is enabled, as there is no API available. It does not currently support other source hosting repositories (forges) other than GitHub.
Risk: `High` (vulnerable to malicious code additions)
This check determines whether the project's automated workflows tokens are set
to read-only by default. It is currently limited to repositories hosted on
GitHub, and does not support other source hosting repositories (i.e., Forges).
Setting token permissions to read-only follows the principle of least privilege.
This is important because attackers may use a compromised token with write
access to push malicious code into the project.
The highest score is awarded when the permissions definitions in each workflow's
yaml file are set as read-only at the
[top level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#permissions)
and the required write permissions are declared at the
[run-level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idpermissions).
The check cannot detect if the "read-only" GitHub permission setting is
enabled, as there is no API available.
**Remediation steps**
- Set permissions as `read-all` or `contents: read` as described in GitHub's [documentation](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#permissions).
## Vulnerabilities
This check determines if the project has open, unfixed vulnerabilities using the [OSV](https://osv.dev) service.
An existing vulnerability is can readily be used by attackers, so a low score is considered `High` risk.
Risk: `High` (known vulnerabilities)
This check determines whether the project has open, unfixed vulnerabilities
using the [OSV (Open Source Vulnerabilities)](https://osv.dev/) service. An open
vulnerability is readily exploited by attackers and should be fixed as soon as
possible.
**Remediation steps**
- Fix the vulnerabilities. The details of each vulnerability can be found on <https://osv.dev>.

802
docs/checks/internal/checks.yaml
View File

@ -19,101 +19,102 @@ checks:
risk: High
tags: supply-chain, security
short: Determines if the project is "actively maintained".
description: >-
This check tries to determine if the project is "actively maintained".
description: |
Risk: `High` (possibly unpatched vulnerabilities)
This check determines whether the project is actively maintained. If the project
is archived, it receives the lowest score. If there is at least one commit per
week during the previous 90 days, the project receives the highest score.
A project which is not active might not be patched, might not have its
dependencies patched, or might not be actively tested and used.
A low score is therefore considered `High` risk.
A lack of active maintenance is not necessarily a problem.
Some software, especially smaller utility functions, does not
normally need to be maintained. For example, a library that
determines if an integer is even would not normally need maintenance
(it might only need to be modified if its underlying implementation
language definition changed, e.g., by adding a new type). A lack
of active maintenance simply suggests that potential users
should investigate further.
The check currently works by looking whether the repo is archived or not.
If it is archived, it returns the minimum score. If it is not,
the check looks for commits within the last 90 days, and
outputs the highest score if there are at least 1 commit/week during this period.
A project which is not active might not be patched, have its
dependencies patched, or be actively tested and used. However, a lack
of active maintenance is not necessarily always a problem. Some software,
especially smaller utility functions, does not normally need to be maintained.
For example, a library that determines if an integer is even would not normally
need maintenance unless an underlying implementation language definition
changed. A lack of active maintenance should signal that potential users should
investigate further to judge the situation.
remediation:
- >-
There is *NO* remediation work necessarily needed here from projects;
this just indicates the project activity and maintenance commitment.
External users should determine if this is the
kind of software that would not normally need active maintenance.
There is no remediation work needed from projects with a low score; this
check simply provides insight into the project activity and maintenance
commitment. External users should determine whether the software is the type
that would not normally need active maintenance.
Dependency-Update-Tool:
risk: High
tags: supply-chain, security, dependencies
short: Determines if the project uses a dependency update tool.
description: >-
This check tries to determine if the project uses a dependency update tool.
description: |
Risk: `High` (possibly vulnerable to attacks on known flaws)
Not updating dependencies makes a project vulnerable to known flaws and prone to attacks.
A low score is therefore considered `High` risk.
This check tries to determine if the project uses a dependency update tool,
specifically [dependabot](https://dependabot.com/docs/config-file/) or
[renovatebot](https://docs.renovatebot.com/configuration-options/). Out-of-date
dependencies make a project vulnerable to known flaws and prone to attacks.
These tools automate the process of updating dependencies by scanning for
outdated or insecure requirements, and opening a pull request to update them if
found.
The checks looks for [dependabot](https://dependabot.com/docs/config-file/) or
[renovatebot](https://docs.renovatebot.com/configuration-options/). This check only looks if
it is enabled and does not ensure that it is run and pull requests are merged.
A project may meet this criterion yet have a failing scorecard report;
there are many ways to implement this criterion, and it's challenging
for an automated tool (like scorecard) to detect them all.
This check can determine only whether the dependency update tool is enabled; it
does not ensure that the tool is run or that the tool's pull requests are
merged.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement dependency updates,
and it is challenging for an automated tool like Scorecard to detect them all. A
low score is therefore not a definitive indication that the project is at risk.
remediation:
- >-
Signup for automatic dependency updates with dependabot or renovatebot and place the config
file in the locations that are recommended by these tools.
Signup for automatic dependency updates with
[dependabot](https://dependabot.com/docs/config-file/) or
[renovatebot](https://docs.renovatebot.com/configuration-options/) and place
the config file in the locations that are recommended by these tools.
Binary-Artifacts:
risk: High
tags: supply-chain, security, dependencies
short: Determines if the project has generated executable (binary) artifacts in the source repository.
description: >-
This check tries to determine if the project has generated executable (binary) artifacts in the source repository.
description: |
Risk: `High` (non-reviewable code)
Including generated executables in the source repository
increases user risks.
Many programming language systems can generate
executables from source code (e.g., C/C++ generated machine code,
Java `.class` files, Python `.pyc` files, and minified JavaScript).
Users will often directly use executables if they are included in the
source repository, leading to many dangerous behaviors.
One problem is that reviews become ineffective and thus enable the
use of obsolete or maliciously-subverted executables.
Reviews generally review source code, not executables, since
it's difficult to audit executables to ensure that they correspond.
Since it's hard to audit this, over time
the included executables might not correspond to the source code.
Another problem is that it allows the executable generation process to
atrophy, which can lead to an inability to create working
executables. These problems can be countered with verified
reproducible builds, but it's easier to implement verified
reproducible builds when executables are not included in the
source repository (since the executable generation process is less
likely to have atrophied).
This check determines whether the project has generated executable (binary)
artifacts in the source repository.
It's fine to include, in the source repository,
files that are simultaneously reviewable source code and executables,
since these are reviewable.
(Some interpretive systems, such as many operating system shells,
don't have a mechanism for storing generated executables that are
different from the source file.)
Similarly, we allow including in the source repository
source code generated by other tools
(e.g., by bison, yacc, flex, and lex).
There are potential downsides to generated source code, but
generated source code tends to be much easier to review and thus
presents a lower risk. Generated source code is also often difficult
for external tools to detect.
Including generated executables in the source repository increases user risk.
Many programming language systems can generate executables from source code
(e.g., C/C++ generated machine code, Java `.class` files, Python `.pyc` files,
and minified JavaScript). Users will often directly use executables if they are
included in the source repository, leading to many dangerous behaviors.
We allow including generated documentation in source repositories.
Generated documentation is intended for use by humans
(not computers), and humans can take context into account.
Thus, generated documentation doesn't pose the same level of risk.
Problems with generated executable (binary) artifacts:
- Binary artifacts cannot be reviewed, allowing possible obsolete or
maliciously subverted executables. Reviews generally review source code, not
executables, since it's difficult to audit executables to ensure that they
correspond to the source code. Over time the included executables might not
correspond to the source code.
- Generated executables allow the executable generation process to atrophy,
which can lead to an inability to create working executables. These problems
can be countered with verified reproducible builds, but it's easier to
implement verified reproducible builds when executables are not included in
the source repository (since the executable generation process is less
likely to have atrophied).
Allowed by Scorecards:
- Files in the source repository that are simultaneously reviewable source
code and executables, since these are reviewable. (Some interpretive
systems, such as many operating system shells, don't have a mechanism for
storing generated executables that are different from the source file.)
- Source code in the source repository generated by other tools (e.g., by
bison, yacc, flex, and lex). There are potential downsides to generated
source code, but generated source code tends to be much easier to review and
thus presents a lower risk. Generated source code is also often difficult
for external tools to detect.
- Generated documentation in source repositories. Generated documentation is
intended for use by humans (not computers) who can evaluate the context.
Thus, generated documentation doesn't pose the same level of risk.
A low score is therefore considered `High` risk.
remediation:
- >-
Remove the generated executable artifacts from the repository.
@ -123,44 +124,49 @@ checks:
risk: High
tags: supply-chain, security, source-code, code-reviews
short: Determines if the default and release branches are protected with GitHub's branch protection settings.
description: >-
[Branch protection](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches)
can enable various rules to enforce certain workflows for
branches, such as preventing rewriting of public history
(e.g., a *force push*),
requiring review before acceptance into a main branch,
or passing certain status checks before acceptance into a main branch.
description: |
Risk: `High` (vulnerable to intentional malicious code injection)
This check determines whether a project's default and release branches are
protected with GitHub's
[branch protection](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches)
settings. Branch protection allows maintainers to define rules that enforce
certain workflows for branches, such as requiring review or passing certain
status checks before acceptance into a main branch, or preventing rewriting of
public history.
Branch protection can reduce the risk of unintentional or malicious
code from entering the "main" branch.
Branch protection rules that prevent rewriting of public history
(e.g., preventing *force push* of public branches) prevent history
from changing without external notice.
Branch protection rules that require status checks ensure that
at least those checks are met before a change is accepted.
Branch protection rules that require at least one other reviewer
reviews greatly reduces the risk that a compromise of a contributor's
account will lead to injection of malicious code. Review also increases
the likelihood that an unintentional vulnerability in a contribution
will be detected and fixed before the change is accepted.
A low score is therefore considered `High` risk.
Note: The following settings queried by the Branch-Protection check require an admin token: `DismissStaleReviews`, `EnforceAdmin`, and `StrictStatusCheck`. If
the provided token does not have admin access, the check will query the branch
settings accessible to non-admins and provide results based only on these settings.
Even so, we recommend using a non-admin token, which provides a thorough enough
result to meet most user needs.
Note, however, that requiring reviews for every proposed change
is impractical for many projects, since many projects simply don't
have that many active participants. For more discussion, see
[Code Reviews](#code-reviews).
Different types of branch protection protect against different risks:
In some cases these rules will need to be suspended.
For example, if a past commit includes illegal content
(such as child pornography), it may be impractical to hide the commit
and in such cases the history may need to be rewritten.
- Require code review: requires at least one reviewer, which greatly
reduces the risk that a compromised contributor can inject malicious code.
Review also increases the likelihood that an unintentional vulnerability in
a contribution will be detected and fixed before the change is accepted.
- Prevent force push: prevents use of the `--force` command on public
branches, which overwrites code irrevocably. This protection prevents the
rewriting of public history without external notice.
- Require
[status checks](https://docs.github.com/en/github/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks):
ensures that all required CI tests are met before a change is accepted.
Although requiring code review can greatly reduce the chance that
unintentional or malicious code enters the "main" branch, it is not feasible for
all projects, such as those that don't have many active participants. For more
discussion, see
[Code Reviews](https://github.com/ossf/scorecard/blob/main/docs/checks.md#code-reviews).
Additionally, in some cases these rules will need to be suspended. For example,
if a past commit includes illegal content such as child pornography, it may be
necessary to use a force push to rewrite the history rather than simply hide the
commit.
This check determines if the default and release branches are
protected with GitHub's branch protection settings.
The check only works when the token has [Admin
access](https://github.community/t/enable-branch-protection-get-api-without-admin/14197)
to the repository. This check determines if the default and release branches are
protected.
remediation:
- >-
Enable branch protection settings in your source hosting provider to
@ -172,24 +178,28 @@ checks:
risk: Low
tags: supply-chain, testing
short: Determines if the project runs tests before pull requests are merged.
description: >-
This check tries to determine if the project runs tests before pull
requests are merged.
description: |
Risk: `Low` (possible unknown vulnerabilities)
Running tests helps developers catch mistakes early on.
A low score is considered 'Low' risk.
This check tries to determine if the project runs tests before pull requests are
merged. It is currently limited to repositories hosted on GitHub, and does not
support other source hosting repositories (i.e., Forges).
Running tests helps developers catch mistakes early on, which can reduce the
number of vulnerabilities that find their way into a project.
The check works by looking for a set of CI-system names in GitHub `CheckRuns`
and `Statuses` among the recent commits (~30). A CI-system is considered
well-known if its name contains any of the following: appveyor, buildkite,
circleci, e2e, github-actions, jenkins, mergeable, test, travis-ci.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement CI testing, and it is
challenging for an automated tool like Scorecard to detect them all. A low score
is therefore not a definitive indication that the project is at risk.
The check works by looking for a set of CI-system
names in GitHub `CheckRuns` and `Statuses` among the recent commits (~30).
A CI-system is considered well-known if its name contains any of the
following: appveyor, buildkite, circleci, e2e, github-actions, jenkins,
mergeable, test, travis-ci.
It does not currently support other source hosting repositories (forges).
A project may meet this criterion yet have a failing scorecard report;
there are many ways to implement this criterion, and it's challenging
for an automated tool (like scorecard) to detect them all.
If a project's system was not detected and you think it should be, please
[open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
[open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
remediation:
- Check-in scripts that run all the tests in your repository.
- >-
@ -201,42 +211,37 @@ checks:
risk: Low
tags: security-awareness, security-training, security
short: Determines if the project has a CII Best Practices Badge.
description: >-
This check tries to determine if the project has earned a
[CII Best Practices Badge](https://bestpractices.coreinfrastructure.org/).
description: |
Risk: `Low` (possibly not following security best practices)
This badge tells us if the project is applying a particular
set of security-focused best development practices
for open source software.
The CII Best Practices badge has 3 tiers (passing, silver, and gold);
we give full credit if the
[passing criteria are met](https://bestpractices.coreinfrastructure.org/criteria/0),
as achieving a passing badge is a significant achievement
for many projects.
We give a little credit if the project is at least working to achieve
a badge, and increasingly more as more criteria are met.
This check determines whether the project has earned a [CII Best Practices
Badge](https://bestpractices.coreinfrastructure.org/), which indicates that the
project uses a set of security-focused best development practices for open
source software. The check uses the URL for the Git repo and the CII API.
For example, to earn the passing badge,
the project MUST publish the process for
reporting vulnerabilities on the project site,
it MUST provide a working build system that can automatically rebuild
the software from source code (where applicable),
it MUST have a general policy that tests
will be added to an automated test suite
when major new functionality is added,
it MUST meet various cryptography criteria where applicable,
it MUST have at least one primary developer who knows how to
design secure software,
at least one of the project's primary developers MUST know of
common kinds of errors that lead to vulnerabilities in this kind
of software (as well as at least one method to counter or mitigate
each of them), and at least
one static code analysis tool (beyond compiler warnings and "safe"
language modes) MUST be applied to any proposed major production
release.
The CII Best Practices badge has 3 tiers: passing, silver, and gold. We give
full credit to projects that meet the [passing
criteria](https://bestpractices.coreinfrastructure.org/criteria/0), which is a
significant achievement for many projects. Lower scores represent a project that
is at least working to achieve a badge, with increasingly more points awarded as
more criteria are met.
A low score is considered 'Low' risk.
The check uses the URL for the Git repo and the CII API.
To earn the passing badge, the project MUST:
- publish the process for reporting vulnerabilities on the project site
- provide a working build system that can automatically rebuild the software
from source code (where applicable)
- have a general policy that tests will be added to an automated test suite
when major new functionality is added
- meet various cryptography criteria where applicable
- have at least one primary developer who knows how to design secure software
- have at least one primary developer who knows of common kinds of errors
that lead to vulnerabilities in this kind of software (and at least one
method to counter or mitigate each of them)
- apply at least one static code analysis tool (beyond compiler warnings and
"safe" language modes) to any proposed major production release.
Some of these criteria overlap with other Scorecards checks.
remediation:
- >-
Sign up for the [CII Best Practices
@ -245,119 +250,110 @@ checks:
risk: High
tags: supply-chain, security, source-code, code-reviews
short: Determines if the project requires code review before pull requests (aka merge requests) are merged.
description: >-
This check tries to determine if the project requires code review before
pull requests (aka merge requests) are merged.
description: |
Risk: `High` (unintentional vulnerabilities or possible injection of malicious
code)
Reviewing code improves the quality of code in general,
because reviews may detect various unintentional
problems that can be fixed immediately before they are merged.
Such problems include unintentional vulnerabilities, so unintentional
vulnerabilities can be reduced through review.
Reviews also make it
more difficult for an attacker to insert malicious code
(either as a malicious contributor or as an attacker who has
subverted a contributor's account), because a reviewer might detect
the subversion.
This increased difficulty can even deter attackers.
A low score is therefore considered `High` risk.
This check determines whether the project requires code review before pull
requests (merge requests) are merged.
However, requiring that all proposed changes be reviewed by someone
else before merging them is impractical for some projects.
A project with only one active participant cannot practically
enforce multi-person review, and even a project with multiple
active participants may not have enough active participation to be able
to require review of all proposed changes.
Such projects will not be able to practically enforce rules such as
requiring a minimum number of reviewers, and administrators in
practice will be exempt from reviews.
Projects with a small number of active participants, but more than one,
instead sometimes aim for a review of a percentage of proposals
(e.g., "at least half of all proposed changes are reviewed").
Reviews detect various unintentional problems, including vulnerabilities that
can be fixed immediately before they are merged, which improves the quality of
the code. Reviews may also detect or deter an attacker trying to insert
malicious code (either as a malicious contributor or as an attacker who has
subverted a contributor's account), because a reviewer might detect the
subversion.
In addition, note that requiring review does not eliminate all risks.
The other reviewer(s) might fail to notice unintentional vulnerabilities
or malicious code. They might be colluding with a malicious developer.
The "other" reviewers might even be the same person
(aka a "sock puppet").
The check first tries to detect whether [Branch-Protection](checks.md#branch-protection) is enabled on the
default branch with at least one required reviewer. If this fails, the check
determines whether the most recent (~30) commits have a Github-approved review
or if the merger is different from the committer (implicit review). It also
performs a similar check for reviews using
[Prow](https://github.com/kubernetes/test-infra/tree/master/prow#readme) (labels
"lgtm" or "approved") and [Gerrit](https://www.gerritcodereview.com/) ("Reviewed-on" and "Reviewed-by").
In short, requiring others' review before accepting a change
reduces risks to users,
as those additional reviewers may detect problems early,
but it is not a practical requirement for some projects.
Note: Requiring reviews for all changes is infeasible for some projects, such as
those with only one active participant. Even a project with multiple active
contributors may not have enough active participation to be able to require
review of all proposed changes. Projects with a small number of active
participants instead sometimes aim for a review of a
percentage of proposals (e.g., "at least half of all proposed changes are
reviewed").
Requiring review does not eliminate all risks. The other reviewers might fail to
notice unintentional vulnerabilities or malicious code, be colluding with a
malicious developer, or even be the same person (using a "[sock
puppet](https://en.wikipedia.org/wiki/Sock_puppet_account)" account).
The check first tries to detect if Branch-Protection is enabled
on the default branch ,and the number of reviewers is at least 1. If this
fails, it checks if the recent (~30) commits have a Github-approved
review or if the merger is different from the committer (implicit review).
It also performs similar check for reviews using
[Prow](https://github.com/kubernetes/test-infra/tree/master/prow#readme)
(labels "lgtm" or "approved") and Gerrit ("Reviewed-on" and "Reviewed-by").
remediation:
- >-
If the project has only one contributor, or does not have enough
reviewers to practically require that all contributions be reviewed,
try to recruit more maintainers to the project who will be
willing to review others' work.
Ideally at least some of these people will
be from different organizations.
If the project has very limited utility, consider expanding its
intended utility so more people will be interested in improving
it, and make that larger scope clear to potential contributors.
reviewers to practically require that all contributions be reviewed, try to
recruit more maintainers to the project who will be willing to review
others' work. Ideally at least some of these people will be from different
organizations (see [Contributors](checks.md#contributors)). If the project has very limited utility, consider expanding
its intended utility so more people will be interested in improving it, and
make that larger scope clear to potential contributors.
- >-
Follow security best practices by performing strict code reviews for
every new pull request / merge request.
Follow security best practices by performing strict code reviews for every
new pull request / merge request.
- >-
Make "code reviews" mandatory in your repository configuration. E.g.
[GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging).
Make "code reviews" mandatory in your repository configuration.
([Instructions for GitHub.](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging))
- >-
Enforce the rule for administrators / code owners as well. E.g.
[GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
Enforce the rule for administrators / code owners as well.
([Instructions for GitHub.](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators))
Contributors:
risk: Low
tags: source-code
short: Determines if the project has a set of contributors from multiple organizations (e.g., companies).
description: >-
This check tries to determine if the project has a set of contributors from
multiple organizations (e.g., companies).
description: |
Risk: `Low` (lower number of trusted code reviewers)
Low score has 'Low' risk.
This check tries to determine if the project has recent contributors from
multiple organizations (e.g., companies). It is currently limited to
repositories hosted on GitHub, and does not support other source hosting
repositories (i.e., Forges).
Some projects cannot meet this requirement.
Obviously projects with only one active participant
cannot meet it.
Many projects have a narrow scope and may not be able to attract
the interest of multiple organizations.
See [Code Reviews](#code-reviews) for added notes about
projects with a small number of participants.
The check looks at the `Company` field on the GitHub user profile for authors of
recent commits. To receive the highest score, the project must have had
contributors from at least 3 different companies in the last 30 commits; each of
those contributors must have had at least 5 commits in the last 30 commits.
The check works by looking at the authors of recent commits
and checking the `Company` field on the GitHub user profile. A contributor
must have at least 5 commits in the last 30 commits.
The highest score is achieved when there are contributors from
at least 3 different companies in the last 30 commits.
It does not currently support other source hosting repositories (forges).
Note: Some projects cannot meet this requirement, such as small projects with
only one active participant, or projects with a narrow scope that cannot attract
the interest of multiple organizations. See
[Code Reviews](https://github.com/ossf/scorecard/blob/main/docs/checks.md#code-reviews)
for more information about evaluating projects with a small number of
participants.
remediation:
- >-
There is *NO* remediation work needed here. This is to provide some
insights on which organization(s) have contributed to the project and
making trust decisions based on that. But you can ask your contributors
to join their respective organizations.
Ask contributors to
[join their respective organizations](https://docs.github.com/en/organizations/managing-membership-in-your-organization/inviting-users-to-join-your-organization),
if they have not already. Otherwise, there is no remediation for this check;
it simply provides insight into which organizations have contributed so that
you can make a trust-based decision based on that information.
Fuzzing:
risk: Medium
tags: supply-chain, security, testing
short: Determines if the project uses fuzzing.
description: >-
This check tries to determine if the project uses fuzzing.
description: |
Risk: `Medium` (possible vulnerabilities in code)
Fuzzing is important to reduce the number of vulnerabilities in code.
A low score is considered 'Medium' risk.
This check tries to determine if the project uses
[fuzzing](https://owasp.org/www-community/Fuzzing) by checking if the repository
name is included in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project
list.
The check currently works by checking if the repo name is in the
[OSS-Fuzz](https://github.com/google/oss-fuzz) project list.
A project may meet this criterion yet have a failing scorecard report;
there are many ways to implement this criterion, and it's challenging
for an automated tool (like scorecard) to detect them all.
Fuzzing, or fuzz testing, is the practice of feeding unexpected or random data
into a program to expose bugs. Regular fuzzing is important to detect
vulnerabilities that may be exploited by others, especially since attackers can
also use fuzzing to find the same flaws.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement fuzzing, and it is
challenging for an automated tool like Scorecard to detect them all. A low score
is therefore not a definitive indication that the project is at risk.
remediation:
- >-
Integrate the project with OSS-Fuzz by following the instructions
@ -366,146 +362,154 @@ checks:
risk: Medium
tags: supply-chain, security, releases
short: Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.
description:
This check tries to determine if the project is published as a package that others can easily download, install, easily update, and uninstall.
description: |
Risk: `Medium` (users possibly missing security updates)
It's important that the project provide an easy way to
download, install, update, and uninstall the software.
It's particularly important to make it easy for users to
receive security patches as updates.
This check tries to determine if the project is published as a package. It is
currently limited to repositories hosted on GitHub, and does not support other
source hosting repositories (i.e., Forges).
This is often done by creating a "package" that is
easy to install and uninstall by a package manager.
Many program language ecosystems have a generally-used packaging format
supported by a language-level package manager
tool and public package repository.
Many operating system platforms also have at least one package format,
tool, and public repository (in some cases the source repository
generates system-independent source packages, which are then
used by others to generate system executable packages).
Container images are yet another way to package software.
In some situations packaging is not sensible, but it's wise to package
software in so many circumstances that it's worth checking for it.
Packages give users of a project an easy way to download, install, update, and
uninstall the software by a package manager. In particular, they make it easy
for users to receive security patches as updates.
A low score is considered `Medium` risk.
The check currently looks for
[GitHub packaging workflows](https://docs.github.com/en/packages/learn-github-packages/publishing-a-package)
and language-specific GitHub Actions that upload the package to a corresponding
hub, e.g., [Npm](https://www.npmjs.com/). We plan to add better support to query
package manager hubs directly in the future, e.g., for
[Npm](https://www.npmjs.com/), [PyPi](https://pypi.org/).
The check currently looks for [GitHub packaging workflows]( https://docs.github.com/en/packages/learn-github-packages/publishing-a-package)
and language-specific GitHub Actions that upload the package to a corresponding hub, e.g., [Npm](https://www.npmjs.com/).
It does not currently support other source hosting repositories (forges).
There is a plan to add better support to query package manager hubs directly in the future, e.g., for [Npm](https://www.npmjs.com/), [PyPi](https://pypi.org/).
A project may meet this criterion yet have a failing scorecard report;
some widely-used tools are relatively easy to detect, but it's
challenging for an automated tool (like scorecard) to detect
them all.
If scorecard fails to detect the way you publish a package and
you think scorecard should support your use case, please
[open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
You can create a package in several ways:
- Many program language ecosystems have a generally-used packaging format
supported by a language-level package manager tool and public package
repository.
- Many operating system platforms also have at least one package format,
tool, and public repository (in some cases the source repository generates
system-independent source packages, which are then used by others to
generate system executable packages).
- Using container images.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to package software, and it is
challenging for an automated tool like Scorecards to detect them all. A low
score is therefore not a definitive indication that the project is at risk. If
Scorecards fails to detect the way you publish a package and you think we should
support your use case, please let us know by [opening an
issue](https://github.com/ossf/scorecard/issues/new/choose).
remediation:
- Publish your project as a downloadable package, e.g., if hosted on GitHub, use [GitHub's mechanisms for publishing a package]](https://docs.github.com/en/packages/learn-github-packages/publishing-a-package).
- Publish your project as a downloadable package, e.g., if hosted on GitHub, use [GitHub's mechanisms for publishing a package](https://docs.github.com/en/packages/learn-github-packages/publishing-a-package).
- If hosted on GitHub, use a GitHub action to release your package to language-specific hubs.
Pinned-Dependencies:
risk: Medium
tags: supply-chain, security, dependencies
short: Determines if the project has declared and pinned its dependencies.
description: >-
This check tries to determine if the project is an application that
has declared and pinned its dependencies.
A "pinned dependency" is a dependency that is explicitly set to a
specific version instead of allowing a range of versions.
If this project is a library (not an application), this check should
automatically pass (but see limitations below).
description: |
Risk: `Medium` (possible compromised dependencies)
It's important for applications to pin dependencies
to ensure that checking and deployment are
all done with the same software, reducing deployment risks, simplifying
debugging, and enabling reproducibility.
In some cases pinning dependencies can help
mitigate compromised dependencies
from undermining the security of the project (in the case where
you've evaluated the pinned dependency, you are confident
it's not compromised,
and a later version is released that is compromised).
In particular, pinning dependencies is one way to
[counter dependency confusion (aka substitution) attacks](https://azure.microsoft.com/en-us/resources/3-ways-to-mitigate-risk-using-private-package-feeds/).
In a dependency confusion attack, an application uses multiple feeds
to acquire software packages (a "hybrid configuration"),
and attackers can fool the user into using a malicious package
via a feed that was not expected for that package.
A risk of pinning dependencies is that it can inhibit software updates
(e.g., because of a security vulnerability or because the pinned version
is compromised);
this can be mitigated by
[having applications and *not* libraries pin to specific versions](https://jbeckwith.com/2019/12/18/package-lock/),
using automated tools to notify applications when their dependencies are
outdated, and by applications that *do* pin dependencies update quickly.
Low score is therefore considered `Medium` risk.
This check tries to determine if the project is an application that has declared
and pinned its dependencies. A "pinned dependency" is a dependency that is
explicitly set to a specific version instead of allowing a range of versions. It
is currently limited to repositories hosted on GitHub, and does not support
other source hosting repositories (i.e., Forges). If this project is a library
(not an application), this check should automatically pass (but see limitations
below).
The checks works by (1) looking for the following files in the root
directory: go.mod, go.sum (Golang), package-lock.json, npm-shrinkwrap.json
(Javascript), requirements.txt, pipfile.lock (Python), gemfile.lock
(Ruby), cargo.lock (Rust), yarn.lock (package manager), composer.lock
(PHP), vendor/, third_party/, third-party/; (2) looks for
unpinned dependencies in Dockerfiles, shell scripts and GitHub workflows.
It does not currently support other source hosting repositories (forges)
other than GitHub.
The check works by looking for:
*Limitations:*
This check should only apply to applications, as
libraries shouldn't normally have enforced pinned dependencies.
Unfortunately, Scorecard currently can't detect if a project
is a library or application.
Even when
- the following files in the root directory: go.mod, go.sum (Golang),
package-lock.json, npm-shrinkwrap.json (Javascript), requirements.txt,
pipfile.lock (Python), gemfile.lock (Ruby), cargo.lock (Rust), yarn.lock
(package manager), composer.lock (PHP), vendor/, third_party/, third-party/;
- unpinned dependencies in Dockerfiles, shell scripts and GitHub workflows.
Pinned dependencies reduce several security risks:
- They ensure that checking and deployment are all done with the same
software, reducing deployment risks, simplifying debugging, and enabling
reproducibility.
- They can help mitigate compromised dependencies from undermining the
security of the project (in the case where you've evaluated the pinned
dependency, you are confident it's not compromised, and a later version is
released that is compromised).
- They are one way to
[counter dependency confusion (aka substitution) attacks](https://azure.microsoft.com/en-us/resources/3-ways-to-mitigate-risk-using-private-package-feeds/),
in which an application uses multiple feeds to acquire software packages (a
"hybrid configuration"), and attackers fool the user into using a malicious
package via a feed that was not expected for that package.
However, pinning dependencies can inhibit software updates, either because of a
security vulnerability or because the pinned version is compromised. Mitigate
this risk by:
- [having applications and _not_ libraries pin to specific versions](https://jbeckwith.com/2019/12/18/package-lock/);
- using automated tools to notify applications when their dependencies are
outdated;
- quickly updating applications that do pin dependencies.
_Limitations:_ This check should apply only to applications, as libraries
shouldn't normally have enforced pinned dependencies. Unfortunately, Scorecards
currently can't detect if a project is a library or application. Even when
[application detection is added](https://github.com/ossf/scorecard/issues/689),
it's always possible for an automated tool like Scorecard to
incorrectly categorize software
(especially in projects that include both libraries and applications).
If the project is hosted on GitHub,
You can learn more about dependencies for projects using
it's always possible for an automated tool like Scorecards to incorrectly
categorize software (especially in projects that include both libraries and
applications). For projects hosted on GitHub, you can learn more about
dependencies using the
[GitHub dependency graph](https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph).
remediation:
- >-
First determine if your project is producing a library or
application. If it is a library, you generally don't want to pin dependencies of
library users, and should not follow any remediation steps.
- >-
First, determine if your project is producing a library or application.
If it is a library, then generally you don't want to pin dependencies
of library users. If your project is producing an application,
consider the remediations below.
- >-
Declare all your dependencies with specific versions in your package
format file (e.g. `package.json` for npm, `requirements.txt` for
python). For C/C++, check in the code from a trusted source and add a
`README` on the specific version used (and the archive SHA hashes).
If your project is producing an application, declare all your dependencies with specific versions in your package
format file (e.g. `package.json` for npm, `requirements.txt` for python).
For C/C++, check in the code from a trusted source and add a `README` on the
specific version used (and the archive SHA hashes).
- >-
If the package manager supports lock files (e.g. `package-lock.json` for
npm), make sure to check these in the source code as well. These files
maintain signatures for the entire dependency tree and saves from future
exploitation in case the package is compromised.
- >-
For Dockerfiles and GitHub workflows, pin dependencies by hash. See example
For Dockerfiles and GitHub workflows, pin dependencies by hash. See
[main.yaml](https://github.com/ossf/scorecard/blob/f55b86d6627cc3717e3a0395e03305e81b9a09be/.github/workflows/main.yml#L27)
and [Dockerfile](https://github.com/ossf/scorecard/blob/main/cron/worker/Dockerfile) examples.
and
[Dockerfile](https://github.com/ossf/scorecard/blob/main/cron/worker/Dockerfile)
for examples.
- >-
To help update your dependencies after pinning them, use tools such as
Github's [dependabot](https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/)
Github's
[dependabot](https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/)
or [renovate bot](https://github.com/renovatebot/renovate).
SAST:
risk: Medium
tags: supply-chain, security, testing
short: Determines if the project uses static code analysis.
description: >-
This check tries to determine if the project uses static code analysis.
description: |
Risk: `Medium` (possible unknown bugs)
SAST tool may prevent known classes of bugs to be inadvertently
introduced in the codebase. A low score is considered `Medium` risk.
This check tries to determine if the project uses Static Application Security
Testing (SAST), also known as
[static code analysis](https://owasp.org/www-community/controls/Static_Code_Analysis).
It is currently limited to repositories hosted on GitHub, and does not support
other source hosting repositories (i.e., Forges).
The checks currently looks for known Github apps
such as [github-code-scanning](https://securitylab.github.com/tools/codeql)
(codeql) and sonarcloud in the recent (~30) merged PRs. The
check also looks for the use of "github/codeql-action" in a GitHub workflow.
It does not currently support other source hosting repositories (forges)
other than GitHub.
A project may meet this criterion yet have a failing scorecard report;
there are many ways to implement this criterion and it's especially
difficult for an automated tool (like scorecard) to detect them all.
SAST is testing run on source code before the application is run. Using SAST
tools can prevent known classes of bugs from being inadvertently introduced in the
codebase.
The checks currently looks for known Github apps such as
[CodeQL](https://codeql.github.com/) (github-code-scanning) and
[SonarCloud](https://sonarcloud.io/) in the recent (~30) merged PRs, or the use
of "github/codeql-action" in a GitHub workflow.
Note: A project that fulfills this criterion with other tools may still receive
a low score on this test. There are many ways to implement SAST, and it is
challenging for an automated tool like Scorecard to detect them all. A low score
is therefore not a definitive indication that the project is at risk.
remediation:
- >-
Run CodeQL checks in your CI/CD by following the instructions
@ -514,10 +518,16 @@ checks:
risk: Medium
short: Determines if the project has published a security policy.
tags: supply-chain, security, policy
description: >-
This check tries to determine if the project has published a security
policy. It works by looking for a file named `SECURITY.md`
(case-insensitive) in a few well-known directories.
description: |
Risk: `Medium` (possible insecure reporting of vulnerabilities)
This check tries to determine if the project has published a security policy. It
works by looking for a file named `SECURITY.md` (case-insensitive) in a few
well-known directories.
A security policy (typically a `SECURITY.md` file) can give users information
about what constitutes a vulnerability and how to report one securely so that
information about a bug is not publicly visible.
remediation:
- >-
Place a security policy file `SECURITY.md` in the root directory of your
@ -526,29 +536,39 @@ checks:
The file should contain information on what constitutes a vulnerability
and a way to report it securely (e.g. issue tracker with private issue
support, encrypted email with a published public key).
- >-
For GitHub, see more information
[here](https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.).
Signed-Releases:
risk: High
tags: supply-chain, security, releases
short: Determines if the project cryptographically signs release artifacts.
description: >-
description: |
Risk: `High` (possibility of installing malicious releases)
This check tries to determine if the project cryptographically signs release
artifacts.
artifacts. It is currently limited to repositories hosted on GitHub, and does
not support other source hosting repositories (i.e., Forges).
Signed releases attest to the provenance of the artifact.
A low score is considered 'High' risk.
It works by looking for filenames: *.minisig
(https://github.com/jedisct1/minisign), *.asc (pgp), *.sign. for the last
5 releases if it's hosted on GitHub.
The check does not verify the signatures.
It does not currently support other source hosting repositories (forges)
other than GitHub.
This check looks for the following filenames in the project's last five
releases: [*.minisig ](https://github.com/jedisct1/minisign), *.asc (pgp),
*.sign.
Note: The check does not verify the signatures.
remediation:
- Publish the release.
- Generate a signing key.
- Download the release as an archive locally.
- Sign the release archive with this key (should output a signature file).
- Attach the signature file next to the release archive.
- >-
Publish the release.
- >-
Generate a signing key.
- >-
Download the release as an archive locally.
- >-
Sign the release archive with this key (should output a signature file).
- >-
Attach the signature file next to the release archive.
- >-
If the source is hosted on GitHub, check out the steps
[here](https://wiki.debian.org/Creating%20signed%20GitHub%20releases).
@ -556,22 +576,25 @@ checks:
risk: High
tags: supply-chain, security, infrastructure
short: Determines if the project's workflows follow the principle of least privilege.
description: >-
This check tries to determine if the project's automated workflows
follow the principle of least privilege, i.e. if the tokens
are set read-only by default.
description: |
Risk: `High` (vulnerable to malicious code additions)
Attackers may use a compromised token with write access to push malicious code into the project.
A low score is therefore considered `High` risk.
This check determines whether the project's automated workflows tokens are set
to read-only by default. It is currently limited to repositories hosted on
GitHub, and does not support other source hosting repositories (i.e., Forges).
The check looks at the GitHub workflows.
For each workflow yaml file,
for the permission definitions. To obtain the highest score, the permissions should be set as
read-only at the [top level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#permissions)
and the required write permissions should be declared at the [run-level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idpermissions).
The check cannot detect if the "read-only" GitHub permission settings is enabled, as there is no API available.
It does not currently support other source hosting repositories (forges)
other than GitHub.
Setting token permissions to read-only follows the principle of least privilege.
This is important because attackers may use a compromised token with write
access to push malicious code into the project.
The highest score is awarded when the permissions definitions in each workflow's
yaml file are set as read-only at the
[top level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#permissions)
and the required write permissions are declared at the
[run-level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idpermissions).
The check cannot detect if the "read-only" GitHub permission setting is
enabled, as there is no API available.
remediation:
- >-
Set permissions as `read-all` or `contents: read` as described in
@ -580,12 +603,13 @@ checks:
risk: High
tags: supply-chain, security, vulnerabilities
short: Determines if the project has open, known unfixed vulnerabilities.
description: >-
This check determines if the project has open, unfixed
vulnerabilities using the [OSV](https://osv.dev) service.
An existing vulnerability is can readily be used by attackers,
so a low score is considered `High` risk.
description: |
Risk: `High` (known vulnerabilities)
This check determines whether the project has open, unfixed vulnerabilities
using the [OSV (Open Source Vulnerabilities)](https://osv.dev/) service. An open
vulnerability is readily exploited by attackers and should be fixed as soon as
possible.
remediation:
- >-
Fix the vulnerabilities. The details of each vulnerability can be found

12
docs/checks/internal/generate/main.go
View File

@ -48,13 +48,11 @@ func main() {
<!-- Do not edit this file manually! Edit checks.yaml instead. -->
# Check Documentation
This page contains information on how each check works and provide remediation
steps to fix the failure. All of these checks are basically "best-guesses"
currently, and operate on a set of heuristics.
They are all subject to change, and have room for improvement!
If you have ideas for things to add, or new ways to detect things,
please contribute!
This page describes each Scorecard check in detail, including scoring criteria,
remediation steps to improve the score, and an explanation of the risks
associated with a low score. The checks are continually changing and we welcome
community feedback. If you have ideas for additions or new detection techniques,
please [contribute](CONTRIBUTING.md)!
`)
if err != nil {
panic(err)