Make checks documentation machine readable. (#345)

*  Make checks documentation machine readable.

Make checks.yaml as a machine and human readable source of truth of
checks documentation.

A tiny Python script is also added to generate checks.json and checks.md
from this file.

* move checks scripts and files
This commit is contained in:
Oliver Chang 2021-04-17 04:15:56 +10:00 committed by GitHub
parent 1d3821e08c
commit df27afd3b3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 323 additions and 164 deletions

View File

@ -285,7 +285,7 @@ The following checks are all run against the target project:
| Packaging | Does the project build and publish official packages from CI/CD, e.g. [GitHub Publishing](https://docs.github.com/en/free-pro-team@latest/actions/guides/about-packaging-with-github-actions#workflows-for-publishing-packages) ? |
To see detailed information about each check and remediation steps,
check out the [checks documentation page](checks.md).
check out the [checks documentation page](checks/checks.md).
If you'd like to add a check, make sure it is something that meets the following criteria:

163
checks.md
View File

@ -1,163 +0,0 @@
# Check Documentation
This page contains information on how each check works and provide remediation
steps to fix the failure. All of these checks are basically "best-guesses"
currently, and operate on a set of heuristics.
They are all subject to change, and have room for improvement!
If you have ideas for things to add, or new ways to detect things,
please contribute!
## Security-Policy
This check tries to determine if a project has published a security policy.
It works by looking for a file named `SECURITY.md` (case-insensitive) in a
few well-known directories.
**Remediation steps**:
- Place a security policy file `SECURITY.md` in the root directory of your
repository. This makes it easily discoverable by a vulnerability reporter.
- The file should contain information on what constitues a vulnerability and
a way to report it securely (e.g. issue tracker with private issue support,
encrypted email with a published public key).
## Contributors
This check tries to determine if a project has a set of contributors from
multiple companies. It works by looking at the authors of recent commits and
checking the `Organization` field on the GitHub user profile.
**Remediation steps**:
- There is *NO* remediation work needed here. This is just to provide some
insights on which organization(s) have contributed to the project and
making trust decision based on that.
## Frozen-Deps
This check tries to determine if a project has declared and pinned its
dependencies. It works by looking for a set of well-known package management
lock files.
**Remediation steps**:
- Declare all your dependencies with specific versions in your package format
file (e.g. `package.json` for npm, `requirements.txt` for python). For C/C++,
check in the code from a trusted source and add a `README` on the specific
version used (and the archive SHA hashes).
- If the package manager supports lock files (e.g. `package-lock.json` for npm),
make sure to check these in the source code as well. These files maintain
signatures for the entire dependency tree and saves from future exploitation
in case the package is compromised.
## Signed-Tags
This check looks for cryptographically signed tags in the git history.
**Remediation steps**:
- Generate a new signing key.
- Add your key to your source hosting provider.
- Configure your key and email in git.
- Publish the tag and then sign it with this key.
- For GitHub, check out the steps
[here](https://docs.github.com/en/github/authenticating-to-github/signing-tags#further-reading).
## Signed-Releases
This check tries to determine if a project cryptographically signs release
artifacts. It works by looking for well-known filenames within recently
published GitHub releases.
**Remediation steps**:
- Publish the release.
- Generate a signing key.
- Download the release as an archive locally.
- Sign the release archive with this key (should output a signature file).
- Attach the signature file next to the release archive.
- For GitHub, check out the steps
[here](https://wiki.debian.org/Creating%20signed%20GitHub%20releases).
## CI-Tests
This check tries to determine if the project run tests before pull requests are
merged. It works by looking for a set of well-known CI-system names in GitHub
`CheckRuns` and `Statuses`.
**Remediation steps**:
- Check-in scripts that run all the tests in your repository.
- Integrate those scripts with a CI/CD platform that runs it on every pull
request (e.g.
[GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/introduction-to-github-actions),
[Prow](https://github.com/kubernetes/test-infra/tree/master/prow),
etc).
## Code-Review
This check tries to determine if a project requires code review before
pull requests are merged. It works by looking for a set of well-known code
review system results in GitHub Pull Requests.
**Remediation steps**:
- Follow security best practices by performing strict code reviews for every
new pull request.
- Make "code reviews" mandatory in your repository configuration.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging).
- Enforce the rule for administrators / code owners as well.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
## CII-Best-Practices
This check tries to determine if the project has a [CII Best Practices Badge](https://bestpractices.coreinfrastructure.org/en).
It uses the URL for the Git repo and the CII API.
**Remediation steps**:
- Sign up for the [CII Best Practices program](https://bestpractices.coreinfrastructure.org/en).
## Pull-Requests
This check tries to determine if the project requires pull requests for all
changes to the default branch. It works by looking at recent commits and using
the GitHub API to search for associated pull requests.
**Remediation steps**:
- Always open a pull request for any change you intend to make, big or small.
- Make "pull requests" mandatory in your repository configuration.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging)
- Enforce the rule for administrators / code owners as well.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
## Fuzzing
This check tries to determine if the project uses a fuzzing system.
It currently works by checking if the repo name is in the
[OSS-Fuzz](https://github.com/google/oss-fuzz) project list.
**Remediation steps**:
- Integrate the project with OSS-Fuzz by following the instructions
[here](https://google.github.io/oss-fuzz/).
## SAST
This check tries to determine if the project uses static code analysis systems.
It currently works by looking for well-known results ([CodeQL](https://securitylab.github.com/tools/codeql), etc.) in GitHub pull requests.
**Remediation steps**:
- Run CodeQL checks in your CI/CD by following the instructions
[here](https://github.com/github/codeql-action#usage).
## Active
This check tries to determine if the project is still "actively maintained".
It currently works by looking for commits within the last 90 days.
**Remediation steps**:
- There is *NO* remediation work needed here. This is just to indicate your
project activity and maintenance commitment.
## Branch-Protection
This check tries to determine if the project has branch protection enabled.
**Remediation steps**:
- Enable branch protection settings in your source hosting provider to avoid
force pushes or deletion of your important branches.
- For GitHub, check out the steps
[here](https://docs.github.com/en/github/administering-a-repository/managing-a-branch-protection-rule).

1
checks/checks.json Normal file

File diff suppressed because one or more lines are too long

106
checks/checks.md Normal file
View File

@ -0,0 +1,106 @@
<!-- Do not edit this file manually! Edit checks.yaml instead. -->
# Check Documentation
This page contains information on how each check works and provide remediation
steps to fix the failure. All of these checks are basically "best-guesses"
currently, and operate on a set of heuristics.
They are all subject to change, and have room for improvement!
If you have ideas for things to add, or new ways to detect things,
please contribute!
## Security-Policy
This check tries to determine if a project has published a security policy. It works by looking for a file named `SECURITY.md` (case-insensitive) in a few well-known directories.
**Remediation steps**:
- Place a security policy file `SECURITY.md` in the root directory of your repository. This makes it easily discoverable by a vulnerability reporter.
- The file should contain information on what constitutes a vulnerability and a way to report it securely (e.g. issue tracker with private issue support, encrypted email with a published public key).
## Contributors
This check tries to determine if a project has a set of contributors from multiple companies. It works by looking at the authors of recent commits and checking the `Organization` field on the GitHub user profile.
**Remediation steps**:
- There is *NO* remediation work needed here. This is just to provide some insights on which organization(s) have contributed to the project and making trust decision based on that.
## Frozen-Deps
This check tries to determine if a project has declared and pinned its dependencies. It works by looking for a set of well-known package management lock files.
**Remediation steps**:
- Declare all your dependencies with specific versions in your package format file (e.g. `package.json` for npm, `requirements.txt` for python). For C/C++, check in the code from a trusted source and add a `README` on the specific version used (and the archive SHA hashes).
- If the package manager supports lock files (e.g. `package-lock.json` for npm), make sure to check these in the source code as well. These files maintain signatures for the entire dependency tree and saves from future exploitation in case the package is compromised.
## Signed-Tags
This check looks for cryptographically signed tags in the git history.
**Remediation steps**:
- Generate a new signing key.
- Add your key to your source hosting provider.
- Configure your key and email in git.
- Publish the tag and then sign it with this key.
- For GitHub, check out the steps [here](https://docs.github.com/en/github/authenticating-to-github/signing-tags#further-reading).
## Signed-Releases
This check tries to determine if a project cryptographically signs release artifacts. It works by looking for well-known filenames within recently published GitHub releases.
**Remediation steps**:
- Publish the release.
- Generate a signing key.
- Download the release as an archive locally.
- Sign the release archive with this key (should output a signature file).
- Attach the signature file next to the release archive.
- For GitHub, check out the steps [here](https://wiki.debian.org/Creating%20signed%20GitHub%20releases).
## CI-Tests
This check tries to determine if the project run tests before pull requests are merged. It works by looking for a set of well-known CI-system names in GitHub `CheckRuns` and `Statuses`.
**Remediation steps**:
- Check-in scripts that run all the tests in your repository.
- Integrate those scripts with a CI/CD platform that runs it on every pull request (e.g. [GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/introduction-to-github-actions), [Prow](https://github.com/kubernetes/test-infra/tree/master/prow), etc).
## Code-Review
This check tries to determine if a project requires code review before pull requests are merged. It works by looking for a set of well-known code review system results in GitHub Pull Requests.
**Remediation steps**:
- Follow security best practices by performing strict code reviews for every new pull request.
- Make "code reviews" mandatory in your repository configuration. E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging).
- Enforce the rule for administrators / code owners as well. E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
## CII-Best-Practices
This check tries to determine if the project has a [CII Best Practices Badge](https://bestpractices.coreinfrastructure.org/en). It uses the URL for the Git repo and the CII API.
**Remediation steps**:
- Sign up for the [CII Best Practices program](https://bestpractices.coreinfrastructure.org/en).
## Pull-Requests
This check tries to determine if the project requires pull requests for all changes to the default branch. It works by looking at recent commits and using the GitHub API to search for associated pull requests.
**Remediation steps**:
- Always open a pull request for any change you intend to make, big or small.
- Make "pull requests" mandatory in your repository configuration. E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging)
- Enforce the rule for administrators / code owners as well. E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
## Fuzzing
This check tries to determine if the project uses a fuzzing system. It currently works by checking if the repo name is in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project list.
**Remediation steps**:
- Integrate the project with OSS-Fuzz by following the instructions [here](https://google.github.io/oss-fuzz/).
## SAST
This check tries to determine if the project uses static code analysis systems. It currently works by looking for well-known results ([CodeQL](https://securitylab.github.com/tools/codeql), etc.) in GitHub pull requests.
**Remediation steps**:
- Run CodeQL checks in your CI/CD by following the instructions [here](https://github.com/github/codeql-action#usage).
## Active
This check tries to determine if the project is still "actively maintained". It currently works by looking for commits within the last 90 days.
**Remediation steps**:
- There is *NO* remediation work needed here. This is just to indicate your project activity and maintenance commitment.
## Branch-Protection
This check tries to determine if the project has branch protection enabled.
**Remediation steps**:
- Enable branch protection settings in your source hosting provider to avoid force pushes or deletion of your important branches.
- For GitHub, check out the steps [here](https://docs.github.com/en/github/administering-a-repository/managing-a-branch-protection-rule).

168
checks/checks.yaml Normal file
View File

@ -0,0 +1,168 @@
# Copyright 2021 Security Scorecard Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is the source of truth for all check descriptions and remediation steps.
# Run `python3 generate_checks.py` to generate `checks.json` and `checks.md`.
checks:
Security-Policy:
description: >-
This check tries to determine if a project has published a security
policy. It works by looking for a file named `SECURITY.md`
(case-insensitive) in a few well-known directories.
remediation:
- >-
Place a security policy file `SECURITY.md` in the root directory of
your repository. This makes it easily discoverable by a vulnerability
reporter.
- >-
The file should contain information on what constitutes a vulnerability
and a way to report it securely (e.g. issue tracker with private issue
support, encrypted email with a published public key).
Contributors:
description: >-
This check tries to determine if a project has a set of contributors from
multiple companies. It works by looking at the authors of recent commits
and checking the `Organization` field on the GitHub user profile.
remediation:
- >-
There is *NO* remediation work needed here. This is just to provide some
insights on which organization(s) have contributed to the project and
making trust decision based on that.
Frozen-Deps:
description: >-
This check tries to determine if a project has declared and pinned its
dependencies. It works by looking for a set of well-known package
management lock files.
remediation:
- >-
Declare all your dependencies with specific versions in your package
format file (e.g. `package.json` for npm, `requirements.txt` for
python). For C/C++, check in the code from a trusted source and add a
`README` on the specific version used (and the archive SHA hashes).
- >-
If the package manager supports lock files (e.g. `package-lock.json`
for npm), make sure to check these in the source code as well. These
files maintain signatures for the entire dependency tree and saves from
future exploitation in case the package is compromised.
Signed-Tags:
description: >-
This check looks for cryptographically signed tags in the git history.
remediation:
- Generate a new signing key.
- Add your key to your source hosting provider.
- Configure your key and email in git.
- Publish the tag and then sign it with this key.
- >-
For GitHub, check out the steps
[here](https://docs.github.com/en/github/authenticating-to-github/signing-tags#further-reading).
Signed-Releases:
description: >-
This check tries to determine if a project cryptographically signs
release artifacts. It works by looking for well-known filenames within
recently published GitHub releases.
remediation:
- Publish the release.
- Generate a signing key.
- Download the release as an archive locally.
- Sign the release archive with this key (should output a signature file).
- Attach the signature file next to the release archive.
- >-
For GitHub, check out the steps
[here](https://wiki.debian.org/Creating%20signed%20GitHub%20releases).
CI-Tests:
description: >-
This check tries to determine if the project run tests before pull
requests are merged. It works by looking for a set of well-known
CI-system names in GitHub `CheckRuns` and `Statuses`.
remediation:
- Check-in scripts that run all the tests in your repository.
- >-
Integrate those scripts with a CI/CD platform that runs it on every pull
request (e.g.
[GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/introduction-to-github-actions),
[Prow](https://github.com/kubernetes/test-infra/tree/master/prow),
etc).
Code-Review:
description: >-
This check tries to determine if a project requires code review before
pull requests are merged. It works by looking for a set of well-known
code review system results in GitHub Pull Requests.
remediation:
- >-
Follow security best practices by performing strict code reviews for
every new pull request.
- >-
Make "code reviews" mandatory in your repository configuration.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging).
- >-
Enforce the rule for administrators / code owners as well.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
CII-Best-Practices:
description: >-
This check tries to determine if the project has a
[CII Best Practices Badge](https://bestpractices.coreinfrastructure.org/en).
It uses the URL for the Git repo and the CII API.
remediation:
- Sign up for the [CII Best Practices program](https://bestpractices.coreinfrastructure.org/en).
Pull-Requests:
description: >-
This check tries to determine if the project requires pull requests for
all changes to the default branch. It works by looking at recent commits
and using the GitHub API to search for associated pull requests.
remediation:
- Always open a pull request for any change you intend to make, big or small.
- >-
Make "pull requests" mandatory in your repository configuration.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging)
- >-
Enforce the rule for administrators / code owners as well.
E.g. [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
Fuzzing:
description: >-
This check tries to determine if the project uses a fuzzing system. It
currently works by checking if the repo name is in the
[OSS-Fuzz](https://github.com/google/oss-fuzz) project list.
remediation:
- >-
Integrate the project with OSS-Fuzz by following the instructions
[here](https://google.github.io/oss-fuzz/).
SAST:
description: >-
This check tries to determine if the project uses static code analysis
systems. It currently works by looking for well-known results
([CodeQL](https://securitylab.github.com/tools/codeql), etc.) in GitHub
pull requests.
remediation:
- >-
Run CodeQL checks in your CI/CD by following the instructions
[here](https://github.com/github/codeql-action#usage).
Active:
description: >-
This check tries to determine if the project is still "actively
maintained". It currently works by looking for commits within the last
90 days.
remediation:
- >-
There is *NO* remediation work needed here. This is just to indicate
your project activity and maintenance commitment.
Branch-Protection:
description: >-
This check tries to determine if the project has branch protection enabled.
remediation:
- >-
Enable branch protection settings in your source hosting provider to avoid
force pushes or deletion of your important branches.
- >-
For GitHub, check out the steps
[here](https://docs.github.com/en/github/administering-a-repository/managing-a-branch-protection-rule).

47
checks/generate_checks.py Normal file
View File

@ -0,0 +1,47 @@
# Copyright 2021 Security Scorecard Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import yaml
with open('checks.yaml') as f:
checks = yaml.safe_load(f)
with open('checks.json', 'w') as f:
f.write(json.dumps(checks))
with open('checks.md', 'w') as f:
f.write("""
<!-- Do not edit this file manually! Edit checks.yaml instead. -->
# Check Documentation
This page contains information on how each check works and provide remediation
steps to fix the failure. All of these checks are basically "best-guesses"
currently, and operate on a set of heuristics.
They are all subject to change, and have room for improvement!
If you have ideas for things to add, or new ways to detect things,
please contribute!
""")
for check, info in checks['checks'].items():
f.write('## ' + check + '\n')
f.write(info['description'] + '\n\n')
f.write('**Remediation steps**:\n')
for remediation in info['remediation']:
f.write('- ' + remediation + '\n')
f.write('\n')