* convert Signed Releases to probes
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Specify that probe is for Github and Gitlab only
Signed-off-by: AdamKorcz <adam@adalogics.com>
* use in loop instead of
Signed-off-by: AdamKorcz <adam@adalogics.com>
* fix linter issues
Signed-off-by: AdamKorcz <adam@adalogics.com>
* fix more linter issues
Signed-off-by: AdamKorcz <adam@adalogics.com>
* specify Github and Gitlab in provenance def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Add link to slsa-github-generator
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Add instructions on signing with Cosign
Signed-off-by: AdamKorcz <adam@adalogics.com>
* refactor evaluation
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* debug failing integration test
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove unused nolints
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* expose release name asset names in finding values
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* fix failed integration test
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove 'totalReleases' value from findings
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove left-over cases of "totalReleases" values in findings
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove remaining totalReleases values
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* use const probe names instead of hard-coded strings
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove totalReleases from test helper arguments
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* merge test helpers
Signed-off-by: Adam Korczynski <adam@adalogics.com>
---------
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* revert the change which made RequiredPullRequestReviews a pointer
While the current approach works with the tiered scoring,
it wont work for probes or if we remove tiers. Making the struct nil to
signal that PRs aren't required hides some of the data we do have.
This is especially problematic for repo rules, where we can infer all
settings by what we see or dont see.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* add helper to deref pointers
Signed-off-by: Spencer Schrock <sschrock@google.com>
* clarify comments and keep code consistent
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* feat(branch-protection): consider if project requires PRs prior to make changes
As discussed at the issue #2727, we're adding the "require PRs prior
to make changes" as another requirement to tier 2. In addition to that,
we're changing the weight of the tier 2 requirements so that
"requiring 1 reviewer" has weight 2, while the other tier 2 requirements
have weight 1
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): increment and adapt testing
1. Adapt previous test cases to consider that now we'll have an aditional
Info log telling that the project requires PRs to make changes.
2. Add more cases to test relevant use cases on the tier 2 level of
branch protection
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection-check): adapt check description to consider requirement of require PRs to make changes
It adds the new tier 2 requirement, but also specify that the
"require at least 1 reviewer" will have doubled weight.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection-check): avoid duplicate funcions and enhance readability
Made some nice-to-have improvements on project readability,
making it easier easier to understand how the branch-protection
score is computed. Also unified 8 different functions that were
doing basically the same thing.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): standardize values received on evaluation
Previously, at the evaluation part of branch protetion, the
values nil and false or zero were sort of interchangeble. This commit
changes the code to set as nil only the data that could not be retrieved
from github -- all the others would have values as false, zero, true, etc
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(github-client): adapt and add tests to check if nil values are coherent
1. Add new test to evaluate how we're interpreting a rule with all
checkboxes unchecked (most shouldn't be nil)
2. Adapt existent tests to expect non-nil values for unchecked
checkboxes
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(client-github): avoid reusing bool pointers
Changes some pieces of code to prefer using pointers of
bool instantiated independently. If reusing bool pointers, at some piece
of code the value of the bool could inadvertently changed and it would change the
value of all other fields reusing that pointer.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): enhance evaluation if scorecard was run by admin
At the evaluation step we were using some non untrusted fieldds of the
resposte to evaluate if Scorecard was run as admin or not. Now we're
using a field provided directly from the client file.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt testings to say if they have admin info or not
After last commit, the client will tell the evaluation files if
Scorecard was run by administrator or not (i.e., if we have all the
infos). This commit adapts the testings to also provide this info.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(e2e-branch-protection): adapt number of logs after changes
- 2 warns (for 'last push approval' and 'codeowners review' disabled) were added because now those informations come as 'not-nil' at the evaluation part.
- 1 info was added to say that PRs are required to make changes
- 1 debug was removed because it said that we couldn't retrieve 'last push approval' information, but we actually can. It was just incorrectly set as nil
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* Revert the 2 commits with changes around how Scorecard detects admin run
Reverts commit 64c3521d89a6493e0d8c7527aa011f98c3e35719 and commit e2662b7173ef90b44b2d72c37614230440e8a919.
Both had chances around using clients/branch.go scructur to store the
information of whether Scorecard was being run by admin or not. We
decided to not change this structure for this purpose.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): change data structure to use pointer instead of value
At clients.BranchProtectionRule struct, changing
RequiredPullRequestReviews to be a pointer instead of a struct value.
This will allow the usage of the nil value of this structure to mean
that we can't say if the repository requires reviews or not.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): use nil pointer on reviewers struct to mean
we don't know if they require PRs
The nil value of the struct RequiredPullRequestReviews will now mean
that we can't tell whether the project requires PRs to make changes or not.
When we get this case, we're printing a debug informing that we don't have
this data, but also printing a warn saying that they don't require
reviews, because that will be true at this case.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): if we're setting the reviewers struct to nil
when needed
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* doc(branch-protection): add code comment explaining different weight on tier 2 scores
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): avoid duplicate if branches on reviewers num comparation
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): clarify commentings around data structure
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: clean code on parsing GitHub BP data
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): ressignify the nil PullRequestReviewRule to mean PR not required
Adapt translation of data from GitHub API, now for our internal data
modeling, having a nil PullRequestReviewRule structure will mean that
PRs are not required on the repo (can also mean we don't have data to
ensure that).
It also changes the order of the calls of copyNonAdminSettings and
copyAdminSettings to make the first one be called first. This eases the
code because the PullRequestReviewRule can be always instantiated at
this function.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): ensure we translate GitHub BP data as expected
Ensure we're correctly translating GitHub data from the old Branch
Protection config.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): adapt score evaluation after 2efeee6512
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt testings to changes of last commits
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): add TODO comments pointing refactor opportunities
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix: avoid penalyzing non-admin for dismissStaleReview
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): prevent false value from API field to become nil
When translating the API results, if the specific field `DismissesStaleReviews`
had a false value, it was not being initiated in our data model and was
remaining nil.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: clarify different weight on first reviewer
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: enhance clarity of loggings and comments
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): new test to cover different rules affecting same branch
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): change requirements ordering to keep admin ones together
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): simplify auxiliary function
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): fix code format to linter requirements
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): avoid unnecessary initializations and rename function
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt test that was forgotten on commit 6858790a3e
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): use enums to represent tiers
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): remove nil fields of struct initialization when they dont contribute for clarification
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): simplify functions by using generics
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): update docs after generate-docs run
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): fix duplicated line on code
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): stop exporting Tier enum
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): changing unchanged var to const
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): Rename test and adapt it to be consistent with its purpose
I also changed the test to not require PRs, as it's how it is when a new GitHub
Branch Protection config is created. The changes on the loggings numbers are due
to:
1. A warning for not having DismissStaleReviews became a debug
2. Removed the warning we had for not requiring CodeOwners
3. Have a new warning for not requiring PRe
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
---------
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* 🌱 convert Webhook check to probes
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Add test + nits
Signed-off-by: AdamKorcz <adam@adalogics.com>
* replace probe with OutcomeNotApplicable
Signed-off-by: AdamKorcz <adam@adalogics.com>
* return one finding per webhook
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* change wording in def.yml
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* change wording in def.yml and checks.md
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove unused struct in test
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* align checks.md with checks.yaml
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* bring back experimental for webhooks
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* change 'token' to 'secret' in probe
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* use checker.MinResultScore instead of 0
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* Change test name
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* use checker.MinResultScore instead of 0
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* fix typo
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* Use checker.MaxResultScore instead of 10
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* rename probe
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove the 'totalWebhooks' value from findings
Signed-off-by: Adam Korczynski <adam@adalogics.com>
---------
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* 🌱 convert binary artifact check to probe
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Reword motivation
Signed-off-by: AdamKorcz <adam@adalogics.com>
* remove unused variable in test
Signed-off-by: AdamKorcz <adam@adalogics.com>
* remove positiveOutcome() and length check
Signed-off-by: AdamKorcz <adam@adalogics.com>
* fix wrong check name
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Split into two probes: One with and one without gradle-wrappers
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Add description about what Scorecard considers a verified binary
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* change 'trusted' to 'verified'
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove nil check
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove filtering
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* use const scores in tests
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* rename test
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* add sanity check in loop
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* rename binary file const
Signed-off-by: Adam Korczynski <adam@adalogics.com>
---------
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* 🌱 convert CII Best Practices check to probes
Signed-off-by: AdamKorcz <adam@adalogics.com>
* change 'NOT' to 'not'
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Change wording in probes
Signed-off-by: AdamKorcz <adam@adalogics.com>
* add links to text
Signed-off-by: AdamKorcz <adam@adalogics.com>
* fix typo
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Edit text in def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* remove hasBadgeNotFound probe
Signed-off-by: AdamKorcz <adam@adalogics.com>
* remove 'that' from text
Signed-off-by: AdamKorcz <adam@adalogics.com>
* use CreateMinScoreResult instead of CreateResultWithScore
Signed-off-by: AdamKorcz <adam@adalogics.com>
* use MaxResultScore instead of maxScore
Signed-off-by: AdamKorcz <adam@adalogics.com>
* return CreateRuntimeErrorResult sooner rather than later
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Combine probes into one
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove minScore variable
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove 'hasInProgressBadge' probe
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* make badge levels global variables
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* return -1 for unsupported badge
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* change text for unknown and unsupported badges
Signed-off-by: Adam Korczynski <adam@adalogics.com>
---------
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* upgrade go.mod to 1.21
Signed-off-by: Spencer Schrock <sschrock@google.com>
* use slices from stdlib
Signed-off-by: Spencer Schrock <sschrock@google.com>
* use max/min builtins
Signed-off-by: Spencer Schrock <sschrock@google.com>
* multierrors
possibly spin this off into its own PR
Signed-off-by: Spencer Schrock <sschrock@google.com>
* dont call rand.Seed
As of Go 1.20, the generator is seeded randomly at startup.
https://pkg.go.dev/math/rand#Seed
Signed-off-by: Spencer Schrock <sschrock@google.com>
* update minimum Go version in documentation
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* Continue on error detecting OS
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add tests for error detecting OS
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add ElementError to identify elements that errored
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add Incomplete field to PinningDependenciesData
Will store all errors handled during analysis, which may lead to incomplete results.
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Register job steps that errored out
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add tests that incomplete steps are caught
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add warnings to details about incomplete steps
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add tests that incomplete steps generate warnings
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Register shell files skipped due to parser errors
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add tests showing when parser errors affect analysis
Dockerfile pinning is not affected.
Everything in a 'broken' Dockerfile RUN block is ignored
Everything in a 'broken' shell script is ignored
testdata/script-invalid.sh modified to demonstrate the above
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Incomplete results logged as Info, not Warn
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Remove `Type` from logging of incomplete results
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Update tests after rebase
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add Unwrap for ElementError, improve its docs
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Add ElementError case to evaluation unit test
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Move ElementError to checker/raw_result
checker/raw_result defines types used to describe analysis results.
ElementError is meant to describe potential flaws in the analysis
and is therefore a sort of analysis result itself.
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Use finding.Location for ElementError.Element
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Use an ElementError for script parser errors
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Replace .Incomplete []error with .ProcessingErrors []ElementError
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* Adopt from reviewer comments
- Replace ElementError's `Element *finding.Location`
with `Location finding.Location`
- Rename ErrorJobOSParsing to ErrJobOSParsing to satisfy linter
- Fix unit test
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
---------
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
* 🌱 convert vulnerabilities check to probe
Signed-off-by: AdamKorcz <adam@adalogics.com>
* rename probe + nits
Signed-off-by: AdamKorcz <adam@adalogics.com>
* edit def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Add vuln ID dynamically to def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Elaborate the purpose of test data in unit test
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Move logging out of loop and change logic of negativeFindings()
Signed-off-by: AdamKorcz <adam@adalogics.com>
* preserve number of vulns found in output
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Preserve grouping of vulns
Signed-off-by: AdamKorcz <adam@adalogics.com>
* fix linter issues
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Add remediation data
Signed-off-by: AdamKorcz <adam@adalogics.com>
* use checker.LogFindings()
Signed-off-by: AdamKorcz <adam@adalogics.com>
---------
Signed-off-by: AdamKorcz <adam@adalogics.com>
* 🌱 convert packaging check to probe
Signed-off-by: AdamKorcz <adam@adalogics.com>
* amend text in def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Correct short description in def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* log negative findings
Signed-off-by: AdamKorcz <adam@adalogics.com>
* rename probe
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Fix the broken e2e test: The probe returned minimum score instead of inconclusive score which was not consistent with the previous scoring. This commit also removes the debug statements
Signed-off-by: AdamKorcz <adam@adalogics.com>
* change score text
Signed-off-by: AdamKorcz <adam@adalogics.com>
* include file details. process all packaging workflows
Signed-off-by: AdamKorcz <adam@adalogics.com>
---------
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Fix SAST no longer working for CodeQL
The app slug for CodeQL appears to have changed from `github-advanced-security` to `github-code-scanning`, causing the SAST rule to false-negative on commits.
Signed-off-by: martincostello <martin@martincostello.com>
* Fix lint warning
Fix lint warning.
Signed-off-by: martincostello <martin@martincostello.com>
---------
Signed-off-by: martincostello <martin@martincostello.com>
* feat: Define if dependency is pinned or unpinned
Add a field Pinned to Dependency structure.
Update to save Dependencies pinned and unpinned. Not only unpinned ones.
All download then run executions are considered unpinned. Because there is no remediation to pin them.
For package manager downloads: add early return if there are no commands, separate package manager identification (go, npm, choco, pip) from decision if installation is pinned or unpinned.
Change Go case "go get -d -v" considered pinned, to any Go installations containing "-d" to be considered pinned.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Convert diff var types to pointer
We need to add a new conversion of boolean to pointer. Currently, we had string and int conversions named asPointer but not used in the same file. In order to know when we are using which conversion and considering bool and string would have to be used in the same file, it was needed to differentiate the method names. New method names are asIntPointer, asStringPointer and soon asBoolPointer.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Pinned Dependency field type
Field needs to be a pointer to work when accessing values on evaluation.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Count pinned and unpinned deps
We're changing the ecossystems result structure. The result structure previously stored if the ecossystem is fully pinned or not. The new result structure can tell how many dependencies of that ecossystem were found and how many were pinned. This change is necessary to ignore not applicable ecossystems on the final aggregated score. When iterating the dependencies, now we go through pinned and unpinned dependencies, not only unpinned, and in each iteration we update the result. We kept the behavior of only log warnings for unpinned dependencies.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Flag not applicable ecossystems
If no dependencies of an ecossystem are found, it results in an inconclusive score (-1). As in other checks, this means here that the ecossystem scoring is not applicable in this case. At the same time, we are keep the scoring criteria the same. If all dependencies are pinned, it results in maximum score (10) and if 1 or more dependencies are unpinned, it results in a minimum score (0) for that ecossystem. GitHub workflow cases are handled differently but the idea is the same. We are also adding a log to know when an ecossystem was not found.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Score only applicable ecossystems
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: If no dependencies then create inconclusive score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: GitHub Actions score and logs
Change test from `createReturnValuesForGitHubActionsWorkflowPinned` function to `createReturnForIsGitHubActionsWorkflowPinned` wrapper function so we can test logs. We have adjusted the existing test cases and included new test cases.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Pinned dependencies score
Break "various warnings" tests into smaller tests for pinned and unpinned dependencies and how they react to warn and debug messages. Plus add tests for how the score is affected when all dependencies are pinned, when no dependencies are pinned, when there are no dependencies, and partial dependencies pinned. Also, how dependencies unpinned in 1 or multiple ecossystems affect the warn messages, add one unpinned case for each ecossystem to see if they are being detected and separate the download then run 2 possible cases, there are currently scoring and logging wrong due to a bug.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Ecossystems score and logs
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Remove deleted maxScore function test
When we changed the scoring method to ignore not applicable scores, we removed the normalization of inconclusive scores to 0. The normalization was done by `maxScore` function, that was deleted in the process.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Adding GitHub Actions dependencies to result
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Update GitHub Actions result
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Update pip installs result
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Handle if nuget dependency is pinned or unpinned
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* tests: Fix check warnings for unpinned dependencies
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Linter errors
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: GitHub Actions pinned log
If, for example, you have GitHub-owned actions and none Third-party actions, you should receive a "no Third-party actions found" log and don't receive a "all Third-party actions are pinned" log. At the same time, you deserve the score of pinning Third-party to complement the GitHub-owned score.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "ossf-tests/scorecard-check-pinned-dependencies-e2e"
The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has no Third-party actions only GitHub-owned actions, that are unpinned, no npm installs, multiple go installs all pinned, and all other dependencies types are unpinned. This gives us 8 for actionScore, -1 for npm score, 10 for goScore, and 0 for all other scores. Previously the total score was 28/7 =~ 4, and now the total score is 18/6 =~ 3. The number of logs remain the same. The "all Third-party actions are pinned" will be replaced by "no Third-party actions found", which is a more realistic info and same thing for npm installs.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* Revert rename `asPointer` to `asStringPointer`
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Handle deps with parsing error and undefined pinning
When a dependency has a parsing error it ends up with a `Msg` field. In this case, the dependency should not count in the final score, so we should not `updatePinningResults` in this case. Also, to continue with the evaluation calculation, we need to make sure the dependencies have a `Pinned` state. Here we are adding this validation for it along with a debug log.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Delete unecessary test
We already have separate test for if 1 unpinned dependency shows a warn message, and 2 cases for when dependencies have errors and show a debug message.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Add missing dep Location cases
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Simplify Dockerfile pinned as name logic
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: If ecossystem is not found show debug log
If ecossystem is not found show debug log, not info log. This affects the tests, all not found ecossystems will "move" from info logs to debug logs. We are also complementing the `all dependencies pinned` and `all dependencies unpinned` cases so we have the max score case and the min score case using all kinds of dependencies.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix e2e tests and more unit tests
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Iterate all dependency types for final score
Now we iterate all existing dependency types in the final score. This will fix the problem of new ecossystems not being count in the final score because we needed to update the evaluation part. This also fixes the problem of download then run being counted twice for the score. Now, we only have debug logs when there are errors with the dependency metadata. That means we don't log anymore when dependencies of an ecossystem are not found. We changed the info log format when dependencies are all pinned. We simplified the calculation of the scores. We removed unused error returns. And now we only iterate existing ecossystems. If an ecossystem is not found we will not iterate it.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Proportional score
We count all pinned dependencies over the total found dependencies of all ecossystems for the final score. But, we still want to give low prioritity to GHA GitHub-owned dependencies over GHA third-party dependencies. That's why we are doing a weighted proportional score, all ecossystems have a normal weight of 10 but GHAs have a weight. If you only have GitHub-owned, it will count as 10, because GHA don't weight less then other ecossystems. Same for GHA third-party, if you only have GHA third-party, it will also count as 10, because GHAs don't weight less then other ecossystems. But if you have both GHA GitHub-owned and third-party, GitHub-owned count less then third-party. Trying to keep the same weight as before, GitHub-owned weights 8 and third-party weights 2. These weights will make the score be more penalized if you have unpinned third-party and less penalized if you have unpinned GitHub-owned.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: GHA weights in proportional score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix scores and logs checking
Add new cases for GHA scores since it's weighted differently now. Remove `createReturnValues` test since the function was removed. Fix current tests to adjust number of logs since we don't log if all dependencies are pinned or not anymore. Fix partially pinned score.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix e2e test
The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has no Third-party actions only GitHub-owned actions, that are unpinned, no npm installs, multiple go installs all pinned, and all other dependencies types are unpinned. This gives us 8 for GHA ecossytem, -1 for npm score, 10 for goScore, and 0 for all other scores. Previously the total score was 18/6 =~ 3. Now, we count 5/6 GitHub-owned GHA pinned, 23/36 containerImage pinned, 0/88 downloadThenRun pinned, 2/49 pipCommand pinned, 17/17 goCommand pinned. This results in 47/186 pinned dependencies which results in 2.5 score, that is rounded down to 2. Plus, the number of info was reduced since we don't log info for "all pinned dependencies in X ecossystem" anymore.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Rename to ProportionalScoreWeighted
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Var declarations to create proportional score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Remove unnecessary pointer
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Dependencies priority declaration
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Ecosystem spelling
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Handle 0 weight and 0 total when creating proportional weighted score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Revert -d flag identification change
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: npm ci command is npm download and is pinned
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Linter errors
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Unexport error variable to other packages
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Simplify no score groups condition
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Log proportion of dependencies pinned
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix unit tests to include info logs
The number of info logs should be same number of identified ecossystems. GitHub-owned GitHubAction and third-party GitHubAction count as different ecossytems.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix e2e tests to include info logs
The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has GitHub-owned GitHubActions, containerImage, downloadThenRun, pipCommand and goCommand dependencies. Therefore it will have 5 Info logs, one for each ecossystem.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Linter error
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
---------
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* Remove EnforceAdmins from tier 1.
Scores in some tests either increase to 3, or 4, since EnfroceAdmins no longer keeps them in tier 1.
The number of Debug, Info, and Warn messages will decrease by 1 per branch, since we're no longer logging them.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* move enforce admins to tier 5.
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* feat: Add go install to pinned dependencies score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix info logs count
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests generate one more Info log for "go installs are all pinned".
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "download then run pinned debug and warn"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests have to weight 7 scores instead of 6.
For "download then run pinned debug and warn", we have a 0 for 2 groups, `dockerDownloadScore` and `scriptScore`. Previously, it scored 4/6 =~ 6, and now it scores 5/7 =~ 7.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "various warnings"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests have to weight 7 scores instead of 6.
For "various warnings", we have a 0 for 4 groups, `pipScore`, `dockerDownloadScore`, `scriptScore` and `dockerFromScore`. Previously, it scored 2/6 =~ 3, and now it scores 3/7 =~ 4.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "Validate various warnings and info"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests have to weight 7 scores instead of 6.
For "Validate various warnings and info", we have a 0 for 4 groups, `pipScore`, `dockerDownloadScore`, `scriptScore` and `dockerFromScore`. Previously, it scored 2/6 =~ 3, and now it scores 3/7 =~ 4.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "ossf-tests/scorecard-check-pinned-dependencies-e2e"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has third-party GitHub actions pinned, no npm installs, multiple go installs all pinned, and all other dependencies types are unpinned. This gives us 8 for actionScore, 10 for npm score, 10 for goScore, and 0 for all other scores. Previously the total score was 18/6 =~ 3, and now the total score is 28/7 =~ 4. Since all go installs are pinned, there's an additional info log for "go installs are pinned".
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Unpinned go install score
When having one unpinned go install and all other dependencies pinned, the score should be 60/7 =~ 8. Also, it should raise 1 warning for the unpinned go install, 7 infos saying the other dependency types are pinned (2 for GHAs, 2 for dockerfile image and downdloads, 1 for script downdloads, 1 for pip installs and 1 for npm installs), and 0 debug logs since the go install dependency does not have an error message.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
---------
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* invert workflow check and explain early exit.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* make workflow run validation optional.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* mark binary artifacts as local file friendly.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* add test for gradle wrapper without workflow run support
Signed-off-by: Spencer Schrock <sschrock@google.com>
* fix policy tests and make their names more clear.
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
