* add config e2e test
Signed-off-by: Spencer Schrock <sschrock@google.com>
* update readme syntax
The old syntax was changed so the README was out of date.
This was exposed when setting up the e2e repo.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* fix rename
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* 🌱 Add branch protection evaluation
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* make helper for getting the branchName
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* move check for branch name
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* define size of slice
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* add probe for protected branches.
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* change 'basicNonAdminProtection' to 'deleteAndForcePushProtection'
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* fix markdown in text field in def.yml
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove duplicate conditional
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove redundant 'protected' value from 'requiresCodeOwnersReview' probe
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove protected values from probes
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* Bring back negative outcome in case of 0 codeowners files
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* log based on whether branches are protected
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* remove unnecessary test
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* debug failing tests
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* Fix failing tests
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* rename test
Signed-off-by: Adam Korczynski <adam@adalogics.com>
* update to with latest upstream changes
Signed-off-by: AdamKorcz <adam@adalogics.com>
* fix linting issues
Signed-off-by: AdamKorcz <adam@adalogics.com>
* remove tests that represent impossible scenarios
Signed-off-by: AdamKorcz <adam@adalogics.com>
* remove protected finding value
This was discussed previously, but accidentally reverted
Signed-off-by: Spencer Schrock <sschrock@google.com>
* Revert "debug failing tests"
This reverts commit 00acf66ea6.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* use branchName key for branch name
Signed-off-by: Spencer Schrock <sschrock@google.com>
* include number of reviews in INFO
this was previously included by the old evaluation code
Signed-off-by: Spencer Schrock <sschrock@google.com>
* reduce info count by 1
requiring codeowners without a corresponding file used to give 1 INFO and 1 WARN
now it only gives 1 WARN
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Adam Korczynski <adam@adalogics.com>
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: Spencer Schrock <sschrock@google.com>
Co-authored-by: Spencer Schrock <sschrock@google.com>
* test failures should print the details they receive
this makes debugging failing tests easier.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* use GinkgoTB so the test helpers work instead of panicing
Signed-off-by: Spencer Schrock <sschrock@google.com>
* ValidateTestReturn will fail the test directly, no need for the bool return
Signed-off-by: Spencer Schrock <sschrock@google.com>
* clarify diff details
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* feat(branch-protection): consider if project requires PRs prior to make changes
As discussed at the issue #2727, we're adding the "require PRs prior
to make changes" as another requirement to tier 2. In addition to that,
we're changing the weight of the tier 2 requirements so that
"requiring 1 reviewer" has weight 2, while the other tier 2 requirements
have weight 1
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): increment and adapt testing
1. Adapt previous test cases to consider that now we'll have an aditional
Info log telling that the project requires PRs to make changes.
2. Add more cases to test relevant use cases on the tier 2 level of
branch protection
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection-check): adapt check description to consider requirement of require PRs to make changes
It adds the new tier 2 requirement, but also specify that the
"require at least 1 reviewer" will have doubled weight.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection-check): avoid duplicate funcions and enhance readability
Made some nice-to-have improvements on project readability,
making it easier easier to understand how the branch-protection
score is computed. Also unified 8 different functions that were
doing basically the same thing.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): standardize values received on evaluation
Previously, at the evaluation part of branch protetion, the
values nil and false or zero were sort of interchangeble. This commit
changes the code to set as nil only the data that could not be retrieved
from github -- all the others would have values as false, zero, true, etc
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(github-client): adapt and add tests to check if nil values are coherent
1. Add new test to evaluate how we're interpreting a rule with all
checkboxes unchecked (most shouldn't be nil)
2. Adapt existent tests to expect non-nil values for unchecked
checkboxes
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(client-github): avoid reusing bool pointers
Changes some pieces of code to prefer using pointers of
bool instantiated independently. If reusing bool pointers, at some piece
of code the value of the bool could inadvertently changed and it would change the
value of all other fields reusing that pointer.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): enhance evaluation if scorecard was run by admin
At the evaluation step we were using some non untrusted fieldds of the
resposte to evaluate if Scorecard was run as admin or not. Now we're
using a field provided directly from the client file.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt testings to say if they have admin info or not
After last commit, the client will tell the evaluation files if
Scorecard was run by administrator or not (i.e., if we have all the
infos). This commit adapts the testings to also provide this info.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(e2e-branch-protection): adapt number of logs after changes
- 2 warns (for 'last push approval' and 'codeowners review' disabled) were added because now those informations come as 'not-nil' at the evaluation part.
- 1 info was added to say that PRs are required to make changes
- 1 debug was removed because it said that we couldn't retrieve 'last push approval' information, but we actually can. It was just incorrectly set as nil
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* Revert the 2 commits with changes around how Scorecard detects admin run
Reverts commit 64c3521d89a6493e0d8c7527aa011f98c3e35719 and commit e2662b7173ef90b44b2d72c37614230440e8a919.
Both had chances around using clients/branch.go scructur to store the
information of whether Scorecard was being run by admin or not. We
decided to not change this structure for this purpose.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): change data structure to use pointer instead of value
At clients.BranchProtectionRule struct, changing
RequiredPullRequestReviews to be a pointer instead of a struct value.
This will allow the usage of the nil value of this structure to mean
that we can't say if the repository requires reviews or not.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): use nil pointer on reviewers struct to mean
we don't know if they require PRs
The nil value of the struct RequiredPullRequestReviews will now mean
that we can't tell whether the project requires PRs to make changes or not.
When we get this case, we're printing a debug informing that we don't have
this data, but also printing a warn saying that they don't require
reviews, because that will be true at this case.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): if we're setting the reviewers struct to nil
when needed
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* doc(branch-protection): add code comment explaining different weight on tier 2 scores
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): avoid duplicate if branches on reviewers num comparation
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): clarify commentings around data structure
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: clean code on parsing GitHub BP data
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): ressignify the nil PullRequestReviewRule to mean PR not required
Adapt translation of data from GitHub API, now for our internal data
modeling, having a nil PullRequestReviewRule structure will mean that
PRs are not required on the repo (can also mean we don't have data to
ensure that).
It also changes the order of the calls of copyNonAdminSettings and
copyAdminSettings to make the first one be called first. This eases the
code because the PullRequestReviewRule can be always instantiated at
this function.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): ensure we translate GitHub BP data as expected
Ensure we're correctly translating GitHub data from the old Branch
Protection config.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): adapt score evaluation after 2efeee6512
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt testings to changes of last commits
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): add TODO comments pointing refactor opportunities
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix: avoid penalyzing non-admin for dismissStaleReview
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): prevent false value from API field to become nil
When translating the API results, if the specific field `DismissesStaleReviews`
had a false value, it was not being initiated in our data model and was
remaining nil.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: clarify different weight on first reviewer
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: enhance clarity of loggings and comments
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): new test to cover different rules affecting same branch
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): change requirements ordering to keep admin ones together
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): simplify auxiliary function
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): fix code format to linter requirements
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): avoid unnecessary initializations and rename function
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt test that was forgotten on commit 6858790a3e
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): use enums to represent tiers
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): remove nil fields of struct initialization when they dont contribute for clarification
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): simplify functions by using generics
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): update docs after generate-docs run
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): fix duplicated line on code
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): stop exporting Tier enum
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): changing unchanged var to const
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): Rename test and adapt it to be consistent with its purpose
I also changed the test to not require PRs, as it's how it is when a new GitHub
Branch Protection config is created. The changes on the loggings numbers are due
to:
1. A warning for not having DismissStaleReviews became a debug
2. Removed the warning we had for not requiring CodeOwners
3. Have a new warning for not requiring PRe
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
---------
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
14f864bfea not only fixed the --commit-depth option,
but also fixed the default commit depth for GitLab repos. Previously GitLab repos looked
back 20 commits because that was GitLab's default for the commits API. Now, GitLab repos
look back 30 commits, so the proportions of this e2e test changed.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* switch ossfuzz test to smaller repo
tensorflow/tensorflow is huge, and this causes the test to take forever.
locally this reduces the test time from 17 to 2.4 seconds
Signed-off-by: Spencer Schrock <sschrock@google.com>
* reuse scorecard results for scorecard attestor policies
previously this test took 27 seconds locally, and now takes 8.
which is split across 3 subtests:
good repos: 1s
bad repos: 5s
code review policies: 2s
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* 🌱 convert packaging check to probe
Signed-off-by: AdamKorcz <adam@adalogics.com>
* amend text in def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Correct short description in def.yml
Signed-off-by: AdamKorcz <adam@adalogics.com>
* log negative findings
Signed-off-by: AdamKorcz <adam@adalogics.com>
* rename probe
Signed-off-by: AdamKorcz <adam@adalogics.com>
* Fix the broken e2e test: The probe returned minimum score instead of inconclusive score which was not consistent with the previous scoring. This commit also removes the debug statements
Signed-off-by: AdamKorcz <adam@adalogics.com>
* change score text
Signed-off-by: AdamKorcz <adam@adalogics.com>
* include file details. process all packaging workflows
Signed-off-by: AdamKorcz <adam@adalogics.com>
---------
Signed-off-by: AdamKorcz <adam@adalogics.com>
* feat: Define if dependency is pinned or unpinned
Add a field Pinned to Dependency structure.
Update to save Dependencies pinned and unpinned. Not only unpinned ones.
All download then run executions are considered unpinned. Because there is no remediation to pin them.
For package manager downloads: add early return if there are no commands, separate package manager identification (go, npm, choco, pip) from decision if installation is pinned or unpinned.
Change Go case "go get -d -v" considered pinned, to any Go installations containing "-d" to be considered pinned.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Convert diff var types to pointer
We need to add a new conversion of boolean to pointer. Currently, we had string and int conversions named asPointer but not used in the same file. In order to know when we are using which conversion and considering bool and string would have to be used in the same file, it was needed to differentiate the method names. New method names are asIntPointer, asStringPointer and soon asBoolPointer.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Pinned Dependency field type
Field needs to be a pointer to work when accessing values on evaluation.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Count pinned and unpinned deps
We're changing the ecossystems result structure. The result structure previously stored if the ecossystem is fully pinned or not. The new result structure can tell how many dependencies of that ecossystem were found and how many were pinned. This change is necessary to ignore not applicable ecossystems on the final aggregated score. When iterating the dependencies, now we go through pinned and unpinned dependencies, not only unpinned, and in each iteration we update the result. We kept the behavior of only log warnings for unpinned dependencies.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Flag not applicable ecossystems
If no dependencies of an ecossystem are found, it results in an inconclusive score (-1). As in other checks, this means here that the ecossystem scoring is not applicable in this case. At the same time, we are keep the scoring criteria the same. If all dependencies are pinned, it results in maximum score (10) and if 1 or more dependencies are unpinned, it results in a minimum score (0) for that ecossystem. GitHub workflow cases are handled differently but the idea is the same. We are also adding a log to know when an ecossystem was not found.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Score only applicable ecossystems
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: If no dependencies then create inconclusive score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: GitHub Actions score and logs
Change test from `createReturnValuesForGitHubActionsWorkflowPinned` function to `createReturnForIsGitHubActionsWorkflowPinned` wrapper function so we can test logs. We have adjusted the existing test cases and included new test cases.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Pinned dependencies score
Break "various warnings" tests into smaller tests for pinned and unpinned dependencies and how they react to warn and debug messages. Plus add tests for how the score is affected when all dependencies are pinned, when no dependencies are pinned, when there are no dependencies, and partial dependencies pinned. Also, how dependencies unpinned in 1 or multiple ecossystems affect the warn messages, add one unpinned case for each ecossystem to see if they are being detected and separate the download then run 2 possible cases, there are currently scoring and logging wrong due to a bug.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Ecossystems score and logs
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Remove deleted maxScore function test
When we changed the scoring method to ignore not applicable scores, we removed the normalization of inconclusive scores to 0. The normalization was done by `maxScore` function, that was deleted in the process.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Adding GitHub Actions dependencies to result
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Update GitHub Actions result
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Update pip installs result
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Handle if nuget dependency is pinned or unpinned
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* tests: Fix check warnings for unpinned dependencies
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Linter errors
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: GitHub Actions pinned log
If, for example, you have GitHub-owned actions and none Third-party actions, you should receive a "no Third-party actions found" log and don't receive a "all Third-party actions are pinned" log. At the same time, you deserve the score of pinning Third-party to complement the GitHub-owned score.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "ossf-tests/scorecard-check-pinned-dependencies-e2e"
The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has no Third-party actions only GitHub-owned actions, that are unpinned, no npm installs, multiple go installs all pinned, and all other dependencies types are unpinned. This gives us 8 for actionScore, -1 for npm score, 10 for goScore, and 0 for all other scores. Previously the total score was 28/7 =~ 4, and now the total score is 18/6 =~ 3. The number of logs remain the same. The "all Third-party actions are pinned" will be replaced by "no Third-party actions found", which is a more realistic info and same thing for npm installs.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* Revert rename `asPointer` to `asStringPointer`
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Handle deps with parsing error and undefined pinning
When a dependency has a parsing error it ends up with a `Msg` field. In this case, the dependency should not count in the final score, so we should not `updatePinningResults` in this case. Also, to continue with the evaluation calculation, we need to make sure the dependencies have a `Pinned` state. Here we are adding this validation for it along with a debug log.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Delete unecessary test
We already have separate test for if 1 unpinned dependency shows a warn message, and 2 cases for when dependencies have errors and show a debug message.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Add missing dep Location cases
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Simplify Dockerfile pinned as name logic
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: If ecossystem is not found show debug log
If ecossystem is not found show debug log, not info log. This affects the tests, all not found ecossystems will "move" from info logs to debug logs. We are also complementing the `all dependencies pinned` and `all dependencies unpinned` cases so we have the max score case and the min score case using all kinds of dependencies.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix e2e tests and more unit tests
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Iterate all dependency types for final score
Now we iterate all existing dependency types in the final score. This will fix the problem of new ecossystems not being count in the final score because we needed to update the evaluation part. This also fixes the problem of download then run being counted twice for the score. Now, we only have debug logs when there are errors with the dependency metadata. That means we don't log anymore when dependencies of an ecossystem are not found. We changed the info log format when dependencies are all pinned. We simplified the calculation of the scores. We removed unused error returns. And now we only iterate existing ecossystems. If an ecossystem is not found we will not iterate it.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Proportional score
We count all pinned dependencies over the total found dependencies of all ecossystems for the final score. But, we still want to give low prioritity to GHA GitHub-owned dependencies over GHA third-party dependencies. That's why we are doing a weighted proportional score, all ecossystems have a normal weight of 10 but GHAs have a weight. If you only have GitHub-owned, it will count as 10, because GHA don't weight less then other ecossystems. Same for GHA third-party, if you only have GHA third-party, it will also count as 10, because GHAs don't weight less then other ecossystems. But if you have both GHA GitHub-owned and third-party, GitHub-owned count less then third-party. Trying to keep the same weight as before, GitHub-owned weights 8 and third-party weights 2. These weights will make the score be more penalized if you have unpinned third-party and less penalized if you have unpinned GitHub-owned.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: GHA weights in proportional score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix scores and logs checking
Add new cases for GHA scores since it's weighted differently now. Remove `createReturnValues` test since the function was removed. Fix current tests to adjust number of logs since we don't log if all dependencies are pinned or not anymore. Fix partially pinned score.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix e2e test
The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has no Third-party actions only GitHub-owned actions, that are unpinned, no npm installs, multiple go installs all pinned, and all other dependencies types are unpinned. This gives us 8 for GHA ecossytem, -1 for npm score, 10 for goScore, and 0 for all other scores. Previously the total score was 18/6 =~ 3. Now, we count 5/6 GitHub-owned GHA pinned, 23/36 containerImage pinned, 0/88 downloadThenRun pinned, 2/49 pipCommand pinned, 17/17 goCommand pinned. This results in 47/186 pinned dependencies which results in 2.5 score, that is rounded down to 2. Plus, the number of info was reduced since we don't log info for "all pinned dependencies in X ecossystem" anymore.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Rename to ProportionalScoreWeighted
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Var declarations to create proportional score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Remove unnecessary pointer
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Dependencies priority declaration
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Ecosystem spelling
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Handle 0 weight and 0 total when creating proportional weighted score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Revert -d flag identification change
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: npm ci command is npm download and is pinned
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Linter errors
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Unexport error variable to other packages
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* refactor: Simplify no score groups condition
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* feat: Log proportion of dependencies pinned
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix unit tests to include info logs
The number of info logs should be same number of identified ecossystems. GitHub-owned GitHubAction and third-party GitHubAction count as different ecossytems.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix e2e tests to include info logs
The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has GitHub-owned GitHubActions, containerImage, downloadThenRun, pipCommand and goCommand dependencies. Therefore it will have 5 Info logs, one for each ecossystem.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: Linter error
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
---------
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* repo rulesets via v4 api
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* good enough fnmatch implementation.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* good enough rulesMatchingBranch
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* apply matching repo rules to branch protection settings
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* rules: consider admins and require checks
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* non-structural chanages from PR feedback
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* fetch default branch name during repo rules query
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* Testing applyRepoRules
Tests assume a single rule is being applied to a branch, which might be
guarded by a legacy branch protection rule.
I think this logic gets problematic when there are multiple rules
overlaid on the same branch: the "the existing rules does not enforce
for admins, but i do and therefore this branch now does" will give
false-positives.
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* Test_applyRepoRules: builder and standardize names
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* attempt to upgrade/downgrade EnforceAdmins as each rule is applied
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
* simplify enforce admin for now.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* handle merging pull request reviews
Signed-off-by: Spencer Schrock <sschrock@google.com>
* handle merging check rules
Signed-off-by: Spencer Schrock <sschrock@google.com>
* handle last push approval
Signed-off-by: Spencer Schrock <sschrock@google.com>
* handle linear history
Signed-off-by: Spencer Schrock <sschrock@google.com>
* use constants for github rule types.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* add status check test.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* add e2e test for repo rules.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* handle nil branch name data
Signed-off-by: Spencer Schrock <sschrock@google.com>
* add tracking issue.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* fix precedence in if statement
Signed-off-by: Spencer Schrock <sschrock@google.com>
* include repo rules in the check docs.
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Peter Wagner <1559510+thepwagner@users.noreply.github.com>
Signed-off-by: Spencer Schrock <sschrock@google.com>
Co-authored-by: Spencer Schrock <sschrock@google.com>
* feat: Add go install to pinned dependencies score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix info logs count
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests generate one more Info log for "go installs are all pinned".
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "download then run pinned debug and warn"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests have to weight 7 scores instead of 6.
For "download then run pinned debug and warn", we have a 0 for 2 groups, `dockerDownloadScore` and `scriptScore`. Previously, it scored 4/6 =~ 6, and now it scores 5/7 =~ 7.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "various warnings"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests have to weight 7 scores instead of 6.
For "various warnings", we have a 0 for 4 groups, `pipScore`, `dockerDownloadScore`, `scriptScore` and `dockerFromScore`. Previously, it scored 2/6 =~ 3, and now it scores 3/7 =~ 4.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "Validate various warnings and info"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests have to weight 7 scores instead of 6.
For "Validate various warnings and info", we have a 0 for 4 groups, `pipScore`, `dockerDownloadScore`, `scriptScore` and `dockerFromScore`. Previously, it scored 2/6 =~ 3, and now it scores 3/7 =~ 4.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "ossf-tests/scorecard-check-pinned-dependencies-e2e"
Considering the new go installs dependencies in Pinned-Dependencies score, there are some changes. The repo being tested, `ossf-tests/scorecard-check-pinned-dependencies-e2e`, has third-party GitHub actions pinned, no npm installs, multiple go installs all pinned, and all other dependencies types are unpinned. This gives us 8 for actionScore, 10 for npm score, 10 for goScore, and 0 for all other scores. Previously the total score was 18/6 =~ 3, and now the total score is 28/7 =~ 4. Since all go installs are pinned, there's an additional info log for "go installs are pinned".
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Unpinned go install score
When having one unpinned go install and all other dependencies pinned, the score should be 60/7 =~ 8. Also, it should raise 1 warning for the unpinned go install, 7 infos saying the other dependency types are pinned (2 for GHAs, 2 for dockerfile image and downdloads, 1 for script downdloads, 1 for pip installs and 1 for npm installs), and 0 debug logs since the go install dependency does not have an error message.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
---------
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
- Add 2 tests for searching commits in e2e/searchCommits_test.go
- Fix errors in e2e/searchCommits_test.go when not using HEAD or when user does not exist
[e2e/searchCommits_test.go]
- Add 2 tests for searching commits
- Fix error when not using HEAD
- Fix error when user does not exist
Signed-off-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com>
- Ensure that only head queries are supported in workflow tests
- Add a test to detect when a non-existent workflow file is used
[e2e/workflow_test.go]
- Add a test to check that only head queries are supported
- Add a test to check that a non-existent workflow file returns an error
Signed-off-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com>
- Add an e2e test for searching commits by author
- Search commits by author `dependabot[bot]` and expect results
Signed-off-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com>
- Add e2e test for workflow runs
- Retrieve successful runs of the scorecard-analysis.yml workflow
Signed-off-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com>
* feat: Add npm install to pinned dependencies score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix pinned dependencies evaluation tests
Considering the new npm installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests generate one more Info log for "npm installs are all pinned". Also, for "various wanrings" test, the total score has to weight now 6 scores instead of 5. The new score counts 10 for actionScore, 0 for dockerFromScore, 0 for dockerDownloadScore, 0 for scriptScore, 0 for pipScore and 10 for npm score, which gives us 20/6~=3.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix pinned dependencies e2e tests
Considering the new npm installs dependencies in Pinned-Dependencies score, there are some changes. The repo being tested, ossf-tests/scorecard-check-pinned-dependencies-e2e, has third-party GitHub actions pinned, no npm installs, and all other dependencies types are unpinned. This gives us 8 for actionScore, 10 for npmScore and 0 for all other scores. Previously the total score was 8/5~=1, and now the total score is 18/6=3. Also, since there are no npm installs, there's one more Info log for "npm installs are pinned".
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix typo
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Unpinned npm install score
When having one unpinned npm install and all other dependencies pinned, the score should be 50/6~=8. Also, it should raise 1 warning for the unpinned npm install, 6 infos saying the other dependency types are pinned (2 for GHAs, 2 for dockerfile image and downdloads, 1 for script downdloads and 1 for pip installs), and 0 debug logs since the npm install dependency does not have an error message.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Undefined npm install score
When an error happens to parse a npm install dependency, the error/debug message is saved in "Msg" field. In this case, we were not able to define if the npm install is pinned or not. This dependency is classified as pinned undefined. We treat such cases as pinned cases, so it logs as Info that npm installs are all pinned and counts the score as 10. Then, the final score makes it to 10 as well. Since it logs the error/debug message, the Debug log goes to 1.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix typo
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix "validate various warnings and info" test
Considering the new npm installs dependencies in Pinned-Dependencies score, there are some changes. Now, all tests generate one more Info log for "npm installs are all pinned". Also, this test total score has to weight now 6 scores instead of 5. The new score counts 10 for actionScore, 0 for dockerFromScore, 0 for dockerDownloadScore, 0 for scriptScore, 0 for pipScore and 10 for npm score, which gives us 20/6~=3.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* fix: npm dependencies pinned log
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Remove test of error when parsing an npm dependency
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
---------
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* Move tests that connect to GitLab out of e2e
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
* update
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
* mark as pat test
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
---------
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
tensorflow/tensorflow is huge and was slowing down tests.
Also removed the rust e2e tests because they're already present as unit tests.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* fix tests
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
* use projectID instead of project where applicable
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
* pass ref as listcommitoption
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
* update tests
* CI-Tests: check if score > 0. pull request client is limited and can't
go back to arbitrary pull requests. CI-Tests don't run on forks, so this
can't be pinned either. But, for active repositories, we typically
expect *some* tests to be run
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
* fix commitshandler commitSHA tests
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
* update tests
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
---------
Signed-off-by: Raghav Kaul <raghavkaul@google.com>
Signed-off-by: raghavkaul <8695110+raghavkaul@users.noreply.github.com>
* feat: Add pip install to pinned dependencies score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix pinned dependencies evaluation tests
Considering the new pipInstalls dependencies in Pinned-Dependencies score, there are some changes. The "all dependencies pinned" test generates now one more Info log for "pip installs are all pinned". The "has 2 dependencies unpinned" test now one more Info log for "pip installs are all pinned" too, plus the total score has to weight now 5 scores instead of 4. The previous score was 10 for actionScore, 10 for dockerFromScore, 0 for dockerDownloadScore, because it's a downloadThenRun problem, another 0 for scriptScore, because it's a downloadThenRun problem, an here we have a bug of the same problem counting twice, but that gives a 20/4=5 for total score. Since we have pip install score, the current score counts a 10 for pipScore and that gives a 30/5=6 for total score.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Fix pinned dependencies e2e tests
Considering the new pipInstalls dependencies in Pinned-Dependencies score, there are some changes. The repo being tested, ossf-tests/scorecard-check-pinned-dependencies-e2e, has only third-party GitHub actions pinned. All other dependencies types are unpinned, including pip installs. This gives us 8 for actionScore and 0 for all other scores. Previously the total score was 8/4=2, and now the total score is 8/5=1.6, which rounds down to 1.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Unpinned pip install score
When having one unpinned pip install and all other dependencies pinned, the score should be 40/5=8. Also, it should raise 1 warning for the unpinned pip install, 5 infos saying the other dependency types are pinned (2 for GHAs, 2 for dockerfile image and downdloads and 1 for script downdloads), and 0 debug logs since the pip install dependency does not have an error message.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: Undefined pip install score
When an error happens to parse a pip install dependency, the error/debug message is saved in "Msg" field. In this case, we were not able to define if the pip install is pinned or not. This dependency is classified as pinned undefined. We treat such cases as pinned cases, so it logs as Info that pip installs are all pinned and counts the score as 10. Then, the final score makes it to 10 as well. Since it logs the error/debug message, the Debug log goes to 1.
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* test: All dependencies pinned score
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
* 🌱 Bump github/codeql-action from 2.3.0 to 2.3.1 (#2920)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.3.0 to 2.3.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](b2c19fb9a2...8662eabe0e)
---
updated-dependencies:
- dependency-name: github/codeql-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---------
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Naveen <172697+naveensrinivasan@users.noreply.github.com>
