We still run into the 429 GCS responses due to the lower limits on the same file.
All of the projects without a repo_url are being mapped to the same
object and leading to rate limiting.
"Maximum rate of writes to the same object name: One write per second"
https://cloud.google.com/storage/quotas#objects
Signed-off-by: Spencer Schrock <sschrock@google.com>
We are getting connection reset requests from bestpractices.dev and 429
errors from our GCS bucket for too many writes. The GCS limit (1000 QPS)
is much higher, so just use the bestpractices.dev limit of 1 QPS.
https://github.com/coreinfrastructure/best-practices-badge/blob/main/docs/api.md
The construct was taken from https://go.dev/wiki/RateLimiting which "works well
for rates up to tens of operations per second."
Signed-off-by: Spencer Schrock <sschrock@google.com>
* update workflows to use go 1.22
Signed-off-by: Spencer Schrock <sschrock@google.com>
* update tools go.mod to 1.22.
no one imports this, so we can bump it now and
avoid issues in the future where we need to upgrade.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* bump docker files
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* 🌱 Bump github.com/google/osv-scanner from 1.6.1 to 1.6.2
Bumps [github.com/google/osv-scanner](https://github.com/google/osv-scanner) from 1.6.1 to 1.6.2.
- [Release notes](https://github.com/google/osv-scanner/releases)
- [Changelog](https://github.com/google/osv-scanner/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google/osv-scanner/compare/v1.6.1...v1.6.2)
---
updated-dependencies:
- dependency-name: github.com/google/osv-scanner
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* specify go patch version
go mod tidy requires this. I was able to delete the toolchain directive,
and it wasn't added back.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* bump dockerfiles to 1.21.6 so the build works
Signed-off-by: Spencer Schrock <sschrock@google.com>
* bump go version used in codeql workflow
github runners currently use Go 1.20 by default,
which doesn't understand 1.21.x format.
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Spencer Schrock <sschrock@google.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Spencer Schrock <sschrock@google.com>
the cron has witnessed a roughly 15% reduction in repo throughput,
this is partly due to increased osv.dev latency, increasing the Vulnerabilities check.
the pinned-dependencies check has also increased after 6d35c865e6.
Signed-off-by: Spencer Schrock <sschrock@google.com>
Adding the Required field to PullRequestReviewRule made BranchRef slightly too big for the linter.
This code isn't highly used, so just ignoring the inefficiency for now.
Not sure why the staticcheck linter started complaining about the date error checking,
but fixed it while I was here.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* revert the change which made RequiredPullRequestReviews a pointer
While the current approach works with the tiered scoring,
it wont work for probes or if we remove tiers. Making the struct nil to
signal that PRs aren't required hides some of the data we do have.
This is especially problematic for repo rules, where we can infer all
settings by what we see or dont see.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* add helper to deref pointers
Signed-off-by: Spencer Schrock <sschrock@google.com>
* clarify comments and keep code consistent
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* feat(branch-protection): consider if project requires PRs prior to make changes
As discussed at the issue #2727, we're adding the "require PRs prior
to make changes" as another requirement to tier 2. In addition to that,
we're changing the weight of the tier 2 requirements so that
"requiring 1 reviewer" has weight 2, while the other tier 2 requirements
have weight 1
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): increment and adapt testing
1. Adapt previous test cases to consider that now we'll have an aditional
Info log telling that the project requires PRs to make changes.
2. Add more cases to test relevant use cases on the tier 2 level of
branch protection
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection-check): adapt check description to consider requirement of require PRs to make changes
It adds the new tier 2 requirement, but also specify that the
"require at least 1 reviewer" will have doubled weight.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection-check): avoid duplicate funcions and enhance readability
Made some nice-to-have improvements on project readability,
making it easier easier to understand how the branch-protection
score is computed. Also unified 8 different functions that were
doing basically the same thing.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): standardize values received on evaluation
Previously, at the evaluation part of branch protetion, the
values nil and false or zero were sort of interchangeble. This commit
changes the code to set as nil only the data that could not be retrieved
from github -- all the others would have values as false, zero, true, etc
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(github-client): adapt and add tests to check if nil values are coherent
1. Add new test to evaluate how we're interpreting a rule with all
checkboxes unchecked (most shouldn't be nil)
2. Adapt existent tests to expect non-nil values for unchecked
checkboxes
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(client-github): avoid reusing bool pointers
Changes some pieces of code to prefer using pointers of
bool instantiated independently. If reusing bool pointers, at some piece
of code the value of the bool could inadvertently changed and it would change the
value of all other fields reusing that pointer.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): enhance evaluation if scorecard was run by admin
At the evaluation step we were using some non untrusted fieldds of the
resposte to evaluate if Scorecard was run as admin or not. Now we're
using a field provided directly from the client file.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt testings to say if they have admin info or not
After last commit, the client will tell the evaluation files if
Scorecard was run by administrator or not (i.e., if we have all the
infos). This commit adapts the testings to also provide this info.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(e2e-branch-protection): adapt number of logs after changes
- 2 warns (for 'last push approval' and 'codeowners review' disabled) were added because now those informations come as 'not-nil' at the evaluation part.
- 1 info was added to say that PRs are required to make changes
- 1 debug was removed because it said that we couldn't retrieve 'last push approval' information, but we actually can. It was just incorrectly set as nil
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* Revert the 2 commits with changes around how Scorecard detects admin run
Reverts commit 64c3521d89a6493e0d8c7527aa011f98c3e35719 and commit e2662b7173ef90b44b2d72c37614230440e8a919.
Both had chances around using clients/branch.go scructur to store the
information of whether Scorecard was being run by admin or not. We
decided to not change this structure for this purpose.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): change data structure to use pointer instead of value
At clients.BranchProtectionRule struct, changing
RequiredPullRequestReviews to be a pointer instead of a struct value.
This will allow the usage of the nil value of this structure to mean
that we can't say if the repository requires reviews or not.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): use nil pointer on reviewers struct to mean
we don't know if they require PRs
The nil value of the struct RequiredPullRequestReviews will now mean
that we can't tell whether the project requires PRs to make changes or not.
When we get this case, we're printing a debug informing that we don't have
this data, but also printing a warn saying that they don't require
reviews, because that will be true at this case.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): if we're setting the reviewers struct to nil
when needed
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* doc(branch-protection): add code comment explaining different weight on tier 2 scores
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): avoid duplicate if branches on reviewers num comparation
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): clarify commentings around data structure
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: clean code on parsing GitHub BP data
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): ressignify the nil PullRequestReviewRule to mean PR not required
Adapt translation of data from GitHub API, now for our internal data
modeling, having a nil PullRequestReviewRule structure will mean that
PRs are not required on the repo (can also mean we don't have data to
ensure that).
It also changes the order of the calls of copyNonAdminSettings and
copyAdminSettings to make the first one be called first. This eases the
code because the PullRequestReviewRule can be always instantiated at
this function.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): ensure we translate GitHub BP data as expected
Ensure we're correctly translating GitHub data from the old Branch
Protection config.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* feat(branch-protection): adapt score evaluation after 2efeee6512
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt testings to changes of last commits
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): add TODO comments pointing refactor opportunities
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix: avoid penalyzing non-admin for dismissStaleReview
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): prevent false value from API field to become nil
When translating the API results, if the specific field `DismissesStaleReviews`
had a false value, it was not being initiated in our data model and was
remaining nil.
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: clarify different weight on first reviewer
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor: enhance clarity of loggings and comments
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): new test to cover different rules affecting same branch
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): change requirements ordering to keep admin ones together
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): simplify auxiliary function
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): fix code format to linter requirements
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): avoid unnecessary initializations and rename function
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): adapt test that was forgotten on commit 6858790a3e
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): use enums to represent tiers
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): remove nil fields of struct initialization when they dont contribute for clarification
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): simplify functions by using generics
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* docs(branch-protection): update docs after generate-docs run
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): fix duplicated line on code
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* fix(branch-protection): stop exporting Tier enum
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* refactor(branch-protection): changing unchanged var to const
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* test(branch-protection): Rename test and adapt it to be consistent with its purpose
I also changed the test to not require PRs, as it's how it is when a new GitHub
Branch Protection config is created. The changes on the loggings numbers are due
to:
1. A warning for not having DismissStaleReviews became a debug
2. Removed the warning we had for not requiring CodeOwners
3. Have a new warning for not requiring PRe
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
---------
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
* upgrade go.mod to 1.21
Signed-off-by: Spencer Schrock <sschrock@google.com>
* use slices from stdlib
Signed-off-by: Spencer Schrock <sschrock@google.com>
* use max/min builtins
Signed-off-by: Spencer Schrock <sschrock@google.com>
* multierrors
possibly spin this off into its own PR
Signed-off-by: Spencer Schrock <sschrock@google.com>
* dont call rand.Seed
As of Go 1.20, the generator is seeded randomly at startup.
https://pkg.go.dev/math/rand#Seed
Signed-off-by: Spencer Schrock <sschrock@google.com>
* update minimum Go version in documentation
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* enable bugs preset
Signed-off-by: Spencer Schrock <sschrock@google.com>
* fix noctx linter
Signed-off-by: Spencer Schrock <sschrock@google.com>
* fix bodyclose linter
Signed-off-by: Spencer Schrock <sschrock@google.com>
* fix contextcheck linter
Signed-off-by: Spencer Schrock <sschrock@google.com>
* This ignores all existing cases of musttag linter complaints.
This analyzer seems useful in the future, but some of this code
is old and I don't want to change it for existing code now.
Signed-off-by: Spencer Schrock <sschrock@google.com>
* ignore existing nilerr lints.
This behavior is from the initial commit, and primarily affects metrics.
Leaving as is, and hope to benefit from the linter in the future.
Signed-off-by: Spencer Schrock <sschrock@google.com>
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
* Bump dockerfiles to 1.21
* Go minimum version should match our go.mod
* Bump GitHub action go version to 1.21 and ensure all workflows use env variable.
---------
Signed-off-by: Spencer Schrock <sschrock@google.com>
