Feature: Scoring for individual releases

[nadgowdas]

Is your feature request related to a problem? Please describe.

Currently, scorecard runs against the repository settings (typically main branch or commit-sha). Malicious developer can circumvent the scoring by implementing bad practice between two runs as shown below:

This is possible, because SCM/github do not maintain immutable ledger (like commit hash) of repository setting changes. Therefore, the scoring at the repository may not be applicable to the individual releases from that repository.

Can we add support for scoring individual releases ?

Describe the solution you'd like For a given release of a given repo, we discover all the PRs/commits that went into that release (since the baseline of previous release or any given release). Then collect the aggregated metrics for a given release that indicates whether all PRs/commits were peer reviewed, how many unique developers contributed in this release, whether there were any commits directly to the main branch, what "kinds" of changes went into the release (based on labels of PRs and linked issues), etc.

Additional context There is an open-source tool I have built to illustrate such use-cases. The tool is called gauge.

E.g. when I ran it against open-source python package flask it reports as follows.

./gauge package -p flask -e python -t 2.1.1 -r https://github.com/pallets/flask
------------------------------------------------------------------------------
Gauge Report for package `flask`
------------------------------------------------------------------------------
Release Measures:
	Current version: 2.1.1
	Latest version: 2.1.2
	Release lag (versions): 1
	Release lag (days): 28 days
--------------------------------------------------------------------------------
		Recommended update
		 Version - 2.1.2
		 Release Time - 2022-04-28 17:48:24 +0000 UTC
		 Num of unique contributors - 7
		 Num of unique reviewers - 0
		 Non peer reviewed changes - 0
		 Num of zombie commits - 15
		 Change annotations - ['docs','typing','testing']
--------------------------------------------------------------------------------

The core motivation behind such feature would be to allow users to evaluate respective release versions for OSS artifacts before consuming and provide them some insights to make an informed decision to upgrade.

[azeemshaikh38]

A wild idea - what if we collaborate with Dependabot/Renovatebot folks here so that every version bump PR by these tools provides a summarized view of diff between new version and existing one? It'll be super impactful and will raise OSS-wide awareness if all OSS developers who use these tools today start thinking about security of their dependencies.

@rarkins @josepalafox what do you folks think?

[rarkins]

For npm releases, Renovate currently provides a link in PRs to a diff of the two release artifacts (ie comparing npm tarballs). You can't trust that the source on GitHub is what was used to create the tarballs on npm

[nadgowdas]

@azeemshaikh38 I liked the idea, would be super useful.

I think in addition to providing link to PR diffs, if we can curate some of the information and provide developers with actionable insights, they can easily put it in their automated workflows.

[azeemshaikh38]

@rarkins makes sense for npm but this could be useful for non-npm ecosystems like Golang? What do you think about adding such diffs in Renovatebot?

[rarkins]

We plan to add more over time. Golang is not the best example because it's one of the few where "what you see is what you get" (on github). Registries where the artifact may differ from the source - for both legitimate and illegitimate reasons - are the most challenging for the user and benefit from this type of diff.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.