Validate the scorecard config file

[asergienk]

Feature Request

Describe the problem you need a feature to resolve.

CVP team would want the scorecard config.yaml file to be validated with operator-sdk bundle validate command. The only two folders that are getting validated are metadata and manifests, so, tests/scorecard/config.yaml is not validated.

Example: ocs-osd-deployer-operator-bundle-container-2.0.4-1 has an invalid config.yaml:

apiVersion: scorecard.operatorframework.io/v1alpha3
kind: Configuration
metadata:
  name: config
stages:
- parallel: true
  tests:
  - entrypoint:
    - scorecard-test
    - basic-check-spec
    image: quay.io/operator-framework/scorecard-test:v1.2.0
    labels:
      suite: basic
      test: basic-check-spec-test
    storage:
      spec:
        mountPath: {}
  - entrypoint:
    - scorecard-test
    - olm-bundle-validation
    image: quay.io/operator-framework/scorecard-test:v1.2.0
    labels:
      suite: olm
      test: olm-bundle-validation-test
    storage:
      spec:
        mountPath: {}
  - entrypoint:
    - scorecard-test
    - olm-crds-have-validation
    image: quay.io/operator-framework/scorecard-test:v1.2.0
    labels:
      suite: olm
      test: olm-crds-have-validation-test
    storage:
      spec:
        mountPath: {}
  - entrypoint:
    - scorecard-test
    - olm-crds-have-resources
    image: quay.io/operator-framework/scorecard-test:v1.2.0
    labels:
      suite: olm
      test: olm-crds-have-resources-test
    storage:
      spec:
        mountPath: {}
  - entrypoint:
    - scorecard-test
    - olm-spec-descriptors
    image: quay.io/operator-framework/scorecard-test:v1.2.0
    labels:
      suite: olm
      test: olm-spec-descriptors-test
    storage:
      spec:
        mountPath: {}
  - entrypoint:
    - scorecard-test
    - olm-status-descriptors
    image: quay.io/operator-framework/scorecard-test:v1.2.0
    labels:
      suite: olm
      test: olm-status-descriptors-test
    storage:
      spec:
        mountPath: {}
storage:
  spec:
    mountPath: {}
  - entrypoint:
      - /test.sh
      - test_1
    image: quay.io/apodhrad/scorecard-tests:0.9
    labels:
      cluster-product: ocp
      cluster-size: small
      phase: msp-main
      test: foo1

The spec block of the config.yaml should not contain

 - entrypoint:
      - /test.sh
      - test_1
    image: quay.io/apodhrad/scorecard-tests:0.9
    labels:
      cluster-product: ocp
      cluster-size: small
      phase: msp-main
      test: foo1

The above block should be inside tests. Also, the indentation of image and labels is not correct.

The error we got is:

--> message : while parsing a block mapping
14:14:52   in 'reader', line 69, column 3:
14:14:52        spec:
14:14:52        ^
14:14:52  expected <block end>, but found BlockEntry
14:14:52   in 'reader', line 71, column 3:
14:14:52        - entrypoint:
14:14:52        ^

Describe the solution you'd like.

CVP team would like the operator-sdk bundle validate command to validate tests/scorecard/config/yaml.

[asergienk]

c/c @camilamacedo86

[camilamacedo86]

Hi @asergienk,

I think a possible check here would ensure that the config is parsable such as we do with the CSV spec. But would still not be possible to ensure that the info added on it is valid.

I think we could make the Scorecard command ( which was used to run this config ) first check if the config file is parsable if not then raise a more friendly error message that helps us users know that fails because of it.

c/c @jmrodri @theishshah

[theishshah]

This would be a great first issue for contributors interested in Scorecard, please reach out to me for any guidance I can help provide :D

[apodhrad]

Hi @theishshah do you already use some kind of yaml validation in operator-sdk? I was thinking of creating some schema which could be used the scorecard tool itself as well as separately with other languages/systems.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci[bot]]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.