test selection in kuttl scorecard image is ignored and all the test are actually run

[bdattoma]

Bug Report

What did you do?

I am using the kuttl scorecard image (i.e., quay.io/operator-framework/scorecard-test-kuttl:v2.0.0) and in my config.yaml I have two test suites. When I run the scorecard command using the test selection, the scorecard actually runs all the tests regardless the selection I specify. operator-sdk scorecard ./bundle --selector='suite=sanity-suite' --namespace=<ns_name> --skip-cleanup --wait-time <timeout> --service-account <SA_name>

This is the config.yaml:

apiVersion: scorecard.operatorframework.io/v1alpha3
kind: Configuration
metadata:
  name: config
stages:
- parallel: true
  tests:
  - entrypoint:
    - entrypoint
    - 00-ods-ci-settings
    - 01-smoke
    image: quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
    labels:
      cluster-product: ocp
      cluster-size: small
      phase: msp-main
      test: smoke
      suite: smoke-suite
    storage:
      spec:
        mountPath: {}
  - entrypoint:
    - entrypoint
    - 00-ods-ci-settings
    - 02-sanity
    image: quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
    labels:
      cluster-product: ocp
      cluster-size: small
      phase: msp-main
      test: sanity
      suite: sanity-suite
    storage:
      spec:
        mountPath: {}

What did you expect to see?

I would expect to see only 00-ods-ci-settings and 02-sanity running

What did you see instead? Under which circumstances?

I see instead both the test suites running:

Environment

Operator type:

Kubernetes cluster type:

Openshift 4.10.12

$ operator-sdk version

operator-sdk version: "v1.19.1", commit: "079d8852ce5b42aa5306a1e33f7ca725ec48d0e3"

$ go version (if language is Go)

go version: "go1.17.8"

$ kubectl version

kubectl version: "1.21"

Possible Solution

One possiblesolution would invole replacing lines from 6 to 9 in the entrypoint script with:

if [ $# -eq 0 ]
  then
    kubectl-kuttl test ${KUTTL_PATH} \
      --config=${KUTTL_CONFIG} \
      --namespace=${SCORECARD_NAMESPACE} \
      --report=JSON --artifacts-dir=/tmp > /tmp/kuttl.stdout 2> /tmp/kuttl.stderr
  else
    for i in $*; do 
      kubectl-kuttl test ${KUTTL_PATH} \
      --test $i  \
      --config=${KUTTL_CONFIG} \
      --namespace=${SCORECARD_NAMESPACE} \
      --report=JSON --artifacts-dir=/tmp > /tmp/kuttl.stdout 2> /tmp/kuttl.stderr
    done
fi

One problem to address would be to make the scorecard able to handle more than one kuttl result file

Additional context

[asmacdo]

Thanks!

[jmrodri]

Took this ticket over from Ish.

[apodhrad]

I have created also a Jira issue for a better tracking in some Jira dashboards https://issues.redhat.com/browse/OSDK-2479

[jmrodri]

So if I do not pass in a selector, both tests are run. I have some debug logging added hence the XXX output. But I get two different tests being run (ignore the Results section for now.

[jesusr@transam memcached-operator{kuttl}]$ operator-sdk scorecard ./bundle  --service-account kuttl-user --wait-time 60s 
XXX before: selector = []
XXX after: selector = []
XXX test [{quay.io/operator-framework/scorecard-test-kuttl:v2.0.0  [entrypoint 00-smoke] map[cluster-product:ocp cluster-size:small phase:msp-main suite:smoke-suite test:smoke] {{{}}}}]
XXX o.Selector == ""
XXX appending test to selected listXXX test [{quay.io/operator-framework/scorecard-test-kuttl:v2.0.0  [entrypoint 00-sanity] map[cluster-product:ocp cluster-size:small phase:msp-main suite:sanity-suite test:sanity] {{{}}}}]
XXX o.Selector == ""
XXX appending test to selected listXXX selected list: [{quay.io/operator-framework/scorecard-test-kuttl:v2.0.0  [entrypoint 00-smoke] map[cluster-product:ocp cluster-size:small phase:msp-main suite:smoke-suite test:smoke] {{{}}}} {quay.io/operator-framework/scorecard-test-kuttl:v2.0.0  [entrypoint 00-sanity] map[cluster-product:ocp cluster-size:small phase:msp-main suite:sanity-suite test:sanity] {{{}}}}]
--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
Entrypoint: [entrypoint 00-sanity]
Labels:
        "suite":"sanity-suite"
        "test":"sanity"
        "cluster-product":"ocp"
        "cluster-size":"small"
        "phase":"msp-main"
Results:
        Name: sanity
        State: pass


        Name: smoke
        State: pass



--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
Entrypoint: [entrypoint 00-smoke]
Labels:
        "cluster-size":"small"
        "phase":"msp-main"
        "suite":"smoke-suite"
        "test":"smoke"
        "cluster-product":"ocp"
Results:
        Name: sanity
        State: fail

        Errors:
                namespaces "sanity" not found

        Name: smoke
        State: fail

        Errors:
                namespaces "smoke" not found

I see Entrypoint: [entrypoint 00-sanity] and Entrypoint: [entrypoint 00-smoke] which are the two tests I have in my config.

[jmrodri]

Passing in a selector causes it to only run that test that I select, so I think the selector portion is actually working.

[jesusr@transam memcached-operator{kuttl}]$ operator-sdk scorecard ./bundle  --service-account kuttl-user --wait-time 60s --selector='suite=sanity-suite'
XXX before: selector = [suite=sanity-suite]
XXX after: selector = [suite=sanity-suite]
XXX test [{quay.io/operator-framework/scorecard-test-kuttl:v2.0.0  [entrypoint 00-smoke] map[cluster-product:ocp cluster-size:small phase:msp-main suite:smoke-suite test:smoke] {{{}}}}]
XXX o.Selector = [suite=sanity-suite]
XXX matches? [false]
XXX test [{quay.io/operator-framework/scorecard-test-kuttl:v2.0.0  [entrypoint 00-sanity] map[cluster-product:ocp cluster-size:small phase:msp-main suite:sanity-suite test:sanity] {{{}}}}]
XXX o.Selector = [suite=sanity-suite]
XXX matches? [true]
XXX appending test to selected listXXX selected list: [{quay.io/operator-framework/scorecard-test-kuttl:v2.0.0  [entrypoint 00-sanity] map[cluster-product:ocp cluster-size:small phase:msp-main suite:sanity-suite test:sanity] {{{}}}}]
--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
Entrypoint: [entrypoint 00-sanity]
Labels:
        "cluster-product":"ocp"
        "cluster-size":"small"
        "phase":"msp-main"
        "suite":"sanity-suite"
        "test":"sanity"
Results:
        Name: smoke
        State: pass


        Name: sanity
        State: pass

What's mysterious is why are there Results for both every time.

[jmrodri]

And it looks like the test pod is returning a result for all of the tests regardless of which one we run.

[jesusr@transam operator-sdk{kuttl}]$ k logs scorecard-test-swwh -n default -c scorecard-test
{
    "results": [
        {
            "name": "sanity",
            "state": "pass"
        },
        {
            "name": "smoke",
            "state": "pass"
        }
    ]
}

[bdattoma]

@jmrodri in my tests, I create a pod for each test suite and when I run it with a selector, I see all the pods being created instead of the only one I requested.

the command: So the logs report correctly my selection in the Entrypoint section (i.e., 00-ods-ci-settings, 01-smoke).

But on the cluster, I found both the pods from 01-smoke and 04-sanity-pt3

Could you pls check on your cluster if you get a similar situation?

[jmrodri]

@bdattoma yes that is happening. What I was trying to say in my previous comments about the selector "working" was that scorecard is parsing it correctly and thinking it is running just one test. But the kuttl image isn't respecting that hence we see the duplicate pods.

[jmrodri]

@bdattoma wow this is really an annoying bug :) So I took the suggested change to the entrypoint which is enough for the use case specified here where you only want to run one test out of n tests. But if you want to run 2 of n tests, this fails because kuttl will only keep the output of the last test which is written to the JSON report /tmp/kuttl-test.json. It would be easier if kuttl would output a different json per test name but it doesn't. And it will not append to the existing file so if I try to run sanity and smoke it will report just smoke.

I'm looking to see what I can do in the entrypoint to handle that json file. Maybe I need to move it after each test run. But then we need a way to merge it back together or have scorecard-test-kuttl binary collect them all, that might be better. So I haven't abandoned this issue, just a lot trickier than I thought.

[jmrodri]

Looks like the entrypoint change and a tweak to the configuration would work. Here's what I have so far:

[jesusr@transam memcached-operator{kuttl}]$ operator-sdk scorecard ./bundle  --service-account kuttl-user --wait-time 60s --selector='suite=sanity-suite' --skip-cleanup
--------------------------------------------------------------------------------
Image:      quay.io/jesusr/scorecard-test-kuttl:dev
Entrypoint: [entrypoint sanity]
Labels:
        "test":"sanity"
        "cluster-product":"ocp"
        "cluster-size":"small"
        "phase":"msp-main"
        "suite":"sanity-suite"
Results:
        Name: sanity
        State: pass

[jmrodri]

But now that I look at your config it may not work.

apiVersion: scorecard.operatorframework.io/v1alpha3
kind: Configuration
metadata:
  name: config
stages:
- parallel: true
  tests:
  - entrypoint:
    - entrypoint
    - 00-ods-ci-settings
    - 01-smoke # THIS WOULD HAVE TO MATCH THE kuttle TEST DIR
...
  - entrypoint:
    - entrypoint
    - 00-ods-ci-settings
    - 02-sanity # THIS WOULD HAVE TO MATCH THE kuttle TEST DIR
...

From the docs example, the parameter to the entrypoint would have to be list-pods or list-other. Because 00-pod won't match anything.

$ tree ./bundle
./bundle
├── manifests
│   ├── cache.example.com_memcacheds_crd.yaml
│   └── memcached-operator.clusterserviceversion.yaml
├── metadata
│   └── annotations.yaml
└── tests
    └── scorecard
        ├── config.yaml
        └── kuttl
            ├── kuttl-test.yaml
            └── list-pods
                ├── 00-assert.yaml
                └── 00-pod.yaml
            └── list-other
                ├── 00-assert.yaml
                └── 00-pod.yaml

[bdattoma]

m the docs example

hi @jmrodri thanks for investigating.

Which doc examples are you referring to? Anyway, 00- and 01- match a kuttl directory in my scorecard. I don't have list-pods and list-other folders, but I have a folder for each entrypoint.

[jmrodri]

@bdattoma I rebuilt the scorecard-test-kuttl image quay.io/operator-framework/scorecard-test-kuttl:v2.1.0

[bdattoma]

thanks @jmrodri I will give a try as soon as I can