crossplane-conformance icon indicating copy to clipboard operation
crossplane-conformance copied to clipboard
verified publisher icon cncf

Submit initial provider-ibm-cloud conformance results

Open pdettori opened this issue 4 years ago • 10 comments

We are interested in having provider-ibm-cloud to be one of the first certified providers for Crossplane. This is a tracking issue to submit the initial test results and provide feedback on the conformance test suite.

pdettori avatar Apr 29 '21 14:04 pdettori

Thanks @pdettori! I'm working on the conformance suite this week and will let you know once there's something ready to test out.

negz avatar Apr 29 '21 20:04 negz

Hi @pdettori! We'd love for you to be one of the first providers to submit a conformance request. The instructions are now available at https://github.com/cncf/crossplane-conformance/blob/main/instructions.md. Is this something you'd be interested in trying out?

negz avatar May 25 '21 19:05 negz

Hi @negz, great, and I am interested in giving it a try, thank you

pdettori avatar May 25 '21 19:05 pdettori

@negz I was able to run the test following the instructions. Some comments:

  • The first time I tried the sonobuoy run ... command complained that I should have had a Kubernetes version 1.17 or higher (my cluster was at version 1.16). I think the instructions should specify this requirement. Note also that the latest Crossplane install instructions for self-hosted still use kind create cluster --image kindest/node:v1.16.15 --wait 5m, so it would be helpful to update the Crossplane docs as well to recommend a more recent version of kube (I used v1.20.2 with success).
  • After updating my kube, I run the sonobuoy run ... command for the IBM Cloud provider. The first time the command was apparently stuck, after CTRL+C and re-running the command this time it run through completion.
  • I think the instructions assume that the conformance test passes, but do not seem to indicate what to do when the conformance test does not pass; in my case I've got:
$ sonobuoy status
                            PLUGIN     STATUS   RESULT   COUNT
   crossplane-provider-conformance   complete   failed       1

but I was not sure on how to find out which non-conformant items I had

  • I did follow the instructions to extract the report in ./results, then I browsed there and I looked into results\plugins\crossplane-provider-conformance\results\global\report.log - is this the right place to look ?
  • In that file, I noticed the following warnings:
  provider_test.go:144: Testing that managed resources defined by ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" appear conformant.
    provider_test.go:183: At least one "groupmemberships.iamaccessgroupsv2.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "customroles.iampolicymanagementv1.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "scalinggroups.ibmclouddatabasesv5.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "policies.iampolicymanagementv1.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "autoscalinggroups.ibmclouddatabasesv5.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "resourceinstances.resourcecontrollerv2.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "resourcekeys.resourcecontrollerv2.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "whitelists.ibmclouddatabasesv5.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "accessgroups.iamaccessgroupsv2.ibmcloud.crossplane.io" managed resource must be manually created in advance.
    provider_test.go:183: At least one "accessgrouprules.iamaccessgroupsv2.ibmcloud.crossplane.io" managed resource must be manually created in advance.

does it mean that I have to create an instance of each managed resource before running the tests ?

  • I have got also other issues, such as all the CRDs should be at version v1 rather than v1beta1
  • There are also other warnings in the TestPackage/DeploysConformantCRDs section I am not too clear about:
 provider_test.go:131: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" must define at least one conformant managed resource
    provider_test.go:135: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" must define at least one conformant provider config
    provider_test.go:139: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" must define at least one conformant provider config usage

pdettori avatar May 26 '21 01:05 pdettori

Thanks @pdettori for this great feedback - and thanks also for being a guinea pig for this process.

I did follow the instructions to extract the report in ./results, then I browsed there and I looked into results\plugins\crossplane-provider-conformance\results\global\report.log - is this the right place to look ?

It is - we can capture this in the instructions.

does it mean that I have to create an instance of each managed resource before running the tests ?

It does. This is captured in the crossplane/conformance docs but we should mention it here too. Do you think the error could be worded more explicitly?

I have got also other issues, such as all the CRDs should be at version v1 rather than v1beta1

Hrm this is surprising to me - what makes you say this?

There are also other warnings in the TestPackage/DeploysConformantCRDs section I am not too clear about:

I’m not at my laptop right now but I think these are aggregate failures; i.e. they’re failing because of the more specific issues with no managed resources existing. I’ll double check this tomorrow.

negz avatar May 26 '21 02:05 negz

@negz thanks for your response, it helps. Re:

Hrm this is surprising to me - what makes you say this?

This is what I see in that section of the report:

=== RUN   TestPackage/DeploysConformantCRDs
    provider_test.go:93: Testing that ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" delivers only conformant CRDs.
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[0]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[1]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[2]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[3]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[4]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[5]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[6]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[7]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[8]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[9]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[10]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:104: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" status.objectRefs[11]: -want, +got:
          schema.GroupVersionKind{
          	Group:   "apiextensions.k8s.io",
        - 	Version: "v1",
        + 	Version: "v1beta1",
          	Kind:    "CustomResourceDefinition",
          }
    provider_test.go:131: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" must define at least one conformant managed resource
    provider_test.go:135: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" must define at least one conformant provider config
    provider_test.go:139: ProviderRevision "crossplane-provider-ibm-cloud-e0a75509c392" must define at least one conformant provider config usage

pdettori avatar May 26 '21 03:05 pdettori

@pdettori Ah - got it. That makes sense. Yes, we require Crossplane providers to install v1 CRDs. This is because we want the definition of a structural schema to be mandatory, and only v1 CRDs require this. Some of the common subtypes in crossplane-runtime also assume support for defaulting which is also only supported by v1 CRDs.

https://github.com/crossplane-contrib/provider-ibm-cloud/blob/c033b/apis/generate.go#L26

The good news is that updating to v1 CRDs should be an easy and transparent change as far as users are concerned. It involves changing the above line, and running make generate to generate new v1 CRDs.

- //go:generate go run -tags generate sigs.k8s.io/controller-tools/cmd/controller-gen object:headerFile=../hack/boilerplate.go.txt paths=./... crd:trivialVersions=true output:artifacts:config=../package/crds
+ //go:generate go run -tags generate sigs.k8s.io/controller-tools/cmd/controller-gen object:headerFile=../hack/boilerplate.go.txt paths=./... crd:trivialVersions=true,crdVersions=v1 output:artifacts:config=../package/crds

negz avatar May 26 '21 19:05 negz

@negz thank you, will update the CRDs and report back on the testing results

pdettori avatar May 26 '21 20:05 pdettori

Hey @negz I did some more progress after updating the CRDs to v1. Now I get this report: report.log

I see now that for most CRDs there is a message saying:

provider_test.go:343: managed resources must be in the 'crossplane' CRD category
provider_test.go:347: managed resources must be in the 'managed' CRD category

Is this because I miss this in the CRDs ? :

names:    
  categories:    
   - crossplane    
   - managed
   - ibmcloud

if that is the case, how do I make the generator generate those tags ?

pdettori avatar Jun 05 '21 03:06 pdettori

Never mind, I found the annotation to use and I was able to get past that point. At this point I will just need to create an instance of each resource to complete the testing.

pdettori avatar Jun 05 '21 03:06 pdettori