feat: add Kerberos authentication for Kafka

[adwk67]

Fixes https://github.com/stackabletech/kafka-operator/issues/655 Pending decision: https://github.com/stackabletech/decisions/issues/28 ( :heavy_check_mark: closed/done)

Openshift/OKD tests :green_circle: :

--- PASS: kuttl (520.16s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-true (41.05s)
        --- PASS: kuttl/harness/kerberos_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-true_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit (100.10s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-true (71.71s)
        --- PASS: kuttl/harness/smoke_kafka-3.8.0_zookeeper-3.9.2_use-client-tls-true_openshift-true (107.86s)
        --- PASS: kuttl/harness/logging_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-true (117.43s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-true (130.04s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-true_use-client-auth-tls-true_openshift-true (164.68s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-true_use-client-auth-tls-true_openshift-true (151.38s)
PASS

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• [x] Changes are OpenShift compatible
• [x] CRD changes approved. Awaiting https://github.com/stackabletech/operator-rs/pull/880.
• [x] CRD documentation for all fields, following the style guide.
• [x] Helm chart can be installed and deployed operator works
• [x] Integration tests passed (for non trivial changes)
• [x] Changes need to be "offline" compatible

Reviewer

• [ ] Code contains useful comments
• [ ] Code contains useful logging statements
• [ ] (Integration-)Test cases added
• [ ] Documentation added or updated. Follows the style guide.
• [x] Changelog updated
• [ ] Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• [ ] Feature Tracker has been updated
• [ ] Proper release label has been added
• [ ] Roadmap has been updated

[adwk67]

@sbernauer I tried to use one principal for all brokers, as we discussed, but could not quite get it to work. This variation works with the brokers (they all come up): principal=\"kafka/test-kafka-broker-default.kuttl-test-glorious-airedale.svc.cluster.local@$KERBEROS_REALM\" but I was not able to do the same for the client job. I set BROKER=test-kafka-broker-default.$NAMESPACE.svc.cluster.local:9093 which allows me to query the topics but it fails with authentication errors when in producer/consumer mode.

[adwk67]

Ran nightly suite again locally following latest changes :green_circle:

--- PASS: kuttl (884.94s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false (73.35s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-true_use-client-auth-tls-false_openshift-false (177.87s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-true_use-client-auth-tls-false_openshift-false (178.71s)
        --- PASS: kuttl/harness/kerberos_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit (95.47s)
        --- PASS: kuttl/harness/kerberos_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit (105.52s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-true_use-client-auth-tls-true_openshift-false (125.30s)
        --- PASS: kuttl/harness/smoke_kafka-3.8.0_zookeeper-3.9.2_use-client-tls-true_openshift-false (141.44s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-false_use-client-auth-tls-true_openshift-false (148.59s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-false_use-client-auth-tls-false_openshift-false (65.18s)
        --- PASS: kuttl/harness/logging_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false (91.36s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-false (88.81s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-false (28.44s)
        --- PASS: kuttl/harness/smoke_kafka-3.8.0_zookeeper-3.9.2_use-client-tls-false_openshift-false (58.76s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-false_use-client-auth-tls-false_openshift-false (120.38s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-true_use-client-auth-tls-true_openshift-false (160.60s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-false_use-client-auth-tls-true_openshift-false (82.92s)

[adwk67]

Re-tested :green_circle:

[adwk67]

Re-testing before merge:

• AKS :green_circle: https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/kafka-operator-it-custom/12/
• OKD :green_circle: https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/kafka-operator-it-custom/13/