Is there any way to configure the security options of connecting to a ZooKeeper cluster?

[johnnyleitrim]

I'm trying to connect my ClickHouse cluster to an existing ZK cluster (which has TLS enabled), but the ClickHouse servers always throw the following errors:

Received exception from server (version 21.3.20):
Code: 999. DB::Exception: Received from service-demo-01.clickhouse.svc.test01.k8s.run:9000. DB::Exception: All connection tries failed while connecting to ZooKeeper. nodes: 172.25.20.51:2181, 172.25.11.2:2181
Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper (version 21.3.20.1 (official build)), 172.25.20.51:2181
Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper (version 21.3.20.1 (official build)), 172.25.11.2:2181
Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper (version 21.3.20.1 (official build)), 172.25.20.51:2181
Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper (version 21.3.20.1 (official build)), 172.25.11.2:2181
Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper (version 21.3.20.1 (official build)), 172.25.20.51:2181
Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper (version 21.3.20.1 (official build)), 172.25.11.2:2181
 (Connection loss). (KEEPER_EXCEPTION)

The ZK nodes have the following errors:

2022-02-09 14:46:18,267 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-3]
2022-02-09 14:46:18,267 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-3]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002c0000000000000000000000000000753000000000000000000000001000000000000000000000000000000000
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
...
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002c0000000000000000000000000000753000000000000000000000001000000000000000000000000000000000
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1269)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1337)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
	... 17 more

I couldn't find a way to configure the ClickHouse cluster to use secure connections when talking to ZK. Here's my existing yaml file:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  defaults:
    templates:
      serviceTemplate: service-template
      replicaServiceTemplate: replica-service-template
  configuration:
    profiles:
      # Configure max memory usage for queries for 512MB
      default/max_memory_usage: "536870912"
      default/max_memory_usage_for_user: "536870912"
    clusters:
      - name: "demo-01"
        zookeeper:
          nodes:
          - host: my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.clickhouse.svc.test01.k8s.run
          - host: my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.clickhouse.svc.test01.k8s.run
        templates:
          podTemplate: pod-template
          clusterServiceTemplate: cluster-service-template
        layout:
          shardsCount: 2
          replicasCount: 1
  templates:
    podTemplates:
      - name: pod-template
        spec:
          containers:
          - name: clickhouse
            image: yandex/clickhouse-server:21.3
            resources:
              requests:
                memory: 2Gi
                cpu: "2"
              limits:
                memory: 2Gi
                cpu: "2"
          namespaceDomainPattern: "%s.test01.k8s.run"
    serviceTemplates:
      - name: service-template
        generateName: "service-{chi}"
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
          type: ClusterIP
          clusterIP: None
      - name: cluster-service-template
        generateName: "service-{cluster}"
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
          type: ClusterIP
          clusterIP: None
      - name: replica-service-template
        generateName: "service-{chi}-{shard}-{replica}"
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
          type: ClusterIP
          clusterIP: None

[mbkh68]

Any update on this?

[alex-zaitsev]

@mbkh68 , check https://github.com/Altinity/clickhouse-operator/blob/master/tests/e2e/manifests/chi/test-041-secure-zookeeper.yaml