rocketmq icon indicating copy to clipboard operation
rocketmq copied to clipboard
verified publisher icon apache

[Bug] 4.9.7版本多主多从集群

Open wolf27w opened this issue 1 year ago • 2 comments

Before Creating the Bug Report

  • [X] I found a bug, not just asking a question, which should be created in GitHub Discussions.

  • [X] I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

  • [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

运行平台是kubernetes中部署的多主多从集群

RocketMQ version

rocketmq集群版本4.9.7

JDK Version

openjdk1.8

Describe the Bug

我在kubernetes中部署了一个rocketmq多主多从集群，在集群内部访问没有任何问题，但是我通过代理访问集群内部的rocketmq集群就会出现消息重复，线看一下我的集群 image 我的多主多从配置 broker-a.conf

brokerClusterName=DefaultCluster
brokerName=broker-a
brokerEnableProxy=true
brokerId=0
namesrvAddr=mq-ns.ops.svc.cluster.local:9876
brokerIP1=192.168.6.93
defaultTopicQueueNums=4
#Whether to allow the broker to automatically create topics, it is recommended to open offline and close online
autoCreateTopicEnable=true
#Whether to allow the broker to automatically create a subscription group, it is recommended to start offline and close online
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=300000
destroyMapedFileInterval=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000
checkTransactionMessageEnable=false
sendMessageThreadPoolNums=16
pullMessageThreadPoolNums=16
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH

broker-b.conf

brokerClusterName=DefaultCluster
brokerName=broker-b
brokerId=0
brokerEnableProxy=true
brokerIP1=192.168.6.93
namesrvAddr=mq-ns.ops.svc.cluster.local:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=300000
destroyMapedFileInterval=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000
checkTransactionMessageEnable=false
sendMessageThreadPoolNums=16
pullMessageThreadPoolNums=16
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH

先看一下我的nodeport配置 image

我的nginx代理配置，在93上面

stream {
    upstream prod-eft-link-mq9876 {
        least_conn;
        server 192.168.6.237:30740;
        server 192.168.6.236:30740;
    }
    upstream prod-eft-link-mq10909 {
        least_conn;
        server 192.168.6.236:30532;
        server 192.168.6.237:32727;
    }
    upstream prod-eft-link-mq10911 {
        least_conn;
        server 192.168.6.236:31213;
        server 192.168.6.237:32742;
    }
    server {
        listen 9876;#9876
        proxy_pass prod-eft-link-mq9876;
    }
   server {
        listen 10909;#10909
        proxy_pass prod-eft-link-mq10909;
    }
   server {
        listen 10911; #10911
        proxy_pass prod-eft-link-mq10911;
    }
}

Steps to Reproduce

如果我在broker-a和broker-b的主节点里增加了brokerIP1=192.168.6.93配置，我就会消息里出现重复 image 如果我不在配置文件里增加brokerIP1=192.168.6.93，消息不重复了，但是我客户端通过nginx无法直接和rocketmq集群通信

What Did You Expect to See?

我希望rocketmq集群可以实现客户端通过代理连接到rocketmq集群，并且消息也不会重复

What Did You See Instead?

这里是我的yaml文件

 cat broker-a.yaml
---
apiVersion: v1
data:
  broker.conf: |-
    brokerClusterName=DefaultCluster
    brokerName=broker-a
    brokerId=0
    namesrvAddr=mq-ns.ops.svc.cluster.local:9876
    defaultTopicQueueNums=4
    #Whether to allow the broker to automatically create topics, it is recommended to open offline and close online
    autoCreateTopicEnable=true
    #Whether to allow the broker to automatically create a subscription group, it is recommended to start offline and close online
    autoCreateSubscriptionGroup=true
    listenPort=10911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    destroyMapedFileInterval=120000
    redeleteHangedFileInterval=120000
    diskMaxUsedSpaceRatio=88
    maxMessageSize=65536
    flushCommitLogLeastPages=4
    flushConsumeQueueLeastPages=2
    flushCommitLogThoroughInterval=10000
    flushConsumeQueueThoroughInterval=60000
    checkTransactionMessageEnable=false
    sendMessageThreadPoolNums=16
    pullMessageThreadPoolNums=16
    brokerRole=ASYNC_MASTER
    flushDiskType=ASYNC_FLUSH
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: broker-a
cat broker-a-s.yaml
---
apiVersion: v1
data:
  broker.conf: |-
    brokerClusterName=DefaultCluster
    brokerName=broker-a
    brokerId=1
    namesrvAddr=mq-ns.ops.svc.cluster.local:9876
    defaultTopicQueueNums=4
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    listenPort=10911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    destroyMapedFileInterval=120000
    redeleteHangedFileInterval=120000
    diskMaxUsedSpaceRatio=88
    maxMessageSize=65536
    flushCommitLogLeastPages=4
    flushConsumeQueueLeastPages=2
    flushCommitLogThoroughInterval=10000
    flushConsumeQueueThoroughInterval=60000
    checkTransactionMessageEnable=false
    sendMessageThreadPoolNums=16
    pullMessageThreadPoolNums=16
    brokerRole=SLAVE
    flushDiskType=ASYNC_FLUSH
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: broker-a-s
cat broker-b.yaml
---
apiVersion: v1
data:
  broker.conf: |-
    brokerClusterName=DefaultCluster
    brokerName=broker-b
    brokerId=0
    namesrvAddr=mq-ns.ops.svc.cluster.local:9876
    defaultTopicQueueNums=4
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    listenPort=10911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    destroyMapedFileInterval=120000
    redeleteHangedFileInterval=120000
    diskMaxUsedSpaceRatio=88
    maxMessageSize=65536
    flushCommitLogLeastPages=4
    flushConsumeQueueLeastPages=2
    flushCommitLogThoroughInterval=10000
    flushConsumeQueueThoroughInterval=60000
    checkTransactionMessageEnable=false
    sendMessageThreadPoolNums=16
    pullMessageThreadPoolNums=16
    brokerRole=ASYNC_MASTER
    flushDiskType=ASYNC_FLUSH
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: broker-b
cat broker-b-s.yaml
---
apiVersion: v1
data:
  broker.conf: |-
    brokerClusterName=DefaultCluster
    brokerName=broker-b
    brokerId=1
    namesrvAddr=mq-ns.ops.svc.cluster.local:9876
    defaultTopicQueueNums=4
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    listenPort=10911
    deleteWhen=04
    fileReservedTime=120
    mapedFileSizeCommitLog=1073741824
    mapedFileSizeConsumeQueue=300000
    destroyMapedFileInterval=120000
    redeleteHangedFileInterval=120000
    diskMaxUsedSpaceRatio=88
    maxMessageSize=65536
    flushCommitLogLeastPages=4
    flushConsumeQueueLeastPages=2
    flushCommitLogThoroughInterval=10000
    flushConsumeQueueThoroughInterval=60000
    checkTransactionMessageEnable=false
    sendMessageThreadPoolNums=16
    pullMessageThreadPoolNums=16
    brokerRole=SLAVE
    flushDiskType=ASYNC_FLUSH
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: broker-b-s

pod的yaml文件信息

cat broker-a-pods.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-a
  name: broker-a
spec:
  # type: NodePort
  type: ClusterIP
  ports:
  - port: 10911
    targetPort: 10911
    name: broker-port
  - port: 10909
    targetPort: 10909
    name: broker-vip
  - port: 10912
    targetPort: 10912
    name: broker-hp
  selector:
    app: broker-a
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-a
spec:
  serviceName: broker-a
  replicas: 1
  selector:
    matchLabels:
      app: broker-a
  template:
    metadata:
     labels:
       app: broker-a
    spec:
      # affinity:
      #   #podAntiAffinity 反亲合性
      #   podAntiAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       - labelSelector:
      #           matchExpressions:
      #             - key: "app"
      #               operator: In
      #               values:
      #                 - broker-a
      #         topologyKey: "kubernetes.io/hostname"
      containers:
      - name: broker-a
        image: apache/rocketmq:4.9.7
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: NAMESRV_ADDR
            value: "mq-ns.ops.svc.cluster.local:9876"
          - name: JAVA_OPT_EXT
            value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m"
        #command: ["sh","-c","mqbroker -c /home/broker.conf"]
        command:
          - /bin/sh
        args:
          - mqbroker
          - '-c'
          - /home/broker.conf
        volumeMounts:
          - name: rocketmq-data
            mountPath: /home/rocketmq/logs
          - name: rocketmq-data
            mountPath: /home/rocketmq/store
          - name: broker-config
            mountPath: /home/broker.conf
            subPath: broker.conf
        readinessProbe:
          tcpSocket:
            port: 10911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: broker-a
      #启动依赖
      initContainers:
      - name: init-mq-ns
        image: busybox:latest
        command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"]

  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
cat broker-a-s-pods.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-a-s
  name: broker-a-s
spec:
  # type: NodePort
  type: ClusterIP
  ports:
  - port: 10911
    targetPort: 10911
    name: broker-port
  - port: 10909
    targetPort: 10909
    name: broker-vip
  - port: 10912
    targetPort: 10912
    name: broker-hp
  selector:
    app: broker-a-s
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-a-s
spec:
  serviceName: broker-a-s
  replicas: 1
  selector:
    matchLabels:
      app: broker-a-s
  template:
    metadata:
     labels:
       app: broker-a-s
    spec:
      # affinity:
      #   #podAntiAffinity 反亲合性
      #   podAntiAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       - labelSelector:
      #           matchExpressions:
      #             - key: "app"
      #               operator: In
      #               values:
      #                 - broker-a-s
      #         topologyKey: "kubernetes.io/hostname"
      containers:
      - name: broker-a-s
        image: apache/rocketmq:4.9.7   #下面使用的是用官方地址自行构建的镜像，推荐直接使用这个
        #image: apacherocketmq/rocketmq:4.8.0
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: NAMESRV_ADDR
            value: "mq-ns.ops.svc.cluster.local:9876"
          - name: JAVA_OPT_EXT
            value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m"
        #command: ["sh","-c","mqbroker -c /home/broker.conf"]
        command:
          - /bin/sh
        args:
          - mqbroker
          - '-c'
          - /home/broker.conf
        volumeMounts:
          - name: rocketmq-data
            mountPath: /home/rocketmq/logs
          - name: rocketmq-data
            mountPath: /home/rocketmq/store
          - name: broker-config
            mountPath: /home/broker.conf
            subPath: broker.conf
        readinessProbe:
          tcpSocket:
            port: 10911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: broker-a-s
        #启动依赖
      initContainers:
      - name: init-mq-ns
        image: busybox:latest
        command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"]

  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
cat broker-b-pods.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-b
  name: broker-b
spec:
  # type: NodePort
  type: ClusterIP
  ports:
  - port: 10911
    targetPort: 10911
    name: broker-port
  - port: 10909
    targetPort: 10909
    name: broker-vip
  - port: 10912
    targetPort: 10912
    name: broker-hp
  selector:
    app: broker-b
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-b
spec:
  serviceName: broker-b
  replicas: 1
  selector:
    matchLabels:
      app: broker-b
  template:
    metadata:
     labels:
       app: broker-b
    spec:
      # affinity:
      #   #podAntiAffinity 反亲合性
      #   podAntiAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       - labelSelector:
      #           matchExpressions:
      #             - key: "app"
      #               operator: In
      #               values:
      #                 - broker-b
      #         topologyKey: "kubernetes.io/hostname"
      containers:
      - name: broker-b
        image: apache/rocketmq:4.9.7   #下面使用的是用官方地址自行构建的镜像，推荐直接使用这个
        #image: apacherocketmq/rocketmq:4.8.0
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: NAMESRV_ADDR
            value: "mq-ns.ops.svc.cluster.local:9876"
          - name: JAVA_OPT_EXT
            value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m"
        #command: ["sh","-c","mqbroker -c /home/broker.conf"]
        command:
          - /bin/sh
        args:
          - mqbroker
          - '-c'
          - /home/broker.conf
        volumeMounts:
          - name: rocketmq-data
            mountPath: /home/rocketmq/logs
          - name: rocketmq-data
            mountPath: /home/rocketmq/store
          - name: broker-config
            mountPath: /home/broker.conf
            subPath: broker.conf
        readinessProbe:
          tcpSocket:
            port: 10911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: broker-b
        #启动依赖
      initContainers:
      - name: init-mq-ns
        image: busybox:latest
        command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"]

  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
cat broker-b-s-pods.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-b-s
  name: broker-b-s
spec:
  # type: NodePort
  type: ClusterIP
  ports:
  - port: 10911
    targetPort: 10911
    name: broker-port
  - port: 10909
    targetPort: 10909
    name: broker-vip
  - port: 10912
    targetPort: 10912
    name: broker-hp
  selector:
    app: broker-b-s
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-b-s
spec:
  serviceName: broker-b-s
  replicas: 1
  selector:
    matchLabels:
      app: broker-b-s
  template:
    metadata:
     labels:
       app: broker-b-s
    spec:
      # affinity:
      #   #podAntiAffinity 反亲合性
      #   podAntiAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       - labelSelector:
      #           matchExpressions:
      #             - key: "app"
      #               operator: In
      #               values:
      #                 - broker-b-s
      #         topologyKey: "kubernetes.io/hostname"
      containers:
      - name: broker-b-s
        image: apache/rocketmq:4.9.7   #下面使用的是用官方地址自行构建的镜像，推荐直接使用这个
        #image: apacherocketmq/rocketmq:4.8.0
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: NAMESRV_ADDR
            value: "mq-ns.ops.svc.cluster.local:9876"
          - name: JAVA_OPT_EXT
            value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m"
        #command: ["sh","-c","mqbroker -c /home/broker.conf"]
        command:
          - /bin/sh
        args:
          - mqbroker
          - '-c'
          - /home/broker.conf
        volumeMounts:
          - name: rocketmq-data
            mountPath: /home/rocketmq/logs
          - name: rocketmq-data
            mountPath: /home/rocketmq/store
          - name: broker-config
            mountPath: /home/broker.conf
            subPath: broker.conf
        readinessProbe:
          tcpSocket:
            port: 10911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: broker-b-s
        #启动依赖
      initContainers:
      - name: init-mq-ns
        image: busybox:latest
        command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"]

  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
cat mq-console.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mq-console
  name: mq-cs
spec:
  type: ClusterIP
  ports:
  - port: 8080
    targetPort: 8080
    name: console-port
  selector:
    app: mq-console
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mq-console
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mq-console
  template:
    metadata:
     labels:
       app: mq-console
    spec:
      # affinity:
      #   #podAntiAffinity 反亲合性
      #   podAntiAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       - labelSelector:
      #           matchExpressions:
      #             - key: "app"
      #               operator: In
      #               values:
      #                 - mq-console
      #         topologyKey: "kubernetes.io/hostname"
      containers:
      - name: mq-console
        image: gguilin/rocketmq-console-ng
        imagePullPolicy: IfNotPresent
        env:
          - name: JAVA_OPTS
            value: "-Drocketmq.namesrv.addr=mq-ns.ops.svc.cluster.local:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false"
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
      #启动依赖
      initContainers:
      - name: init-mq-ns
        image: busybox:latest
        command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"]
cat mq-nmserv.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mq-namesrv
  name: mq-ns
spec:
  type: ClusterIP
  ports:
  - port: 9876
    targetPort: 9876
    name: namesrv-port
  selector:
    app: mq-namesrv
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mq-namesrv
spec:
  serviceName: mq-namesrv
  replicas: 1
  selector:
    matchLabels:
      app: mq-namesrv
  template:
    metadata:
     labels:
       app: mq-namesrv
    spec:
      # affinity:
      #   #podAntiAffinity 反亲合性
      #   podAntiAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       - labelSelector:
      #           matchExpressions:
      #             - key: "app"
      #               operator: In
      #               values:
      #                 - mq-namesrv
      #         topologyKey: "kubernetes.io/hostname"
      containers:
      - name: mq-namesrv
        image: apache/rocketmq:4.9.7
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: JAVA_OPT_EXT
            value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m"
        #command: ["sh","-c","mqnamesrv"]
        command:
          - /bin/sh
        args:
          - mqnamesrv
        readinessProbe:
          tcpSocket:
            port: 9876
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20

Additional Context

正常在rocketmq的集群就应该可以在集群外部可以接受访问的，而不是只是集群内部访问

wolf27w avatar Jun 05 '24 01:06 wolf27w

升级到5.x，使用proxy。 或者换到Pulsar，已实现使用LB(nginx或f5)到多个proxy，通过域名单端口暴露到公网使用

huxiyu avatar Jul 26 '24 09:07 huxiyu

我这里使用的是nginx，但是发现一个问题，我需要弄两个nginx分别代理两个broker，如果只用一个nginx代理两个broker会出现消息重复，如果分开mq会有两个消息，但是应用不会消费两次同一个消息，不分开就会消费两次同一个消息

wolf27w avatar Jul 26 '24 09:07 wolf27w