EdgeCase: docker-mailserver on bare-metal kubernetes with metallb as loadbalancer

[tibeer]

Hi there!

I know my scenario is kind of an edge-case, but maybe someone can help me. I am running Kubernetes on bare metal as a single node cluster. The machine itself has one public IP natted to its private IP address (e.g. 1.2.3.4 is natted to 10.0.0.20). Thus, my metallb ipaddresspool looks like this:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  creationTimestamp: "2024-06-22T05:05:20Z"
  generation: 1
  name: pool
  namespace: metallb-system
  resourceVersion: "1033"
  uid: b545786c-a08b-4e8e-a643-72c35a0c837c
spec:
  addresses:
  - 10.0.0.20/32
  autoAssign: true
  avoidBuggyIPs: false

This works fine for all applications that use a service of type LoadBalancer with externalTrafficPolicy: Cluster, but this is of course not working for docker-mailserver, as it requires the Local externalTrafficPolicy. Since I only have one public IP available in my metallb Pool, I need to use certain metallb annotations (allow-shared-ip) to share the IP between services:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: docker-mailserver
    meta.helm.sh/release-namespace: docker-mailserver
    metallb.universe.tf/address-pool: pool
    metallb.universe.tf/allow-shared-ip: default
    prometheus.io/path: /metrics
    prometheus.io/port: "9102"
    prometheus.io/probe: "false"
    prometheus.io/scrape: "true"
  creationTimestamp: "2024-09-10T05:06:39Z"
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: docker-mailserver
    chart: docker-mailserver-4.0.5
    heritage: Helm
    release: docker-mailserver
  name: docker-mailserver
  namespace: docker-mailserver
  resourceVersion: "22750686"
  uid: 040776d5-ac86-4a96-ad07-50425d46eaaf
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.96.49.23
  clusterIPs:
  - 10.96.49.23
  externalTrafficPolicy: Local
  healthCheckNodePort: 32252
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: smtp
    nodePort: 31950
    port: 25
    protocol: TCP
    targetPort: smtp
  - name: submissions
    nodePort: 31416
    port: 465
    protocol: TCP
    targetPort: submissions
  - name: submission
    nodePort: 31180
    port: 587
    protocol: TCP
    targetPort: submission
  - name: smtp-proxy
    nodePort: 31784
    port: 12525
    protocol: TCP
    targetPort: smtp-proxy
  - name: subs-proxy
    nodePort: 32593
    port: 10465
    protocol: TCP
    targetPort: subs-proxy
  - name: sub-proxy
    nodePort: 31094
    port: 10587
    protocol: TCP
    targetPort: sub-proxy
  - name: imap
    nodePort: 31984
    port: 143
    protocol: TCP
    targetPort: imap
  - name: imaps
    nodePort: 31504
    port: 993
    protocol: TCP
    targetPort: imaps
  - name: imap-proxy
    nodePort: 30708
    port: 10143
    protocol: TCP
    targetPort: imap-proxy
  - name: imaps-proxy
    nodePort: 30947
    port: 10993
    protocol: TCP
    targetPort: imaps-proxy
  - name: rspamd
    nodePort: 32494
    port: 11334
    protocol: TCP
    targetPort: rspamd
  selector:
    app.kubernetes.io/name: docker-mailserver
    release: docker-mailserver
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

But since the externalTrafficPolicy is required to be local for docker-mailserver, this conflicts with sharing the IP. Has anybody an idea how I can solve that situation?