Can't revert sidecar containers and extra volumes on vitess operator

[naama-connecteam]

We have an issue in our vitess operator deployment. We are running vitess/lite:v12.0.0-mysql80. I added in the operator two sidecar containers and two extra volume to the gateway section. It looks like that -

  cells:
  - name: {{ app_cell }}
    gateway:
      sidecarContainers:
      - image: docker.elastic.co/beats/filebeat:8.3.1
        name: filebeat
        args: [
            "-c", "/etc/filebeat.yml",
            "-e",
        ]
        env:
        - name: LOGSTASH_URL
          value: "XXX"
        volumeMounts:
        - name: filebeat-config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: logs-dir
          mountPath: /var/logs/vitess
          readOnly: true
      - image: blacklabelops/logrotate:1.3
        name: logrotate
        volumeMounts:
        - name: logs-dir
          mountPath: /var/logs/vitess
        lifecycle:
          postStart:
            exec:
              command:
              - sh
              - -c
              - echo "XXX"
        env:
        - name: LOGS_DIRECTORIES
          value: "/var/logs/vitess"
        - name: LOG_FILE_ENDINGS
          value: "json"
        - name: LOGROTATE_CRONSCHEDULE
          value: "*/1 * * * *"
        - name: LOGROTATE_COPIES
          value: "2"
        - name: LOGROTATE_SIZE
          value: "50M"
      extraVolumes:
      - name: logs-dir
        emptyDir:
          medium: "Memory"
          sizeLimit: "0.5Gi"
      - name: filebeat-config
        configMap:
          name: vitess-filebeat-config
      extraVolumeMounts:
      - name: logs-dir
        mountPath: /var/logs/vitess

All of the changes are applied nicely, but once I applied them, I can’t revert to the previous configuration. Applying the yaml again without the sidecar containers and the extra volumes doesn’t do anything in itself, and I’m getting this error message in the operator pod:

{
  "level": "error",
  "ts": XXX,
  "logger": "controller.vitesscell-controller",
  "msg": "Reconciler error",
  "name": "vitess-app-XXX",
  "namespace": "vitess",
  "error": "Deployment.apps \"vitess-app-vtgate-XXX\" is invalid: [spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"filebeat-config\", spec.template.spec.containers[1].volumeMounts[1].name: Not found: \"logs-dir\", spec.template.spec.containers[2].volumeMounts[0].name: Not found: \"logs-dir\"]",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214"
}

The only way I found to apply the changes was to delete the vitess-app-vtgate deployment - and obviously I don’t wanna do that in my production environment.

Can you help?

[Verolop]

hello @naama-connecteam , Thank you for your report.

How are you using your sidecar containers? is this for vtgate or vttablet resource optimization? We've noticed some behaviours with sidecars interacting with the operator, in some cases they are expected(antipatterns), but some can be fixed or worked around. I'll take a look :)