operator Resource memory limits should apply to `minio` process as well

We are trying to limit the memory consumption of Minio pods, but of course also not want to see "random" OOM kills of the minio containers. Our intuitive approach was to set the common Kubernetes resource requests and limits for the memory inside the resources property of the Pool definition for a Tenant. Now if there's enough load on the cluster (and/or over time) we see the minio containers of the pods getting OOM killed.

Researching a little bit, we found certain issues trying to fiddle with GOMEMLIMIT and MINIO_MEMLIMIT (or --memlimit parameter) which is not (really) documented in the Minio documentation (another issue to be addressed?).

Our last attempt was to set the two environment variables from above which let the minio container fail on startup when specifying the MINIO_MEMLIMIT (either in numeric bytes or with alphanumeric MiB units). The only error logged was, before the container restarted in a loop:

minio FATAL Unable to prepare the list of endpoints: strconv.ParseFloat: parsing "": invalid syntax

We assume that even numeric values of environment variables are always strings.

On the other hand, even if this (manually configuring MINIO_MEMLIMIT variable) would solve our actual issue, we rather like to see it more convenient if the pure configuration of resources.limits.memory will apply automatically and properly to the minio process/runtime configuration as well.

Related Github issues we found

https://github.com/minio/operator/issues/2131
https://github.com/minio/minio/issues/20437
- Led to discussion: https://github.com/minio/minio/discussions/20438
https://github.com/minio/minio/issues/18849

Expected Behavior

We are not sure how and if it is actually intended to define a memory limit for the MinIO process itself. But setting memory limits in the resources section of a Pool definition is leading to OOM kills of the pod container. We'd rather see an automatic adoption of the GOMEMLIMIT and MINIO_MEMLIMIT

Current Behavior

No (documented) way to properly configure of memory limits of the actual minio process.
OOM kills of containers, when resource memory limits reached

Possible Solution

see: "Expected Behavior" above.

Steps to Reproduce (for bugs)

This is a minimal sample Tenant specification which is not working for us:

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: s3-test
spec:
  ### ↓ Some hackish attempts to configure the mem limits of the `minio` process via environment variables according to some Github issues/discussions we also tried.
  # env:
  #   - name: GOMEMLIMIT
  #     value: "1073741824" # no impact (also with "1GiB")
  #   - name: MINIO_OPTS
  #     value: "--memlimit=1073741824" # also not working
  #   - name: MINIO_MEMLIMIT
  #     value: "1073741824" # will lead to "strconv.ParseFloat" error (also "1GiB")
  configuration:
    name: s3-test-admin
  image: quay.io/minio/minio:RELEASE.2025-01-18T00-31-37Z
  pools:
    - name: test
      resources:
        requests: { memory: 128Mi }
        limits: { memory: 128Mi }
      servers: 4
      volumesPerServer: 1
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 1Gi
  requestAutoCert: false
  subPath: /data
  buckets: [ name: test ]
  users: [ name: s3-test ]

With such a setup, we just needed to copy a large file (~1GB .iso) to the test bucket, which killed the container after some seconds. Without the resources.limits config we see that memory consumption is going above the 200Mi mark, which explains the OOM kill when limits are enabled.

Context

no particular context

Regression

No - at least not for us.

Your Environment

Version used:
- minio-operator: 7.0.0
- minio container image: RELEASE.2025-01-18T00-31-37Z
Environment name and version: kubernetes v1.28.11
Operating System and version (uname -a): Linux 5.15.0-117-generic on Ubuntu 22.04.4 LTS
Link to your deployment file: → see "Steps to reproduce"

Jan 30 '25 10:01 mhoyer

I am also experiencing this error after upgrading to version 7.1.0

GOMEMLIMIT is not a problem, and can be set without error MINIO_MEMLIMIT makes Minio crash, with same error as described in the issue. It does not matter if i write it as XXGi or as just a number representing the bytes.

Aug 29 '25 19:08 LuxTheDude

Further info: it appears there is no auto gomemlimit feature in minio itself yet.

Unfortunate since it means a memory limit must be calculated as 90% or whatever of the container memory limit, and kube downward api divisor only supports powera of 10 for some crazy reason.

Minio operator silently ignores Tenant's .spec.env if a .spec.configuration is provided, making this even more difficult to configured in a sensible manner. If you think you set GOMEMLIMIT but minio doesn't behave like it, check /proc/1/environ within the container to confirm if a GOMEMLIMIT has actually taken effect.

It looks like the hidden CLI option --memlimit does in fact apply a GOMEMLIMIT. Unfortunately, the minio-operator doesn't appear to expose a way to set additional args via the tenants.minio.min.io (Tenant) CR.

Code reading supports this - the construction of the minio container arguments doesn't take any external or configured inputs.

I've tried setting the GOMEMLIMIT by adding a line like

export GOMEMLIMIT="3500MiB"

to the file pointed to by my .spec.configuration. I can see the added line in the /tmp/minio/config.env inside the resulting Pod.

But I do not see it in the actual running go program's environment in /proc/1/environ:

$ kubectl exec -it -n ${TENANT}-minio-tenant-default -c minio pod/${TENANT}-pool-0-0 -- cat /proc/1/environ | xargs -0 -n 1 echo | grep -i MEM
$

Something may be sanitizing/filtering it.

To confirm, I examine the metrics from the tenant pod's /minio/v2/metrics/node endpoint I see:

$ curl -sSLf1 http://localhost:9000/minio/v2/metrics/node | egrep ^go_gc_gomemlimit_bytes
go_gc_gomemlimit_bytes 4.831838208e+09

What's odd is that this value doesn't correspond to either the Pod's .resources.limits.memory of 5 GiB (5.369e+9 bytes) or the configured GOMEMLIMIT of 3500 MiB (3.67e+9). It's almost exactly 90% of the container memory limit though.

It is unclear how this GOMEMLIMIT is getting set, as the args does not contain any --memlimit, nor is there one in the environment. The minio component does have a cgroupMemLimit function but it appears to be dead code. The only place that calls debug.SetMemoryLimit is here. I don't see anything in the logic that handles that setting that would assign a 90% value.

It's apparent that 90% isn't an aggressive enough setting for my workload anyway, and I can't seem to override it.

I'm only using this for a test workload in kind anyway, but I'm kind of astonished that it just OOMs when I copy 70 GiB of Thanos metrics TSDB archives into minio; with a 5 GiB memory limit it OOMs around the 15 GiB mark during the copy.

Nov 20 '25 01:11 ringerc