minio
Cluster domain from operator is not being used to proxy requests
The cluster domain setting from the operator has no effect and gets defaulted to "cluster.local" when I try to manage a tenant from the operator ui.
I found that operatorapi/proxy.go#L95 calls function GetClusterDomain(). I've tried setting the environment variable clusterDomain that is used by v2.GetClusterDomain() manually in the deployment.yaml but it doesn't work.
Logs from console pod:
2022/01/13 20:50:50 Post "https://minio-console.minio.svc.cluster.local:9443/api/v1/login": dial tcp: lookup minio-console.minio.svc.cluster.local on 10.233.0.3:53: no such host
2022/01/13 20:50:52 Post "https://minio-console.minio.svc.cluster.local:9443/api/v1/login": dial tcp: lookup minio-console.minio.svc.cluster.local on 10.233.0.3:53: no such host
I found that operatorapi/proxy.go#L95 calls function
GetClusterDomain(). I've tried setting the environment variableclusterDomainthat is used by v2.GetClusterDomain() manually in the deployment.yaml but it doesn't work.
Please share how you set it @vincentbaeten
I found that operatorapi/proxy.go#L95 calls function
GetClusterDomain(). I've tried setting the environment variableclusterDomainthat is used by v2.GetClusterDomain() manually in the deployment.yaml but it doesn't work.Please share how you set it @vincentbaeten
I've used the krew plugin kubectl minio init --cluster-domain <clusterdomain> to init the operator. After that I manually changed the deployment file in k8s and added an env in the container spec.
@vincentbaeten do you see the CLUSTER_DOMAIN environment variable set on the console deployment in the minio-operator namespace?
No I do not. I've tried setting the env variable to CLUSTER_DOMAIN and CLUSTERDOMAIN manually too but without any luck.
Also I forgot to mention that I'm using operator v4.4.3 which uses console v.0.13.2
I see that operator has it set properly
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
operator.min.io/authors: MinIO, Inc.
operator.min.io/license: AGPLv3
operator.min.io/support: https://subnet.min.io
name: minio-operator
namespace: minio-operator
spec:
replicas: 2
selector:
matchLabels:
name: minio-operator
template:
metadata:
annotations:
operator.min.io/authors: MinIO, Inc.
operator.min.io/license: AGPLv3
operator.min.io/support: https://subnet.min.io
labels:
name: minio-operator
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: name
operator: In
values:
- minio-operator
topologyKey: kubernetes.io/hostname
containers:
- env:
- name: CLUSTER_DOMAIN
value: test.local
image: minio/operator:v4.4.3
imagePullPolicy: IfNotPresent
name: minio-operator
resources:
requests:
cpu: 200m
ephemeral-storage: 500Mi
memory: 256Mi
securityContext:
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: minio-operator
I've used the krew plugin
kubectl minio init --cluster-domain <clusterdomain>to init the operator. After that I manually changed the deployment file in k8s and added an env in the container spec.
Why did you change manually? @vincentbaeten
@harshavardhana The cluster domain is properly set on the operator deployment but the problem is that the cluster domain does not get set in the console deployment like @dvaldivia asked. I haven't touched the operator deployment by hand only the console deployment.
Operator deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"operator.min.io/authors":"MinIO, Inc.","operator.min.io/license":"AGPLv3","operator.min.io/support":"https://subnet.min.io"},"name":"minio-operator","namespace":"minio-operator"},"spec":{"replicas":2,"selector":{"matchLabels":{"name":"minio-operator"}},"template":{"metadata":{"annotations":{"operator.min.io/authors":"MinIO, Inc.","operator.min.io/license":"AGPLv3","operator.min.io/support":"https://subnet.min.io"},"labels":{"name":"minio-operator"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"name","operator":"In","values":["minio-operator"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"env":[{"name":"CLUSTER_DOMAIN","value":"bookm-hetzner-fsn1-eu-central-prod-cluster1"}],"image":"minio/operator:v4.4.3","imagePullPolicy":"IfNotPresent","name":"minio-operator","resources":{"requests":{"cpu":"200m","ephemeral-storage":"500Mi","memory":"256Mi"}},"securityContext":{"runAsGroup":1000,"runAsNonRoot":true,"runAsUser":1000}}],"serviceAccountName":"minio-operator"}}}}
operator.min.io/authors: MinIO, Inc.
operator.min.io/license: AGPLv3
operator.min.io/support: https://subnet.min.io
creationTimestamp: "2022-01-07T16:55:58Z"
generation: 2
name: minio-operator
namespace: minio-operator
resourceVersion: "4510446"
uid: d0744b48-d81b-4a9b-aedc-5b57c7978ff5
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
name: minio-operator
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
operator.min.io/authors: MinIO, Inc.
operator.min.io/license: AGPLv3
operator.min.io/support: https://subnet.min.io
creationTimestamp: null
labels:
name: minio-operator
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: name
operator: In
values:
- minio-operator
topologyKey: kubernetes.io/hostname
containers:
- env:
- name: CLUSTER_DOMAIN
value: bookm-hetzner-fsn1-eu-central-prod-cluster1
image: minio/operator:v4.4.3
imagePullPolicy: IfNotPresent
name: minio-operator
resources:
requests:
cpu: 200m
ephemeral-storage: 500Mi
memory: 256Mi
securityContext:
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: minio-operator
serviceAccountName: minio-operator
terminationGracePeriodSeconds: 30
status:
availableReplicas: 2
conditions:
- lastTransitionTime: "2022-01-07T16:56:08Z"
lastUpdateTime: "2022-01-07T16:56:08Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2022-01-07T16:55:58Z"
lastUpdateTime: "2022-01-13T14:23:18Z"
message: ReplicaSet "minio-operator-85b7757cd7" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 2
readyReplicas: 2
replicas: 2
updatedReplicas: 2
Console deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"operator.min.io/authors":"MinIO, Inc.","operator.min.io/license":"AGPLv3","operator.min.io/support":"https://subnet.min.io"},"name":"console","namespace":"minio-operator"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"console"}},"template":{"metadata":{"annotations":{"operator.min.io/authors":"MinIO, Inc.","operator.min.io/license":"AGPLv3","operator.min.io/support":"https://subnet.min.io"},"labels":{"app":"console"}},"spec":{"containers":[{"args":["server"],"env":[{"name":"CONSOLE_OPERATOR_MODE","value":"on"}],"image":"minio/console:v0.13.2","imagePullPolicy":"IfNotPresent","name":"console","ports":[{"containerPort":9090,"name":"http"},{"containerPort":9433,"name":"https"}],"securityContext":{"runAsGroup":1000,"runAsNonRoot":true,"runAsUser":1000}}],"serviceAccountName":"console-sa"}}}}
operator.min.io/authors: MinIO, Inc.
operator.min.io/license: AGPLv3
operator.min.io/support: https://subnet.min.io
creationTimestamp: "2022-01-13T20:16:04Z"
generation: 3
name: console
namespace: minio-operator
resourceVersion: "4715100"
uid: 7bf0003b-469d-4e50-83d0-e1d3f84ef2f3
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: console
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
operator.min.io/authors: MinIO, Inc.
operator.min.io/license: AGPLv3
operator.min.io/support: https://subnet.min.io
creationTimestamp: null
labels:
app: console
spec:
containers:
- args:
- server
env:
- name: CONSOLE_OPERATOR_MODE
value: "on"
- name: CLUSTERDOMAIN
value: bookm-hetzner-fsn1-eu-central-prod-cluster1
image: minio/console:v0.13.2
imagePullPolicy: IfNotPresent
name: console
ports:
- containerPort: 9090
name: http
protocol: TCP
- containerPort: 9433
name: https
protocol: TCP
resources: {}
securityContext:
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: console-sa
serviceAccountName: console-sa
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2022-01-13T20:16:04Z"
lastUpdateTime: "2022-01-13T20:41:11Z"
message: ReplicaSet "console-69fb4486f8" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2022-01-13T20:50:22Z"
lastUpdateTime: "2022-01-13T20:50:22Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 3
readyReplicas: 1
replicas: 1
updatedReplicas: 1
The CLUSTERDOMAIN env is manually added but doesn't work.
Can you try using CLUSTER_DOMAIN for console deployment as well, that is the correct environment variable so that we honor the customer cluster domain. Please try that and re-open if the problem persists
@vincentbaeten Did it work for you though? Having the same issue here
